Which design? An introduction to Split A/B testing

Following on from any website usability study a number of usability problems are usually found. There can often be debate within any organisation as to the best solution for each problem, with no one really knowing the optimal solution. Rather than letting the person that shouts the loudest get his or her own way, a better solution can be to test two solutions in a live environment. Whichever performs the best is clearly the superior solution. Welcome to split A/B testing!

BURIDAN'S ASS AND A/B TESTING
Did you hear the story of the donkey stood in the middle of two equally appealing stacks of hay? He spent so long trying to decide which one to start eating first that the poor animal starved to death without having moved a single inch. This paradox, first discussed by Aristotle is known as 'Buridan's ass' and is an often discussed psychological phenomenon.

If you're managing a website you might face similar situations when you need to decide which of two different designs to opt for. Nowadays, you need to continuously improve and evolve your site by making small frequent adjustments. But how do you know which change will have the highest impact on the customer experience?

Split A/B testing is a way of finding out which changes help your users' performance. It provides a controlled method of measuring the effectiveness (or not) of alterations to your site. It's often used for small tweaks (eg 'Is this style heading clearer than the original?') but can also be used to test bigger wholesale changes (eg 'Is this new 1-click checkout process better?'). In essence, it involves running two different versions side by side to see which is more effective.

A TYPICAL A/B SCENARIO
You feel your users might not be finding the 'Proceed to checkout' button hidden below the fold of the page and think this might be causing people to leave your site before making a purchase. You've created a new design which you feel is more effective but you'd like to know for sure.

To A/B test your new page, you serve the regular page to say 90% of your visitors as usual, but a randomly selected 10% would be shown your new design. Then you sit back, pour yourself an ice cold gin and tonic, wait and watch your web statistics.

If your new page has a positive effect as you suspect, then you should see an improvement in the 10% group, as measured by conversion rates. Proof that you should publish the more successful page to all visitors.

ADVANTAGES OF SPLIT A/B TESTING
There are a number of benefits to A/B testing:
- Low risk approach
- Cheaper than other methods such as focus groups
- Provides proof
- Invisible to most of your users
- Great way to do 'test run' new designs before full roll-out (to avoid negative surprises on the launch day)
- Can solve internal disputes

DISADVANTAGES OF SPLIT A/B TESTING
A/B testing isn't always suitable. Some of its disadvantages include:
- New designs might have to 'wear in' before you can measure their real performance (visitors' initial response might be negative because they're used to the old solution)
- You can only compare two versions with a single factor that differentiates both designs (for more factors / variations you need to deploy multivariate testing which is more difficult to analyse)
- It takes technical know-how to set up and analyse the results
- You're testing in a live environment so external factors might have an impact on the outcome
- If your site is really in bad shape, you'll still need to do a proper overhaul

IN A NUTSHELL
A/B testing is powerful stuff and a useful method for a quick comparison for two different designs. However, bear in mind that it's no substitute for getting proper feedback from your users. Only this will give you the whole picture - straight from the ass's mouth.

Associated Link: Webcredible - usability & accessibility consultancy

Why 'The 10 Commandments of Web Design' are Complete Baloney
Source: SitePoint, 5 July 2008
 
Further to our earlier discussion...

Local Council Websites: Good, but no Cigar
Source: Webcredible, 4 July 2008
 
The 'Local Council Websites: Good, But No Cigar' report evaluates the top 20 Socitm council websites against 20 best practice usability guidelines.

Why does the OK Button say OK?
Source: Gerry McGovern, 3 July 2008
 
Most times I come across the OK button, something not-OK has happened.

The 10 Commandments of Web Design
Source: Business Week, 2 July 2008
 
To try and make sense of it all, BusinessWeek.com canvassed a broad range of Internet luminaries to discover the design rules they live by right now.

Bill Gates' email rant at the Usability Team
Source: Crunch Gear, 1 July 2008
 
The Seattle P-I has tracked down an old e-mail from 2003 that Bill Gates sent out to members of the Windows Usability team.

Can a better User Experience also be a Greener Experience?
Source: Catalyst Resources, 30 June 2008
 
By improving the user experience of its web collaboration software, Catalyst made it easier for clients to work in productive teams without travelling the globe.

The Spread of Telepresence Technology
Source: Tech News World, 28 June 2008
 
Rising energy costs and green concerns are fuelling a huge increase in demand for videoconferencing, collaborative software and virtual telepresence technology.

UK Usability market worth more than £200 million by end of 2008
Source: eConsultancy, 27 June 2008
 
The UK Usability market will grow by an estimated 20% in 2008 to a value of £214 million, according to research published this week by e-consultancy.

Research suggests growing importance of Usability to UK Businesses
Source: Business Strata, 26 June 2008
 
Integrated software for lead generation and marketing is valuable, but only as long as it's usable.

Web 2.0: Too Smart by a Half?
Source: Computer Technology Review, 25 June 2008
 
Will the Web 2.0 generation’s epitaph be, “They died with their options open?”