Caroline's Corner: Process or Outcome? Measuring the Success of Usability

A friend has been working with a client for 9 months. She’s sort of been ‘usability midwife’ to the birth of their redesigned web site: a major development, loads of nasty back-end integration problems, the effort of many teams. Her role was to conduct a series of usability tests on different iterations. And she told me it had finally launched.

“So, are you pleased with it?” I asked.

“No, it’s horrible” she said. “But it’s much better than it might have been”.

Which led me to brooding on how we measure our successes.

THE THEORY OF MEASURING USABILITY SUCCESS
Measuring the usability of something is quite easy: define your product, users, goals and context. Run a measurement evaluation (also known as ‘summative evaluation’). Publish your report. You’re done. There’s even an ISO standard for reporting these results: the "Common Industry Format". And you’ll find discussion of the details of how to do it in any of the standard textbooks: my own, for example.

But that’s a different matter from measuring the success of “usability”, meaning a single or series of user-centred design activities. I’m going to call this a ‘usability intervention’.

To measure the success of a usability intervention, we need to know:
- what the starting point was, ideally by measuring the usability of the product before the usability intervention
- how the usability intervention affected the product
- what the finishing point was, by measuring the usability of the product again.

Do some sums, and bingo: you’ve got a return-on-investment success story.

THE CHALLENGE OF THE SUCCESS STORY
The problem is real life, as usual.

The starting point is often unclear. Was there a particular product that could be measured? Was the whole thing just a sketch on a whiteboard? Was the organisation willing to allow any statement of the start point to be created, even for internal use?

The effect of the usability intervention is often unclear. Many people worked on the product. Some of them were influenced by the usability intervention; others were not. Even if we take the simplest case, a single usability test with a bunch of findings and recommendations, the influence can be hard to pin down. The team working on one area of the product might fight you line by line and reluctantly implement the least invasive of the suggestions. The team working in another area suddenly “gets it” and starts behaving in a wholly user-centred manner, coming up with lots of brilliant ideas of their own that markedly improve the product. It’s easier to measure the effect of the intervention on the ‘reluctant’ team (but it won’t be much). The ‘get it’ team will achieve a more usable product: but was that solely the result of the intervention?

Even the finishing point is often unclear. Final release? The version immediately after the round of changes following one specific activity? The whole product, or just the bit of it that was worked on intensively in the intervention?

THE MEASUREMENT OF PROCESS IMPROVEMENT
So we often fall back on intangibles, like measuring the ‘user-centredness’ of an organisation.

A while ago, a large organisation that I worked with decided to try to do this. While a few of us where fighting in the trenches of some tough product development, the consultants came in. They ‘measured’ the ‘user-centredness’ of the organisation by doing a few interviews here and there. They did a bit of proselytising of user-centred design methods, and then they repeated the measurements. Bingo, things had improved.

But what had actually happened was that our line-by-line, recommendation-by-recommendation fights for change had been going on at the same time. And we’d had enough successes that the products had in fact got better. And a few teams had ‘got it’ because they’d seen what was going on.

The consultants published their success. Really, it was our success. But it was a very, very complex situation and we never published:
- The start point wasn’t obvious. There were many good reasons why the organisation didn’t want to state in public how bad the situation was before we began.
- Many, many changes happened all at once. Some of them were influenced by usability activities, others weren’t. It almost seemed random: we’d make some recommendations, things would happen, no obvious connection.
- And like my friend, we ended with something that was much, much better than it could have been. But still horrible.

CELEBRATING THE CAPTURE OF HEARTS AND MINDS
In the end, I don’t think either type of measurement really works. I think the real successes come person by person, as attitudes change. And sometimes that will have no effect on the current product at all. I know my friend has been successful on her project, because I know that she’s created usability evangelists from a whole slew of people who are now inspired to keep working to turn their horrible web site into a better one, release by release. And that’s what we’ve got to celebrate.


If you have any comments or suggestions about this article then please contact Caroline at:
Caroline.Jarrett@Effortmark.co.uk
Caroline Jarrett is a usability consultant specialising in forms and improving web content.

©2007 Caroline Jarrett, all rights reserved.

Associated Link: Effortmark website

All change at the top for System Concepts
Source: System Concepts Ltd, 3 July 2009
 
Leslie Fountain has been promoted to joint Managing Director of leading usability consultancy System Concepts.

Life in UCD immortalised in fiction: you couldn't make it up
Source: UN, 2 July 2009
 
Sarah Herman's fictitious book on life in a user-centred design company has hit the shelves and The Guardian's book pages...

Interfaces Magazine - Issue 79: The Education Issue
Source: Interaction Group, 1 July 2009
 
The latest issue of Interfaces is now available as a free download from the Interaction Website.

Two new Behavioural research Tools from Noldus
Source: UN, 30 June 2009
 
Tool updates make on-site behavioural data collection easier.

Cell Phones that Listen and Learn
Source: MIT Technology Review, 29 June 2009
 
New software tracks a user's behavior by monitoring everyday sounds.

Top Six Don’ts for Usability Testing
Source: FutureNow Inc., 27 June 2009
 
Six tips for creating quality usability tests to ensure useful feedback from testers.

Usability: ‘Lovely software. But I can’t work it’
Source: FT.com, 26 June 2009
 
In a recent survey by Global Graphics, 77 per cent of office workers estimate they lose up to one hour a week because business software is difficult to use.

And what do you do?
Source: Dexo Design, 25 June 2009
 
How do you describe your job role? Here are the results of a recent 'Preferred UX/UI Title' Poll.

Most Doctors cite Usability as critical to Electronic Health Record Adoption
Source: TMCNet, 24 June 2009
 
It's all about 'meaningful use'.

Glossy monitors look good but can hurt
Source: QUT, 23 June 2009
 
A new advisory cites research which suggests high gloss monitors make users sit awkwardly.