Caroline's Corner: Teaching Beginners to find Usability Problems

My sister lives in Melbourne and that naturally makes me want to get to Australia as often as I can. So I was especially pleased when a few things came together and I was able to make it to OzCHI, the Australian equivalent of the British HCI Group conference. It's a small conference so you can feel that you've had a chance to talk to most people, but even so they managed to put together a really interesting programme with plenty in it for practitioners. The proceedings will be available shortly (try the ACM digital library).

I'd like to pick out one paper that will change my approach to teaching usability testing.

THE PAPER
Mikael Skov and Jan Stage are at Aalborg University. They teach HCI, and they think about how to teach it better. One problem they encountered was that although it's pretty easy to get students to perform competently as facilitators in usability tests, it seems to be a lot harder to get them to analyse the data that they gather in the tests.

So they tried a small-scale experiment. They got a tape of a typical usability test. Each participant in their experiment reviewed the tape and recorded the usability problems. 8 of the participants did this review without being taught how to analyse data. 6 of the participants did their review after being taught the use of a simple 'conceptual tool'.

The participants who had been taught the use of the tool found more problems and were more consistent in what they found.

Mikael Skov, in the modest way of academics, pointed out a few deficiencies in their research. For example, they did not consider the capture of positive findings. The groups were quite small. But still, the results are worth considering. I'm sufficiently convinced by Skov and Stage's work to take the view that thinking about severity will help students to find more problems and be more accurate in their analysis of those problems.

THE ANALYSIS "TOOL"
So what's in the magic 'conceptual tool' that made so much difference?

The primary dimension is severity. They chose three levels: critical, serious and cosmetic.

The secondary dimension (and I suspect the basis of much of the strength of the tool) is four ways of explaining what we might mean by 'critical', 'serious' or 'cosmetic': "Slowed down relative to normal work speed", "Understanding", "Frustration" and whether the participant needed to get help from a "Test monitor".

So, for example, a 'serious' problem is characterised as:

- Slowed down: Delayed for several seconds

- Understanding: Does not understand how specific functionality operates or is activated. Cannot explain the functioning of the system.

- Frustration: Is clearly annoyed by something that cannot be done or remembered or something illogical that you must do. Believes has damaged something.

- Test monitor: Receives a hint.

TEACH SEVERITY SCALES AND THEIR IMPACT AS PART OF ANALYSIS
Now, those of us who do this stuff for a living might argue somewhat with aspects of this conceptual tool. For example, I might look for a 'minor' category to mop up problems that are a bit more important than 'cosmetic' but don't quite make it into 'serious'. And I might argue that you need to be slowed down for a bit longer than a 'several seconds' before a problem becomes serious'.

But I think the big lesson here isn't the precise details of the tool. I think it's the importance of teaching beginners an explicit, described way of categorising problems before they try looking for problems. And up to now, I've been doing it the other way around: getting them to look for problems, then asking them to determine how severe the problems might be.

I'd probably call it a 'severity scale' rather than a 'conceptual tool', but I'd aim to include both the classification of the problem and Skov and Stage's secondary dimensions, which I'll probably call something like 'impact' (suggestions, please?).

REFERENCE

Skov, M. B. and Stage, J. (2005) Supporting Problem Identification in Usability Evaluations.In Proceedings of the Australian Computer-Human Interaction Conference 2005 (OzCHI’05), ACM Press,

If you have any comments or suggestions about this article then please contact Caroline at:

Caroline.Jarrett@Effortmark.co.uk

Caroline Jarrett is a usability consultant specialising in forms, questionnaires and data capture.

© 2005 Caroline Jarrett, all rights reserved.

Associated Link: Effortmark

Why 'The 10 Commandments of Web Design' are Complete Baloney
Source: SitePoint, 5 July 2008
 
Further to our earlier discussion...

Local Council Websites: Good, but no Cigar
Source: Webcredible, 4 July 2008
 
The 'Local Council Websites: Good, But No Cigar' report evaluates the top 20 Socitm council websites against 20 best practice usability guidelines.

Why does the OK Button say OK?
Source: Gerry McGovern, 3 July 2008
 
Most times I come across the OK button, something not-OK has happened.

The 10 Commandments of Web Design
Source: Business Week, 2 July 2008
 
To try and make sense of it all, BusinessWeek.com canvassed a broad range of Internet luminaries to discover the design rules they live by right now.

Bill Gates' email rant at the Usability Team
Source: Crunch Gear, 1 July 2008
 
The Seattle P-I has tracked down an old e-mail from 2003 that Bill Gates sent out to members of the Windows Usability team.

Can a better User Experience also be a Greener Experience?
Source: Catalyst Resources, 30 June 2008
 
By improving the user experience of its web collaboration software, Catalyst made it easier for clients to work in productive teams without travelling the globe.

The Spread of Telepresence Technology
Source: Tech News World, 28 June 2008
 
Rising energy costs and green concerns are fuelling a huge increase in demand for videoconferencing, collaborative software and virtual telepresence technology.

UK Usability market worth more than Ł200 million by end of 2008
Source: eConsultancy, 27 June 2008
 
The UK Usability market will grow by an estimated 20% in 2008 to a value of Ł214 million, according to research published this week by e-consultancy.

Research suggests growing importance of Usability to UK Businesses
Source: Business Strata, 26 June 2008
 
Integrated software for lead generation and marketing is valuable, but only as long as it's usable.

Web 2.0: Too Smart by a Half?
Source: Computer Technology Review, 25 June 2008
 
Will the Web 2.0 generation’s epitaph be, “They died with their options open?”