When is Time a useful Metric?

By Dr. Alex DeWitt
Senior Research Consultant, fhios


In usability studies, time is an obvious metric. It is absolute, easy to measure, and easy to interpret. Recording how long it takes a user to complete a process or activity gives a concrete figure which is sometimes easier for clients to accept than qualitative findings. Time is also a metric which fits into the ‘efficiency’ category, which, along with ‘effectiveness’ and ‘satisfaction’ makes up the ISO9241-11 standard for measuring usability. However, there are several considerations to make before time can be used as a valid usability metric:

- Methodology
If you are going to use ‘think aloud’ techniques, timing the participant is out of the question as the result will be skewed by moderator questions, time required to think about and respond to these questions, and time required to re-focus back to the original task at hand. Timing sessions requires using alternative methods.

- Context
‘Time taken’ has different meanings depending on the context. Some activities, like game playing, are used as a ‘time killer’, so although interesting, ‘time taken’ is not a useful usability metric. Information seeking is equally hindered when the user is confused, interested, or distracted, causing them to spend more time on an activity. Arguably the most useful context in which to record time taken is when a user is completing a registration process and moving through fields; here we can be fairly certain that the user does not enjoy this process, but is required to move through it to progress to a subsequent goal (banking, blogging etc). Therefore the time taken should be as low as possible.

- Number of participants
Nielson recommends testing with 20 users when measuring time or other quantitative metrics to ensure that the confidence level is acceptable. In practice though, timing even a few users can provide a frame of reference when comparing multiple sites or versions of a site.

- Actionable data
If you know that it took users a mean time of 4 minutes to complete a process… is that good? Bad? Better than average? How much should you aim to reduce that by? What was the factor that caused them to take that amount of time? Having a comparison is necessary (either your site over time, or your site versus competitor sites.)

If the study is a benchmarking or performance assessment type, a time metric would be useful to indicate general efficiency. Whereas if the goal of the study is to uncover why the interface is performing badly, or to uncover usability problems and how to fix them, timing the session may prove less fruitful. In either case, the decision whether to time a usability session must be made very early in the process such as shortly after the kick off meeting; this way all involved parties are clear about how to prepare the test materials and what is expected of the reports. If the session is to be timed, consider using retrospective think aloud which asks the user to explain their thought processes after the event. Alternatively, ‘stimulated retrospective think aloud’ uses a memory aid such as replaying a video of the actions they took on screen, to elicit the participants’ thoughts on what they did. This approach is more thorough but takes more time and resources.

Ultimately it should be understood that the usability session is a qualitative event and therefore the implicit and explicit actions of users can lead to more insight and actionable recommendations than a measure of how long users spent to complete a task.

Associated Link: fhios

'Internet addiction' linked to Depression
Source: BBC, 9 February 2010
 
There is a strong link between heavy internet use and depression, UK psychologists have said.

Could *You* be more Usable?
Source: UN, 8 February 2010
 
Bet you could.

Stowe Boyd on 'Steampunk' thinking about the Future of Computing
Source: Stowe Boyd's blog via Experientia, 6 February 2010
 
Are established metaphors of user experience holding us back from new ways of structuring our interaction through computers?

Nokia's User Experience Programme
Source: UN, 5 February 2010
 
Nokia has put together a rich and informative website covering the key elements of user experience.

Interfaces magazine: latest issue available now
Source: HCI News Service, 4 February 2010
 
The latest issue of Interfaces is now available in pdf format, free from the Interaction Website.

A Lighter Brigade of Chargers
Source: UN, 3 February 2010
 
Lots of gadgets, one charger. At last.

Mobile Touch Screens could soon Feel the Pressure
Source: MIT Technology Review, 2 February 2010
 
A quantum switch could add pressure sensing to mobile screens.

Usability, Usability, Usability: why the iPad will Succeed
Source: Econsultancy, 1 February 2010
 
The tech critics love it, hate it, love it again, shrug it off. What do usability experts say?

British Airways - at last some good news
Source: Loop11, 30 January 2010
 
In a recent website usability study for the world's leading airlines, the British Airways website proved to be the most user friendly, with Malaysia Airlines and Virgin Atlantic having the lowest user experience rating.

Computation of Emotions in Man and Machine
Source: Royal Society, 29 January 2010
 
Advances in computer technology now allow machines to recognise and express emotions, paving the way for improved human-computer and human-human communications.