HCI2002/EUPA: Evaluation takes Centre Stage

This HCI2002 session included six short papers, presenting late breaking research and initial studies on a variety of topics, all focusing on evaluation.

The first two papers reported aspects of a project evaluating educational multimedia applications, an area with which Helen Sharp is, of course, familiar through her work at the OU.

[Educational multimedia cognitive walkthrough M. Dimitrova, S. Wilson & H. Sharp] The first presentation, by Maia Dimitrova, focused on measuring the effectiveness of an educational multimedia (EMM) application. Their framework enables the evaluation of EMM design solutions, both in terms of usability, and users' comprehension. The framework allowed analysts to make better predictions of usability problems.

She reported an application of the cognitive walkthrough technique, in which guidelines were also used, to enable more complete explanation of the problems found. Chris Evans made an interesting comment that the addition of guidelines to the cognitive walkthrough was a major factor in the increased effectiveness of the technique.

[Categorising and reporting usability problems of educational multimedia interfaces M. Dimitrova, S. Wilson & H. Sharp] The talk, by Helen Sharp, focused specifically on the reporting of usability problems. As in the previous paper, the problems were categorised as user interaction, and comprehension problems.

Helen pointed out that the key aspect of evaluating educational multimedia was not only a usability issue, but also about learning efficacy. Comprehension problems were defined as specifically relating to student learning, in other words, those relating to the specific domain, as separate from usability problems encountered during use of the interface.

The problem report format was based on a model by Lavery et al. Usability problems were reported in a standard format, under the headings:
* context,
* cause,
* misconception and
* comprehension outcome.
Comprehension problems were categorised as:
* translation,
* interpretation or
* extrapolation problems.

[Testing a conjecturee based on the DR-AR model of UIM effectiveness A. Woolrych & G. Cockton] Alan Woolrych noted that heuristic evaluation remains the most popular usability technique, and this paper investigated the main failings of HE studies, in particular, missed problems, and false alarms. Problems, which were correctly identified, but attributed to the wrong heuristic, were also investigated. Indeed, in their study, only 39% of problems were attributed to the correct heuristic.

He noted that the process of HE comprised two distinct stages – discovery and analysis. He examined Nielsen's assertion that more analysts will result in a greater number of problems, highlighting that more analysts may well dilute the overall elimination resources (result in more false alarms). So, he asked, what if we eliminated all unique predictions? His answer was that approximately 1/3 of both the usability problems and false alarms would be eliminated. This is clearly a case of throwing the baby out with the bath water. The work highlighted the conclusion that care must be taken with merging predictions, and that unique problems are not necessarily false alarms.

[Interaction design issues for car navigation systems P. Curzon, R. Butterworth, R. Bhogal & A. Blandford]

The aim of the research was to investigate the need for better interaction design of in-car navigation systems, given their relative novelty, and the safety critical context in which they are used. The talk, by Paul Curzon, evaluated an in-car satellite navigation system, which he encountered when driving a hire car on his family holiday, and recorded critical incidents during a weekend, then a one-week trip. Cognitive walkthroughs were then performed on each of these critical incidents, to explore why they might have occurred.

He found the system very intuitive to use in general, with good speech synthesis, clear display, and none of the ergonomic problems identified by various models, such as Egenhofer. However, he still recorded a large number of critical incidents, in particular, when taking a sharp corner, which appeared to confuse the unit, as it treated this as a junction, and instructed him to make a turn.

He noted the delay required to input the destination into the system caused some irritation for his fellow travellers (who had already planned their route using a paper map), and complained that salient information about the environment (waypoints) was not integrated into the system, such as 'turn left at the Red Lion pub'. This reduced his confidence in the system, as the instructions were sometimes counter-intuitive. His main issue with the system was that there was no scope for adding his own knowledge to the system (such as his preferred route).

[Diary study of payment systems D. Abrazhevich & P. Markopoulos] The talk, by Dennis Abrazhevich, explored usability issues in the design of online payment systems – an important class of systems, as they underpin e-commerce interactions. Users filled in paper diaries after using internet-based payment systems (online banks, and Paypal). The study explored the major risks, as perceived by users during the interaction, and their attitudes to the information requested by the systems (was this treated properly? Did they ask for too much information?)

Some users had problems even registering, due to lengthy security codes, and gave up before even accessing the functionality available through the system. Whilst security is undoubtedly a key factor in banking systems, if it prevents users from registering, the entire development effort is wasted, as least on these users.

They also found many issues of trust and privacy affecting the user experience, with users being sensitive to giving any information which they deemed unnecessary for completing their tasks.

[Using interfaces and liking interaction C. Roast & B. Steele] This talk, by Chris Roast, explored subjective reactions to aesthetics and usability of widgets for web-based interaction – e.g. the 'navihedron' – a whizzy way of navigating websites, which, although 'cool', might not result in improved usability. The study focused only on users' evaluations of these two qualities, rather than any deconstruction of the concepts themselves.

The study identified a range of web-based navigation widgets, and asked users to rate them for attractiveness and usability – both before and after repeated use. Whilst the moving 3D spheres were appreciated aesthetically, in contrast to a static menu, they were not predicted to be as usable – a conclusion which did not change after use – although the static menu was rated even more highly on usability.

Aesthetic appreciation appeared to be linked to dynamic behaviour, particularly when users could control this dynamic movement. Chris thinks that the concept of 'Flow' may well be a useful one for future analysis in this work.


Whilst all the papers in this session explored interesting topics, it was difficult to find much in the way of knowledge to take away and apply during future work. This may be due to the fact that the papers reported preliminary studies, rather than completed projects. However, it would have been useful if potential insights were to have been made more explicit, albeit with the qualification that the sample sizes were often too small to give confidence in the insights at this stage.

Steve Cummaford,
amberlight partners

See also A Day at HCI2002/EUPA: a Practitioner's View for a different perspective on some of the same papers.

All change at the top for System Concepts
Source: System Concepts Ltd, 3 July 2009
 
Leslie Fountain has been promoted to joint Managing Director of leading usability consultancy System Concepts.

Life in UCD immortalised in fiction: you couldn't make it up
Source: UN, 2 July 2009
 
Sarah Herman's fictitious book on life in a user-centred design company has hit the shelves and The Guardian's book pages...

Interfaces Magazine - Issue 79: The Education Issue
Source: Interaction Group, 1 July 2009
 
The latest issue of Interfaces is now available as a free download from the Interaction Website.

Two new Behavioural research Tools from Noldus
Source: UN, 30 June 2009
 
Tool updates make on-site behavioural data collection easier.

Cell Phones that Listen and Learn
Source: MIT Technology Review, 29 June 2009
 
New software tracks a user's behavior by monitoring everyday sounds.

Top Six Don’ts for Usability Testing
Source: FutureNow Inc., 27 June 2009
 
Six tips for creating quality usability tests to ensure useful feedback from testers.

Usability: ‘Lovely software. But I can’t work it’
Source: FT.com, 26 June 2009
 
In a recent survey by Global Graphics, 77 per cent of office workers estimate they lose up to one hour a week because business software is difficult to use.

And what do you do?
Source: Dexo Design, 25 June 2009
 
How do you describe your job role? Here are the results of a recent 'Preferred UX/UI Title' Poll.

Most Doctors cite Usability as critical to Electronic Health Record Adoption
Source: TMCNet, 24 June 2009
 
It's all about 'meaningful use'.

Glossy monitors look good but can hurt
Source: QUT, 23 June 2009
 
A new advisory cites research which suggests high gloss monitors make users sit awkwardly.