HCI2002/EUPA: Evaluation takes Centre Stage

This HCI2002 session included six short papers, presenting late breaking research and initial studies on a variety of topics, all focusing on evaluation.

The first two papers reported aspects of a project evaluating educational multimedia applications, an area with which Helen Sharp is, of course, familiar through her work at the OU.

[Educational multimedia cognitive walkthrough M. Dimitrova, S. Wilson & H. Sharp] The first presentation, by Maia Dimitrova, focused on measuring the effectiveness of an educational multimedia (EMM) application. Their framework enables the evaluation of EMM design solutions, both in terms of usability, and users' comprehension. The framework allowed analysts to make better predictions of usability problems.

She reported an application of the cognitive walkthrough technique, in which guidelines were also used, to enable more complete explanation of the problems found. Chris Evans made an interesting comment that the addition of guidelines to the cognitive walkthrough was a major factor in the increased effectiveness of the technique.

[Categorising and reporting usability problems of educational multimedia interfaces M. Dimitrova, S. Wilson & H. Sharp] The talk, by Helen Sharp, focused specifically on the reporting of usability problems. As in the previous paper, the problems were categorised as user interaction, and comprehension problems.

Helen pointed out that the key aspect of evaluating educational multimedia was not only a usability issue, but also about learning efficacy. Comprehension problems were defined as specifically relating to student learning, in other words, those relating to the specific domain, as separate from usability problems encountered during use of the interface.

The problem report format was based on a model by Lavery et al. Usability problems were reported in a standard format, under the headings:
* context,
* cause,
* misconception and
* comprehension outcome.
Comprehension problems were categorised as:
* translation,
* interpretation or
* extrapolation problems.

[Testing a conjecturee based on the DR-AR model of UIM effectiveness A. Woolrych & G. Cockton] Alan Woolrych noted that heuristic evaluation remains the most popular usability technique, and this paper investigated the main failings of HE studies, in particular, missed problems, and false alarms. Problems, which were correctly identified, but attributed to the wrong heuristic, were also investigated. Indeed, in their study, only 39% of problems were attributed to the correct heuristic.

He noted that the process of HE comprised two distinct stages – discovery and analysis. He examined Nielsen's assertion that more analysts will result in a greater number of problems, highlighting that more analysts may well dilute the overall elimination resources (result in more false alarms). So, he asked, what if we eliminated all unique predictions? His answer was that approximately 1/3 of both the usability problems and false alarms would be eliminated. This is clearly a case of throwing the baby out with the bath water. The work highlighted the conclusion that care must be taken with merging predictions, and that unique problems are not necessarily false alarms.

[Interaction design issues for car navigation systems P. Curzon, R. Butterworth, R. Bhogal & A. Blandford]

The aim of the research was to investigate the need for better interaction design of in-car navigation systems, given their relative novelty, and the safety critical context in which they are used. The talk, by Paul Curzon, evaluated an in-car satellite navigation system, which he encountered when driving a hire car on his family holiday, and recorded critical incidents during a weekend, then a one-week trip. Cognitive walkthroughs were then performed on each of these critical incidents, to explore why they might have occurred.

He found the system very intuitive to use in general, with good speech synthesis, clear display, and none of the ergonomic problems identified by various models, such as Egenhofer. However, he still recorded a large number of critical incidents, in particular, when taking a sharp corner, which appeared to confuse the unit, as it treated this as a junction, and instructed him to make a turn.

He noted the delay required to input the destination into the system caused some irritation for his fellow travellers (who had already planned their route using a paper map), and complained that salient information about the environment (waypoints) was not integrated into the system, such as 'turn left at the Red Lion pub'. This reduced his confidence in the system, as the instructions were sometimes counter-intuitive. His main issue with the system was that there was no scope for adding his own knowledge to the system (such as his preferred route).

[Diary study of payment systems D. Abrazhevich & P. Markopoulos] The talk, by Dennis Abrazhevich, explored usability issues in the design of online payment systems – an important class of systems, as they underpin e-commerce interactions. Users filled in paper diaries after using internet-based payment systems (online banks, and Paypal). The study explored the major risks, as perceived by users during the interaction, and their attitudes to the information requested by the systems (was this treated properly? Did they ask for too much information?)

Some users had problems even registering, due to lengthy security codes, and gave up before even accessing the functionality available through the system. Whilst security is undoubtedly a key factor in banking systems, if it prevents users from registering, the entire development effort is wasted, as least on these users.

They also found many issues of trust and privacy affecting the user experience, with users being sensitive to giving any information which they deemed unnecessary for completing their tasks.

[Using interfaces and liking interaction C. Roast & B. Steele] This talk, by Chris Roast, explored subjective reactions to aesthetics and usability of widgets for web-based interaction – e.g. the 'navihedron' – a whizzy way of navigating websites, which, although 'cool', might not result in improved usability. The study focused only on users' evaluations of these two qualities, rather than any deconstruction of the concepts themselves.

The study identified a range of web-based navigation widgets, and asked users to rate them for attractiveness and usability – both before and after repeated use. Whilst the moving 3D spheres were appreciated aesthetically, in contrast to a static menu, they were not predicted to be as usable – a conclusion which did not change after use – although the static menu was rated even more highly on usability.

Aesthetic appreciation appeared to be linked to dynamic behaviour, particularly when users could control this dynamic movement. Chris thinks that the concept of 'Flow' may well be a useful one for future analysis in this work.


Whilst all the papers in this session explored interesting topics, it was difficult to find much in the way of knowledge to take away and apply during future work. This may be due to the fact that the papers reported preliminary studies, rather than completed projects. However, it would have been useful if potential insights were to have been made more explicit, albeit with the qualification that the sample sizes were often too small to give confidence in the insights at this stage.

Steve Cummaford,
amberlight partners

See also A Day at HCI2002/EUPA: a Practitioner's View for a different perspective on some of the same papers.

Online videos: Engaging your users
Source: Webcredible, 6 September 2008
 
A guide to using online video on your site to offer more enticing content and provide a more compelling user experience.

The Office of the Future?
Source: Occupational Hazards via Ergonomics in the News, 5 September 2008
 
The office of the future is less about technology and more about physiology.

This year's 10 best-designed Application User Interfaces
Source: NNg, 4 September 2008
 
Nielsen Norman Group recently announced the winners in its inaugural competition to find the 10 best-designed application user interfaces of the Application Design Annual 2008.

Caroline's Corner: Buttons on Forms - where to put them, and what to call them
Source: Caroline Jarrett, 3 September 2008
 
Should you put the ‘OK’ button to the left or the right of the ‘Cancel’ button? Like so much in forms, the simple answer isn’t really appropriate. And yet, who needs another ‘it depends’?

Three Usability Gurus
Source: avangate blog, 2 September 2008
 
A personal view, but lots of good links, especially if you're new to usability.

Ofcom publishes report on Usability Event
Source: Ofcom, 1 September 2008
 
Independent regulator and competition authority for the UK communications industries, Ofcom, has published its report on usability issues.

Get Fit for Remote Working
Source: UN, 30 August 2008
 
BT Business has announced the launch of a practical guide 'Get fit for mobile working' designed to help mobile workers avoid back, neck and arm problems.

Site Visit Interviews: from Good to Great
Source: User Focus, 29 August 2008
 
For those of you for whom the Basic Introduction to User Interviews wasn't quite enough.

Six Metrics for Managing UI Design
Source: Russell Wilson, 28 August 2008
 
A proposal of six metrics to be used for managing a user interface design department.

Don't Judge a Form by its Cover
Source: Formulate Information Design, 27 August 2008
 
The saying "don't judge a book by its cover" reminds us that looks are deceptive. It turns out that this idiom applies to forms too.