Evaluating Design Effectiveness sparks Debate

Some of the recent raft of design awards appear to reward websites that exhibit serious design concerns, according to AIGA organiser Nico Macdonald. Habitat.net, which has recently been awarded a BAFTA Interactive Entertainment Award, is billed by BAFTA as 'a true replacement for the showroom', but in fact is frustrating to use, he says.

So how are we to evaluate designs? How can we talk about why a website works and why it doesn't? Can we talk about what's good and what's bad? And how does a designer convince someone to hire them based on their history of design effectiveness?

Macdonald drew together a wide range of viewpoints to discuss these issues and others at the Design Council, with a designer, a usability academic and writer, a lecturer in graphic communication and a design consultancy practitioner on the panel.

Awards such as the BAFTA award to the Habitat site were unhelpful and even counterproductive for those teaching design, according to Gerry Leonidas, a lecturer in graphic communication at Reading University.

'These kind of awards teach students that design is something other than what it is. Then we have to undo that message,' argued Leonidas.

He argued that one of the problems facing designers and their clients is nomenclature: there are no clear demarcations of expertise, with self-described design consultants, graphic designers and interaction designers all swimming around in the same design pond. To make matters worse, designers do not themselves use a common language, and concepts are not uniformly understood, so there is little basis for expressing evaluations. He said that graduates from his courses sometimes had a hard time explaining to clients what exactly it is they do and the value being added.

In contrast, designer Jack Schulze claimed that we don't really need tools to tell us if design is any good. But one problem Schulze pointed to was the practice of design agencies 'borrowing the language of design to support business decisions'. The graphic language of the Bauhaus is bandied about without any of the original principles being applied, for example.

People will use things despite poor design, Schulze argued. 'Often things that work on the web are not "designed" at all,' claimed Schulze, pointing to the success of eBay, Hotmail and Amazon. Schulze perceived poor usability to be an engineering issue, not a design problem. Although he conceded that both functions were on occasion the responsibility of the designer: 'I would like to believe that as a designer that is not my role'.

Perhaps this perspective arose out of a narrow view of the meaning of usability, a merely mechanical application of deterministic criteria. The apex of usability, according to Schulze, would be that 'the same product would emerge from two agencies' given the same brief, whereas an indication of good design is that 'you can see something of the designer in the product'.

Schulze saw little sign of consumers picking up the usability agenda. Continued poor usability had not halted the sales of that most notorious of badly designed consumer goods, the video recorder, he argued, with most consumers continuing to choose recorders based on functions offered. The grounds for improving such designs are not in place if consumers continue to buy these products, he said.

Suddenly finding herself described as involved in engineering, Usability News editor Ann Light was quick to return the goalposts to their former position: 'I've come to talk about Design, and what Jack's been talking about is Art,' said Light, who is a judge in the Brighton and Hove Virtual Web Awards as well as being an academic researcher.

While evaluation of websites is very important to their owners, Light believes that it is more important that designers evaluate their own work than for outsiders like herself to do so.

Light argued that breaking down design into its component parts is essential if we are to learn from experience. 'The moment you want to repeat the success of a design, then you need to know which parts of the design contribute to or achieve that and which are just there for the ride.'

She also described the role of the researchers – who unlike designers has the time to consider these issues – as key in the development of tools to evaluate effectiveness: 'But they need direction, and there needs to be more co-operation between designers and researchers.' Furthermore, better paths are needed to deliver the results of research back to the world of practice, enabling designers to pick up on and apply the findings.

Following up Light's point about the importance of designers evaluating their own work, Meriel Lenfestey, founder and MD of Flow Interactive argued for internal evaluation based on a consideration of the quality of the design process.

'We need to define what we mean by design: the verb design versus the noun with a capital D,' she argued. For the former, there are usability methodologies and benchmarking against set criteria. The latter can be evaluated with design awards, and achievements against stated objectives within the team.

Lenfestey also contrasted the positions of the designer per se, and the design or development team, incorporating a range of job roles. Being a good designer involves being able to produce ideas, facilitate and problem solve, she said; the team process is more concerned with communication. But nobody is evaluating the quality of the team process, she said, echoing a point made by Schulze that the dialogue around the product is more important than the product itself.

A point made by several speakers was that design needs to be evaluated against the brief. Light accepted that external evaluators would not get the design brief or sales figures, but regarded it as inevitable that in evaluating a website, the evaluator would be considering what the producer had intended or what the evaluator thinks the producer had intended and what the user wanted to do. Echoing this, Lenfestey argued that design succeeds in relation to criteria set by the sponsor, whether this is to make money, to entertain or to inform.

Money talks and the real measures that we should be using to evaluate are sales and financial returns, argued Schulze. But Phil Barrett of Flow Interactive questioned the ability of such measures to assess effectiveness of experience design involved in such products as pop videos, which perhaps – according to Light – might need to be judged as a thing of beauty rather than products requiring a direct financial return.

As Macdonald pointed out in his introduction, the debate was lacking a client perspective – unfortunately, none that were approached had stepped forward to provide their views. And, ultimately, it's the client with his or her chequebook that ultimately stands in judgement of the success or failure of design.

The London Hopper Colloquium – here come the Girls
Source: UN, 17 May 2008
 
Falling numbers of women in computing is causing headscratching across the industry, but you wouldn’t think it judging by attendance at the recent Hopper Colloquium in London.

How can we improve Mobile User Experience?
Source: Mobile Entertainment, 16 May 2008
 
A roundtable of forward thinkers convened in London last month to debate mobile user experience in advance of the forthcoming MEX conference.

HCI for Community and International Development
Source: UN, 15 May 2008
 
Emerging economies, developing countries and the challenges of designing across cultures came under scrutiny at a CHI 2008 workshop on Community and International Development.

The Future of social networking: Mobile Phones
Source: Times Online, 14 May 2008
 
The future of social networking is the coming together of internet-connected mobile phones and location or proximity technology.

Usability for Fun and Profit
Source: The Herald, 13 May 2008
 
An interview with Chris Rourke, owner and director of Edinburgh-based consultancy, UserVision.

Caroline's Corner: Usability of Content is Plain Language
Source: Caroline Jarrett, 12 May 2008
 
An exciting thing happened in the USA on 14th April 2008. It didn't quite manage to make it onto the national news - that day, we were mostly hearing about the Pope's visit to the USA. Any ideas? Any clue from my title? Give up?

HP Labs opens doors to Academia
Source: VNUNet, 10 May 2008
 
HP has unveiled an initiative allowing academic institutions to collaborate with HP Labs in joint research through an open and competitive process.

Email: is it time to get some Training?
Source: UN, 9 May 2008
 
How hard can it be? Apparently lack of email training is costing employers dearly.

Book Review: Mental Models by Indi Young
Source: UN, 8 May 2008
 
A practical, readable and relevant account of key processes in the vital task of researching, building and applying mental models to product design.

CHI '08: What makes a really Great Designer?
Source: System Concepts, 7 May 2008
 
What qualities do elite designers share, and what do they know that other members of the HCI community could benefit from?