Feature: Assessing Mobile Devices more Effectively

Most user studies in the past have focused on the use of static devices. What are the new challenges to us, as designers and evaluators, on how to approach the design and evaluation of multimodal mobile devices and applications? In other words, function, value and meaning are relational and not absolute, as the applications and services reside on the network and not on the device. From the point of view of design and evaluation this presents new challenges. But why should we be interested? Perhaps because as IEEE computer magazine recently stated: 'Mobile computing using handheld devices has matured and will become the dominant computing paradigm'(IEEE Computer Magazine, April, 2003).

The future is 'mobiquitous' and that we will be using our mobile devices everywhere and we will be always 'on'. Does this mean, then, that context is no longer important, as we can use our devices in any context, at any time? Not at all, in fact, it is quite clear that 'mobiquity' will have the opposite effect, ie. that the context in which we use a device will become one of the most important factors in determining our interaction with that device. Studies of context are essential to the successful future design of mobiquitous applications and devices, however, what is not clear is how such studies can be undertaken and how the results of these studies can be fed into the design or redesign of an application, service or device.

If one of the main tenets of usability is to understand the context of use, there are additional problems while the user is using a multimodal mobile device, as we don't know when an event will occur, what problems there may be with this setting and what mode of interaction the user will choose. It is also necessary, to understand what the users’ goals are, how they achieve those goals at the moment, and what they require from a future device or application.

Methods
The methods used in our evaluation were chosen because they reflected the current research knowledge in this area. That is not to say that these methods are the only applicable methods for evaluating a prototype application of this nature only that they were thought to be the most appropriate. After looking at several possible methods and techniques for involving users throughout the design process, scenario and participatory design techniques were settled upon. The reason for this was that these techniques are user driven rather than designer driven and have proved successful in the past in unusual and mobile settings

MODE framework
The MODE (Multimodal Mobile Design) framework was set up to manage the processes involving users. The framework has now been trialled on three separate occasions in Scotland and Austria. The framework provides a way in which to involve users quickly and effectively in the design process, including participatory design workshops.

The MODE framework phases and the methods used at each stage:

Phase I Discovering current practices: Reconnaissance
Then, using workshops:
Phase II Facilitating discussion with users: Conceptual Scenario Discussion
Phase III Gathering data about the user wants and modes of interaction: Questions
Phase IV Investigating current problems and future possibilities: Action Scenarios
Phase V Contextualising ideas for the application: User Sketches

Evaluation of a Route Finder Application
The overall aim of one evaluation was to assess the usability of a multimodal route finder application. The device used to access the application was a mobile phone. The evaluation aimed, in the first instance, to assess whether or not the user could obtain the correct routing information. The second part of the study investigated whether the user could undertake the task as easily in a natural setting as they could in a controlled environment.

The application was built as part of the B2 project at Forschungszentrum Telekommunikation Wien (ftw). The project works on the development and evaluation of man-machine interfaces combining voice and visual elements with a view to enabling attractive services in GPRS and UMTS infrastructures.

The route finder works in the following way:
Step 1: The user enters an address using either voice or text.
Step 2: The application locates the address for them.
Step 3: A map is then delivered to the user's device.

The number of users who took part in the study was twelve. Many usability experts believe that twelve is an adequate sample size for an evaluation. The users were split evenly along gender lines. The user's percentage preferred walking speed (PPWS) was first of all captured as it had been found in previous studies that this measurement is very important to the evaluation of a mobile device in context.

The subjects were given four tasks to complete while static and mobile.
Task 1: Please choose any modality you like to find an address.
Task 2: Please use voice interaction to find an address.
Task 3: Please use text input to find an address.
Task 4: Please choose any modality you like to find an address.

The evaluation took place at the research centre; there were two settings in which the user undertook the tasks:
1) In a room free from interruptions and noise (the user could sit or stand whilst undertaking a task).
2) In the research centre building and adjacent to it.

It was hoped that we would discover additional information about the user's interaction with the application by undertaking part of the study in a natural setting. It was also anticipated that we might discover differences between the user's choices of interaction with the device when they are moving, as opposed to when the user is static.

The users, in the main, completed all the tasks given to them. There were some problems with the voice recogniser taking slightly longer to recognise the commands from a non-native speaker. There was no significant time difference between those undertaking the tasks in the laboratory than in the natural setting. If a user had taken longer than average to complete a task we studied the video and notes taken during the evaluation to try and understand what was happening at that time. The number and type of errors made by each user also helped us to understand more clearly the problems a user was having with the application.

Undertaking part of the evaluation in a natural setting turned out to be more of a challenge to the evaluators than to the users, as they had to carry cameras, notes and so on. The users felt comfortable wandering through the research centre building and wandering outside. The users were free to wander where they pleased and some wandered around quite a lot and some only a little, a few even decided to walk to the nearest underground station! Another finding was that users became more quickly frustrated with the application in the more natural setting than in the controlled setting.

One of the reasons for undertaking part of the evaluation in a natural setting was to see if users used different modalities in this setting than in the constrained one. We found that the users did use different interaction modes when mobile than when static and could articulate the reason for their choice:

Researcher: Can you say why you chose that mode of interaction just then?
User: Hm..m it just seemed to me that when I was walking down the stairs, that I wanted to be able to see fully my surroundings and not be trying to text at the same time as walk, so I chose to use the voice mode instead.
(Excerpt from Evaluations CM.1.4.03)

This was a typical comment made by the users. This was an interesting finding as we had not been sure how and in what way the more noisy environment, ie. lifts going up and down, people walking in and out of offices and going past on bicycles, would effect the users choice of modality. Our finding here was that the mode of interaction was heavily dependent on the users surrounding terrain.

Action Scenarios
In an effort to elicit more information from the users we gave them the following scenario: 'One of your colleagues at work is having a party in their apartment. While trying to make your way there you realise that you are lost. You need help to try and find the apartment and think that the multimodal route finder on your mobile device may be able to help.'

By linking a scenario with physical activity we hoped to connect the users learning of a new tool (the mobile application) with a practical activity. We encouraged the users to move freely while undertaking this task. We hoped that by doing this the user would be able to provide a richer description of their interaction with the application.

After they had finished working through the scenario we asked them to show and describe to us the different menus, icons and so on that had annoyed or hindered them. We also asked them to tell us how they would structure the task, menus and icons differently.

The user was videoed and the evaluator took additional notes with paper and pencil. The video was analysed using conversational analysis techniques and evaluators noted incidents.

The Action Scenarios were the most useful part of the evaluation in terms of aiding us in our understanding of the user's view of the application. The users could clearly articulate to us the problems they were having with the menu structure. For example, the application structured the input form in the following way: district, street and house number. Whereas the users wanted the order to be: street, house number and district. The users also pointed out that some landmarks that they were used to seeing in real life would help to orientate them when using the maps, such as underground stations or chemist shops. In the voice recognition part of the application, the users wanted to be able to say 'help' after each sub-task rather than wait till they had completed the whole task to say that part of the address was wrong. These and other problems highlighted by the users aided us in our thinking when we came to redesign the application.

Discussion
Undertaking the investigations took much time and effort from the HCI researcher, as well as the more engineering orientated staff (eg programmers and systems architecture engineers). However, it was felt that this time was well spent as we, as a project group, had some very fixed ideas about what a user would want. It took the results of the framework to convince us that in, most cases, we were wrong. However, turning the participants' comments, descriptions, sketches and action scenarios into concrete design decisions was challenging. This is something that has proved problematic for the HCI research community.

The findings helped us to create the first scenario and application design document. For example, from the user data we were able to elicit the following:
• A list of the essential elements of the applications
• A list of the optional elements (we decided that we would aim to provide these elements in the 2nd iteration of our applications)
• The user goals
• An iteration road map (what should be included and working at the first stage and the second stage of the application development)
• Preferred modalities in certain contexts.

We were also able to move from the conceptual scenario phase to the concrete scenario phase. The list above aided us in our writing of use cases.

It can be concluded from the findings that users can take an active part in an evaluation of a multimodal mobile application and can provide valuable insights into its use. That undertaking a study in a natural setting is more of a challenge for the evaluator than for the user. We would conclude, however, that it is still necessary to carry out studies in a natural setting as it gives valuable insights into possible uses and interactions with the application. The action scenario descriptions aided us in our thinking about the redesign and proved to be a valuable tool.

We hope that the studies we are conducting at ftw are going some way to include notions of context in the design of applications, however, we realise that these are quite experimental and limited studies. We are currently grappling with two problems in this area:
1. How do we conduct useful and usable evaluation studies of how users learn to use our applications when we cannot be in the context or an approximation of it to see how the user deals with 'real' problems when they encounter them with our application?
2. How could the results be formulated so that they can inform others how to design a multimodal mobile application, device or service?

Drawn from Lynne Baillie's work at Forschungszentrum Telekommunikation Wien, as written up in "Gathering Requirements for the Design of Multimodal Mobile Applications", presented at the Mobile and Ubiquitous Computing Workshop: Designing for Ubiquitous Computing in the Wild: Methods for Exploring the Design of Mobile and Ubiquitous Services, and a short paper for HCI2003.

Associated Link: More details of Lynne's work

User Interviews - Analysis Simplified
Source: Webcredible, 6 January 2009
 
You’ve conducted your user interviews, but now you need to make sense of all that information you’ve gathered.

Why Products Fail
Source: ComputerWorld, 5 January 2009
 
Most gadget and software makers don't understand what users want most: control.

How to Design Websites for Mobile Phones
Source: stemkoski.com, 3 January 2009
 
Tips from Ryan Stemkoski's web design blog.

Pioneer of Cyberspace honoured
Source: BBC, 2 January 2009
 
A professor who invented a forerunner of the world wide web has been made a dame in the New Year Honours.

2008 in Review: Developments that rocked the world of User Experience
Source: Catalyst Resources, 1 January 2009
 
A look back at 2008 highlights some of the key developments that rocked the world of user experience.

2008: The Year Online
Source: MIT Technology Review, 31 December 2008
 
The business of social networking, cloud computing, and a flaw in the fabric of the Internet top the most notable stories of 2008.

Shoveling through the Spamalanche
Source: UN, 30 December 2008
 
A ‘Spamalanche’ of 3,000 emails will be waiting in your inbox by the time you get back to work. What can you do besides 'delete all'?

New guidelines boost Web Access
Source: BBC, 29 December 2008
 
The World Wide Web Consortium (W3C) has announced a new standard to make sites more accessible to older and disabled people.

Measuring Emotional Engagement
Source: UN, 27 December 2008
 
New applications of electroencephalography (EEG) provide the means to gather detailed information on a user’s emotional relationship with a brand or service.

UXalliance thrives on Research strengths
Source: UN, 24 December 2008
 
The UXalliance provides clients with access to over 220 user experience professionals and 40 test labs worldwide.