Feature: The Return on Investment (ROI) for Personas

Extract from "The Persona Lifecycle: Keeping People in Mind throughout Product Design" by John Pruitt and Tamara Adlin. Printed with permission from Morgan Kaufmann, a division of Elsevier. Copyright 2006. (Certain boxes and other material and references to these have been removed for ease of presentation in the UN format.)


The complete persona lifecycle positions your persona team as the “first in/last-out” members of the product development team. You will be first in as you collect and express data about target user populations to your executive team to support their strategic work. You will be last out as you help manage the transition from the end of one project to the beginning of the next. In this sense, this last phase of the persona lifecycle is both critical and, at present, too often ignored.

For a variety of reasons, persona efforts tend to peter out rather than end in a managed, measured, and organized manner. Consultants are usually not paid to stick around long enough to manage the personas at the end of a project and in-house teams are usually more concerned with ramping up for the next project than they are with tidying up loose ends from the previous one. Being first-in/last-out on projects means that you will probably end up with responsibilities that straddle two projects. You will be completing your work on project A even after you have begun your work on project B. That is no simple task. It is certainly easier to simply move on to project B. However, we argue that an organized approach to measuring and managing the end of a project can yield significant benefits.

The final persona lifecycle phase is about measurement, regaining control of the persona effort as a whole, and preparing for the future. As the leader of your persona core team, you have two primary tasks at the end of your persona effort:
* Measure the lifetime achievement of your personas (their value), including the return on investment (ROI) of the persona effort
* Manage the organization’s transition to a new project with regard to UCD and target audiences, which will involve reusing, retiring, or in some way reincarnating your personas.

LIFETIME ACHIEVEMENT: MEASURE THE RETURN ON INVESTMENT (ROI) OF YOUR PERSONA EFFORT
One of the most common questions we are asked about personas is if they actually work. And if they do, how can you tell if they were worth the effort? Answering these questions is difficult. What counts as proof of the method is different for different people, products, and companies. There is no single case study or research study that proves their effectiveness rigorously. However, as you have seen throughout this book, there are hundreds of little examples of their effectiveness and value scattered about the industry. This is why this section includes so many stories from the field describing success stories from persona practitioners.

Measuring the ROI of user experience work
Products fail for many reasons, and it is the same for their success. As Mike Kuniavsky [2003] points out, no one would argue against the assertion that in principle you should “build products with users in mind.” However, how that principle is translated into reality can cause quite a bit of discussion and disagreement. Whether or not the end product does better in the marketplace because you had the user in mind is difficult to verify.

Measuring the ROI of user experience work, and particularly of personas, is tricky, but it is not impossible. Measuring ROI depends on being able to clearly express the costs and benefits of the effort, neither of which is easy to express when it comes to most user experience work.

In many cases, UCD professionals know we have done well when the solution we helped to design looks obvious to others. In a sense, we know we have done our best work when it looks like we were never involved at all—when it does not seem like the product should (or could) have been designed any other way. It is difficult to imagine, let alone measure, the problems that might have arisen had the design been done differently. In addition, because successful user experiences support user goals, exploit and are constrained by technology, and promote business goals, it is difficult to isolate the ROI of any one factor that influenced the final design.

Bias and Mayhew differentiate between trying to cost justify any usability work versus trying to influence which usability work will be the most effective. They claim that in “more enlightened” organizations the question is not whether to do usability work but “how much resource to expend and how to apply that resource (i.e., what sort of usability engineering techniques, employed when and where, will prove most cost-effective?)” [Mayhew and Bias 1994, pp. 321–322]. If you have used personas in conjunction with other UCD techniques (which we hope you have), this perspective is very helpful.

There are many good resources to help you measure the internal benefits of your entire UCD initiative. We have cited several already. For a few additional resources, check out the lists supplied at:
* www.deyalexander.com/resources/roi.html
* www.rashmisinha.com/useroi.html
Forrester Research has also published a series of reports on design and ROI that you can purchase. See, for example, the following:
* How to Measure What Matters
* Get ROI from Design

As these and other resources point out, measures of product success can come from many sources. Obvious metrics include revenue or sales, abandonment rates, and completion of specific strategically defined user scenarios. However, many of the resources on usability and ROI also recommend measuring increased productivity and other benefits experienced by end users and listing these as proof of the worth of the initiative. If your organization is already dedicated to creating products that are easy to use, measurable improvements in the user experience stand on their own as an expression of the value of your efforts. However, these measures of increased ease of use are usually much more powerful if they are related to the fiscal bottom line.

The importance of being able to measure and articulate the ROI for UCD methods will only increase as the field of UCD continues to mature. Measuring ROI is not easy in any discipline, and it is particularly challenging (and context dependent) in fields such as UCD, which produce important results that are inherently difficult to measure and quantify (e.g., communication and process improvements). We hope that the suggestions following will be useful whatever the situation you find yourself in, whether you are using personas to introduce UCD methods into your organization or attempting to change the way your organization approaches UCD.

Measuring the ROI of the persona effort
The work you do during this phase to prove the ROI of the persona effort, both quantitatively and qualitatively, will empower you as you continue to introduce customer-centered methods into your organization. Now is your chance to help your organization understand why it was so critically important to understand users and their goals before diving into product design or redesign (and thus why your work as a user experience professional is valuable). In the new edition of Cost-Justifying Usability [Bias and Mayhew 2005], contributors Wilson and Rosenbaum refer to three distinct measures of ROI:
* External ROI: Measurable ways UCD work helped make the company money
* Internal ROI: Measurable process improvements and savings in the organization
* Social ROI: Perception that the UCD team and their methods were helpful during product development, whether or not this helpfulness is measurable.

We think this is a helpful way of thinking about the ROI of personas, because we believe personas can help you create a product that is likely to:
* Be more successful and require fewer support costs (bottom-line improvements)
* Help you streamline your design and development efforts (process improvements)
* Improve the way your company communicates about and focuses on your users.

The work you did during the family planning phase will help you measure ROI [see book]. During the family planning phase of the persona lifecycle we recommended that you do some “organizational introspection” to understand current product- and process-related problems you might be able to solve with your persona effort. In addition to identifying ways in which your products could be more user centered, we suggested that you ask the following questions:
* How user focused is your company?
* How does your organization think and communicate about users?
* How is user information incorporated into the product design and development process?

In the process of answering these questions, you probably collected information about how your company measures the success of its products, projects, and people. During your ROI examination you can research these questions again, now that the project is finished, to discover changes that occurred as a result of the persona effort.

You can also use aspects of your persona action plan from family planning to identify good questions to ask as you evaluate the ROI of your personas. For example, to create your action plan, you and your core team decided:
* What resources do we have for personas and other UCD activities?
* Which product problems do we want to solve with personas?
* Which process problems do we want to solve with personas?
You also defined the scope and goals for your persona effort by filling in the blanks to phrases such as the following:
* The personas we build will help the organization to ______.
* The personas we build will help the persona core team members by ______.
* The personas we build will be used by ______.
* The personas we build will be used for ______.
* The personas we build cannot possibly solve ______.
* We will know that the persona effort was successful if ______.

The work you did during the family planning phase is highly related to what you will do in to assess the lifetime achievement of your personas. If you can apply quantitative values to the answers, you can illustrate the ROI of your personas.

It is time to decide whether or not you satisfied the goals you established in your action plan. You will need to go a bit deeper than just stating whether or not you met your goals. If you did satisfy some (or, even better, all!) of the goals in your action plan, you can work to express the benefits of achieving these goals in quantitative and/or qualitative terms. We believe you can express the full range of persona-related ROI by answering the following questions.
* How much did your persona effort cost?
* In what measurable ways did the personas improve your product? (related to Bias and Mayhew’s external ROI)
* In what measurable ways did the personas improve your design and development processes? (related to Bias and Mayhew’s internal ROI)
* Is your company more user centered with personas than it was before? (related to Bias and Mayhew’s social ROI)

Consider these four questions while looking back at the resources and goals your team identified for your persona effort during family planning.


References
Bias, RG and Mayhew, DJ (eds) (1994 and 2005), Cost-Justifying Usability, Boston: Harcourt Brace and co.

Kuniavsky, M. (2003). Observing the User Experience: A Practitioner’s Guide to User Research. San Francisco: Morgan Kaufmann.

Why 'The 10 Commandments of Web Design' are Complete Baloney
Source: SitePoint, 5 July 2008
 
Further to our earlier discussion...

Local Council Websites: Good, but no Cigar
Source: Webcredible, 4 July 2008
 
The 'Local Council Websites: Good, But No Cigar' report evaluates the top 20 Socitm council websites against 20 best practice usability guidelines.

Why does the OK Button say OK?
Source: Gerry McGovern, 3 July 2008
 
Most times I come across the OK button, something not-OK has happened.

The 10 Commandments of Web Design
Source: Business Week, 2 July 2008
 
To try and make sense of it all, BusinessWeek.com canvassed a broad range of Internet luminaries to discover the design rules they live by right now.

Bill Gates' email rant at the Usability Team
Source: Crunch Gear, 1 July 2008
 
The Seattle P-I has tracked down an old e-mail from 2003 that Bill Gates sent out to members of the Windows Usability team.

Can a better User Experience also be a Greener Experience?
Source: Catalyst Resources, 30 June 2008
 
By improving the user experience of its web collaboration software, Catalyst made it easier for clients to work in productive teams without travelling the globe.

The Spread of Telepresence Technology
Source: Tech News World, 28 June 2008
 
Rising energy costs and green concerns are fuelling a huge increase in demand for videoconferencing, collaborative software and virtual telepresence technology.

UK Usability market worth more than £200 million by end of 2008
Source: eConsultancy, 27 June 2008
 
The UK Usability market will grow by an estimated 20% in 2008 to a value of £214 million, according to research published this week by e-consultancy.

Research suggests growing importance of Usability to UK Businesses
Source: Business Strata, 26 June 2008
 
Integrated software for lead generation and marketing is valuable, but only as long as it's usable.

Web 2.0: Too Smart by a Half?
Source: Computer Technology Review, 25 June 2008
 
Will the Web 2.0 generation’s epitaph be, “They died with their options open?”