Half Web Searchers enter One Query, look at One Page of Results

Researchers at Pennsylvania State University have been exploring the evaluation of search engines, driven by findings about how users typically search. Specific to the latest study, "An Analysis of Web Documents Retrieved and Viewed", Bernard J Jansen and Amanda Spink asked:
(1) How many pages of results do Web search engine users' examine?
(2) How many Web documents do Web search engine users' view when searching the Web?
(3) How relevant are the Web documents that they are viewing?

Actual queries submitted to FAST's All the Web.com from a sample of 1M collected on one day were analysed to determine the number of results pages the searcher viewed. The team also captured the Web document that the user viewed for each query. Then a subset of 530 queries from this transaction log was submitted to a major Web search engine. The retrieved sites were evaluated to determine whether or not they contained relevant materials.

Some clear patterns concerning the number of results pages viewed by FAST users. Approximately 54% of the users view only one page of results. This result is similar to the percentage of users that enter only one query (53%) and the percentage of relevant documents (52%).

The closeness of these percentages indicates several things, say the researchers. 'One, the information needs of a majority of Web searchers are not extremely complex, given they require only one query. Two, Web search engines appear to do a good job of indexing and ranking Web documents in response to queries, based on the majority of users viewing only one results page. Three, it appears that on average about 50% of the documents viewed will be relevant, implying that the typical Web user will have to view about two Web documents. This is supported by our analysis of Web documents viewed, with 43% of users in our sample viewing two or fewer Web documents.'

Further results showed that these Web search engine users on average viewed about 8 Web documents. 'However, our analysis shows that over 66% of searchers examine fewer than five with more than one in three Web searchers viewing only on one document in a given session. Users on average view about 2 to 3 documents per query. Over 55% of Web users view only one result per query.'

The mean time spent viewing a particular Web document was just over 16 minutes. However, 70% of the users spent less than 15 minutes viewing the retrieved Web document. And 30% of the Web users viewed a Web document for less than a minute. These results would seem to indicate that the initial impression of a Web document is extremely important as Web searchers are typically not going to spent a great deal of time combing the document to find the relevant information.

It was also clear that the searchers were making relevance determination based solely on the document summary that is displayed in the search engines results page. One out of every two of the Web documents viewed will be relevant to their query. 'Given the large number of documents that most Web search engines retrieve, fifty percent is rather high. However, note that this analysis is for Web documents viewed, not documents retrieved. This is has great implications for Web search engines and Web page designer,' say the researchers.

The researchers point out the limitations of their work: the sample data comes from one major Web search engine, introducing the possibility that the queries do not represent the queries submitted by the broader Web searching population. However, Jansen and Pooch have shown that characteristics of Web sessions, queries, and terms are very consistent across search engines. Lack of information about the demographic characteristics of the users who submitted queries meant they had to infer their characteristics from the demographics of Web searchers as a whole.

But there is a message from the information gathered. While the searchers' strategies are not those traditionally regarded as successful using the standard metric for information retrieval system performance, it appears that Web search engine users are finding relevant information with this searching strategy, concludes the team. This may indicate the need for new metrics to evaluate Web information systems.

"An Analysis of Web Documents Retrieved and Viewed" by Bernard J Jansen and Amanda Spink was presented at the 2003 Internet Computing Conference in Las Vegas, USA.

User Interviews - Analysis Simplified
Source: Webcredible, 6 January 2009
 
You’ve conducted your user interviews, but now you need to make sense of all that information you’ve gathered.

Why Products Fail
Source: ComputerWorld, 5 January 2009
 
Most gadget and software makers don't understand what users want most: control.

How to Design Websites for Mobile Phones
Source: stemkoski.com, 3 January 2009
 
Tips from Ryan Stemkoski's web design blog.

Pioneer of Cyberspace honoured
Source: BBC, 2 January 2009
 
A professor who invented a forerunner of the world wide web has been made a dame in the New Year Honours.

2008 in Review: Developments that rocked the world of User Experience
Source: Catalyst Resources, 1 January 2009
 
A look back at 2008 highlights some of the key developments that rocked the world of user experience.

2008: The Year Online
Source: MIT Technology Review, 31 December 2008
 
The business of social networking, cloud computing, and a flaw in the fabric of the Internet top the most notable stories of 2008.

Shoveling through the Spamalanche
Source: UN, 30 December 2008
 
A ‘Spamalanche’ of 3,000 emails will be waiting in your inbox by the time you get back to work. What can you do besides 'delete all'?

New guidelines boost Web Access
Source: BBC, 29 December 2008
 
The World Wide Web Consortium (W3C) has announced a new standard to make sites more accessible to older and disabled people.

Measuring Emotional Engagement
Source: UN, 27 December 2008
 
New applications of electroencephalography (EEG) provide the means to gather detailed information on a user’s emotional relationship with a brand or service.

UXalliance thrives on Research strengths
Source: UN, 24 December 2008
 
The UXalliance provides clients with access to over 220 user experience professionals and 40 test labs worldwide.