Amberlight's Hierarchy HCI Briefings Library Amberlight Whitepapers

Testing with Real Users

Usability Testing with real users involves observing actual target users or existing users of an interactive system interacting with it by performing a set of realistic or representative tasks. This is done to evaluate whether the system works: whether it has met its design goals, its performance targets and, at the simplest level, whether users can actually complete their tasks.

Benefits of testing with real users

Usability Testing leads to changes in system architecture and functionality, which can quantifiably contribute to business goals such as customer reach, retention and conversion. There exists a general set of benefits related to Usability Testing for interactive B2C systems (e.g. websites):

1. Improved navigation to help users complete key tasks.
2. Increased conversion rates with the removal of barriers to purchase/registration.
3. Improved customer experience of the client brand as a whole.

Usability Testing with real users can also help to reduce costs by detecting problems early in the development process, identify issues related to different user groups and help to avoid internal, departmental biases affecting important design decisions.

Types of testing

Not all usability tests are the same. The recommended approach to testing real users depends on the goals that are set. This goal may be to collect quantitative performance data (workload, error rates, task completion times, etc.) in order to compare competing designs or user performance over time, or it may be to determine whether any problems or issues exist on a poorly performing system (e.g. poor Web sales). The structure of the user sessions needs to be designed in a way that best achieves the testing goals. The testing approach also depends on what is being evaluated (a user’s workflow on a legacy system, a paper-based prototype, a high fidelity prototype or a fully functional system, and so on).

With some products, such as mobile phones, it may be more appropriate to conduct a longitudinal study to collect data about usage and related difficulties over time and in the natural context of use. At other times, discrete sessions may be sufficient (e.g. can a new user locate the phonebook application and enter the name and number of a contact on their first attempt?). Some types of testing can be carried out in custom-built labs, where the facilitator has more control over the testing environment, while others require a more realistic setting (e.g. in the user’s home on their own computer).

General information on selecting the most appropriate methods for Usability Testing:

• http://www.userdesign.com/usability_uem.html
• http://www.usabilitynet.org/tools/methods.htm

Why use professionals?

Any type of real-user testing is better than doing nothing or second-guessing user behaviour. However, there are certain types of biases and distortions that occur in various testing scenarios, which the facilitator must be familiar with in order to accurately interpret the session data with confidence. A user might spend 10 minutes trying to complete a simple task, get lost, confused and frustrated, and fail to complete the task at all, but when asked about how they felt the task could be improved answer that they are completely happy with it as it is. There are also cultural effects associated with some of these biases. For example, in Korea users are reluctant to publicly criticise the work of others, which gives rise to inaccurate data. There are practical tips and tricks for dealing with most of these issues, and interpreting, analysing and prioritising user responses. In most cases it is best to leave the planning, facilitating and analysis of real-user testing to trained professionals.

Outputs of testing

The most appropriate combination of Usability Testing outputs or deliverables depends on the goals of the project and how the information will be used within the client organisation. Common deliverables include comprehensive problem logs, prioritised findings, proposed solutions, summative performance metrics (e.g. ISO 9241, error rates, task completion times) and picture-in-picture video clips (simultaneously showing the user and their on-screen behaviour).

Standards and guidelines

Ergonomic requirements for office work with visual display terminals ISO 9241 (Part 11: Guidance on Usability): provides guidance on defining and measuring the usability of a system. These metrics can be applied to score task performance during Usability Testing.

Ease of operation of everyday products ISO 20282-1 (proposed standard): guidance on identifying context of use and variance of abilities within the intended user population for the purposes of designing and evaluating everyday products.

Ease of operation of everyday products ISO 20282-2 (proposed standard): describes a test method for measuring the extent to which an everyday product is easy to operate.

UK Usability Professionals Association: provides guidelines on how to select the most appropriate Usability Testing supplier.

ANSI/NCITS 354-2001 Common Industry Format for Usability Test Reports. This document outlines a standard format for presenting the results of usability tests.

Technologies for user testing

A number of technology solutions exist for supporting the delivery, recording, facilitation and analysis of usability sessions, and the communication of results. These include video capture of screen and user, eye-tracking software, biometrics (heart rate, pupil dilation, skin conductance, muscular tension, etc.), remote Usability Testings and a host of specialised software packages for capturing and analysing behavioural data.

Further reading

Rubin, J. (1994) Handbook of Usability Testing. John Wiley & Sons, Inc

Dumas, J. S. & Redish, J. C. (1999) A Practical Guide to Usability Testing (revised edition). Exeter: Intellect Books