What are usability tests?
Testing the user experience: how working with real people can improve your website.
Anyone who designs a digital product will, sooner or later, need to find people who use it. That is the moment to put the design to the test with real users, a moment that should never be missing from the development of a product, because the improvements that follow make it more usable and closer to people.
Before we go on, a few clarifications are needed. When people talk about “testing the user experience (UX)”, they end up using many definitions (first among them, “usability test”) and referring to many different kinds of test, depending on what is being tested and how the test is built.
To avoid confusion, at Conflux we prefer less common but more precise definitions, such as “interactive test”: a research tool that puts an interactive product in front of people, for example a high-fidelity prototype or a site or app that is already live. In this way we avoid any mix-up with tests on static screens (for example a first click test or a first impression test), where interaction with the product is not possible.
“User test” is also a very common phrase, but it risks suggesting that the object of the study is the person, when in fact it is the product (and the people taking part should be reassured of this throughout).
The goals of a usability test
The ultimate goal of a usability test is to find as many critical issues as possible in the product, thanks to feedback from people who are asked to try using, for example, a prototype.
The issues that emerge become pointers for fixes and opportunities for improvement.
“Usually, in a usability test, there is a participant who carries out tasks and a user experience researcher who moderates the session.”
How do you run a usability test?
Usually, in a usability test, there is a participant who carries out tasks and a user experience researcher who moderates the session, observes how people behave and collects their feedback. In an ideal setting there is also an observer who focuses entirely on collecting feedback. All of the data is tracked and recorded, including with dedicated software.
“How many people should you involve?” is a question on which dozens of articles could be — and have been — written.
Staying with our own practice, we like to involve ten participants, for several reasons:
- The chance to split the group into two segments of five people and compare them
- The chance to treat the first session as a “pilot” and still obtain sound results
- The reassurance of protecting the research even if some participants drop out
The structure of an interactive test is fairly flexible and can change with the research objectives. It is important to keep in mind that a test can produce both quantitative and qualitative insights, and that, depending on the goals, you can give more weight to one or the other with different methods and tools.
Methods of quantitative research:
- Task analysis: we ask the person to complete a series of activities using the product. This is the method that gives us measurable results: the task success rate, completion time and the number of errors
- Standardised questionnaires at the end of the test or after each task (such as the Single Ease Question (SEQ) or the System Usability Scale (SUS)), which use shared benchmarks from the research community and are therefore solid points of reference
Methods of qualitative research:
- We can ask the participant to comment on what they see and what they are doing (so-called “think aloud”), so that we can record spontaneous reactions and reconstruct their mental model
- Retrospective: at the end of the test we go back with the person over the moments we most want to understand and discuss the results. A retrospective does not focus on the context of use or the user journey; it is a good alternative to think aloud when we do not want to affect the metrics of the task analysis
- We can add an interview at the beginning or at the end. This part matters a great deal if the focus of the study is the context of use of the product or the user journey. Listening to people also makes it possible to gather insight on brand perception, on how pleasant the user interface (UI) or the chosen tone of voice feels, and on how to improve the relationship between the brand and the people who use it.
In our experience, a usability test should include an opening interview, a central task analysis on the prototype or product, and a closing round of standardised questionnaires.
The whole session should not run longer than an hour, which is why it is essential to identify the most important tasks to complete. We try never to go beyond eight tasks.
The insights collected become pointers for a redesign aimed at improving the new product. The next step should be a new test: a clear example of an iterative process.
In-person tests or online tests?
Technology gives us many tools for gathering information and running an interactive test. Another important distinction is whether to run the test in person or online.
In-person tests
A more direct relationship can make it easier to break the ice, and it also offers the chance to study the person’s body language up close. The eye-tracking tools available today are also more effective when used in person.
Online tests
The option to run tests online is now widespread. This reduces costs and makes it much easier to recruit participants, who, taking part from the comfort of their own home, can feel more at ease than in a more sterile laboratory setting. In these cases there is still the issue of a possible digital barrier to overcome, and this is especially true for tests that involve older people or, more generally, people with a lower level of digital literacy.
In conclusion: what does a usability test give you?
However skilled a user experience design team may be, it is only by involving the people who will use the product that you obtain confirmation or contradiction of its actual usability.
Tests with users always hold surprises, which is why research should always be part of a project and should never be ignored or left until last: a usability test on a product in a late stage of development can imply a heavy rework effort, which earlier tests could have avoided. Interactive tests are the most important and effective tool for finding opportunities to improve the product in line with the expectations of the people who will use it.
We also recommend
