One of the most commonly cited rules of UX research is that just five or so users can uncover 80% of usability issues. I’ve often wondered if that’s really true given the different ways various users interact with their devices.
I’m lucky to sometimes get five users to test a new site’s navigation so I’d eagerly embrace this rule if it’s true. But is it?
In an article titled “A Fundamental Mind Shift for Usability Testing,” by Jared Spool, he notes the rule comes from an article published back in 1990 by the Nielsen-Norman group, and a paper by Bob Virzi called, “Streamlining the Design Process: Running Fewer Subjects.” To quote from the paper directly:“The basic findings are that (1) with between 4 and 5 subjects, 80% of the usability problems are detected and (2) that additional subjects are less and less likely to reveal new information.”
Turns out this rule was derived from looking at 5 user testing studies a loooong time ago. A relatively small number of studies were used to draw this conclusion well before the Internet era exploded. Is this conclusion statistically significant? Turns out, it’s not.
This begs the question: “How many users DO we need to conduct comprehensive user testing?” It’s not wrong to ask that question, but perhaps it’s also not right.
Maybe we should get more curious:
Why are design professionals so quick to buy into this thinking? And what about various user types who use a digital product—wouldn’t different customers have different needs? How are we making sure we cover all of those users? And what kinds of user testing and user research are the most useful? Is this the best way to test and learn?
What if it’s more important to test ideas and concepts, and research new ways of solving problems than trying to catalogue all of the flaws in an existing product or service?
Jared Spool talks a lot about design maturity. It turns out that post-launch user testing is, well, kind of basic. There are many research approaches an organization can take earlier in the design process other than looking at what they already have once it’s done.
We should think more broadly about how to create products and services that people will want to adopt, and how to design them so that they are customer-informed, customer-focused and easy to use.
If you want to have millions of customers, talk to as many customers as you can:
- Conduct research earlier than you think you need to—start by using techniques like surveys and contextual inquiries to pinpoint needs, remotely, and in person
- Test concepts, and prototypes before you bring anything to market
- Use online research tools that allow you to reach scores if not hundreds of test subjects, especially early on
- Refine the concepts, and prototypes, and test againMonitor user behavior after launch, and refine as needed
But if you only have five — that’s better than nothing.