Identifying the best design option like a scientist
A case study of using unmoderated click tests to evaluate the usability of alternate feature designs, co-authored by Alessia Romano and me.
Small design choices can make or break a solution. The right icon, placement or copy can help users find your feature and understand what it might do for them. Bad choices consign your hard-discovered solution to confusion and lack of adoption.
In the UI below, we showcase a ‘’filter’’ option on the menu in three different ways. One of the designs is the most discoverable and well-understood by users by far. Can you guess which one?
A few months ago, Alessia, a Designer, established best-practice at Gousto for quantifying a feature’s visibility and understandability. Building on the great visual testing thinking found across the world, she turned the process of choosing the most impactful design from many similar options from gut-feel into a science. And the best thing, you can get great signal on the most impactful design within a day!
The design challenge we were facing
Gousto is the UK’s largest recipe-box company. We sold over 1m+ meals per week in 2020. Customers can choose from 60+ recipes every week, and we’ll send their choices to them in a beautiful red box. Our digital product team’s primary goal is to help customers navigate this choice and pick the recipes they want to build a perfect box.
One of the top opportunities we explored this year was how to improve our “find a specific recipe” experience — where customers look for recipes with specific criteria in mind. When looking to “find a specific recipe”, customers wanted to access certain groupings (like cuisine, dish-type, etc.) they cared about or did not see recipes that included disliked ingredients. But we could not offer this via the current categories (for reasons we won’t get into now).
We believed we could solve the problem by introducing filtering functionality on the menu — i.e. by toggling drop-downs, they could cut the recipes list by criteria like ‘’cuisine-type’’ or “ingredients”.
So far, so good. But what was the right way to surface the filters feature? Our menu experience is already crowded, and people were historically confused when we introduced other new buttons. We had ideated seven different options for visualising the feature. But we were unclear how to identify the most impactful one without running loads of moderated user interviews or an arduous and slow A/B testing process.
Enter the unmoderated multivariate click test™.
We showed a static menu page with one of the seven variants to groups of c.50 users (350 users in total) and asked them to complete just one task. We wanted to choose an easily understandable task to share without giving any hints. So we decided to use an ‘’allergic to mushroom” use case for the study.
“Imagine that you are allergic to mushrooms and you want to exclude all recipes that contain them. Where would you click to do so? You may only click once”
For each design, we identified a ‘’success area’’ for a first click, which Bailey & Wolfson show is a strong indicator of eventual task completion.
We then monitored their task success rate for each group of 50 users and also asked them the question, “How confident are you with your choice? (From 1 (not at all) to 7 (very confident)”. See the results below for the three designs we shared at the start of the blog.
Ding ding ding!!! Our second contestant is a winner! Statistically significant, even with 50 results, the middle design is flying.
If we’d launched the first design instead of the second, our adoption rate might have been up 30% worse than the more optimal design, or more!
Note, sometimes results might not be statistically significant and may only be indicative. But often, we’ve found this click test can eliminate ‘’bad’’ treatments and highlight a ‘’great’’ treatment that stands out from the others. (False positives with less data does apply though, treat results with caution blah blah blah).
NB:// for a more detailed exposé of different types of questions to ask in click tests, check out UserZoom’s helpful blogpost here.
What this method won’t tell you
Unmoderated click tests can help you cut through the noise around design variations in a day. But they are only part of a larger puzzle to solve when creating an easy-to-use product. These tests help you evaluate Visibility, Affordance (partly) and Expectation. But they do not tell you:
- Whether the pop-up or action that happens after users click on your design is apparent to the user (Feedback)
- Whether your solution is the right solution when considering the broader ecosystem of actions people can take on a page (Constraints)
Hence, we consider it just one tool in a much bigger toolbox.
Our love affair with unmoderated testing
And speaking of tools and toolboxes, nothing increased our capabilities more in Product Discovery than getting an unmoderated user testing platform (in our case, UserZoom). We get high-quality, quantified, insights in days, instead of weeks (see why we move fast here). Working at a consumer-facing company without unmoderated testing is like going cave diving without a flashlight. You’re going to be some cave-dwelling barracuda’s lunch.
5-second tests, longer recall tests, and card sorting are all methods we’ve used with great success to zero in on optimal solutions. Without them, we’d be lost.
OK… Love rant over. Stay tuned for more of our applications of unmoderated testing soon!
One final shoutout. A project we undertook with Pattrn Studios was a big catalyst for a lot of the great thinking we use across Gousto (include this!). Would highly recommend them as an agency if you’re looking to step-change your thinking!
