Applying psychology to product experimentation
Are you putting the best foot forward every time you launch a test? Can you be sure that unconscious psychological bias won’t impact your test results?
The answer is almost certainly not.
I’m currently a product manager working on the website at CarMax, but my undergraduate self lived and breathed human psychology — which I’ve now found to be the most applicable part of my formal education. Connecting these two worlds, here are a few things I try to have in mind whenever we are designing a product experiment, keeping those biases in check and improving the chance of success.
1. What are the user’s jobs to be done?
2. What biases and heuristics are users bringing with them?
3. What habits are you trying to change?
4. What does success look like?
What are the user’s Jobs to be Done?
The first question to ask is, “Are we actually solving a problem with this test?” I particularly like to ask myself, “Am I trying to solve for a customer need or a company opportunity cost?” Both are reasonable to tackle — you’re trying to make money after all. The approach, however, is different for each
“Jobs to be done” was coined by Clayton Christianson and they are the reason a customer “hires” your product in their life. E.g., we “hire” Amazon to do the job of getting products to our doorsteps fast and without hassle. Read more about jobs to be done.
- Customer Need: Does this experiment actually solve the need of the customer? Hopefully this is a low-effort and almost unnecessary question to ask of your experiment design, since you should have talked to customers before committing any code to de-risk your experiment. You will nearly always find some small tweaks that will help ensure success by getting designs in front of customers each step of the way. A good way to check yourself is to ask, “Why do we think the customer wants this,” then ask “why” again until you’ve gotten to the core of it. Test along your assumption chain in customer interviews until you have a strong and validated understanding of the user’s true need.
- Business Opportunity Cost: If your experiment assumptions start with a business need rather than a customer’s need, you have a tougher challenge ahead. Your responsibility is to make sure you’re not going against customer jobs to be done, or negatively impacting their ability to complete their goal — either way you are impacting customer experience (e.g., adding friction to the process so you can make more money, reduce fraud, or reduce manual processes). You’ve got to ensure that you position this change in a way that mitigates friction or, ideally, provides true value to customers along their mission.
How can you best help a customer complete their job to be done while also meeting your business need?
What biases and heuristics are users bringing with them?
Sometimes the reason an experiment failed is that elements weren’t grouped or displayed in a way that optimized usability. Put another way, the experiment introduced accidental friction by creating cognitive load. All of experience design is psychology, and there are many heuristics, cognitive biases, and areas of study that can be leveraged when designing high-quality experiments. I like to keep in mind principles that impact decision-making and scannability.
Sunk Cost Phenomenon & Ben Franklin Effect
These are two sides of the same decision-making coin, where effort has already been spent prior to a new decision point.
The Sunk Cost Fallacy describes our tendency to follow through on an endeavor if we have already invested time, effort, or money into it, whether or not the current costs outweigh the benefits. — The Decision Lab
“He that has once done you a Kindness will be more ready to do you another, than he whom you yourself have obliged.” — Ben Franklin
If we take these two phenomena a step further in experience design, the implication is that the amount of effort already invested will impact the likelihood of a user to complete yet another action in service of their goal. If you put a new task toward the end of your experience, it’s less likely to reduce conversion.
This is unfortunately used in a “black hat” fashion frequently.
When helping a user complete their original goal, there is a fine line between ordering the operations in a way that reduces abandonment, and diluting the experience by taking advantage of these psychological rules. Be responsible, be ethical, and don’t do the wrong thing.
The Paradox of Choice & Hicks Law
Especially in eCommerce, the number of choices customers have can be, quite literally, overwhelming. Using these two principles of psychology to frame up your experiments can help you make sure that you don’t put your users in a position where choosing is hard.
The Paradox of Choice states that the more choices people have in front of them, the more likely they are to be overwhelmed, and the less likely they are to choose at all. This concept is complimented by Hicks Law, which theorizes that the length of time it takes someone to make a decision is a direct function of the number of choices they have to filter through.
An experiment is fundamentally an attempt by a business to choose between two experiences by asking the data to decide. When you’re running a simple A/B test, users only ever see one experience. However, if you’re asking users to decide on their favorite option among many, you might sabotage yourself before you ever start. Keep your options limited, frame them appropriately, and do plenty of qualitative research ahead of time. Also consider testing the conversion of individual variants, rather than a single test with multiple options shown to users.
Attentional Bias & Priming
Customers pay attention to the things already on their mind, more than details that seem less relevant. Both explicit and implicit factors have been proven through rigorous psychological research to impact decision-making and focus. There is a ton of literature on this (I recommend Thinking Fast and Slow by Daniel Kahneman as a fun place to start on this topic).
The attentional bias describes our tendency to focus on certain elements while ignoring others.
Priming is the concept that exposing someone to a stimulus, even briefly, may bias their future response to stimuli (e.g. thoughts, associations, behaviors, choices, etc.).
Attentional Bias and Priming impact qualitative experimentation as much or more than quantitative experiments.
Imagine you ask a set of users to read a paragraph of text.
To the first person, you say: “Read this text and tell us how it made you feel.”
To the second: “We’re thinking about showing this to customers, but we’re worried it might make them feel nervous. What do you think?”
To the third: “We’re planning to show this to customers, but we’re worried there might be errors. Can you tell us if you think this is ready for prime time?”
You will, without a doubt, bias the results of your experiment with all of these phrasings. Attentional bias is created by exposing intention, or even by asking the user to read the text (we all know customers don’t always read). Bias cannot be fully avoided, but we can be intentional about the way we ask questions, and filter the resulting insights based upon our understanding of attentional bias. Read more from Teresa Torres about interview technique.
Priming is similar, but generally refers to implicit (or unconscious) associations based on previous experiences. To harness priming, you should understand the end-to-end customer journey and what users have already been through before seeing your experiment.
The last product I worked on included a form with questions, to which accurate answers were incredibly important to the user experience. To guide users toward the right results, we used priming throughout by standardizing the question design, where “no” was the first and default answer in all cases, and “yes” was the deviation from standard. We actually had a failed experiment when we strayed from this pattern — customers kept selecting an unintended answer because they were primed by the format of previous questions. Once we recognized our blunder and reacted, customers were able to accurately complete the form.
Gestalt Psychology
Gestalt is the psychology of processing visual data by the human brain. While often illustrated in optical illusions, the study of Gestalt is also a useful tool in arranging elements in your digital experience.
Make your site more scannable, discoverable, and understandable with the law of proximity and the law of common region. The Law of Proximity suggests that objects that are near each other tend to be grouped together. The Law of Common Region states that objects sharing a clearly defined boundary are perceived to be associated. For example, if you want a chart to be perceived as related to something else on the screen, it doesn’t help to have content between them. Keeping associated content in close proximity reduces cognitive load, improves scannability, and ultimately impacts discoverability.
Read more about applying Gestalt in product design
What habits are you trying to change?
Building on Gestalt with New vs. Recurring users in mind
Having worked on products that see new users every day and on products with small and loyal user bases, the biases to consider are very different and should inform your experimentation design.
First time users are seeing everything fresh — Because they don’t know the feature set like a recurring user, first-time usability is your number one goal. Gestalt is a helpful framework for ensuring there is no learning curve in your application or product. Accomplish this by choosing the right affordances and making sure the most important actions are differentiated and not blending into the background (see Law of Similarity)
An Affordance is the quality or property of an object that defines its possible uses or makes clear how it can or should be used. — Mirriam-Webster
Recurring users form heuristics — When your goal is to help change behavior with customers who are active and recurring users of your product, you need a different game plan. Regular exposure means that recurring users fully understand the current feature set, but might gloss over new features due to “muscle memory”. Creating defined boundaries, interrupting common workflows, and/or introducing new colors are examples of valuable ways to differentiate new features and achieve experimentation success.
Whether you’re building for new or existing users, discoverability is your first assumption to test in an experiment. If people cannot find what you want them to notice, then it might as well not exist. Discoverability is a hard challenge to crack with recurring users. Some best practices exist — using in-app communication with things like tooltips and knowledge bases, using multiple channels of communication to share product updates with users, and placing changes obviously in the path of a common user flow. These are all valuable tools to have in your belt, but always assume that recurring users will gloss over your new feature without noticing something has changed.
With recurring users, consider changing the color of the tooltips, or other indicators over time so that your users never become habituated to automatically dismiss that new feature update.
Opt-in vs. Opt-out Experience Design
Recurring users will not opt-in to change. We get comfortable with the way we are working. People in general are very efficient at working inefficiently — they’ll even tell you they prefer the old way no matter how convoluted it is!
Humans will take the path of least resistance and won’t opt-in to extra work in the short term — even if it means incredible benefit in the long run. This was illustrated in a now-famous study by Fidelity proving that newly hired associates who had to opt-out of a 401k program (un-check the box) participated at higher rates than those who had to actively opt-in to participate (check the box). Another study showed a jump from 37.4% participation to over 85%!
Building an “opt-in” experiment will always have limited success.
Recently, my team ran an experiment where we offered a frequently requested and highly anticipated new feature as a tile on the home page of our app. All of our qualitative feedback suggested that our users would jump up and down in their chairs once we launched this feature…and then, crickets.
We had no usage!
After a week in the market, we talked to some of our customers and heard things like “People around here are creatures of habit. They trust the old way.” and “I’m using it, but other people here aren’t comfortable with anything new.”
This was a lightbulb moment for our team. We re-launched the test with a new group, and this time, didn’t change the user interface at all aside from a tool-tip that new functionality was now included in their experience. A week later, we had higher usage than ever and were getting rave reviews about the new feature. There was no hesitation about it being “new” when it was seamlessly integrated into their muscle-memory (or heuristic-based) workflow.
A final thought — Measuring Success and Confirmation Bias.
When setting up an experiment, decide what success looks like before you ever launch and stick to it! This will help you avoid confirmation bias.
Confirmation bias, a phrase coined by English psychologist Peter Wason, is the tendency of people to favor information that confirms or strengthens their beliefs or values, and is difficult to dislodge once affirmed…Confirmation bias is a result of automatic, unintentional strategies rather than deliberate deception. Confirmation bias cannot be avoided or eliminated, but only managed by improving education and critical thinking skills. — read more
Define your KPI and the threshold of improvement/degradation that determine next steps. For example:
KPI: Improve clickthrough rate
Threshold: 3%
If it passes (+3% or more) — roll out to production
If it’s flat (+/- 2.9% or less) — run another variant
If it fails (-3% or more) — keep the current experience
Understanding these concepts has my team create truly fantastic experiences for users that result in seamless decision making and fulfilling end results. I hope this was a helpful primer on keeping biases and natural human action in mind when designing experiments.
Let me know what you think in the comments!
For further reading on the concepts covered check out:
- Laws of UX — https://lawsofux.com
- The Decision Lab — https://thedecisionlab.com
