Are you testing to test, or testing to prove?
The important difference between these two mentalities
I often hear teams use the term ‘validate’ whenever they conduct evaluative studies (e.g Usability Tests) during the formative stages of the product. This is especially common after investing a substantial amount of time and effort in their work, triggering a natural tendency to defend what they’ve created.
Don’t get me wrong; it’s completely fine to validate and give ourselves a confidence boost for our design directions, provided that it’s intended to. (e.g summative evaluations, utilized after production for benchmarking purposes.)
However, furnishing evaluative study plans with the mindset to validate, when the main goal is to iterate, makes the study lose its purpose. This mentality can be contagious and foster an environment that encourages project team members to plan or discuss things that’ll only legitimize the design deliverables.
That becomes a one-way street toward inhibiting the growth of the product.
Testing to test
UX Research to me have a similar approach to scientific-based studies, which generally involves:
1) Making observations,
2) Formulating a hypothesis,
3) Testing that hypothesis,
4) Draw conclusions based on the data collected.
We’ve got a myriad of tools (e.g. surveys, interviews, usability testing, and observation) each having its pros/cons, and attitudinal/behavioral constructs. These tools help collect relevant data for us to analyze and draw conclusions that drive informed design decisions. This approach allows researchers to investigate human behavior in a systematic, objective, and repeatable way, which are fundamental characteristics of any scientific-based research.
The beauty of UX Research lies in its empirical value; supporting evidence through objective and observable data.
Empirical science sets out to falsify; the capacity for some proposition, statement, theory, or hypothesis to be proven wrong, allowing room for bigger discoveries.
Using the black swan problem as an illustration: If a man lives his whole life seeing swans that are only white, he might assume that all swans are white. For falsifiability, it isn’t necessary to know that black swans exist but to understand the fact that “All swans are white” is just our current beliefs based on observations; the status quo.
One caveat is that while these research techniques have their limitations, it only shows the complexities of studying human behavior, and the need to carefully consider potential sources of bias or errors in the research process.
Testing to prove
To validate something makes it a catalyst for resistance. When that happens, the study loses its scientific value, turning it into pseudoscience.
The word ‘pseudo’ means fake. In proper definition, pseudoscience is a set of statements/beliefs/ideas that presents itself as science but does not meet the criteria of being referred to as such. What’s mostly unfavorable about them is the fact that they’re irrefutable. They cannot be proven wrong, and they exist to validate assumptions.
In this day, it’s hard to deny the common presence of Pseudoscience. From studies like Astrology to metaphysical objects that claim to have certain powers, to certain diets that promise results. Contrary to Science, it attempts to persuade by appealing to emotions, faith, sentiment, anecdotal evidence, or on evidence that is not based on scientific methods.
Conclusion
Now back to the question at hand. When approaching a hypothesis, which mindset should we adopt?
- To test and be open to surprises or discoveries
- To test and ensure that assumptions are correct
In my experience, approaching with the former mindset leads to discoveries and solutions we’ve never thought of before, prior to the study. I for sure am more afraid of the idea of not knowing what I don’t know.
Be it science or design, when conducting research studies, we don’t want to be tainted with the idea of ‘validating’. We ought to attempt to take a neutral stand and show that the hypothesis is just status quo. If test results have nothing to disprove the hypothesis, it thereby strengthens it and boosts our confidence. That’s when we’ll know we’re on the right track.
If you’d take away anything from this read, let them be these:
- If research is conducted with the goal of validating a hypothesis, it will not lead to interesting discoveries.
- This mentality can be contagious, so it’s important to set the tone at the very beginning. If it’s already happening in your team, attempt to shift the mindset.
- Research should be conducted without the intention to prove anything right, and the willingness to iterate with new evidence.
- If we fail to find anything that challenges our hypothesis, it’ll strengthen it instead.
Let’s test to test, not test to validate!
References & Citations
Seeing is not believing
The key lesson from Plato’s Allegory of the Cave is to question every assumption we have about the reality we call “real”. It takes courage to stray away from our comfort zones and break away from status quo.
If you’d like to find out more, do watch the following video!
Thank you for reading my article. My opinions are my own, and I know it’s subjective. Please do show appreciation if you’ve liked what you read! It’ll motivate me to write more.
