Vanity interviews: what can PMs and UX researchers learn from oral historians?

Avoid “vanity interviews” with techniques from academic historians.

Published in

UX Collective

8 min readFeb 6, 2023

From Teresa Torres's Continuous Discovery Habits to Marty Cagan’s Inspired, references to the importance of constant conversations with users abound. The fact is that good product managers interview customers, a lot. But most of them have never been trained to conduct those interviews.

A drawing of two stickman chatting as if in an interview.

This leads to a plethora of vanity interviews around the world. Vanity interviews are done without research and questionnaires. They have poor framing of questions and they annoy interviewees. They are full of false memories and lies. In the end, they are left behind in some random repository or never recorded at all.

As a result, vanity interviews don’t translate into business insights. Worst than that, they lead products to the wrong paths. Here’s where oral historians can help.

First things first: what is oral history? Since I’m not an academic anymore, I now can quote the almighty Wikipedia:

“Oral history is the collection and study of historical information about individuals, families, important events, or everyday life using audiotapes, videotapes, or transcriptions of planned interviews. These interviews are conducted with people who participated in or observed past events and whose memories and perceptions of these are to be preserved as an aural record for future generations. Oral history strives to obtain information from different perspectives and most of these cannot be found in written sources.”

Put into simpler terms, oral history is history made through interviews, akin to journalism (but pickier about details, truth, and consent). Oral historians, then, are well-trained interviewers who have for at least five decades academically discussed the whys and hows of conducting interviews — historians needed recorders before starting to make oral history.

The bad news for PMs is that this is a massive amount of content. The good news is that five decades of research have built some rules of thumb that can instantly be applied to their day-to-day interviews.

The validity of oral evidence

Is oral evidence valid? Are memories true? Can we trust interviewees? Those questions can quickly derail into deep philosophical conundrums. We don’t want that. But still, we need to discuss what we can expect from interviews. Thus, we can stick to the short answers and work from there.

Oral evidence is valid, and great histories have been told because of it.
Memories are not true in the sense that they are not perfect depictions of the past. As Luigi Portelli often quoted sentence says: “What is really important is that memory is not a passive depository of facts, but an active process of creation of meanings” (Portelli, 2010).
No, we cannot trust most interviewees.

Most oral historians would agree with these short answers. As a result, contrary to what their title might suggest, oral historians are extra cautious with oral evidence. The first thing to notice is that oral recollections have proved false several times, and historians have gotten into trouble because of that in the past.

Memories are active beings. They interact with facts that occurred after them and change. False memories, thus, are common. The simple phrasing of a question may induce interviews to have false recollections. The question “how disorganized was your task list before you started your day?” implies disorganization, and it may induce respondents to believe they weren't organized when in fact they were.

There are numerous other mechanisms that induce our brains to create false memories, like trauma for instance. The fact is that everybody has had trouble remembering a name or date at least once, and this is not the only flaw our brains are subject to when trying to remember facts. Consequently, false memories are a reason for historians' distrust of interviewees.

But I don’t want to make a storm in a teacup. Memories may be false, but if we are cautious with framing and are preventively aware of any possible bias our interviewee might be under, we are probably ok. False memories are a big problem for oral historians, but probably not as big for product managers.

Worst than false memories for product interviews, are dishonest interviewees. Dr. House used to say: “Everybody lies”. And this is true. People lie or distort the truth for several reasons, but two special reasons are avoiding embarrassment and creating self-serving truths.

Most products act to solve a pain point. Those pain points might not be the most prideful moments of our customers’ days. For instance, before signing up for a CRM service, most sales teams use Google Sheets or Excel to organize their sales pipeline. An interviewed member of this team could know that their process is flawed and that their conversion rates are below benchmarks, but could opt to inform that they have never had problems with the conversion before. He's just worried about what the interviewer might think. Else, he could just have a rehearsed answer about his process that he tells anyone who asks.

How come oral evidence is valid if memories are false and people are deceptive? Well, they are valid once interviewers take some measures before, during, and after interviews.

Tips on how to conduct interviews

Oral historian share multiple tips online, in books, in academic articles, and in many more places. They spam topics from where to place the recorder to how to dress properly. It’s impressive. Everything matters. The Smithsonian Institution Archives’ tips on how to do oral history count with more than 50 bullet points.

I won't go through all of them. Instead of being exhaustive, I rather focus on a few, but valuable tips I have applied while doing oral history and customer interviews.

Research a lot

Do not ever interview someone without previous research. Without thorough research, you’re an easy target for lies and false memories. Historians normally leave interviews for last. They first go through secondary sources (like books and articles), then they read primary sources (archives and previous interviews for instance). After that - those who have skills in data science - analyze any quantitative data available. And finally, they prepare questionnaires. However, before they start interviewing, they might do it all over again just to be certain they know everything there is to be known without an interview.

You’re not gold digging with interviews, you’re confirming your research hypotheses and finding new hypotheses to work with. Not everything that shines is gold, especially in an interview. The only way to be prepared to distinguish gold from copper is through research.

And if you are a PM who does continuous weekly interviews and is now thinking that you cannot make thorough research before every interview, reevaluate whether your interviews are actually useful or just “vanity interviews”.

Be prepared and organized

After research, prepare yourself for the interview. Two things are important in this step. First, always write down a list of questions and topics — and use it during the interview. Second, test your recording app and/or equipment (you don’t want to have technical problems with the interviewee in front of you).

Every list of questions is different and depends on the intentions of the interviewer. However, every list of questions should follow some logical order. Most historians conduct their interviews in chronological order. This helps interviewees organize their thoughts.

In the supporting document of questions, use the logical order of your preference. However, remember that in most interviews you won't be able to make all the questions you want. Your question list will be just a guide. It will guide you through the topics and have a few prompts to help you during the interview. Then, it is a good strategy to order topics by priority and importance, and order the prompts and questions inside the topics in the same manner.

Respect

If you did your research, prepared everything, and will conduct interviews, just be respectful. Actually, do that even if you did not do any research and have nothing prepared. I participated in usability tests and product interviews on the other side of the table and just felt that my time was being wasted and interviewers were not listening to what I was saying.

Respect in interviews does not mean that you can’t pressure your interviewees when needed or disagree with them. Actually, respect is not NASA science.

Just do the basics and you’ll be alright. Make good rapport by chit-chatting at the beginning of every interview. Inform the reasons for the interview. Ask for permission to record. Make one question at a time. Do not interrupt.

Silence is important too. You don’t need to fill every void in a millisecond. Allow interviewees to think. Actually, awkward silence should always be broken by the interviewees. It is at these moments that they share the most interesting stuff.

Finally, end the interview properly. Thank the interviewee and if possible send a follow-up message informing the results of the interview.

Record and Transcribe

This might be obvious, but sometimes the obvious must be said. Oral historians are not journalists, they need to know sources and have records of everything. They record every interview and then transcribe it to have an easier time searching for topics and quotes. They are very picky about this. Most historians have incredibly complex systems for organizing records and transcriptions. This might be why historians and librarians get along very well.

Anyways, product managers should be as picky as historians in this regard. You don’t want to try to convince your stakeholder by saying you heard something but cannot prove it. I’m not saying PMs or historians are not trustworthy. It’s just that it makes sense to have this documentation and use it for research benefit and then for convincement.

Framing

Framing questions in the right way is essential. False memories result from poor framing. Cognitive biases result from poor framing. Useless responses result from poor framing.

Good framing follows a few rules. First, always prefer open-ended questions, unless you really need confirmation in the form of yes or no.

Second, details are helpful during interviews. Ask questions to help the interviewee feel comfortable and start remembering facts in a more accurate manner. A good manner for achieving more accurate recollections is to elicit memories by asking for descriptions. For instance, most B2B product managers would benefit from asking their interviewees to describe their offices or how their desk is organized (supposing they use the application in the office).

Finally, to stimulate their memory, use “statement questions” such as “I can see in your data, that Friday last week you tried three times to use feature X. What do you remember about that experience?”. It is only important to be careful to not look like a stalker.

Review

In the end, remember to review every interview. Listen again, read the transcription and make notes. Review your research hypotheses and come up with new ideas. Don’t leave your records and transcription alone in some random repository. Reviewing is more important than the interview itself.