Concept Evaluation Test
A UX Researcher's take on interviewing users for an expansive TabBar project
In August of 2021 the UX Research team of the company I work at invited 6 users to perform a Concept evaluation test to best capture their perception of a prototype. The Design and Product teams were eager to know how the new features and improvements on the app were going to be received by customers.
The initial stages
The UX Research team was briefed on a research initiative about to take place. We had to test a concept, about to be implemented on the company’s app. The app’s bottom navigation bar was redrawn to include an expansive menu.
An expansive menu is not a common feature in this market segment, which made the entire team curious about our upcoming research findings.
My role in this context was extensive. In the initial setting, I participated in defining the scope and planning the operational aspect alongside other members of the team. We iterated, tested the prototype, established the right software, ran tests on these platforms, assigned auxiliary roles, and set the schedule. I wrote the script, based on the prototype and the research goals of this test and interviewed the users.
For context — I work as a User Experience Researcher in one of the main reward companies in Brazil, with more than 22 million customers and dozens of reward redemption partners. Clients can exchange the points they accumulate on purchases of products, travel experiences, and, even, exchange them in donation initiatives.
Defining the scope
The Research team decided to assess 4 main aspects:
- Can users understand the expansive bottom navigation bar feature? Are users willing to embrace this change in concept and adopt it?
- Can users make sense of the names and images displayed on the screen and menu? Is the navigation flow adequate?
- Are Designers making changes and using terminology that best suits the user? What feedback can users provide to make changes that are valuable to their experience, especially in taxonomy and iconography?
- Are users interested in a floating button feature on the home page of the app?
Planning the test 📋
After receiving the briefing of the project and defining its scope, the team gathered up to start working on the planning stages. We decided on 6 sessions of 50–60 minutes of individual interviews. We feared a group session could bring bias into the process, as we were focused on users’ perceptions.
We decided to frame the test like a concept evaluation through context analysis. In practical terms, it meant we had a prototype that allowed for navigation but we were going to display this to the users, instead of having the users themselves navigate through the prototype.
Though this wasn't the ideal model for tests, it was feasible. We faced a few problems while testing the tool, and decided to skip the traditional usability testing route to avoid disappointing users, should there be poor connection and software/technical issues. This meant the script would have to be written in a way that could accommodate both formats.
Finding the perfect set of users 🔎
The criteria to screen for users was threefold and simple: To be eligible users had to
- Be familiar with the concept of reward programs and points. No need to be an active user of any program.
- Understand how to navigate on an app and recognize basic features.
- Be at least 18 years old at the time of the test.
A bit of the script
Contextualizing, navigation habits, currently used apps.
- For E.g. What apps do you consider easy to use? Why would you say this app is easy to use?
App usage and preferences, expectations on click response.
- E.g. In which part of your navigation do you use the search bar?
Other reward programs, understanding of company's value proposition.
- E.g. Have you exchanged your points recently? How was this experience?
Screen visualization, navigation, perception on the expansive TabBar model.
- E.g. What is your perception of this menu format? Do you feel you can understand the menu at first sight or do you feel the 'Extra info' click would be useful? What do you believe the app will do when you click the 'Extra info' link?
UI assessment on icons, names, colors, style
- E.g. What is the best visualization of this card — horizontal or vertical display? Why is this format more interesting in your opinion?
Talking to users 🧡
Users were very keen on being a part of the test. They took the test seriously and though there was no requirement, they mentioned they downloaded the app, navigated a bit, and got themselves ready to participate.
I would perform the interviews and other members of the Design and Research team participated either in taking notes or at the operational aspect of the testing.
The interviews happened smoothly, with minor unscripted moments. One interviewee mentioned a Sociological aspect regarding the main color of the theme, asking whether pink would be intimidating to male users.
A senior participant got excited about telling a few life stories, which was addressed by telling them my support team was on a tight schedule but I could stay a little longer after we were done with the questions of the test (turns out that got lost and never happened — still curious to this very day).
Processing raw data
After the sessions, we had a significant amount of recording time to process. The goal was to turn raw data into product insights. Additionally, we had to tailor the results to our initial briefing and archive useful information that was given besides the script.
We worked on the qualitative data by synthesizing key information in bullet points, setting great participant quotes aside, marking down the time of valuable insights in the video, and established metrics of success in anticipating on-click results.
Analyzing hours of sessions in a smaller team on a tight schedule can be very challenging. Team members who were watching the test took notes on the sessions. This made me think of how different the main concerns of members of a UX team are. In this process, those who design on a daily were mostly focused on functionality and UI. UX Research, in this test, was focused mostly on the experience, how it all tied together and whether users would be able to make sense of what they were seeing.
The UXR team created a presentation that comprised two perspectives;
Insights users gave which were not on the script. Reminders of a similar app, familiar navigation journeys, and experiences, ideas they brought up because they understood they'd fit the contextual inquiry, interesting ways of perceiving the experience which was not our design but maybe could be an improved version of the experience we planned. In Summary, how they perceived the company and its services. I will list a few below:
- Suggestions on new options of icons designs and names,
- Adapting the profile/personal information page to a sidebar vertical menu activated by a lateral swap movement.
- The possibility of making more than one floating button available on the Home page for quick access.
- Personalized experience according to their navigation pattern on the app.
- The personalized experience starts at the moment of logging in.
- The inclusion of an icon that could forward promotional products/traveling packages to friends.
- Customizable display of features on the menu, allowing the users to hide features they wouldn't use as frequently.
- The possibility of accessing onboarding screens beyond the first usage.
- Augmentation of the expansive menu by using a Finger and thumb movement to improve readability.
The former insight was given by a senior user. He mentioned he regularly experiences difficulties in reading menu options in apps, as these can be too small for his impaired eye-sight, and how this minor adjustment could help him. Accessibility matters!
Insights by the Design & Research team directed to service design and actionable insights
We planned it should focus on a few guidelines: Findings should be specific and should include redesign recommendations to usability reports. On customization options and personalization of the experience, our report suggested:
- A new round of Usability tests to further evaluate the floating button feature, according to the user's feedback.
- Recommendation on altering the personal information menu from an icon on the TabBar to a lateral swipe menu.
- The QR Code feature was perceived with a lot of confusion by the users. Accumulating and using options were difficult to understand in their current design format.
- A comprehensive study on the perception of the brand to diagnose points that need extra attention on educational initiatives.
On iconography assessment, a surprise was presented. Users were aware and focused on heuristics. Their feedback on the interviewing process revealed that users not only had expectations of the icons that should be visible at that moment of their navigation but, also, they were very vocal in pointing that inconsistency out.
For reference: Consistency and standards, users should not have to wonder whether different words, situations, or actions mean the same thing. Follow real-world conventions, making information appear in a natural and logical order. (From NN Group's official page)
Overall, the users provided an outside view that is valuable to the design process. They mentioned a learning curve on interacting with the expansive menu feature and how it was weird at first. They mentioned the perceived value of this change and how this was unique in its segment.
Final words on participating in this experience. By being a part of this process I learned that no matter how long we have been doing user-oriented research, each test is different, and talking to people always can keep surprising us and expanding our points of view. In a way, researching in the field of user experience emulates much of life's learnings — if one's open to see the nuances.
M.C. Escher's artwork has been used to illustrate this article as it's a close representation of how user’s perception during a test can seem: a set of remotely connected pieces of information. The shifting nature of the images suggests that looking from a close-up perspective could indicate some traits that, if watched at a distance, for a longer period and imaginatively, can convey entirely different information. Thanks to The Diggital Commonweath/Boston Public Library for making amazing art available in a digital format.
