A case study of bringing user research online during a pandemic
Tips and learnings on how to conduct remote research
TEAMS acquired a user research project in December 2019, to collect feedback and comments regarding prepared concepts as well as observe current user behaviors with certain products through user diary studies and in-home visits.
After a successful kickstart of the fieldwork outings before the Chinese New Year, the project was invariably affected by the now infamous epidemic, making in-home visits impossible to carry out as initially planned. The future of the project was uncertain. At that point, after lengthy discussions with the client and making them aware of the situation, it was decided to switch the remainder of the face-to-face interviews to an online approach.
How does one go about changing platform processes on the fly? Here are some takeaways from the project recently wrapped up in March:
Diary Study
Before the interview
Diary studies as a user research method enable respondents to record their behaviors, habits and daily life over a period (usually around seven days to cover both weekdays and weekends). Through this, we can remotely understand how respondents interact with products in real-life scenarios.
At the beginning of this project, we set up the diary session so that the participants could consciously record the user experience and scenarios in their daily routine before being interviewed. This enables the following interview to cover more testing tasks as well as dig more in-depth in specific areas within a limited time frame. After the necessity to switch from an in-home interview to a video interview, the diary study also provided more invaluable information about the environment and daily behaviors. This, in a way, made up for the lack of not being able to conduct on-site observations.
“After the necessity to switch from in-home interview to video interview, the diary study also provided more invaluable information about the environment and daily behaviors. This in a way made up for the lack of not being able to conduct on-site observations.”
We encouraged users to share their diaries through videos or photos, as well as text or audio explanations. Practice has proven that adding non-leading examples to guide the logging process can help respondents understand the amount and level of information we want to collect so that the quality of collected logs could be guaranteed (improved significantly).
From In-home Interview to Video Interview
Choosing the right tool for the video interview
The first question we had to answer was, which software should we select to conduct the online interviews? There are some relatively common criteria for choosing a remote research tool/software. For example, Nate and Tony mentioned some factors a long time ago in their book “Remote Research” to be considered when choosing a remote interview tool:
After a broad comparison of tools available on the market, we finally chose ZOOM. Having built-in video and audio export functions proved to be extremely convenient for recording and archiving the interviews. Besides, there were various good and bad user experiences, listed here for your reference:
Stimuli Preparation and Cards Sorting
Visual stimuli were used for concept and technology demonstration in this project. Meanwhile, more interactive activities such as card sorting were embedded into interviews to immerse the user into the conversation and keep them focused.
“…Another challenge we faced was how to bring these face-to-face activities online…”
After the project turned into a video interview, another challenge we faced was how to bring these face-to-face activities online. In the process of integrating stimuli into remote interviews, we divided them into two types according to the degrees of interaction with the participants, and then use different ways to adapt:
- High Interaction part: it requires the interviewee to take the initiative of the operation.
- Low-interaction part: which the interviewer could conduct, mainly to present.
For the high interaction part, we tried to create an online questionnaire through a survey platform. The platform had a list of options that was intended to have the users sort out. In this way, we hoped to restore some semblance of the face to face interviews by observing any adjustments made by respondents during the ranking process and discovering the underlying reasons.
However, in reality, respondents could not open the questionnaire while sharing the screen at the same time. After balancing the time spent explaining the process and output intended, we subsequently adjusted the online interviews so that the moderator could aid the respondents in the sorting process, through oral communication. This granted us more time for discussion based on the results rather than wasting precious time explaining the software. Even though we lost partial information, we strongly encouraged respondents to express their sorting logics and think out loud verbally. Whenever faced with the situation where online research could not achieve the same results as offline, the researchers would adjust and balance the overall output of the project, looking at the bigger picture.
For the low interaction part, we integrated all slides into a deck. We arranged them one by one according to the flow of the interview, thereby avoiding confusion caused by switching multiple files at the same time.
A point worth mentioning is that it was necessary to conduct a pilot interview before starting formal video interviews. It helped test how the prepared stimuli looked on different platforms (e.g. if the font size was legible etc.). Every so often we had attendees join the meeting through various platforms (e.g. respondents join through mobile phone and clients join through PC).
Remote Room Tour
Having the participants show us around and explain their home environment helped us to have a more comprehensive understanding of their situation. Meanwhile, a different perspective to that of the designer or the client might allow us to discover more topics that the moderator didn’t cover at first.
After changing to video interviews, we still kept the room tour session and observed the participants’ homes through mobile phone cameras.
It is important to note that a “pre-call” had to be made before the formal interview, to communicate with the respondent that a room tour session was scheduled. They needed to be at their residence at that time.
Reflections & Learnings
Although remote research and video interviews have obvious advantages, such as reducing the difficulty of recruitment and saving on travel expense and time, some drawbacks and challenges are hard to avoid. Here is a simple summary of our experiences and what we learned throughout the project:
- The difficulty of network control →prepare for the worst!
- Make sure you communicate clearly with clients and other participants to lower their expectations due to this unavoidable risk.
- As a moderator, while trying to ensure your own network environment, communicate with participants that they should join the meeting 5 mins early or pre-call to test the network. It should be noted that different participants have different network environments at home. Some participants may have uneven coverage of networks in their homes, especially on the balcony or bathroom, which are prone to interference.
- The difficulty of staying focused → time control & design of discussion flow.
- In the video interviews, we found that it was difficult to hold a participant’s attention for the entire duration of the call. A 90min session, especially when the whole family is being isolated at home, was particularly tough. The participants who were parents were very easily distracted by children outside the room, and they were also prone to fatigue caused by constant staring at the screen
- The building of intimacy and comfort → to be improved
- More interactive game/activity design →to be improved
This article mainly focuses on sharing four first-hand experience on how we adapted our user research methods during the epidemic. Also, it is essential to distinguish the short-term and long-term changes in behaviour caused by the outbreak while analyzing the data collected “in-field”. This will be another big challenge for the implementation of user research made in this extra-ordinary time.
