reading time of blog
4 min read

Following our discussion on Jiva’s Remote Usability Testing (UT) Day in the previous blog post, let’s now dive into why UT is critical to do before releasing any app, and what a researcher must do to ensure they conduct seamless UT sessions.

Why is UT important?

It’s not uncommon for people in product development to overlook small, perhaps insignificant issues they initially deem unproblematic. But, such issues could turn out to be a problem when users start to interact with the product. This is why UT is important! UT helps uncover these problems early, allowing for quick iterations and improvements before the product launch.

Having spent hundreds of hours doing interviews, I’d like to share some learnings from conducting remote UT sessions with my fellow researchers. 

Here are my top 6 things to keep in mind when implementing remote UT – from research preparation to execution.

  1. Be mindful of your participant and time spent

When you notice your participant starts to become uncooperative and shows signs of fatigue, it is best to not push the interview to continue. 

I noticed from my experiences that during the interview stage, participants often start off excited but lose interest and stop giving effort once they realise they have to complete several tasks for the usability testing, which could take time. 

When this happens and you decide to push through, it could be challenging for your participants and you as the researcher. Participants could feel tired and frustrated, and you would not obtain any valuable insights when in such situations.

This is common especially with rural users, who often get easily tired or are not as familiar with using gadgets. 

Tips!
→ Conduct participant screening to categorise participants into articulate and less articulate groups while still meeting your main criteria.

  1. Avoid leading questions

Is interviewing participants as simple as asking them questions? Think again. Trust me, it’s not that easy. 

As a researcher, you often expect participants to give their answers exactly like you expect them to. But this is exactly what you need to avoid.

When doing UT, you want to discover overlooked issues, so the one thing you need to avoid is leading questions. You don’t want participants to align their answers with your expectations - this will lead to biased results. 

Tips! 
→ When probing deeper, reframe your questions to avoid leading participants to a specific answer. Instead of asking yes or no questions, opt for open-ended questions to encourage more comprehensive responses.

Example: 

Leading question → Do you think this new feature makes the app easier

Open-ended question → How would you describe your experience navigating through the new feature on the app?

  1. Give your participants ample time to respond

When trying out a product for the first time, users need time to understand it. Make sure they take their time before responding - it will allow them to think about your questions and their interactions with the app. Your participant’s initial responses during the UT may not fully convey their thoughts or experiences. So, ask follow-up questions to gain deeper insights and understand your participant’s perspectives better.

  1. Define a clear success metric for the test

Without defined success metrics, there is a chance that your research will end up evaluating aspects beyond the intended scope. There are a lot of components and potential user scenarios in the app, so doing UT without clear metrics can lead to confusion about the evaluation criteria and outcomes.

Tips!
→ During my note-taking process within the UT, I include success metrics in the form where I do the content analysis. This helps me assess task completion levels easily. I label it as "Task/Step Completion Evaluation and Success Metrics" and divide it into four parts which are applied to the tested app. I also add notes on each metric to track how users complete certain tasks/steps/flows.

  1. Embrace negative feedback

It's important to stay open-minded especially when you receive negative feedback. It might be too much for you to handle at first, but it’s valuable. Negative feedback often pinpoints areas for improvement, enhancing the user experience significantly.

         6. Be flexible! Always prepare a backup plan.

In my experience, users often bail out at the last minute. So, to ensure we have enough participants who meet our criteria, Jiva’s Research Coordinators (on-ground research team) and I always keep an additional list of available participants.

Even when we've made a structured timeline, things often unexpectedly change with rural participants due to factors beyond our control. Therefore we need to be always prepared for such things!

Another challenge I frequently encounter is the unstable and unpredictable internet connections in rural areas. To mitigate this issue, I maintain regular communication with the research coordinators. This helps me stay informed about locations with better connectivity, ensuring sessions are not delayed.

Tips! 
→ Consider adding a “buffer day” to your project plan to accommodate unexpected delays.

Fiona is one of Jiva’s youngest team members in the Design Research team. She specialises in remote Usability Testing and has spent hundreds of hours speaking to rural users in search of ways to make our app more accessible and easier to use. 

Want to read more specialised tips and work-related content? Follow Jiva on Medium, LinkedIn, and visit our website jiva.ag!

More From
Behind Jiva
More like this
Related posts link
Beyond interviews: Reflections from researching our rural users

Beyond interviews: Reflections from researching our rural users

Fathia Ramadina has been in user research for years prior to joining Jiva - but there are things about rural users so distinct that uncovers rich layers of enlightening insights on doing research. In this blog post, she reflects on how she found that after doing many research projects and hundreds of interviews with Sahabat Jiva, a meaningful rural user research goes beyond techniques.

6 Must-Dos to Unlock Seamless Remote Usability Testing

6 Must-Dos to Unlock Seamless Remote Usability Testing

Our Design Researcher Fiona Yasmine shares her top 6 lesson-learned about conducting seamless usability testing session with rural users.

Alief Anindra on leveraging tech to transform the finance function

Alief Anindra on leveraging tech to transform the finance function

Alief Anindra holds a critical position in Jiva’s finance team, but you might not be familiar with his job title. In this article, taken from one of our #WorkWonders episode, Alief tells Nea Ningtyas (Sr. Content Specialist) about his role as a Finance Transformation Lead. Read through to discover more about how, in this role, Alief initiates projects to transform the finance functions.

Dinda Setiawan on rural design research challenges and improving ways to do it

Dinda Setiawan on rural design research challenges and improving ways to do it

Close to a year into the job, Dinda Setiawan, Jiva’s newest addition to the design research team, got acquainted with unique challenges in researching our rural users. When asked if her new journey has led to discovering the number of effective ways to improve our research process, Dinda said yes — the number is infinity.

6 Must-Dos to Unlock Seamless Remote Usability Testing

May 22, 2024

6 Must-Dos to Unlock Seamless Remote Usability Testing

Following our discussion on Jiva’s Remote Usability Testing (UT) Day in the previous blog post, let’s now dive into why UT is critical to do before releasing any app, and what a researcher must do to ensure they conduct seamless UT sessions.

Why is UT important?

It’s not uncommon for people in product development to overlook small, perhaps insignificant issues they initially deem unproblematic. But, such issues could turn out to be a problem when users start to interact with the product. This is why UT is important! UT helps uncover these problems early, allowing for quick iterations and improvements before the product launch.

Having spent hundreds of hours doing interviews, I’d like to share some learnings from conducting remote UT sessions with my fellow researchers. 

Here are my top 6 things to keep in mind when implementing remote UT – from research preparation to execution.

  1. Be mindful of your participant and time spent

When you notice your participant starts to become uncooperative and shows signs of fatigue, it is best to not push the interview to continue. 

I noticed from my experiences that during the interview stage, participants often start off excited but lose interest and stop giving effort once they realise they have to complete several tasks for the usability testing, which could take time. 

When this happens and you decide to push through, it could be challenging for your participants and you as the researcher. Participants could feel tired and frustrated, and you would not obtain any valuable insights when in such situations.

This is common especially with rural users, who often get easily tired or are not as familiar with using gadgets. 

Tips!
→ Conduct participant screening to categorise participants into articulate and less articulate groups while still meeting your main criteria.

  1. Avoid leading questions

Is interviewing participants as simple as asking them questions? Think again. Trust me, it’s not that easy. 

As a researcher, you often expect participants to give their answers exactly like you expect them to. But this is exactly what you need to avoid.

When doing UT, you want to discover overlooked issues, so the one thing you need to avoid is leading questions. You don’t want participants to align their answers with your expectations - this will lead to biased results. 

Tips! 
→ When probing deeper, reframe your questions to avoid leading participants to a specific answer. Instead of asking yes or no questions, opt for open-ended questions to encourage more comprehensive responses.

Example: 

Leading question → Do you think this new feature makes the app easier

Open-ended question → How would you describe your experience navigating through the new feature on the app?

  1. Give your participants ample time to respond

When trying out a product for the first time, users need time to understand it. Make sure they take their time before responding - it will allow them to think about your questions and their interactions with the app. Your participant’s initial responses during the UT may not fully convey their thoughts or experiences. So, ask follow-up questions to gain deeper insights and understand your participant’s perspectives better.

  1. Define a clear success metric for the test

Without defined success metrics, there is a chance that your research will end up evaluating aspects beyond the intended scope. There are a lot of components and potential user scenarios in the app, so doing UT without clear metrics can lead to confusion about the evaluation criteria and outcomes.

Tips!
→ During my note-taking process within the UT, I include success metrics in the form where I do the content analysis. This helps me assess task completion levels easily. I label it as "Task/Step Completion Evaluation and Success Metrics" and divide it into four parts which are applied to the tested app. I also add notes on each metric to track how users complete certain tasks/steps/flows.

  1. Embrace negative feedback

It's important to stay open-minded especially when you receive negative feedback. It might be too much for you to handle at first, but it’s valuable. Negative feedback often pinpoints areas for improvement, enhancing the user experience significantly.

         6. Be flexible! Always prepare a backup plan.

In my experience, users often bail out at the last minute. So, to ensure we have enough participants who meet our criteria, Jiva’s Research Coordinators (on-ground research team) and I always keep an additional list of available participants.

Even when we've made a structured timeline, things often unexpectedly change with rural participants due to factors beyond our control. Therefore we need to be always prepared for such things!

Another challenge I frequently encounter is the unstable and unpredictable internet connections in rural areas. To mitigate this issue, I maintain regular communication with the research coordinators. This helps me stay informed about locations with better connectivity, ensuring sessions are not delayed.

Tips! 
→ Consider adding a “buffer day” to your project plan to accommodate unexpected delays.

Fiona is one of Jiva’s youngest team members in the Design Research team. She specialises in remote Usability Testing and has spent hundreds of hours speaking to rural users in search of ways to make our app more accessible and easier to use. 

Want to read more specialised tips and work-related content? Follow Jiva on Medium, LinkedIn, and visit our website jiva.ag!

Read this post on medium