The rise of remote usability testing

Usability testing is one of the foundational disciplines for anyone championing a user-centered approach, whether your job title is UX Researcher, Product Designer, or something else. The great news is that usability testing is more accessible than ever before! There’s a range of low-cost (or free!) tools enabling you to conduct proper usability testing in various ways.

New technologies have made remote usability testing particularly affordable and easy to do.

In this article, I’ll make a case for remote usability testing, based on my own and others’ experience with various usability testing methods.

I hope I can encourage you to give remote usability testing a try, help you sell the method to your manager and other stakeholders, as well as make you aware of the potential drawbacks that come along with all the benefits.

Benefits of remote usability testing

1. Recruit your participants more easily

Participating in a usability test from the comforts of your own home (or office, or whatever is your natural setting when using a certain product) causes less anxiety and mental strain. It’s much less scary than going into an office or a formal test lab, or even having the test moderator come to you.

Remote testing makes it easier for your participant to pick the right time of day for them, and set aside half an hour or an hour, compared to the extra time and energy that go into scheduling and conducting an in-person test.

Unmoderated testing with participants from an online test panel is even easier because of the established test panel and the asynchronous format. Define your screener criteria, order the number of participants you want, and sit back and wait. This also enables you to quickly and cheaply iterate and improve upon your test script, the value of which I’ve written about before. Check out TryMyUI, Teston, or UserTesting.com.

2. Test with an international user base

Many tech products today are used internationally, and often globally. With in-person usability testing, you’re limited to testing with the users who are where you’re at or can realistically go. Even if you are able to travel for this part of work, it’s certainly a limiting factor.

Remote usability testing will enable you to recruit test participants from your entire user base, regardless of geographical location. A larger pool of potential participants will obviously make it easier for you to get the sample size you need. The true value though is simply getting an international, multi-cultural perspective on what you’re testing.

The Nielsen Norman Group wrote a great article about their experience with international usability testing. The title sums it up nicely: International Usability: Big Stuff the Same, Details Differ.

Even though the universal usability guidelines don’t change, you will likely discover relevant differences from country to country.

Reading direction has an obvious impact on the user experience, and technical aspects like Internet speed, screen sizes, and popularity of different browsers vary a lot.

For your specific product, you may attract different demographic groups in different countries, or see your product being used in different ways across cultures. All this can have a big impact on how your users experience your product or website. You just won’t know until you do your research.

3. Get more realistic data

Getting realistic data from an unrealistic situation is one of the biggest challenges in usability testing. One part of it is the Hawthorne Effect (or Observer Effect) which describes how individuals alter their behavior in response to their awareness of being observed.

The closer you get to emulating your test participant’s natural environment, the less biased your data will be.

With remote usability testing your test participant is able to use their own device, where and how they would normally use it. They won’t have to abstract from a curious moderator peeking over their shoulder, adapting to a new device, or getting comfortable in a test lab.

Take it to the next level with unmoderated remote testing. Unmoderated testing has the added benefit of not requiring you to become a great moderator, which isn’t nearly as easy as you may think.

4. Save money 💰

While in-person testing doesn’t even have to be expensive — as long as you avoid hiring an agency or booking a fancy test lab — remote testing is generally cheaper whether you test with your own users or an online test panel. Expenses for screen recording software and payment to your test participants are negligible and will be there regardless of your method. Video calls can be handled for free.

And that’s just comparing in-person to remote testing on overall terms. With remote testing, you get more for a lower price. For instance, you get the value of an international and multi-cultural perspective without paying extra. Imagine the cost of doing that in-person…

Remote Moderated Usability Tests: Why to Do Them

Under normal circumstances, few teams have enough time and resources to perform as much in-person usability testing as…

www.nngroup.com

5. Scale up your testing and get more data

Easier recruitment, access to a bigger pool of users, and lower costs make remote testing a lot more scalable than in-person testing. Especially when utilizing the test panels on online testing platforms, you’re able to conduct a much higher number of tests for the same investment of time and money.

Use it to rapidly iterate and improve upon your test script at the beginning of a new study. Make usability testing a natural and frequent practice in your Design process. Expand your qualitative test results with quantitative data.

The lower barriers to entry make remote usability testing a great opportunity for any Designer or Researcher. It should first and foremost enable you to make usability testing a habit. Why not get into it right away?

The challenges of remote usability testing

Aah yes, the challenges. Perhaps you thought by now that going remote is the perfect approach to usability testing? Unfortunately, it doesn’t come without some tradeoffs, and remote shouldn’t replace in-person testing in your toolkit.

1. Some information will be missed

Depending on your target group, the device being used, the product you’re testing, and the exact situation, video calls and screen sharing are usually an option for your remote testing. And a very valuable one! However, we don’t communicate everything through our words and facial expressions.

Some information will get lost when you do your tests remotely. Hand gestures, changes in body posture, an involuntary tapping with the foot when something isn’t going as expected.

Practice your ability to pick up the subtle signs and read between the lines, and invite your teammates and other colleagues to analyze the video and screen recordings with you for extra perspectives.

2. Participants might deviate from the script

While in-person usability tests can also be unmoderated, it’s much more common in remote usability tests. Since it’s a method I’ve referred to and recommended throughout this article, I think it’s worth mentioning the associated risk in the context of remote usability testing.

Your test participant might get side-tracked and forget about the task you asked them to complete. They might misunderstand something that sends them down a completely wrong path. Or they might simply get lost inside your product (hopefully not though 🤞) and not be able to find their way back to what you’re really interested in testing.

As the moderator in all these cases, you’ll be able to gently get your test participant back on track, complete the test, and get the insights you wanted. In an unmoderated test, you’d be screwed.

3. Your participants might be unrepresentative or unmotivated

Especially when using an online test panel, you’ll have to accept these two challenges. First of all, they’re not your real users and may not be completely representative of your target group. Second of all, their motivation for “helping you” may vary. Some are simply in it to make some quick money.

I personally haven’t found the latter to be a big issue. The platforms’ rating systems help filter out the worst test participants and increase the overall quality of testers.

Screenshot from TryMyUI.com

4. Technical requirements can cause a bias

Although remote usability testing makes user recruitment easier and lets you avoid a geographical or cultural biases, it might cause another one. The seemingly complex technical setup can stop your least tech-savvy users from participating.

A video call and screen sharing might be completely natural to you, but do your users feel the same way? Are they used to these kinds of software? If not, you risk attracting a certain segment that isn’t representative of your user base as a whole.

Key takeaways

Remote usability testing is an essential tool for all UCD advocates. It lets you save time and resources on recruitment of test participants, get international and multi-cultural perspectives on your product, reduce the Observer Effect for higher-quality data, all at a lower cost than in-person testing. This enables you to scale up your usability testing and get even more value out of it 🚀

Sounds too good to be true? Well, it’s not far off. Just like any other method though, remote usability testing comes with some tradeoffs. No usability testing method is perfect, but remote testing needs to be in your toolkit alongside the in-person approach. It offers too many benefits for you to pass it up. So, what are you waiting for?

Originally published at techmoneyculture.com on June 27, 2020.