Usability tests in an agile world

Some months ago, we had the idea to develop a virtual UX lab that supports us in collecting user feedback on the fly as early as possible. First, we thought about a tool for unmoderated usability tests. Unfortunately, there was no tool that met our requirements. Most tools that do a great job (e.g., http://www.usabilitytools.com) are cloud-based solutions and require public access to the prototype. This is a no-go for our industrial clients that try to get a competitive advantage with the software we develop for them.

Moderating tests avoids information loss

Another reason that let us reorient towards remote moderated usability tests was the loss of information density if the participants do not have to think aloud and if you are not able to ask questions for further insights. If you conduct only a few tests, you want to get as much information as possible. Andreasen, Nielsen, Schrøder and Stage (2007) showed in their paper “What happened to remote usability testing? An empirical study of three methods” that unmoderated usability tests detected less usability issues than moderated remote or lab usability tests.
To dive deeper into the discussion about lab, remote, and unmoderated usability tests, I recommend “Pros and Cons of Remote Usability Testing” by Nate Bolt and “Unmoderated, Remote Usability Testing: Good or Evil?” by Kyle Soucy. Also the book “Remote Research” by Nate Bolt and Tony Tulathimutte is a great resource (corresponding slides are available on http://de.slideshare.net/boltpeters/remote-research-the-book-the-talk).

Advantages of remote usability tests

Remote usability tests have the advantage to test on multiple platforms in a more natural setting, because participants stay in their usual workspace setup. You are also not limited to web applications, so you can test native apps as well. Since you are invisible of the interviewee, it is less socially demanding and you can focus on the interview itself. Finally, it increases the quality of your participants, because it requires less time and no travel cost. Usually, people cannot take off 2-3 hours for a lab test on weekdays.

Keep control and focus on user behavior

The most challenging part was to find the right tool, because the market of conferencing tools is huge. There is already a dedicated website for the evaluation of conferencing tools (http://webconferencing-test.com). Only a usable and performant meeting tool gives you the control over the interview and lets you focus on the user’s experience. We tested many meeting tools and think that GoToMeeting is the best tool on the market for conducting remote usability interviews. In particular its high video and audio quality, the performant integrated recording and the usable meeting controls (e.g., webcam, screen and audio requests, muting, etc.) lead to our decision. A list of conferencing tools that we tested as well, but were rejected (state: June, 2013):

We also want to share our best practices for conducting moderated remote usability, see our blog post A step-by-step tutorial for conducting moderated remote usability tests.