We report our best practices for conducting moderated remote usability tests. Like a step-by-step tutorial, we want to enable you to manage your own remote usability test. We evaluated the tutorial internally and other team members could reproduce a remote usability test with almost no transfer costs.
We chose GoToMeeting for facilitating the test, because it provides a great video and audio quality and it gives you the necessary control to confidently manage an interview.
Prepare the remote usability test
Choose tools for conferencing, screen recording and video editing
We recommend GoToMeeting ($49/month) as conferencing tool, see our blog post Pick the right tools for remote usability tests. Because GoToMeeting does not record the webcam for privacy reasons, we used Camtasia (93,50 €) to record the webcam screen. We also used Camtasia for editing the videos and create highlight videos, but you can also choose other screen recording and video editing tools.
Write a task-based interview script
The script should be a line-up of scenarios and follow-up questions that make the participants comfortable and resample a stereotypical workflow. If you need help, feel free to contact us (nicole.charlier [at] akquinet.de). Nate Bolt shared an interview script here http://remoteresear.ch/samples/moderator-script/. In contrast to Nate’s script sample, I prefer to start a task with a scenario description that helps the participants to get into the situation. Besides, I leave a lot of whitespace between the questions to note answers and observations.
Schedule test sessions
There are several tools that will help you scheduling the test sessions (e.g., doodle, PowWow, iCal, Outlook, etc.). Keep in mind that you need some time before the session to get ready and after the session to save the data and get ready for the next session.
Send an instruction mail to participants
This mail is very important, then it builds the foundation for an successful remote usability test. Explain the procedure and clarify technical requirements (high speed internet, webcam, microphone, speaker/headphone, etc.). Do not forget to make your prototype remotely accessible in advance.
Meet and deliver with GoToMeeting + Camtasia
Edit meeting and recording preferences
- Launch GoToMeeting and go to “Preferences – Meetings” (cmd + ,).
- Disable “View the Attendee List”. (Hereby, we want to prevent that the participants feel uncomfortable by a list of observers.)
- Go to “Preferences – Recordings”.
- Define directory for recordings and set reminder to convert recordings.
- Leave the preferences panel and click “Schedule Meeting”. ( GoToMeeting offers also a web interface for scheduling a meeting. This tutorial bases on the desktop app.)
- Login with your GoToMeeting Account.
- Choose an appropriate subject and appointment.
- We have only positive experience with the built-in audio conferencing, but we choose “Provide both” for the case that it does not work properly.
Customize invitation mails
- Add instructions to the default invitation mail for participants, e.g. “Since you will share your screen with us, we recommend you to close all application windows in advance.”.
- Invite observers (e.g., team members, clients, UX researchers, etc.) and tell them that they mute themselves when they join the meeting.
- Go to “My Meetings” and login with you name and email.
- Check if the attendee list is disabled for attendees. Right click on an attendee in the attendee list and check if tick is gone in front of “Allow Attendee List”. (Unfortunately, I had sometimes the problem that disabling the list in the general settings was not applied.)
- Turn off “On-Hold Beeps” and “Entry/Exit Chimes”.
- Mute all other participants. (New attendees are muted by default, so observers can not disrupt the interview.)
- Check if “Mic & Speakers” is selected and your webcam shared if you want to. (There is also a window with your webcam video.)
- Send Webcam Request to participant. (Participants has to confirm.)
- Make participant presenter to request screen sharing.
- Launch Camtasia and start “New Recording…” for participant’s webcam (shift + cmd + N).
- Select “Custom Region”, microphone icon and speaker icon.
- You may have to install the “System Audio Component” in “Settings – Recording”, before you can select the speaker icon.
- Click the red button to select a recording region.
- Draw the recording region around the webcam in the GoToMeeting Viewer.
- Do not move the GoToMeeting window while recording and keep the webcam image unhidden.
- Click “rec”.
- Start recording in GoToMeeting.
Conduct the usability test
- Start by explaining the purpose of the test and what is going to happen. – “We test software, not users.”
- We usually send the link to the prototype within the GoToMeeting chat.
- While the interview, you can note highlight times. This makes it easier to create a highlight video later.
- Ellen Beldner (June, 2010) shared her learnings about how to manage an interview: Adventures in remote usability, Part 2: GoToMeeting
End meeting and stop recording
- Camtasia – Stop Recording (alt + cmd + 2).
- Recording is imported automatically to current Camtasia project.
- End meeting by closing the GoToMeeting meeting window.
- You will get reminded to “Convert Recording”.
- Select last recording and click “Convert Recordings”.
- Get an espresso or tea while converting.
- Open recordings folder and drag & drop file to Camtasia.
Combine and edit recordings
- Open Camtasia and Adjust Canvas (right click in canvas).
- Arrange clips on canvas by drag & drop.
- You can also change the size of the video elements.
- “Separate Video and Audio” of screen recording (right click on GoToMeeting recording).
- Delete audio track of screen recording. (The webcam recording by Camatasia contains all audio sources. The webcam recording is also synchronous with the participant’s voice. This synchrony has the highest priority, because the participant’s mouth is visible and it appears odd if the voice is out of phase with the speaking lips.)
- Synchronize webcam and screen recording per drag & drop.
- Since you started the webcam recording before the screen recording, you have to play around how much delay suits best. Listen to the audio and try to map it to the screen interaction.
- Export project (cmd + E).
- We recommend to choose the highest quality for the mp4-file.
- Now, you can continue to extract highlight clips or deliver the whole recording.
We would like to hear your best practices for remote usability testing and whether this tutorial is helpful.
If you have any questions or remarks, drop us a line: Martin Grund (martin.grund [at] akquinet.de)
6 thoughts on “A step-by-step tutorial for conducting moderated remote usability tests”
Great post! Very detailed and informative, with helpful supplemental links! The first one is broken (you might want to check and update), but I found it on your blog.
One thing I’m thinking about – all these tools, like ethn.io, the scheduling ones you link to, GoToMeeting, Camtasia… Is it possible that it needs to cost this much? I’m trying to figure out a way to conduct a budget test for my company and I’m thinking Skype screen share + Camtasia or a cheaper screen recorder, and for recruiting a popup on the website itself or something, if the user’s been browsing for long enough. Do you think this make sense to try?
Thank you for writing about the matter!
Thank you so much.
I fixed the link.
Of course you can try other tools. At the time we evaluated the tools (3 years ago) GoToMeeting delivered the best quality (see https://blog.akquinet.de/2014/09/08/pick-the-right-tools-for-remote-usability-testing/). However, Skype improved for instance substantially and offers now even free group video calls with screen sharing.
I have no experience with onsite recruitment. If you can implement it easily, give it a try. I would be very interested in your experience, maybe you can even collect some data.
At the end, it is about answering research questions. If the tools allow you to collect the data that will answer your questions, you made a huge step to inform your design decisions.
Maybe you can also share your experience with other tools and we can include this here.
Great stuff. I didn’t check out how long ago this was published, the field indeed definitely changed. I’ll try this guerilla style and will drop a line or two on how it worked out!
Good luck. Looking forward to your report.
Hi there! In step 7 above “Make participant presenter to request screen sharing.” – I’m wondering how you did this step? I’ve tried this and it requires the participant to download the desktop version of the software, which is of course a pain in the butt for participants and opens up risk (what if they are at work and can’t install it?). Did you ask participants to install the software ahead of time?
The participant has to install the GoToMeeting desktop app or has to use the GoToMeeting web app within Google Chrome, where a plugin can be installed (see http://support.citrixonline.com/en_US/meeting/help_files/G2M030003?title=GoToMeeting+Web+Viewer#Share). We use a customized invitation mail that explains the procedure and clarifies the technical requirements (e.g., installation permissions). This invitation mail also includes a link to download GoToMeeting. So you might run a test before the actual interview. Sometimes the participant has no installation permissions, then we try to switch to the conferencing tool that is used in their company and pre-installed.
Currently, I am not aware of any web app for video conferences that allows you to share the whole screen. Are you?
Comments are closed.