Reviewing WhatUsersDo.com: a UK-based remote unmoderated qualitative usability testing platform

Note: This post is over 4 years old. It was first published in August 2010

WhatUsersDo.com is a UK-based remote, unmoderated, qualitative usability testing platform, somewhat similar to usertesting.com. You pay £25-£30 per participant, and for each one you get back a ≈20 minute screen recording with audio of them thinking aloud during the tasks.

WhatUsersDo.com screen-shot

I’ve been aware of WhatUsersDo for a while now, but I have to admit, I’ve always shied away from using it. I’ve already got a healthy and well established research programme at Madgex – do I really need to use a new web app?

Looking at it from afar, WhatUsersDo seems a bit of a strange beast. Normally I’d carry out qualitative, moderated user research (in which the researcher gets to interview the participants), or quantitative, unmoderated research (Analytics, AB Testing, Surveys, etc). WhatUsersDo is an qualitative, unmoderated tool. Since it’s qualitative, you don’t get aggregated data – so you can’t quickly come to conclusions about the entire data set; and since it’s unmoderated, you don’t get to interview the participants – so you can’t keep them on track and drill into interesting topics as needed. To me, this isn’t an ideal combination.

Another thing that put me off is the fact that with the standard WhatUsersDo offering, your participants are provided from a pre-established panel, and you know very little about them. There’s quite a few new web apps offering panel-based testing nowadays, so let me say this for the record: panels are not good unless you create and control them yourself. An experienced professional researcher will find it almost unthinkable to base any redesign, however small, on feedback from a bunch of unknown people. You should always aim to carry out research on your real target users – people who in real life would actually use your product, for whom the minutiae of your design decisions have real world implications. If you test your site on random, unknown people, at best all you get is insights as to whether a generic human is able to complete tasks without facing insurmountable usability problems.

However, now I’ve seen WhatUsersDo in action, I’ve realised that just because it’s not for me, this doesn’t mean it’s not a useful tool for other people. It’s a good value entry-level tool. If you’re new to user research, and you want a cheap and easy way to get your toe in the water, then Whatusersdo is ideal for you. On the other hand, if you’re an established researcher then you’ll probably come to the same conclusions as me. Perhaps in coming months this will change, as they do have a new “recruit your own users” feature coming out soon.

In order to give it a fair evaluation, I found a volunteer who seemed to be well suited to WhatUsersDo – Ixxy, a brighton-based web development agency run by a friend of mine. Like many small agencies, they are technically capable but they don’t have in-house User Experience researcher.

Setting up the study took about 10 minutes. We filled in one form and wrote the wording for a single task (shown below). This was incredibly easy.

WhatUsersDo screen-shot
Setting up the whatusersdo study

Having set up the study, all we had to do is sit-back and wait. A few days later, the test videos were waiting for us online. As you can see below, there was only one metric for each user: a satisfaction score marked out of 100.

WhatUsersDo screen-shot
Study results overview page

Out of the five videos we got back, a couple of them were duds. The participants either rambled on, or didn’t take the task very seriously (e.g. in a holiday booking task, one participant booked the first property she found on the first dates she chose, and didn’t blink at the price – totally unlike real world holiday booking behaviour). However, 3/5 of the videos revealed useful, actionable findings, and this was enough to made it seem worthwhile (£25 per person is, after all, very cheap).

There was one feature in particular that the tests showed to be problematic – a rather complex calendar view that users tended to get confused by. It was one of those features that was designed to the clients specification, and the developers always had reservations about it. For them, the video footage was great news, as it gave them exactly the evidence they needed to convince their client that the feature needed reworking.

The WhatUsersDo video player offers a nice tagging tool (shown below), so you can add a text notewhenever an interesting event occurs. This becomes a hyperlink and you can use it to jump to that point in the video. I expect this would be very useful for team-based analysis where you each pick a video to analyse, and then later get into a group and rapidly go through the video highlights.

WhatUsersDo screen-shot
Video player and tagging tool

The tagging tool doesn’t give you quite enough functionality to manage the analysis of the video observations. This isn’t a big deal – you can create your own findings log very easily using something like google docs, shown below (Template available here).

WhatUsersDo screen-shot
A very simple findings log template in Google Docs

Conclusions:

If you’re in the UK and you’ve never done any usability testing before, then WhatUsersDo is probably worth trying – it’s so cheap that even if it only reveals two or three usability issues, it’ll be worthwhile. What’s more, it’ll give you a taste for UX research, and that in itself is invaluable. If you’re an experienced researcher in a company that has a regular research budget, then it’s possible you’ll find it a bit low-end for your needs. It’ll be interesting to find out more about their “recruit your own users” feature when it comes out, but personally, as a verteran researcher, I’d always prefer to carry out a live interview than watch a pre-recorded video.

Disclosure: WhatUsersDo is an advertiser on this site and gave us free usage of the platform for the purpose of the review.

6 comments