User research. It’s right there in the name. A user is someone who actually uses your service. Equally valid is the idea of a “target user” – someone who doesn’t yet use your service, but has a genuine need that it would fulfil.
User research has to involve these people. Otherwise, by definition, it’s not user research.
When you use a panel provided by remote usability testing service, you end up gathering data from a bunch of freelancers – professional participants who know the kind of things they’re expected to say. At best, they’ll make a real effort to take the tasks seriously and you feedback. At worst, they’re just actors, going through the motions to get paid.
Real user research involves recruiting participants who honestly care about the problem your service is trying to solve. They’ll engage with the information, weigh up their options in difficult decisions, and carefully consider implications. Most importantly, they’ll draw upon their life experiences to weigh up the benefits of your service against their current practices and other competitors.
One of the biggest strengths of remote research is the fact that you can cast your recruitment net far more widely than with face-to-face research. Remote research is well suited to sourcing real users. If you use remote research techniques and then test irrelevant participants, you’re missing the point of it all.
——
Edit 1: This post isn’t intended as a criticism of remote usability testing services per se. Most services offer a predefined panel alongside the option to recruit your own users. I’m advising you to recruit your own, and to take care in doing so – it’s as simple as that.
Edit 2: Nate Bolt has reminded me that ethnio is a good tool for recruiting your own users, including the ability to intercept users live off your own site. I’ve used it a few times, and it gets my vote.
Related posts:
- Reviewing WhatUsersDo.com: a UK-based remote unmoderated qualitative usability testing platform
- Pidoco° – prototying and remote, moderated user testing – combined!
- Accessibility Field Testing
- Chalkmark: a simple app for testing page mock-ups on real users
- What people say they do vs. what they actually do
Hi Harry,
Interesting post. I agree with you in that whenever we have used remote testing services, it seems that the tests have almost always been completed by professional participants who spend a lot of time testing sites. While this is obviously NOT ideal in terms of usability testing with representative end users, this can still be quite useful in terms of getting early non-domain specific, UI/IxD feedback that can catch any general usability issues. These can then be fixed and the solution tested with REAL end-users, rather than via a remote service – this is useful therefore if your access to real end users is in limited supply. Steve Krug actually makes the case for this approach in his latest usability testing book, Rocket Surgery Made Easy, when getting enough end users is a problem.
Otherwise, I totally agree with you. Testing should be with real end users completing tasks based on real, research-based, user needs identified early in UCD process and our experiences are that this isn’t typically the case with remote testing. Thanks for the link to ethnio though – I will definitely be checking that out!
[...] Why I never use panels provided by remote usability testing services, http://www.90percentofeverything.com [...]
I would also recommend doing your own recruitment and refraining from outsourcing to recruitment agencies.
There’s little point testing with participants that are not the target audience. UX researchers will always have a better understanding of the target audience over an external recruiter. The researcher also has a vested interested in engaging with the right users – as you don’t want to waste your time and money on duff users.