There’s a persistent myth about guerilla user research that it’s perfectly OK to grab just anyone to act as a proxy for your users. Perhaps it’s something to do with the whole low-cost, lo-fi ethos that makes this myth so easy to believe.
Actually if there’s one thing you shouldn’t cut corners on, it’s your recruitment. User testing should mean real end users, not Bob from accounts or some random dude walking down the street. Let me explain why this is so important with a case study.
Picture this: a large institution designs and builds a decision-making app for their customers. Their customers are sales-staff in small, partner companies whose job it is to resell the larger institution’s products to joe public, along with a commission. Now, during the early stages of design they thought they were doing all the right things. Wireframing, low fidelity prototyping, and usability testing – but here’s the crunch – they did their user research on their colleagues within the institution. They included middle management, data entry staff, the receptionists, and even some of the staff in the canteen. In other words, everyone but the sales staff in the partner companies who the product was really for.
And so they ended up with a product that was really usable – even a naive first-time user could use the system to punch in the data and get product recommendations out.
At this point, having spent about half a million dollars, they were sure they were onto a winner. So, they decided to get some real end-users in for face-to-face research sessions. The first few users were very polite since they had strong professional and political relationship with the institution making the product. But then one of the users, an admin assistant, commented that they would never use this system in her workplace. Her boss would do all of this work on paper and in his head. Then, he’d give her his calculations and she’d double check them in Excel. Apparently, his figures were almost always spot on.
Sure enough, when all the other users were asked about their current practices, they admitted that they also did all their decision-making in their heads. In fact 3/4 of them said they’d never use the app, because it would actually increase their workload. The owners of the product suddenly had a crushing realisation. The app was highly usable but simply not useful – it was solving a problem that didn’t exist. In other words, they were screwed.
And so ends today’s lesson. Always carry out research on real users, or you might end up like them.