90 percent of everything : Usability Blog

Written by Harry Brignull

Do you ever think about the impact of the experimenter effect (or Hawthorne effect) when you’re running face to face user research?

Here’s a quick test.

First, go and check your Analytics package to see how many users check your site’s Terms and Conditions before accepting them. My guess is that the number will be roughly 1-3% (maybe lower).

Now, take a look at the notes from your last few usability research projects. How many users diligently looked at the Terms and Conditions while you were watching them over their shoulder? In my last few projects, it’s been 10-30%

So, that’s roughly 10x more in my case. Pretty substantial. This is a perfect example of how people adjust their behaviour in face to face research sessions. As soon as you pay someone to sit in a room with you, give them a task and watch them intently, they will start doing and saying what they think you want them to.

The experimenter effect is unavoidable. I’m a huge advocate of face-to-face research, but this is one of the method’s biggest weaknesses (and in equal parts, it’s one of the biggest strengths of Analytics).

What steps do you take to mitigate the experimenter effect?

Comments below please!

The default widget library supplied with Axure occupies an uncomfortable middle ground – it looks like it’s just badly designed high fidelity, rather than intentionally lo-fi.

This sketchy Axure widget library by Kevin Wick gets around the problem by giving your your prototypes a rough hand-drawn look.

You can see the full widget library here (You’ll need to install Scoder hand font for it to look right).

I have to admit I only tried Pidoco° very briefly at UX Brighton last week, but I was impressed with what I saw. They’ve integrated an Axure-style prototyping tool with a remote, moderated usability testing tool, and the price is very competitive: only $45/month for the entry-level package. For the same service using Protoshare (which only does prototyping) and Uservue (which only does remote moderated testing), you’d be paying almost $200/month in total: almost 3x the price!

Here’s a quick walkthrough the product:

Above you can see the wireframing/prototyping tool. It’s pretty much as you’d expect if you’ve ever used Axure, Protoshare or any of the other similar tools out there at the moment (there were at least 32 last time I checked).

When you view your prototype, you can view it with either a low-fi (hand sketched) or hi-fi skin. Above you can see the low-fi skin. All of the form fields are interactive.

And here’s the hi-fi skin.

You invite participants by sending them an email containing a unique URL. When they click through, they are taken to the page shown above. Pidoco° has a built in VOIP tool (much like Skype), or you can fall back on good old-fashionned telephones. Then you simply run the usability test over the web, much like you would in a face-to-face session. While you are interviewing them, the session is streamed to your web browser. Obviously it’s not going to feel as intimate and easy as a face-to-face test, but it’s far cheaper, particularly when you are testing people spread over large geographic distances.

This is the participant’s screen during a test. The narrow grey bar along the top is inserted over the prototype. The entire session is recorded in the same way as any other screen recording tool (e.g. jing, camtasia, etc), and it includes the VOIP audio. The footage is saved online for you, within your (secure) Pidoco° account area.

Bargain hunters among you may be thinking “I could do all of that using free tools instead of Pidoco!” – This is completely true. You could, for example, use yuuguu for the screensharing (free), hotgloo (still in free beta) for the prototyping and camstudio (free) for the screen recording. However, if you cobble together your own assortment of tools, you wont get the tightly integrated workflow process that Pidoco° offers. For example, on Pidoco°, moving from prototyping to a user test only requires one click. Plus, you can edit a live prototype during a user test, which is trivially easy. It’s these little details that promise to make Pidoco° different to the masses of other prototyping tools on the market today.

Disclosure: I received no incentive for this review (not even a cup of coffee). However, Pidoco° was one of the sponsors of UX Brighton Remote 360, an event I helped organize. UX Brighton is a free event, and sponsorship covers the cost of venue hire and refreshments for the attendees.

Back in the 1990s, when wireframing was a niche activity, you were pretty much limited to Visio or Illustrator. Nowadays there are a huge number of alternatives. If you want an online app, you can choose from Balsamiq, Just in Mind, Jumpchart, iPlots, iZotz, HotGloo, Connect-A-Sketch, ForeUI, Pidoco, Simulify, Mockup screens, Mocklinr, Wireframe sketcher, Gliffy, Lovely Charts, Project Draw, Creately, Napkee among zillions of others. Offline the situation is equally messy, we’ve got Axure, Omnigraffle, Visio, Sketchflow, iRise, Inkscape, Illustrator, Fireworks, Indesign, Pencil, Denim, Serena, Qmockup, Flairbuilder, Photopro, Caretta Studio, and there’s also reams of GUI builder apps if you’re designing desktop apps. The list just goes on and on. How do you know which ones to use and which to avoid?

Has anyone actually tried them all and created an über comparison table? Not as far as I can tell. Instead I’m sort of hoping for some kind of K-T event to kill of all of the weaker ones. Not sure how that would work, though. Any ideas?

Amendment: post has been repeatedly edited to include additional tools.

At UX Brighton last night GiGi Demming (Head of User Testing at SCEE in London) gave a talk on gameplay research involving kids. She neatly summed up one of the problems with interviewing kids – the “I like turtles effect”:

Using a feedreader and can’t see the turtle video?

This video went viral a couple of years ago and has had about 12.5 million views, so you may have seen it before. When interviewed later, little Jonny admitted “I was just having nervous thoughts in my mind and thinking about turtles, I found a snapping turtle there that was really cool and I just wanted to blurt it all out.”. Ah, bless!

Also speaking last night was Gareth White, Director of Vertical Slice, a new Brighton-based gameplay research agency. Among other things, he talked about some of the differences between playability and usability. He showed a clip this hilarious Zero Punctuation video by Ben Croshaw, which highlights the importance of baseline usability in gameplay design. It’s well worth a watch if you haven’t seen it (contains lots of swearing). Fast forward to 2:20 if you’re in a hurry.

Using a feedreader and can’t see the Zero Punctuation video?

If you like that video, there’s loads more here.

Taken from Rapid Problem Solving with Post-it notes by David Straker (1997). Despite its age this book contains some really useful techniques for stakeholder workshops, such as group sorting activities, trees, and maps.