90 percent of everything : Usability Blog

Written by Harry Brignull

I’ve noticed a fair few designers muddle up observations with recommendations when analysing user research findings. This can really screw up your design process, but thankfully it’s quite an easy one to avoid.

It’s important to always state observations separately from your design recommendations, and to try to state them in a pair. Observations are facts like “Participant 3 failed to create an account successfully” or “There’s a 25% drop out rate on step 3 of the wizard”. If you start with these facts, you add structure to the discussion: all the stakeholders can do at this point is acknowledge that there is a problem, or discuss the validity and generalisability of the observation. Once you’ve cleared this hurdle, then you can move onto the fun bit: design recommendations.

One common mistake is to make a blanket statement about user research, followed by a design recommendation e.g. “User research indicated that there were problems with this area, so we should change all the labels as follows…”. This is bad because it’s opaque. What user research? What problems exactly? Without establishing a bedrock of fact, your recommendations cannot be evaluated properly and could lead you into a bad place.

Another common mistake is to suggest the research findings specifically require a certain design change, e.g. “In the usability test 6/8 users failed to notice error feedback in the payment form, so we absolutely must to use red text here”. User research rarely, if ever, necessitates a specific design change – and until you make that change and test the impact, you cannot know how effective it will be. There will always be myriads of design alternatives, and everyone will have their own opinion. Heated discussion may occur at this point, but that’s OK – the solution is to design and test the advocated approaches.

I should add that there’s absolutely nothing wrong with stating your opinion, but it’s important to make it clear that’s what it is. When doing a post-up activity with sticky notes in a workshop, you may want to use the FOG method: mark each note with F (fact), O (opinion) or G (guess). It’s such a simple thing to do, but it adds a great deal of clarity to the decision-making process.

Using a CAPTCHA is a way of announcing to the world that you’ve got a spam problem, that you don’t know how to deal with it, and that you’ve decided to offload the frustration of the problem onto your user-base. As statements go, that’s pretty lame.

If you ran a high street store, you wouldn’t force your customers to mop the floor before you serve them, on account of the people who came in earlier with muddy boots. That mud is your problem, not theirs. The same goes for spam. reCAPTCHA bothers me the most because it tries to sugar coat users’ frustration and make it palatable to site owners. Helping digitise and preserve literature is a worthy goal, but it’s a task that’s utterly at odds with what your users are trying to do at that very moment.

Sometimes site owners seem to think they really need CAPTCHAs, having been hurt by spam in the past. Without hard evidence, it can be difficult to persuade them otherwise. Well, here’s some good news – I recently got chatting to Chris Korhonen of Animoto, who’s kindly shared some data that could help you talk your clients around.

In case you don’t know, Animoto is a web app that allows users to create video compositions from their photos, video clips and music. According to their press releases, their registered user-base grew from 300k users in August 2009 to 2 million users in November 2010. Roughly speaking, that’s 2,400 new registrations a day – giving them plenty of data to run quantitative research.

In Q1 2009 they ran a simple experiment, looking at the impact of CAPTCHA on registration completion. This is what their signup form looked like at the beginning of the study:

Users were directed to the sign-up form direct from the homepage before they could interact with the product. As you can see, there was a CAPTCHA at the bottom of the form (powered by reCAPTCHA). With this design, they had a conversion rate of roughly 48%. They then removed the CAPTCHA, and it boosted the conversion rate up to 64%. In conversion rate lingo, that’s an uplift of 33.3%! They replaced the CAPTCHA with honeypot fields and timestamp analysis, which has apparently proven to be very effective at preventing spam while being completely invisible to the end user.

To quote Chris:

“We left the test running until the results were statistically significant to a 99% confidence level. We’ve followed the same testing methodology with other bits and pieces – removing demographic fields, moving things around, and so on, but nothing has moved things more than a couple of percent.”

Got any evidence of your own about CAPTCHAs and conversion rates? Comments, please…




————–
Edit #1: For some reason this article has hit the front page of Hacker News and is getting quite a lot of traffic. I should mention that yes, I acknowledge CAPTCHAs are of course sometimes unavoidable. That doesn’t mean, however, that we should ever feel good about using them, nor should we fool ourselves that users don’t mind them.

Edit #2: Links added to articles about honeypot fields and timestamp analysis.