90 percent of everything : Usability Blog

Written by Harry Brignull

It’s nice to see that Dilbert’s been taking on a UX focus lately…

Step 1: Realisation

Step 2: Confirmation

Step 3: Response

Amendment: this post has been edited for typos: Thanks Carlo!

Unless you’re a mathematically-minded person, it’s easy to get muddled up when people talk about conversion rates and conversion rate “uplift”. Here’s a quick explanation:

• If you had 1000 users visit your site yesterday, and 1 of them made a purchase, you’d have a conversion rate of 1:1000 (also referred to as “0.1%” or “0.001”).
• If today you had 1000 visitors and 3 of them made a purchase, what’s the uplift? It’s 200%. In other words, the uplift is a percentage of the increase, not the increase itself (which is 2:1000 people more).

200% sounds HUGE, doesn’t it. Companies like to talk about it that way for two reasons. Firstly, if the underlying numbers are poor (as in this example), it makes them sound impressive. It doesn’t sound quite so good to say “1:1000 more users made a purchase today”. Secondly, they might simply want to hide the details from their competitors. They could have great figures, but don’t want their competitors to know exactly how they are doing. This is understandable – it’s often very sensitive information.

Here are some questions to consider when you’re shown percentage uplift figures within your own organization:

1. Establish whether AB / Multivariate testing was carried out. Some people talk about uplift when comparing two time periods, e.g. last month vs. this month. The problem with that is time-based variation. For example, you will sell more ice-cream in the summer months, regardless of your sales pitch.
2. Establish the period of the test. AB / Multivariate testing randomises the samples on the fly, so it removes the effect of “before and after” time-based variations. However, the test itself is still time-bound: if your test was run on christmas day alone, then you should have doubts about whether the findings will generalize to every day of the coming year. Why? Because you will be getting a different sample of users who will be behaving differently to normal.
3. Get access to the real numbers, not just the uplift percentages. Ideally, take a look at the reports produced by the testing platform itself. For example, GWO produces unintimidating reports.
4. Find out what the statistical significance is. If you toss a coin twice today and then do it again tomorrow, you might get a 200% uplift in incidence of heads. On the face of it, you might be tempted to generalize that your technique is much better today. Of course, that’s plain wrong – you just need a much larger sample size. (Check out User Effect’s split test calculator)
5. Find out how the conversion goals are defined. For example, is the goal “the number of people who enter the checkout process” or “the number of people who actually complete a purchase”? There’s a big difference.
6. Find out what the research objectives were. Why did they run the study? What were the hypotheses? Were the alternative designs based upon any prior research or theory? Have the findings confirmed expectations or confused them? Answers to questions like this are solid gold – they will extend your organization’s understanding of the relationship between design, psychology and user behaviour within the unique context of your own offering.

Lets look at an example: 37signals reportedly achieved a 200% uplift in an A/B test on Highrise registrations when they tested a call-to-action that read “Sign up for a Free Trial” against an alternative wording: “See Plans and Pricing”. (Read more on Ryan Carson’s blog.) Because it’s 37signals, we can assume they’ve got ample traffic and good conversion rates to start off with, which implies that their uplift is probably very impressive. But hang on a minute – if we’re put our scientist hats back on, we’ll remember that this is only an assumption. We don’t know the answers to many of the questions above. We haven’t been given enough information to evaluate the meaning of the percentage given. It’s intriguing enough for a press release, but it wont wash when communicating within your own organization.

Oh, and by the way, just because they got a 200% uplift for that copy change, it doesn’t mean you will. You can’t reliably generalize AB test results between sites – you have different users, in different states of mind, looking at different page designs. It might even have the opposite effect in your context, so it’s worth doing some user testing of your own. Don’t forget to report it properly, though!

Theresearchbuyersguide.com looks like a potentially useful site for UX researchers:

Though it’s it’s intended for buyers of generalist market research, there are some useful links on here, such as a list of viewing facilities in the UK (i.e. posh places to run your research if you’re a freelancer or lacking the space in-house).

Date: Tuesday 11th August, 6.30pm – 10pm
Location: iCrossing, Central Brighton, BN1 1ND
Price: Free, but you must book a ticket on stubmatic

This free event will be comprised of three parts – two talks and a demo!

1. Carsten Schmitt & Poppy James are coming down from Bunnyfoot to give a short talk on “How to run successful international user research projects”
2. Rob Gillham of Human Factors International is going to give a talk on “Designing for Persuasion, Emotion & Trust on the web”
3. The Bunnyfoot team will be giving an eye tracking demonstration, over drinks & nibbles

1. How (not) to run successful international user research projects

Speakers: Carsten Schmitt & Poppy James, Bunnyfoot.

• User research across several countries – be it surveys or user tests – has its own challenges. Some of them are well known (but nevertheless often forgotten) others are unexpected.
• Poppy and Carsten – a Briton and a German – do not aim to give you the ultimate 101 of international user research. A lot has been written about that already. What they do want to do instead is to talk about their own experiences to show you which problems you could encounter before you even start your testing abroad and what could happen if you don’t have that foreign language expertise in-house.
• Prepare to be surprised, shocked, and, hopefully, amused. And the end of this talk you should be able to add a few bullet points to your checklist for the next international project (bullet points will be provided).
• Some of the topics include “Foreign affairs – Managing multi-national clients” and “Parlez you Italiano? Language issues beyond ‘translations’”

Carsten Schmitt & Poppy James are consultants at Bunnyfoot, a User Experience Consultancy that’s passionate about Eye Tracking. Bunnyfoot’s clients include the BBC, the COI, Virgin Holidays, and Yell.com to name only a few.

2. Designing for Persuasion, Emotion and Trust On The Web

Speaker: Rob Gillham, HFI

• Since the early ’60s, Social and Experimental Psychology Research that has taught us a great deal about the nature of persuasion in human psychology.
• In recent years, there’s been a flurry of books and articles on the subject – after all, persuasion is part of the elusive magic recipe for success in business. What’s been lacking, however, is a selection of case studies that give a solid understanding of what works and what doesn’t in the context of human-computer interaction.
• This is exactly where Rob Gillham’s talk picks up. Rob will give a talk covering the underpinning Psychological research and insights from HFI’s own research findings.
• Rob’s talk involves a 1 hour sneak peak into HFI’s ‘Psychology, Emotion & Trust’ training course, normally only available for paying customers.

Rob Gillham is a Project Director at Human Factors International (HFI), the world’s largest UX consultancy with offices in UK, US, Germany, India, China and Singapore.

3. Audience Participation Eye tracking demo

• After the talks we’ll be cracking open the beers (and wine and snacks) for an eye tracking demo from the Bunnyfoot crew.
• To make it more fun, we’ll be accepting submissions from attendees. Feel free to email Harry Brignull with your site screengrabs, art, or naked photos (1024×768 PNGs are ideal).
• Submissions will be drawn at random from the pile and used in the demo. (PS no naked photos).

This event is sponsored by Madgex and iCrossing.

There are certain kinds of user research that require extreme video analysis, where you literally have to transcribe every utterance, gesture and movement, and a single hour of footage can take almost a day to analyse. You can imagine this is a painstakingly tiresome process, so this open source app by IBM researcher Eben Haber seems like it would be pretty useful:

The app is called Video Note Taker and it does exactly what it says on the tin. Once you’ve learned a few key combinations you can easily scrub back and forth through the footage at different speeds, bind each line of your notes to a timestamp in the video, and then (here’s the clever bit) jump to the point of in the video that each timestamp corresponds with.

There are a number of other apps that provide similar functionality – including Microsoft’s OneNote, the captioning tool in Techsmith’s Camtasia, and a whole ream of dusty-looking academic apps. However, since Video Note Taker is free, simple, and easy to learn, I can see it being quite popular within its niche.

Video Note Taker requires Quicktime and Java, and runs on both on Windows and OS X.

→ Find out more about Video Note Taker

(Found via Melissa Cefkin on the Athrodesign mailing list).