90 percent of everything : Usability Blog

Written by Harry Brignull

So, according to Jared Spool’s Keynote at IA Summit 08, “User-Centred Design is dead”. There’s a good write up by Mia Northrop of Avenue A | Razorfish here, and another by Molly Anglin of NLC here.

I usually love challenging the status quo and making bold claims to stimulate discussion. But in this case, I think it wasn’t particularly constructive. Why? Because for many people, UCD is a wooly philosophy that sits somewhere between the concepts of “putting users first”, Usability and, perhaps the vaguest of all terms, User Experience. To say UCD is dead raises the question, what exactly do you mean by UCD?

Let’s take a look at some excepts from Jared’s slides:

To say that “UCD never worked” implies that it’s a single thing. It’s not: it’s a container term. It’s a bit like the term ‘Web 2.0′: what goes inside depends on who’s holding it. Generally, though, the agreed constituents look a bit like this:

There are many different flavours, sizes and scales of UCD - different organisations tend to implement it differently in their design processes. So, unsurprisingly, those that love red tape and strict process will implement it in that way, while those that at the other end of the scale will use it as a flexible, lightweight approach.

So, should we really be throwing out a perfectly good container, or should be trying to hone and improve the recipe of things that go inside? Although Jared has overcooked his arguments, he’s done us all a service by stimulating a useful discussion about the recipe of effective design, regardless of what we call it.

Photo credits: Balakov on Flickr

Quietly announced on Twitter - Clearleft’s silverback app is going to be “Usability testing software for web designers”. See, I guessed right back in February!

As a result I’ve ended up on the alpha testing team. My lips are sealed except to say, yes, it is nice.

Ethnio has come on a lot since I last checked - now it’s looking pretty damn good.

Ethnio is a tool for finding participants for user research by enlisting live users from your website. If you’re not sure what I mean by that, take a look at this explanatory video, charmingly made in felt using stop motion.

Ethnio requires you to have a live website that already has a decent user-base, so it’s not suited to early-stage research or non-web products.

It’s strength is in the amount of money it saves you. Normally, recruiting end users for face-to-face user research is hideously expensive. For example, in London, if you have specific criteria for your users, you’re talking £100 for the recruitment plus £50 incentive (i.e. payment for participation).

Ethnio is completely free to use, and it cuts out the recruitment costs - you pay users an incentive, and that’s it.

Last week Clearleft announced that they are building a “revolutionary” new app, called Silverback. As a viral marketing tactic, they haven’t told anyone what it actually does, and have only released a rather lovely splash page. Well, I’ve clearly succumbed to their marketing, because here I am blogging about what I think it’s going to be.

Observation: the Silverback gorilla logo is wearing a lab coat, holding a clipboard.
Interpretation: maybe it’s something to do with analytics or at least some form of research.

Observation: guerilla sounds like gorilla, and nerdy research types (like me) recognise the phase “gorilla usability testing” as a bit of an in-joke.
Intepretation: maybe it’s a tool for usability research.

Observation: Clearleft is moving steadily into user experience. Last years DConstruct conference (organised primarily by Clearleft) was user experience themed. Andy Budd is doing a talk on guerilla user testing at the FOWD conferece this year, which is a substantial departure from his prior fare of CSS and web standards.
Interpretation: to do user experience design properly, you have to pair it with user experience research in an iterative process called User-Centred Design. This process is pretty well known and even has an ISO standard associated with it. With this in mind, it makes sense that Clearleft, moving along their current trajectory, might consider making an app for user research of some kind.

Reasons why I may be wrong:

• Clearleft refer Silverback as a desktop app, not a web app, which probably means no tracking / analytics.
• They say the application is for web designers, as opposed to website owners, managers, or any other likely culprits.
• They have a much stronger background in UI development and web standards than user research. So maybe I’m barking up completely the wrong tree.

This demo video came out in December, but if you haven’t already seen it, you can watch this edited down version (without the warm-up chatter).

Online Videos by Veoh.com

Thermo is basically a tool for interaction designers (rather than developers), to bridge the gap between their photoshop mock-up and a fully interactive user-interface. The demo video implies that Thermo will make excellent prototypes for user testing, and then the UI can be completely re-used by the dev team with little or no tweaking.

Lets hope it lives up to its promises, because if it does, it will rock. No release date has yet been given.

I’ve been delivering a lot of User-Centred Design training lately at Flow, and I’ve noticed that when most people do paper UI sketching, they can’t help going “hi-fi”, and making very precise wireframes.

It’s just too easy to get sucked into the minutiae rather than maintaining a focus on the bigger picture. Plus, precise UI sketches can end up taking hours to make, so then when you begin the evaluation phase, the author is inevitably feeling defensive over their baby. And since the small details are present - the wording, layout, and so on - feedback then ends up focusing on it. You end up drilling further and further into the detail because it’s such a tempting, solid thing to talk about.

Instead of small details, this initial stage of design sketching should concern things like proposition (Does the overall idea seem useful?), concept (How does it deliver it’s value?) and context (Would it fit in with the other things the user is doing, e.g. before and after using it?)

This is where ‘boxing glove’ wireframing technique comes in. You don’t actually wear boxing gloves (sorry for the let-down), the idea is that you take measures to physically compel yourself to do very basic, very quick sketches. It’s a bit like Andy Budd’s idea of “one day design concepts“, but at a scale of minutes rather than days.

It’s simple:

1. Grab a big pad of post-it notes
2. Grab a felt tip pen
3. Sketch each page on a single post-it
4. Draw a single user-journey through the system. Concentrate on the ‘happy path’, i.e. ignore contingencies for now.
5. That’s it!

The constraints of the small paper makes it feel a bit like you’re wearing boxing gloves - it forces you to draw only the most crucial parts of the user interface. (If you want to use bigger paper, just use a fatter pen). This enables you to hammer through a user journey in a few minutes.

Another nice benefit is that this method invites participation in a way that finessed diagrams don’t - anyone can join in, no special skills necessary. If you’ve read Bill Buxton’s Sketching User Experiences, you’ll know exactly what I’m getting at here.

Coming soon - GUI whiteboard magnets from Eight Media in the Netherlands.

These look quite useful for collaborative wireframe sketching sessions in front of the whiteboard. I’ll definitely be ordering some.

The nice folks at Techsmith are giving away free copies of Snagit 7.5.2 and Camtasia 3. Both apps are really useful for UX professionals, and they’d normally set you back a few quid. They aren’t the newest versions but they are still very good.

Get Snagit 7.5.2 here, and get Camtasia 3 here.

There’s been a fair amount of discussion over the past week about eye tracking. As someone who used to do a bit of qualitative eye tracking research at Amberlight, (a great London-based UCD consultancy that I used to work at), I have a few factoids and opinions that I’d like to share.

1. Quant vs Qual eye tracking research - the key differences

Qualitative eye tracking research (“Qual”) involves observation and interpretation. First, the user is given a task while their eyes are tracked. During this time they are not allowed to speak (to discourage them from looking away from the screen). Then, after the task, the user is interviewed about their experience, and depending on your method, you can play the gaze trail video back to them and ask them to describe what was going through their mind. Analysis of Qual research involves touchy-feely “human” skills rather than statistical analysis. With Qual studies, you normally test between 8-20 users (roughly speaking).

Qualitative research is like a form of detective work. You look at the pieces of evidence you have available, and you try to fill in the gaps based on your past experience and expertise. It’s is a bit like looking around a crime scene and saying “Well the window is broken, the room has been upturned, it looks like a robbery.” But – the big but – you never actually get concrete conclusive proof that this is the case. This is the nature of Qual research, but don’t let it put you off - Qual is the cornerstone of most design research.

Quantitative, on the other hand, involves recruiting a large number of volunteers (e.g. 75-200), so you can collect enough data for statistical testing. This costs substantially more, and it takes a lot more time. Also, your research objectives are screwed down much more tightly – you control variables, you have hypotheses, and everything feels a lot more like “laboratory science”. The big differentiator is that although you get statistical evidence, you only get a sense of “what people did” but not “why they did it”. Qual, on the other hand, gives you a lot of “why” findings but no statistical evidence. With Qual, you have to put your trust in the expertise, experience and past portfolio of your researchers. When doing a Quant study, it’s fairly common to have a Qual element bolted onto it (e.g. interviewing the participants as well as tracking their eyes).

In summary: the point I am making here is simply that there are two types of eye-tracking, qualitative and quantitative. It’s important to understand the difference between the two, which leads onto my next point.

2. The “Qual-Quant confusion” problem

One problem with eye tracking is that the people who buy the service often get confused about what type they are getting. The high tech kit, the gaze trails and heatmaps look impressively scientific. It feels like the most rock solid evidence you’re ever going to get. In a qualitative eye tracking study, this just isn’t true. I’m not saying it’s fictional - I’m saying it’s just a picture of what a dozen or so people fixated on. In other words, it can useful but it’s just another piece of supporting evidence, like the verbal statements of your users, or your observations of their behaviour (e.g. task failure rate).

Going back to our “qualitative research ≡ detective work” metaphor, both eye tracking heatmaps and hand-written notes are like ‘clues’. They require human interpretation. The researcher has to sit there, scratch their head and think about what it all might mean.

In summary: in a qualitative study, eye-tracking data is no more valid or conclusive than your handwritten interview notes. Just because heatmaps and gaze trails are shiny, impressive and expensive doesn’t mean they hold any special “weight”.

3. Eye tracking tells you what users looked at, but not why.

Imagine you have run an eye tracking study on your site, and you now have a heatmap showing what 12 users looked at. You notice that there is a lot of heat on your proposition statement. Great, you think to yourself, they read the proposition! This means they understand what we are offering!

Actually, you’ve just made an assumption - that the more someone looks at something, the more they understand it. Actually, the opposite might be true. The heat may be a symptom of confusion - users might be re-reading your statement repeatedly because they found it hard to understand.

Just because someone looks at something doesn’t mean they understand it or like it. This is why eye tracking is almost always paired with interviewing where you try to find out what users were thinking. In short, beware of heatmaps - they can be easy to misinterpret.

4. Eye tracking often doesn’t tell you anything new

Another problem with Qual eye tracking is that often, it doesn’t tell you anything that you wouldn’t have found out through other means. Take a look at the eye tracking evidence provided by Luke Wroblewski in this study here. It’s a great article for educational purposes, but to an experienced eye, it’s quite obvious which would be the winning and loosing design patterns. Theory and principles like “affordances”, “visual hierarchy”, “call to action” and even Nielsen’s heuristics would all point you towards the right approach. And if you use “standard” user testing methods (no eye tracker, just a 1-on-1 interview where the user is given tasks and thinks aloud), you can test your designs without the cost of eye tracking - simply record task time, failure rate, and gather user feedback.

I’m not saying “don’t do qual eye tracking”, since it can be a great educational tool, it can uncover things you may not have otherwise noticed, and it can be a great weapon when trying to fight the user experience corner with a difficult stakeholder. But you’re unlikely to suffer if you don’t do it, and instead you opt for cheaper “standard 1-on-1 user testing” (by this I mean giving users tasks, asking them to think aloud and interviewing them).

In summary: if you don’t have a huge budget, and your research objectives don’t explicitly require eye-tracking, then you should seriously consider other options. Eye tracking can be expensive beyond its usefulness. Instead of doing a single, costly eye tracking study, it can be more rewarding to do multiple rounds of “standard” user testing (iterative design & research), and employ a collaborative process using a war room and design workshops. Involve your design team: research should be an intimate part of the design process, rather than something carried out by strangers and delivered via PowerPoint bullets.

I read a nice factoid on this topic this morning in Eric Schaffer’s Institutionalization of Usability book. It’s a quote from Jared Spool:

In April of 2002, Princeton Survey Research Associates surveyed 1,000 adult Internet users about their concerns with privacy on the Internet. In the survey, only 18% said they never read privacy policies most of the time, or every time they shop.

Yet, in our study of more than 1,000 shopping sessions, where we actually observed what users did while shopping, we noticed that only two users ever checked the privacy policy. And for these two users, it had no effect on their shopping behaviour. This is yet one more case of users doing something different from what they say they do.

To reiterate this point -

What users said they do:
82% of users said they read privacy policies (from survey data)

What they actually did:
0.2% looked at privacy policies (in user tests)

This ‘observational data vs self-report’ argument is a road well trodden (read Jakob’s 2001 alertbox on this topic here, or a great signal vs. Noise post here), but I like this factoid because it sums up the argument so well.

[Note - typos have been removed from the original post. Thanks Jared!]