90 percent of everything : Usability Blog

Written by Harry Brignull

I read a nice factoid on this topic this morning in Eric Schaffer’s Institutionalization of Usability book. It’s a quote from Jared Spool:

In April of 2002, Princeton Survey Research Associates surveyed 1,000 adult Internet users about their concerns with privacy on the Internet. In the survey, only 18% said they never read privacy policies most of the time, or every time they shop.

Yet, in our study of more than 1,000 shopping sessions, where we actually observed what users did while shopping, we noticed that only two users ever checked the privacy policy. And for these two users, it had no effect on their shopping behaviour. This is yet one more case of users doing something different from what they say they do.

To reiterate this point -

What users said they do:
82% of users said they read privacy policies (from survey data)

What they actually did:
0.2% looked at privacy policies (in user tests)

This ‘observational data vs self-report’ argument is a road well trodden (read Jakob’s 2001 alertbox on this topic here, or a great signal vs. Noise post here), but I like this factoid because it sums up the argument so well.

[Note - typos have been removed from the original post. Thanks Jared!]

If you don’t know much about Personas, or have your doubts about them, read this article and the comments thread over at Signal vs. Noise (the 37signals blog). Here’s an excerpt:

We don’t use personas. We use ourselves. I believe personas lead to a false sense of understanding at the deepest, most critical levels.[…]

I’ve never been a big believer in Personas. They’re artificial, abstract, and fictitious. I don’t think you can build a great product for a person that doesn’t exist.

As you’d expect, there was a bit of a backlash in the comments thread. It makes interesting reading, not because it gives 37signals a bashing, but because it clearly articulates some common misunderstandings of Personas, and explains how they are wrong, e.g.

• Personas are “baseless fictions” (Actually, they are the synthesis of your research findings)
• Personas are a replacement for user-research (Actually, they complement user research)
• Personas don’t get frustrated or express opinions (Ok, they don’t really exist, but they are used to simulate exactly this kind of user feedback to help making design decisions)

> Read the Signal vs. Noise article

This post by Christopher Fahey of graphpaper.com got me thinking about the longitudinal nature of usability issues, and what it means for user experience research & design.

Now: the teething problem
This is a usability issue that you experience at first, but eases off as you get used to the product. For example, the first time you used an ipod you probably had at least 30 mins of familiarization and teething problems (”hey, this isn’t like my minidisc player, but it is kind of cool…”).

Later: the ill fitting shoe
This is a usability issue that you don’t notice at first, but it becomes increasingly bothersome as time wears on. For example, some voicemail systems that plays you the same long introductory message every time - educational at first, but after a while you start to notice every second it is stealing from your life.

So, what does this means for evaluation?

User tests often evaluate the first 60-90 minutes of use. This is a defining period but how much time the average person will use the product in total? If they are clocking up many hours of use every week, then the first hour of use is, in a sense, just a edge case.

To evaluate this product’s user experience, you’d have to employ longitudinal techniques - diary studies, repeat testing, telescoping, betas, and of course maintain an open dialogue with your customers.

If you don’t take a longitudinal view, then you are at risk of angling your product too specifically at beginners, and in doing so alienate the experienced users - or vice versa. Research methods are like a narrow beamed flashlight in a dark cave. To get a good view of things you have to expend a fair amount of effort looking at things from different angles.

And what does it mean for innovation?
Sometimes, when you try to avoid teething problems, you conform to the status quo, and you don’t help users in the long run. Imagine if the iPod ended up looking like the Creative Rio Mp3 player, because people in the user test sessions couldn’t quite get their heads around the concept of click-wheel navigation.

But sometimes arrogance can get the better of you. So some teething problems go away - how do you know if your design will do this? And, even if it does, can you justify the pain?

Google’s new “website optimiser” is one of the biggest and most exciting user research tools to emerge on the scene in quite some time.

It’s a bit surprising, then, that hardly anyone in the User Centred Design field is talking about it. Perhaps it’s passed a lot people by – after all, it does have a rather bland name, it’s been released alongside Adwords (their advertising product), and the sales blurb says that it’s for “online marketers”, which is not a title normally associated with UCD.

Basically, Google Website Optimizer allows you to evolve your site using live user data. You give it a number of different page elements and it alternates between combinations of them on your live site. It gathers performance data (i.e. click-throughs) and you get a live report on which combination is the most successful.

It’s like natural selection- the good combinations win, the bad ones die, and you sit back and watch it all happen.

But it won’t solve all your design problems - it doesn’t compete with traditional UCD tools like lab-based user testing, participatory design and so on. Its place is at a very specific part of the design process, when you have some very specific questions to ask. Here’s a quick illustration:

A few months ago I did some user testing for a .com start-up who was focussing on achieving a huge number of sign-ups. We held 8 one-on-one user sessions on a prototype of their website. Afterwards we had some great findings. Users had real difficulties understanding the proposition of their service – it was just so new and weird, they have nothing to compare it to. Through a process of behavioural observation, interviewing and low-fi design exercises, we nailed down where the problems were and how to solve them. By the end of it we had a new tag line and step-by-step explanation of the service that most users grasped instantly.

This is something that Google Website Optimizer would not have been able to do. It can’t talk to your users to find out what they are thinking, and it can’t engage with them creatively. With Google Website Optimizer, all the richness of your customers’ needs, goals and expectancies gets left behind. All you get is some numeric data showing what they clicked.

Going back to the example scenario - by the end of the project debrief meeting we got to a point where many of the big issues had been dealt with, but lots of smaller issues started cropping up. Should the copy on the registration button read “Join” or “Sign up”? Should it have some associated copy like “No credit card needed” or “First month free”? What about the bullet points on the front page? We had 6 good ideas but we only had space for 3 bullet points, so which should we choose? You get the picture - we’d sorted out the big issues, but now we needed to finesse. This is exactly where Google Website Optimizer should come in.

In summary: it’s fantastic, but not a magic bullet. Consider it just another tool in your User-Centred Design toolbox.

I’ve been having a lot of fun doing out of box experience design consultancy over the last few weeks (OOBE as it is pretentiously called by those in the know). If you’ve ever opened an Apple product then you’ll know what an excellent out of box experience is; and, if you’ve ever opened a packaged copy of Windows Vista you’ll know what a bad out of box experience is. [See a nice comparison on Robert Hoekman’s blog, via Reaction.]

If you look around on the web, there isn’t much in the way of out-of-box design guidelines, with the notable exception of IBM’s offering. So I’ve put together a mini set of guidelines based on a metaphor of the bento box.

A bento box:

• Is a joy to use.
• Actively improves your perception of the contents, through attention to every minute detail.
• Encourages an order of consumption - the physical structure affords consumption of the top layer first. This is very useful if you need your user to do things in a certain order.
• Compartments are spacious enough to allow easy access to contents.
• Makes it just as easy to put things in as to take things out.

So next time you’re doing OOBE design with a bunch of non-UX people, introduce this metaphor to your team. It’s quick, it’s easy and it’ll give you a piece of common vocabulary to hang your ideas off.

I’m really interested to hear your comments - so please comment below! >>>

When people normally think about accessibility they normally think about standards compliance, automated tests, and box ticking. This is really important stuff, but it isn’t user-facing. In other words, you don’t get to find out what it is really like for people with accessibility issues to use and experience your product.

Accessibility field testing is a good method to address this. Basically, you recruit a range of users with accessibility issues, visit them in their homes (you have to do this since accessibility set-ups are very bespoke), then you interview them and carry out some user testing.

Locating these users has traditionally been quite difficult, but its great to see some services springing up to fill the gap (in the UK at least): The Shaw Trust offers recruitment and testing services, and so does the RNIB (although only for vision impaired users).

Accessibility field testing is important because there is a difference between ‘looking good on paper’ (standards compliance), and what it is like in practice, i.e. whether your product is joy to use. Bear this in mind next time you are scoping a project with an accessibility component.

[ Source: Boagworld.com ]

There has been a sudden mushroom of posts in response to Donald Norman’s recent essay about simplicity entitled “simplicity is overrated”. (Joel Spolsky, Nick Bradbury and many other bloggers).

I have a feeling that this discussion is getting confusing because we haven’t really pinned down what kind of simplicity we are talking about. It seems to me that Donald Norman (and most of the responses) are talking about “feature simplicity”. You also have “usage simplicity”.

“Feature simplicity” refers to keeping the number of different features a product offers down to a minimum. This keeps menus nice and tidy; and since there are less features, there are less things to mess up, so it’s easier for the designer to be sure they have done a good job at designing the user experience.

“Usage simplicity” is something different. This is a “perceived” simplicity, i.e. how much simpler it makes your life. There might be a hell of a lot of complexity going on back stage to provide this. Consider auto-focus in cameras. It adds to the number of features, but at the same time it makes your life easier. Also consider Gracenote’s CD track identification service that is integrated into iTunes. You stick a CD in, it automatically looks up the names of the tracks online and shows the names. A lot of users probably don’t realize this is happening but still enjoy its benefits.

It’s important we don’t confuse the two. It interests me that this is exactly the kind of point that Donald Norman would normally make, but chose not to in his recent essay.