This isn’t breaking news, but it’s an interesting enough tidbit to deserve being shared around. Back in November ’08, Chris Goward of Wider Funnel noticed that Obama’s team used Google Website Optimizer to run an A/B test on barackobama.com. Unsurprisingly they didn’t publish the results. If you look at variation D, you’ll see that one of the purposes of this test was to ascertain whether omitting the free t-shirt (variation D) would have a effect on the donation rate. Printing and shipping all those t-shirts must have been pretty pricey!
It seems like everyone is doing A/B and multivariate testing these days. If you’re not, then maybe it’s time to get busy?
(Found via WiderFunnel, the Conversion Rate Optimization Agency)
What the team failed to test was an alternative garment offering – perhaps an attractive Obamathong? They’d now have enough donated money left over to fix the floundering US economy.
Haha! This is exactly the limitation of A/B testing – you only ever find out which of your *existing* ideas are most effective…
What’s the alternative to an *existing* idea? An unknown unknown? ;)
A/B testing is evaluative – it allows you to converge on the best item from a predefined set of items. If you have 10 bad ideas, you will only find out which one is the least bad.
Other forms of research are more generative – they stimulate innovative thought and creative design. Like contextual field research / Ethnography, for example.
Pingback: Presidential Promotions « Rpromo’s Blog