I really enjoyed this article on Wired.com this week – The A/B Test: Inside the Technology That’s Changing the Rules of Business – and not just because it introduced me to a term that is going to be a mainstay of my vocabulary for years to come!
That term is HiPPO – highest paid persons opinion – and it is apparently a Googler term for when decisions are made by a senior member of staff based on their opinions rather than making data driven decisions. I really do like this term and it certainly chimes true but over the years my biggest problem has actually been the design by committee/groupthink kind of thing. The desire to be ‘collegiate’ in some of the places I have earned my crust has been crippling at times. Once or twice I could have benefited by some HiPPO action 🙂
The core of the article is around the extensive use of A/B testing by the major online companies and how their decision making is now based increasingly on the results they get from these tests. The anecdote about the Obabma campaign homepage was particularly interesting – especially the fact that the expert assumptions were so often proved wrong when the A/B testing was initiated and also just how considerably wrong they turned out to be;
Had the team listened to instinct—if it had kept “Sign Up” as the button text and swapped out the photo for the video—the sign-up rate would have slipped to 70 percent of the baseline.
As someone starting to seriously think about a major website overhaul and working primarily with people for whom data is golden I have to say I am fascinated with the possibilities – even for a small site like ours. The question is going to be just what is possible financially and time wise but I do think this sort of activity coupled with the analytics data already collected could make a powerful case for change. I am also interested to find out how far my assumptions are incorrect – I certainly have my opinions on how elements of the site could be improved but perhaps the data will prove me as out of touch as those Obama staffers!
[as an aside I wonder if the Cookies law is going to torpedo all of this anyway?]