Test Your Own Damn Hypothesis

My colleague Michel published a post on our company blog detailing a recent experiment which followed in a long-line of published experiments dealing with the efficacy of the hamburger-icon-as-menu-metaphor. 

The ensuing discussion reminded me of a long-standing gripe I have with the fetishisation of A/B testing on the web:

As experimenters we are motivated by the promise of finding evidence to support a hypothesis, but the discussion we have as an industry is never about the merits of testing, but about using other people's tests to validate our pre-standing opinions. If experimentation allows us to eschew expert opinion in favour of evidence, then the very value of that evidence is reduced to zero the minute it changes hands.

Robbed of context, the result of an experiment is a fart in the wind. So when people take to twitter to rejoice that Booking.com has put to bed this months-old UX question, or when they moan in the comments that without published data the findings are unreliable; they completely miss the fundamental principle upon which A/B testing should be built - do it yourself, or don't bother. 

In fact, if you were to create a hierarchy of reliable data sources for decision-making purposes, I think it would look something like this: 

  1. Your own experiment data
  2. Your opinion
  3. Someone else's opinion
  4. Someone else's experiment data

I guess that we as an industry struggle to deal with this because so many people dine out on #3. No-one goes to a conference, buys a book or listens to a podcast only to be told that they are better qualified than the presenter/author/host to decide how best to solve a problem. It would be a rather self-limiting career move to suggest that one is less qualified than everyone else in attempting to solve everyone else's design problems.




I decided this weekend that I would like to try my hand at presenting at an industry conference or two next year. I think that I have some interesting stuff to share when reflecting in particular on the 3.5 years I've spent so far at Booking.com - particularly on my favourite topic of the role of data in design and the challenge this poses to the ego of the designer and his position as magician of our modern web culture. 

I've spoken lots to other kinds of audiences - academic panels and peer groups, internally to my several hundred department colleagues at Booking.com, and at a couple of small events where the feedback has been positive, so I want to challenge myself to see if I can translate that in to an engaged, informative and entertaining presentation. This post is my stick in the ground and a prod to myself to actually do something about this, and hopefully a catalyst for me to start thinking seriously about the story I'd like to tell. 

Wish me luck.