Test Your Own Damn Hypothesis

My colleague Michel published a post on our company blog detailing a recent experiment which followed in a long-line of published experiments dealing with the efficacy of the hamburger-icon-as-menu-metaphor. 

The ensuing discussion reminded me of a long-standing gripe I have with the fetishisation of A/B testing on the web:

As experimenters we are motivated by the promise of finding evidence to support a hypothesis, but the discussion we have as an industry is never about the merits of testing, but about using other people's tests to validate our pre-standing opinions. If experimentation allows us to eschew expert opinion in favour of evidence, then the very value of that evidence is reduced to zero the minute it changes hands.

Robbed of context, the result of an experiment is a fart in the wind. So when people take to twitter to rejoice that Booking.com has put to bed this months-old UX question, or when they moan in the comments that without published data the findings are unreliable; they completely miss the fundamental principle upon which A/B testing should be built - do it yourself, or don't bother. 

In fact, if you were to create a hierarchy of reliable data sources for decision-making purposes, I think it would look something like this: 

  1. Your own experiment data
  2. Your opinion
  3. Someone else's opinion
  4. Someone else's experiment data

I guess that we as an industry struggle to deal with this because so many people dine out on #3. No-one goes to a conference, buys a book or listens to a podcast only to be told that they are better qualified than the presenter/author/host to decide how best to solve a problem. It would be a rather self-limiting career move to suggest that one is less qualified than everyone else in attempting to solve everyone else's design problems.




I decided this weekend that I would like to try my hand at presenting at an industry conference or two next year. I think that I have some interesting stuff to share when reflecting in particular on the 3.5 years I've spent so far at Booking.com - particularly on my favourite topic of the role of data in design and the challenge this poses to the ego of the designer and his position as magician of our modern web culture. 

I've spoken lots to other kinds of audiences - academic panels and peer groups, internally to my several hundred department colleagues at Booking.com, and at a couple of small events where the feedback has been positive, so I want to challenge myself to see if I can translate that in to an engaged, informative and entertaining presentation. This post is my stick in the ground and a prod to myself to actually do something about this, and hopefully a catalyst for me to start thinking seriously about the story I'd like to tell. 

Wish me luck. 

Hoge Veluwe

Victoria arranged for us to spend a few days on the edge of the Hoge Veluwe National Park in Gelderland, the home of the Kröller-Muller Museum, and a forest, and the largest collection of Van Gogh paintings outside of the Van Gogh Museum itself. We stayed in a Finnish-style cabin and had a lovely, relaxing time away from work and the city. The unexpected highlight was the fantastic architecture of the museum itself, which would I wish I'd got more photographs of. It's a late thirties, rather austere construction with a minimum of fuss and lots of big blocks of glass and brick set amongst a landscaped sculpture park in the middle of vast National Park.

The Museum contains a lot of Van Gogh stuff (90 paintings and 180 drawings form the complete collection, a selection of which forms part of the permanent exhibition), but also some unexpected treats from the likes of Mondrian, Monet, Barbara Hepworth and Picasso. We only managed to get around half of the place before we were zapped of energy, having walked across the park to the museum through the wonderful forest. 

Our Cabin - at De Roek in Otterlo. Probably about the same size as our apartment in Amsterdam. 

Our Cabin - at De Roek in Otterlo. Probably about the same size as our apartment in Amsterdam. 


Walking across the park to get to the museum afforded me the opportunity to mess about with my camera again and try through taking endless photographs to get one or two which didn't look like they were taken completely by accident. The forest was beautiful, and totally unexpected in a country I thought was almost completely bereft of this kind of landscape.

On the way back home this morning we stopped off in Utrecht for a mooch around the city centre and some lunch, and I think we've pegged that as the next bit of the Netherlands to spend some time in - the city centre was beautiful, but for whatever reason most of the cafés, shops and museums were closed. There is much to admire a city which refuses to entertain Monday morning.