Four Ways to get Measurement Wrong

Recently, we were asked by a client to  think about best practices in a user experience measurement program. We can’t tell you what we came to recommend, but we can share our starting point – a reflection on several things we’ve seen go wrong in the process of marketing measurement.

1. Only Measure the Things That Matter To You

Now you might be asking yourself, why would you measure anything else? Well, John Sealy Brown, in The Social Life of Information relates a worthwhile story of an innovation overlooked at Xerox Park – a little invention called the graphical user interface. While the inventors felt with some conviction that they were on the cusp of something important, the sponsors of the research effort were unable to discern how the GUI could help Xerox sell more toner cartridges. Had the question been reframed in terms of how the GUI could help Xerox customers create more value with documents, the last two decades of corporate history might have worked out very differently for Xerox, and Apple too.

Take by contrast the nearly obsessive emphasis that Steve Jobs reportedly associated with the sound output levels of Apple’s MP3 player prototypes prior to the launch of the iPod. It seems evident from iPod advertising at launch and ever since that Jobs had a vision of a product experience that would be at once immersive and transporting – and these experiential qualities would be limited by a music level too low to make the ambient soundscape vanish. Toner cartridges were a trailing indicator of marketing success – and the wrong metric for a new innovation. Volume levels were a leading indicator of a sort of experience – and the right metric, or at least one of them, to guide the introduction of a massively successful product. 

2. Treat Research Methods Like a Sports Fan Treats Teams

Sports fans are allowed to regard their team with unconditional  loyalty, under almost any circumstances. It’s kind of an endearing quality, in fact, to see such loyalty among the particularly long-suffering fans. (Those still rooting for the Toronto Maple Leafs come to mind.)

But a “fan-like”  relationship with one class of market research methods can lead to corporate losing streaks almost as protracted as that of the Leafs. While it may still remain fashionable in some business circles to claim a powerful affinity for quantitative research  in favour of the “softer” qualitative research, that’s actually rather like taking the position of preferring your left eye to your right. You have a much better sense of perspective when you use both. 

And as for those who regard the focus group as the essence of qualitative research – it is truly only one branch of a rather rich tree of insight-building methods – many of which may be better suited than focus groups to the particular research problem at hand. 

Qualitative and quantitative research are, in the end, not functional substitutes. The first type of research locates ranges; the second, proportion. Most marketing problems worthy of study require an understanding of both. 

3. Treat Research Like a Drunk Treats a Lamp-post

This is a steal from David Ogilvy – and a tribute to his perspicacity in observing that too many people in business lean on research “more for support than illumination.” 

We are, all of us, terribly susceptible to the bias of selective perception – taking in with greater receptivity that information that tends to confirm our existing models of reality, and blocking out those inconvenient and noisy bits that break our models apart. Research should be, and should be used as an antidote to this bias. Too frequently, it is used to an opposite end, leaving organizations “data-rich and insight poor.”

4. Measure for Measurement’s Sake

A contemporary of mine worked summer jobs at a Ford Motor Company factory back in his university years. Part of his job one summer was to count the bicycles in the parking lot. He dutifully counted away, while asking questions about why this counting was going on. Few in his direct circle of contacts had any satisfying answers, but it was a widespread and long-standing practice to be sure. In the end, he learned that the practice related to Henry Ford’s strategy of making the Model T affordable to the common man. The counting of bicycles was at one time strategically relevant, but had long since ceased to be worth the effort. Can you think of any similar vestigal measures in your business?

We could go on here, but perhaps the pattern of observation underscores the point: Measurement is strategic. The process involves a variety of invitations to get it wrong. And the consequences of getting measurement wrong can be far-reaching.  

Here’s an invitation that could make this post the basis for a lively discussion – why not share with other readers your story of measurement gone wrong?