| |
Measuring Reality or Performance?
This edition of Idea Exchange surveys the broad field of customer experience measurement. Converging trends bring this topic to the forefront of the discourse on marketing and branding today. The business adage that “you can’t manage what you can’t measure,” is a signature concept of the industrial economy, a period when measurement and optimization became defining management processes for value-creation.
If the adage was true then, does its relevance continue today, in the period that Pine and Gilmore dubbed as the “Experience Economy”? Certainly, if we have an experience economy, it stands on the shoulders of the industrial economy, and it’s unlikely we’ll stop measuring the things we believe are most important to competitive performance now. But measurements of customer experience may not be quite as reliable as the tangible metrics that propelled competition in decades past.
It’s an old joke – doing “social science” is like “doing chemistry with dirty test tubes.” And as you read about several of the customer experience measurement methods described in this issue, you will see complexities due to the problems of interpretation, the barriers of language, and differential responses based on personality. Indeed, across the range of the methods described here, the common pattern of our experience as observers of customer experience seems to be this: the more closely we observe, the more complex the phenomenon appears and the more nuanced our cause-effect inferences must become.
One could take such an observation as a reason to abandon hope for reliably measuring customer experience. But another possibility exists too; we can frame our experience-measurement efforts according to a more modest purpose. While we may never get to the bottom of measuring customer experience, still, it seems both possible and practical to develop comparative measurements of our own performance in creating customer experiences. In that spirit, we hope you’ll enjoy the “Measuring Experience” issue of Idea Exchange. However conclusive or inconclusive the methods, this is clearly a growing field of practice – one rich with new techniques, and perhaps some new promises for informing our actions as brand stewards in an experience economy.
Back to top
|
| |
TrendWatch
Actionable Analytics by Paul Reader
|
Paul Reader
Practice Lead: Lead Building Solutions
Quarry Integrated Communications, Inc.
|
As you will read later on in this issue, there are numerous quantitative and qualitative research methods that can be applied to measure experience. At the same time, advancements in technology designed for marketers are making it possible to monitor the behaviour of people at the individual level across communication channels. There are two major ramifications of these technological developments. First, it is possible to gain insight into an individual’s historical experience interacting with your communications. Second, this intelligence may be leveraged to refine marketing practices or to personalize future communications based on what is known about the individual. Consider the following trends in marketing technology:
- Pure play Web analytics companies are maneuvering into the marketing automation space
- Pure play email marketing and tracking companies are maneuvering into the marketing automation space
- Marketing automation plays are influencing the role of analytics
- The trend is towards building more complete and holistic profiles about prospects, leads, and customers by combining explicit data (information that the contact provides to the organization) and implicit data (information gathered by tracking what the contact does - behavior).
- In other words, analytic data has moved beyond anonymous metrics (number of opens, click-throughs, visits, views) and is now being associated with individual people.
- This data is converging into software platforms designed for the marketing organization.
- The marketing organization can then do more sophisticated segmentation for planning better targeted communications programs.
- Rules can also be established for how profile data can be leveraged to assemble personalized content dynamically.
- All these developments are giving marketers more tools needed to move towards “precision” or “one-to-one” marketing communications.
- A/B testing combined with real-time measurement and reporting gives marketers tools for fast and inexpensive iterative fine-tuning.
- This evolution of marketing practices promises more relevant and engaging experiences for customers and improved economic results for businesses that lead the way.
Back to top
|
| |
Tips for Measuring Experience
|
|
| |
“Business success is always defined by the quality of the overall customer experience.”
—Forrester Research, 2001
“The product is the brand. You build brand in our industry through the product and the experience.”
—Jim Wicks, Motorola, 2006
“Education is when you read the fine print. Experience is what you get if you don't.”
—Pete Seeger
- Identify the goals of your product. Define the experience you want for your customers by identifying your representative users, their work environment and how your product will impact their lives.
- Develop your research plan. Target your established goals by identifying appropriate evaluation methods, relevant testing criteria and key goals for a successful user experience. You’re not just testing the features of your product; you’re making sure the customer experience you’re designing is hitting the mark.
- Create your recruiting plan. Match your recruiting criteria with the key characteristics of the customers you identified at the beginning of the process. The cardinal rule of recruiting is to “Know Thy Users.” Recruiting participants who represent your users will ensure that the experience is tailored for the intended audience.
- Test your product. Observe your customers in action. Find out what they like, what they don’t like, where they succeed, and where they experience difficulties with your product. Think about how their experience measured up to their expectations and what can be done to exceed their expectations.
- Refine and retest. Based on your customer feedback, improve your product and repeat the process until you meet or exceed your customers’ goals for a successful experience.
Back to top
|
| |
Practical Wisdom
A Long Look at Eye Tracking by Dan Skeen
|
Dan Skeen
Practice Lead: Traffic Building Solutions
Quarry Integrated Communications, Inc.
|
Despite a long and successful history, the design team at Sensis decided their Yellow Pages directory was ready not just for a new look, but for hundreds of them. They used hardware designed to track a viewer’s eye movements to monitor how readers accessed and navigated the 32-year-old advertiser directory.
The results were (forgive us) eye-opening. It revealed a consistent sequence of visual signposts that help readers become oriented.
“A reader would turn the page, check the high corner, check headings, then bounce out to richer ads and say ‘Yep, I’m in the right place,’ ” says Brett Collinson, design manager at Sensis. “It gave us some real insight into the role of graphical ads compared to straight text listings.” This new insight, and the empirical evidence to back it up, was a welcome finding for the Sensis team. Their sales force can use the navigational role of ads to make a stronger argument for buying large display ads over smaller ones.
Spurred by a number of factors such as improved hardware and software, reduced costs, growing acceptance as a usability technique, and broadening methods of usage, eye tracking is becoming an increasingly valued tool in the product design toolkit.
The gear used in the Yellow Pages example involved a pair of sensor-equipped viewing goggles accompanied by a belt pack that records the data, providing a record of eye activity. This is a far cry from the dreadful chin-rests and bite-bar set-ups of the past. Today’s models are much more unobtrusive. One looks like a typical LCD monitor. With a few simple calibration steps, the user is then free to interact with the screen normally. As the hardware has become less reliant on complex lab settings, innovative practitioners have increasingly moved into more natural environments. Eye tracking has been used outside the lab to study driver safety, point-of-sale displays, billboards, and more.
In terms of feedback, eye movements are typically plotted graphically using reports like a heat map, which uses vivid colours to represent heavily viewed areas, or a gaze trace, a colour-coded sequence of eye activity.
The central appeal of eye tracking is a factual record of user focus. While for years usability specialists have asked questions to try to understand participants’ behaviour, there is a well-recognized gap between what people say and what they do that defies even the most astute facilitator. Eye tracking helps get underneath this to recognize design flaws. For example, when asked about a particular object on a Web site, a participant might say they didn’t look at it. With eye tracking you can go back and actually determine that they spent five seconds looking at that object. The fact that the user looked at the object and it didn’t register might suggest the need for a different visual treatment.
Eye tracking has gained credibility among product designers and usability practitioners. But despite the visual allure of the colourful reports, eye tracking hasn’t replaced tried-and-true facilitation tactics. It’s not a replacement for a traditional one-on-one usability session; it’s very much a complementary practice. Eye tracking is perhaps best used as an extension of the information collected through the observations and inquiries of a skilled facilitator.
The rich reporting features associated with eye tracking can be quite helpful for instances where a quantitative benchmark is required. Eye tracking also often proves valuable where entrenched assumptions cause a design stalemate among team members. It adds an objective viewpoint that can be an effective mediator, helping the design team to quite literally see through the eyes of the customer.
Back to top
|
| |
Feature Interview: Current Research Methods for Measuring Experience
Pierre Jolicoeur, Professor of Psychology, University of Montreal interviewed by Glen Drummond
|
Pierre Jolicoeur
Professor of Psychology, University of Montreal
Canada Research Chair in Experimental Cognitive Science
Pierre is a Professor of Psychology at the University of Montreal, as well as an adjunct Professor at the University of Waterloo. As the Canada Research Chair in Experimental Cognitive Science, Pierre’s research involves the study of human attention (the process of selecting one source of stimulation from among many) and its relation to perception and thinking, seeking to determine how, why, and when attention succeeds or fails. Pierre has pursued research in this field for over 25 years after obtaining his PhD in experimental psychology (Harvard, 1982).
|
| Glen: |
What is the current set of research tools for measuring experience? |
| |
|
| Pierre: |
The first would be reaction time methods. Present stimuli to people, ask them to make speeded responses, and by clever use of different experimental conditions, we can actually infer quite a bit in terms of the underlying mental processing ability to see things. I would call that classical psychophysics or classical cognizant physiology where the main dependent measures are reaction time and patterns of error rates. |
| |
|
| Glen: |
This is the sort of stuff that’s gone into the interface design for flight equipment and so on to make sure that pilots get the right information at the right time and don’t crash their planes and so on. |
| |
|
| Pierre: |
Exactly. But because a subject may have moved his or her eyes on top of a stimulus doesn’t guarantee he’s actually processed that stimulus very deeply. So, you also probe their memory to determine if they can remember it later.
Then we look at the measurements of their physiological responses, such as Galvanic skin response (GSR), heart rate, pupil diameter, and measuring the strength and speed of various kinds of reflexes. These have been used in the past, and are still in use today.
Techniques I know more about would be used in core cognitive neuroscience today, such as functional magnetic resonance imaging (FMRI), which looks at the differences in blood flow. The basic principle is that, in the areas of the brain where the neurons are working harder, they require more energy from glucose in the bloodstream, and the cells metabolize more and produce changes in blood flow and blood oxygen level. This allows us to tie the activity to local structure in the brain; that’s where we’re both understanding which parts of the brain do what, when. This technique has major limitations. One, it is very expensive, with a scanner costing up to $7 million and an hour of the procedure costing up to $3,000. Two, it requires that the subject be very still for quite a long time, perhaps an hour.
|
| |
|
| Glen: |
Now, there’s a body of thinking that has emerged more recently that claims that we respond at an emotional, precognitive, or pre-rational level to stimuli from advertising and marketing and that we rationalize our impulse after the fact. Do you believe that this kind of inference is one that can be made based on the use of this kind of technique? |
| |
|
| Pierre: |
Well, there’s both, I guess, because if I’m watching a TV commercial, and it’s cut in various ways, the information is flying by really quickly. It either grabs me or doesn’t grab me. I may or may not see something or remember something later on based on my current kind of state of readiness to receive that message. That would be one message that I would convey that is actually closely tied to my own research that we process information, sometimes surprisingly conditional, on our current goals and motivation, even at a very microscopic level. |
| |
|
| Glen: |
We absolutely have evidence that says when we do eye tracking followed by an interview, we see that people looked at something that their interview didn’t reveal. |
| |
|
| Pierre: |
That’s correct, but we can’t discount the eye tracker information completely. For example, if you’re looking in a cluttered visual display for a particular piece of information that you know is red and is supposed to be at the center of the screen, then it turns out that other red things will draw your attention involuntarily, in the periphery.
FMRI has one major limitation from a scientific point of view and that is that the response that you measure, the change in blood flow, actually takes quite a while – 4 or 5 seconds – to take place. When your neurons start to work, their activity will ramp up very quickly, with about a tenth of a second, and we can do a very simple task like judge the pitch of a tone or the color of a visual stimulus in well under a second. But this hemodynamic blood flow response will take another 4 or 5 seconds before it peaks and another 5, 6, 7 seconds before it comes back down to baseline. So it is very difficult to determine the order in which the critical cognitive and perceptual events have taken place.
Therefore, you need to use a variety of techniques. One which is gaining in popularity is EEG, in which we put electrodes on the surface of the scalp to measure the electrical activity of the brain underneath. This technique has existed since 1929 or so, but really emerged in the '60s and '70s when people started to discover brain responses and manifestations of that in revoked responses. Now, with faster computers, we can do much more high-quality, high-density recordings of EEG. The big advantage of EEG is that the electrical response occurs at the scalp essentially instantaneously, relative to when the cells start to respond, because electrical signals travel at the speed of light. It’s an immediate measure of what’s happening now. |
| |
|
| Glen: |
Are you able with EEG to locate what area of the brain is making that activity as well as when it’s happening? |
| |
|
| Pierre: |
No, not very well. That’s the drawback of the EEG. I can tell roughly, but with only some degree of precision. Localization of EEG requires a kind of mathematical model, some inverse solution, which is controversial at this point. |
| |
|
| Glen: |
So with our current set of technical apparatus for studying what’s going on in the brain, we have a kind of uncertainty principal. We can find out with FMRI where it’s happening, and we can find out with EEG when it’s happening; the problem is we can’t find out where it’s happening when. |
| |
|
| Pierre: |
We can try, and that brings me to a third technology called MEG, which is magnetoencephalography where we measure magnetic fields produced by the brain. That helps a little bit over EEG because one of the difficulties with EEG is that the signals are distorted by the skull itself, the bone, as a really poor conductor; and the electrical fields get distorted. It turns out bone is essentially transparent for magnetic fields, so they just pass through it without distortion and so that helps. We still have this inverse problem issue.
So yes, your intuition is exactly right that now what we’re going to do is we’re going to combine all these methods and sometimes even use them simultaneously. You can’t use MEG at the same time as FMRI, but we can do EEG at the same time as FMRI and get the initial electrical response and then follow that up with the blood flow response and try to put it all together. |
| |
|
| Glen: |
So, from a marketing discipline standpoint, we ask: Did they see it or not? Did they process it or not? There’s another question: How meaningful was that, how involving, how deeply moving was that particular idea or experience? Do the quantitative sciences bring anything to this? |
| |
|
| Pierre: |
I think they can, but there’s a catch. For both FMRI and EEG, we need to repeat the stimuli over and over because the signal that we’re measuring is relatively small, relative to the noise. So we need to repeat the basic stimulation sequence fifty times, a hundred times, and then we average over those presentations to get a reliable signal. We can modulate it, so we can make it easier or harder to process that stimulus; and then our brain wave response is going to follow, and it will track nicely. We can even tell you whether or not you were successful in coding the semantic content of something. The way we do that is it turns out there’s a special brain wave that appears to be generated only when there’s a mismatch between a semantic context and a currently processed stimulus. That actually gives us a powerful tool to measure when a person has been able to extract a semantic content of the stimuli, but the caveat is that I can’t do that with one trial. I can’t show you one ad and measure this brain wave; I have to show you the stimulus thirty, forty, fifty times. |
| |
|
| Glen: |
As a closing question, what are the natural ranges of the way brains work? Do people process information the same way, or are there really quite significant differences from individual to individual in the way they respond? |
| |
|
| Pierre: |
There are two different layers of response. When we look at early brain responses to auditory, tactile, and visual stimuli, it is uncanny how closely aligned our brain responses are. Then, when I look at brain waves from two people, they actually look pretty similar, as well, so I can average them across people. Every now and then we find an oddball person that has different brain wave; we have some good hypotheses about why, but the basic response looks really similar from person to person.
But the intentions of the observer modulate early brain responses – going back to the red target scenario. You are expecting a red target at the center, and I flash a big green, bright green spot of light, just an inch and a half to the right or left of where you are looking, and it doesn’t do anything. You’ll get a brain response to it, but it won’t capture your attention. If I present the same stimulus, but in red, bang, your attention is drawn over. And if you weren’t paying attention to it, you’re not going to remember it. You’re not going to process it in the same way. If there was an emotional response attached to it, you’re not going to get it or you will, depending on your current state. So I think yes, I think we rapidly diverge; and our intentions and emotional states do modulate what we perceive and remember and process. |
| |
|
| Glen: |
Which, in some respect, brings us from the quantitative back over to the qualitative side of things. If you understand the motivations of the people who are going to be exposed to your content, you can begin to predict the full trajectory of their attention to that. If you don’t understand those intentions and those motivations, you’re likely to miss.
|
| |
|
| Pierre: |
Exactly. |
Back to top
|
