Data "point" of the week: Data alone is not enough

Cheers Forrester for this point of view on data and how to approach it.

by Josh Bernoff

I've been thinking about data lately. This is partly because of Bill's comment on our last Data Chart of The Week (Seniors) and also the responses we got to Part 1 of our recent Webinar series. In the Webinar, I chose to focus almost exclusively on several Social Technographics Profiles -- data about how different groups of people use social technologies -- and while most of the responses were positive, a few people responded that they were unsatisfied with the result.

Confession: I love data. I helped start Forrester's first consumer surveys, called Technographics, in 1997, and I love the idea that we can answer consumer behavior questions around the world now. As an analyst, I love having the ability to dive into data and find the answers to a client's behavioral or attitudinal question. And I find it endlessly fascinating to see how people's attitudes vary with age, or where they live, or from year to year. Having the ability to ask questions of the form "I wonder if people . . ." and actually get the answer is intoxicating. That's why data about People is the first step in the POST method for social technology strategy formation.

But data isn't the whole story. Data looks backward, and answers only questions you thought to ask. Surveys have biases. They have margins of error and sample size issues. I believe our surveys are the best, most comprehensive set of data available anywhere on technology. That said, if you start and end with data alone, you will go wrong.

That's why we always try to provide analysis along with, or driving, or surrounding the data we present, which creates the basis for intelligent decision-making.

There are experts on this stuff, most notably my colleague Brad Bortner. But I've learned a few things myself in a decade of research with consumers. So rather than present more data this week, I will offer up these bits of advice:

• Better to have data than to trust your gut. In the absence of data about your market, youi are driving blind. You intuition about your customers may be right, and it may not. (It's useful to remember that in every stock trade, one side expects the stock to go up and the other does not -- why are you so sure you are smarter than everyone else.) If you can get access to consumer behavioral data about what you're doing, then by all means study it.
• Data is not insight. What does it mean that 40 year-olds are twice as likely to create social content than 20 year olds? That depends on what you want to do about it. Use data to start and bolster your argument. Then ask, "Is there any other way to interpret this?"
• Check the source and the base. All Forrester data charts have a source (typically a survey) and a base (the group of people surveyed). The source will tell you when the survey was conducted -- and data is perishable, since people's attitudes change over time. The base us important to -- is it online consumers, people with mobile phones, or people in metropolitan China? Sample size is also important -- results from 200 people are interesting, but clearly not as as solid at results from 10,000. That's why I can tell you the social Technographics profile of people who own Mercurys, but not people who own Hummers -- we only reach a dozen or so of the latter.
• Protect against bias. All surveys have bias -- most notably, they reach only people willing to take surveys. Another common bias is "social desirability bias" -- people want to believe they are better than they are. This is why PBS gets higher ratings in diaries than when measured with a meter. Always ask "is there a bias here that could reduce the value of this data?"
• Beware of overinterpretation of a question. I will often show survey data in the form of a question and the percent who gave various answers. An audience member will ask "did they mean this, or did they mean that?" It's a hard question to answer, since I wasn't in the brain of the person answering. Strictly speaking, they read the question and they gave the answer -- it's up to you to interpret. Similarly, just because people participate in discussion forums doesn't mean they want to talk about your products. Data gives you a feel for how people are thinking, but it's still up to you to engage them.

I promise to keep showing you data, but always with a context on what to do about it. What bits of advice do you have about working with data?