If You Want "Real Usage" Figures Go Elsewhere!

Most reported "usage" data from qualitative research should be treated as suspect, and I would go so far as to recommend that my clients NOT make decisions based on usage data reported in qualitative research.

Before I go into details, let me provide a key reason for my recommendation. I have a summer cottage, and a few times a year we have myriad visitors, like cousins and friends of cousins who go to the local casino. The funny thing is that most every visitor to this casino comes back "a winner" - they doubled their money, beat the bank, killed at Blackjack and knew just how to bet at Roulette. My question to them is simply this - "What farking casino are you people going to, because every time I go, I lose money." When I challenge them with this, what invariably happens is that these "killer winnings" get whittled down to statements like "well, we won enough to pay for dinner" or "I did well at roulette, but I lost a lot at blackjack."

Now, most people know exactly what's going on. A bunch of beer-drinkin, nature-luvin, Muskoka-bound, middle-aged Canadian "dudes" want to "play up" their winnings so that they can puff-out their chests bigger than the next guy. Of course, we know full-well that the odds of so many people winning so much money are slim to none.

So, how does this relate to reported usage in qualitative research? There are three ways. The first is rather intuitive - many people in a group will "fudge" just a bit to blend-in. The fudging may be to ensure that they are equal with the group, or ensure that what they say corresponds to social norms. Some may just be scared of telling the truth. This forms the basis for why we can't rely on qualitative research to provide us with numbers of any sort of reliability.

However, the next two reasons are where a deeper understanding of psychology comes into play. The second reason my story relates to qualitative research is because it is not the reported numbers or usage that is important to us. What is important to us is the deeper feelings and thoughts that underly the answer. If a moderator's BS detector is going-off full-tilt like mine does when I hear these "casino stories", I know to probe further. What is it about this reporting that is common to most of these people? There is a hope that they will win, and a fear of looking foolish in front of others.

The third way this relates to qualitative research is what I do after the fact. In my casino example, I actually stood quite firm. The way I questioned people about the casino they went to was such that they knew I would likely follow-up with "Exactly how much did you win?" or "Tell me the method you used that night?" When people have to compound their "exaggeration", they usually back-down. Now, in a focus group - it is highly unlikely that I would ask "what casino are you going to..." - mainly because I mentioned that it is not important for me to get the truth in this instance. Instead, what I would likely do is say "Wow - there's a lot of excitement going on in here... Can someone show me what it is like to win a lot of money at a Casino?" Engaging them in their fantasy will show the client how to enhance a brand, experience or communication to play to the perception of excitement that these players have. Attempting to find the "truth" of the matter would serve to ground participants in a reality that they would rather not face, and one that would not be appropriate for a casino message.