It's Time To Stop Using Qualitative Research As A "Pass/Fail" System

Qualitative Research, and in particular, Focus Groups, are the target of a lot of criticism because of their inability to accurately determine the success or failure of what is being tested. If I hear one more New Coke story, or Malcolm Gladwell's Herman Miller example in "Blink", I think I'll puke. However, despite my gastrointestinal reflux reflex, the fact is these stories ring true, and qualitative research (and I would argue Quant as well, but I don't do either Windows or Quant so I won't speak to it) should not be used to pass or fail a product.

A good psychotherapist will know within a second what someone's reaction is to a new product, regardless of what they state. A good researcher will know how to use that reaction to get relevant data for their clients. The main way to do this is to take the "focus" off the "evaluation" and put it more into a moderated discussion, or into a realm where people do not have to justify anything. "Evaluations" and "justifications" (e.g. Why did you react this way? What do you like... what don't you like...) have a very technical term in Gestalt Psychotherapy - they are called Bullshit. What ends up happening is that even though the researcher/therapist can see how a person reacts to a new product within a second, often the person doing the reacting is completely unaware of their reaction. As such, when they are asked to justify or explain, they end up being confused and spewing-out answers in order to please themselves or the moderator. This also has the effect of "fixing" a person onto a specified answer. Participants will not want to be seen as "flip-flopping" in front of a group, so they tend to keep justifying a position that may have changed.

There are very few reactions someone can have when seeing something new - positive, negative or neutral (neutral could also be called confused). What needs to happen in a product evaluation is that even though the moderator knows initial reaction, the moderator must let participants sit with their thoughts and feelings for 30-60 seconds before people have a chance to speak.

From there, the moderator simply probes with a "Well..." and lets the discussion flow from there. What begins to happen in the discussion is that people are much more in-tune with their initial impressions (they've had a chance to sort through their own BS so their clarity of thinking is better), and will often what will happen is that people will begin to refer to their initial reaction, and discuss how it changes as they have had time to sit with their impression, or to hear others in a group.

So, here's where the pass/fail concept gets thrown out the window. Someone could have had a negative initial split second view of the product, but when they sit with their thoughts and feelings, the negative impression can be melted away - and this is what the product test measures - the change in opinion (if any) from initial reaction through to a final opinion. The change can go from negative to positive, positive to negative or from any change in between. The only thing that a moderator needs to be on the lookout for is whether the change in views someone has is a real phenomenon or whether it a result of "group-think", an attempt to please, or an attempt to "kybosh" a good idea. These three behaviours are considered neurotic and unproductive, and a good moderator will know how to get around them, and know the extent of legitimate influence or neurotic influence.

So what we wind-up getting from a concept test is not a pass/fail result at all. Instead what we get is a dynamic result (which the last time I checked is the way a market actually operates). We can measure initial opinions (which, to a good marketer or advertiser should mean very little - their job is to change opinions after-all), but more importantly we can measure how those opinions can be changed and influenced. The group discussion will illuminate what factors changed their initial impressions of the product, or what factors keep people stuck in their initial impressions. The goal for the moderator is to ensure that the conversation is kept free of that very technical Gestalt term - bullshit.

And speaking of that, it is worthwhile to return to the initial split-second reaction observed by the moderator when the product is first exposed - he needs to be keenly aware of it as people are speaking. It is possible that someone could "fudge" their explanations, or change their story based on what they hear in the group. It is important for the moderator to check-out what a participant is saying versus the initial observed reaction. There is nothing wrong with a moderator calling someone and saying "I hear that you initially said you liked Product X, but I happened to see you out of the corner of my eye and I would have made a bet that you didn't like it - just let me know how far off I am." All this does is re-frame the participant back to their initial thoughts so that the data is more accurate, and we can more accurately measure the progression of thoughts and opinions.