Trump vs. Fox News signals a major focus group fail
In a nutshell, focus groups are really lousy at predicting anything and while they can be good at analyzing past behavior, they are so easily manipulated they generally turn out to be less than worthless (because they can lead folks to believe in a world that doesn’t exist). As a result, it is rare you can depend on their results.
I spent a lot of my life doing market studies and I’ve both been in and funded focus group studies. Only in some very narrow instances would I ever consider using them. Let me walk you through why.
A focus group is a way of sampling a population in order to determine why they did something or what they will do in a given situation. In order for it to work it has to be both representative of the population and not be corrupted by any aspect of the focus group process. The goal isn’t lots of drama, the goal is to gain understanding. Its advantage over doing other forms of sampling is you can pick up emotions you wouldn’t get out of a questionnaire and, done right, the responses are typically far more nuanced and granular.
This was primarily what Jobs objected to and he was right. One of my first focus group experiences was for Chrysler. They wanted to know if we would buy a new car they were considering and after showing us the car most of us said absolutely, and then the car came out and it didn’t sell that well. This goes to the heart of why focus groups suck at predicting things. By the time the Chrysler car shipped, Ford and Chevy had both improved their lines so that the alternatives were better and ultimately overshadowed the Chrysler. In addition, Chrysler had gone cheap in areas we weren’t aware of in the focus group. So while the car still looked great the actual driving experience wasn’t that good.
So to make a predictive focus group work you have to emulate the full experience. In this case of a car buying experience, the complete process someone is likely to go through includes other car choices available and unless you’re capable of precognition (in which case you don’t need the focus group) chances are you can’t accurately create an environment that matches the future when the decision final will be made.
The very fact that the focus group was on live TV corrupted the Fox focus group because perspective voters would not be making decisions in that kind of situation. Being on TV has a huge influence on someone if you are asking for an opinion.
In addition, the environment was one where very strong individuals could pressure weaker members into like positions, and if you watched you saw this play out. Not only were there far more people in support of Trump at the beginning than appeared to be the case in the broad audience, but the percentage change of supporters was unreasonably large.
Once someone takes a position it is hard to get him to change it by simply watching a candidate talk because, with confirmation bias, they’ll generally hear what they want to hear particularly early on. Yet the focus group appeared to swing solidly against Trump by the end even though Trump’s poll numbers didn’t shift nearly as much.
In short, at the beginning there were likely folks saying they supported Trump who really didn’t, and at the end there were folks going with the crowd who wouldn’t have without pressure. So you had too much support up front and then too little at the end with the result being a massively inaccurate representation of what had and eventually did happen. In the end, the focus group was worthless other than as an attack vehicle against Trump, which backfired.
The analyst appeared to be pushing his group against Trump and later Trump indicated the two had history and it wasn’t good history. The focus group moderator/analyst has to be unbiased otherwise they will corrupt the results either by influencing the focus group and/or by misinterpreting the results.
Now in the case of Fox it did look like it might be manipulating the results in order to influence the population against Trump, which Trump himself seemed to pick up and there is science here because false analysis results, if presented well, can change minds. And it is possible that some of Trump’s slight initial fall in the polls could have resulted from this.
So you can use a focus group as a competitive weapon but, to be effective, the population has to identify with the group and find the reasons for their positions compelling, both of which require a lot of reach for a large population (which FOX had). You also have to bet the other side won’t be successful in proving your focus group or survey is false, which would destroy your credibility. But there is a big difference between being a representative and being a manipulation tool, and a real danger of forgetting which you are supposed to be.
There are other more accurate ways to sample a population. The cost of using a fake focus group to manipulate a population isn’t only high, it takes a very specialized moderating skill to pull it off. Oh, and if you get caught, it could become a career ending scandal. It is simply too easy for focus groups to generate misleading or totally inaccurate results and this Fox News attempt showcased this in spades.
In this day and age it is almost impossible not to corrupt a focus group, we all are simply too surrounded by technology now, I wouldn’t use one for analysis particularly in a predictive way and I’d always back it up with some kind of validating study if we were looking at historical behavior. But my general and far easier to follow advice is don’t do focus groups and don’t rely on the results that come out of them.
Of course the real interesting result in all of this was that Carly Fiorina was the biggest beneficiary of this event, largely because she was the best prepared to take advantage of it. Not bad for the one tech exec running for the office. It is interesting to note she is likely also the only candidate, because of her marketing background, who really understands the dynamics I’m talking about.