I'm with Mike Murphy
on this one. The presence of on-screen results from dial-testing groups
is something that needs to be reconsidered during future presidential
debates.
It's not that the squiggly lines aren't fun to watch. Rather, they're too much
fun to watch. It's hard to avert your eyes from them. It's hard to
separate your own, independent reaction from theirs. And it's certainly
hard to integrate back into to the non-squiggly universe once you've
gotten hooked on the squigglys.
It was only a matter of time
before one of the networks figured this out and started carrying the
dial-testing results with their live debate feed. That network turned
out to be CNN, which made a terrificly smart programming decision, and
was rewarded with ratings comparable to the major networks. (Which channel do you think I watched last night?)
The
problem is that the squigglys may give thirty random strangers from
Bumbleweed, Ohio just too damned much power to influence public
perception. The squigglys
influence the home viewers, the home viewers participate in the snap
polls, the snap polls influence the pundits, the pundits influence the
narrative and -- voilà! -- perceptions are entrenched.
Mind
you, I'm not complaining about the post-debate snap polls really, like
the ones that CNN and CBS conduct. I'd certainly rather look at those
numbers than watch the pundits babble for hours on end, especially as
pundits tend to watch for all the wrong things during the debates.
But
whereas the snap polls are scientific instruments with sample sizes of
500 or more, the probability of getting an unrepresentative reaction
from a 30-person dial-testing group is much, much higher. First and
foremost is the matter of sample size. You'd never see a poll conducted
with just 30 respondents, because the margins of error would be around 18 (!) points.
In addition, as Mark Blumenthal points out, these focus groups depart significantly from truly random samples. Let me quote from him at length:
Focus
groups do have important limitations that are not well understood.
Although focus group recruiters try to make the participants as
representative as possible, the focus group is not a
projective random sample, like a poll. Participants usually live near
the facility. As the response rates are miniscule given the time
commitment, participants usually receive monetary incentives (usually
$50-$75) to encourage participation. Recruiters also seek to fill
specific quotas for specified demographic characteristics (a mix of
ages, for example). Thus, we simply cannot count answers in a focus
group to estimate the reactions of a larger population. In other words,
if 20 of 30 "undecided voters” react a certain way to the debate
tonight, we cannot conclude that 66% of all undecided voters nationally
feel the same way.
A second limitation is what researchers
call "group dynamic.” In a focus group, participants are often
influenced or cowed by the opinions of others in the group. If one
dominant personality loudly stakes out a position, others tend to hide
or modify their contradictory views. [...]
Finally, the artificial
nature of the focus group is often a poor way to judge how the
information from advertising (or the fallout of a debate) will be
processed in the real world. For example, focus group participants
often express genuine antipathy for negative advertisements and reject
the information contained in them as false and unfair. Yet in the real
world, as the recent campaign has demonstrated powerfully, such
advertising can still communicate negative information with ruthless
effectiveness. Also, People no doubt watch advertising much more
closely and critically in focus group than in their living rooms.
Mark
is talking about interactive focus groups, which are a slightly
different beast from the squiggly-line groups that CNN and the other
networks use, but most of the criticisms carry over. You're not really
getting a random sample when everyone has to be sitting in the same
physical location at the same time; maybe voters in Ohio were really
grooving on Obama's message last night, but voters in Florida weren't.
Moreover, people may react differently when they feel as though they're
being watched, and that their reactions are being broadcast in real
time to 9 million Americans on CNN. The dial-testing groups may also be
paying too much attention to
the debate to mimic real-world conditions ... they're sitting there in
a room with absolutely nothing else to do but watch the debate and
twiddle their knobs. That's not how most people watch the debates. Most
people are flipping channels between the debate and Dodgers-Phillies,
or trying to put their kids to bed, or are chatting on the phone, or
are four beers into a six pack, or all of the above.
What I'd suggest is that the CPD
ask the network to refrain from including focus-group reactions in
their live broadcasts of the debates. If the networks want to include
the squigglys in their re-broadcasts
of the debates, or perhaps on their Internet streams, I'd be all for
that. But I think the viewer should be entitled to formulate her own,
independent reaction to the debate, rather than having to share her
television with Joe the Plumber and some guys from his neighborhood.
--Nate Silver