Two Firms Concede Errors in Exit
statement, "Two Firms Concede Errors in Exit
Polls" was a front page article in the January 20, 2005 Cincinnati
My first reaction was "so what, we knew that." Actually anyone
following the election day polling results knew this. My second
more analytical, that is, what will participants and especially users
research think about our profession? In this polling that lasted about
of 12 hours, the researchers invested 1,400 interviewer days to come up
prediction that picked the wrong man, and had an error of approximately
Unfortunately for the researchers and the clients, validation of the
was almost instantaneous. They knew the true results within 24 hours.
15, 2005 - by
Robert E. Stevens, GENESIS II
Second Beginning) E-Mail: firstname.lastname@example.org
In the analysis of
what went wrong, the researchers cited a number of reasons such as:
voters were more willing to be interviewed.
interviewers were too young.
interviewers were too inexperienced.
was a programming error that led to too many women voters being
researchers, however, only
recommended better training and more monitoring of the interviewers.
totally ignored their sampling model. There are solutions to these
The solutions are not easy and they definitely add to the cost of the
From my perspective, additional information needs to be obtained beyond
the respondents had to say. There are two populations at play, first,
being interviewed and second those voting. The disproportionate
number of women
respondents should have been determined very early in the process so
adjustments could be made to the results before publication. The
be interviewed in my estimation is the difficult problem but still
through sample control and data analysis.
I am amazed at how
infrequently we verify the assumptions we make when designing the
seems to me that if it is important enough to build in a research
should be important that we verify the assumption is actually in-play.
checks and balances to verify your assumptions.
Early in my career, while auditing a research
for another corporate division, I asked the designer why some specific
parameters were not being collected and recorded. His response was that
not want to know and he definitely did not want the client to know if
the test effects were significantly affecting the results. I did not
this individual to be a researcher because of his willingness to ignore
accepted standards, in my mind, he became a clerk collecting numbers
I do have sympathy
pollsters. Their results, unlike those I had to deal with, are
quickly. Personally, considering my history, I do not think I would
being involved in a system whereby validation is almost immediate. I
spending time getting to know and understand my data before drawing
Sponsor: Sorensen Associates Inc
Portland, OR: 800.542.0123
Minneapolis, MN: 888.616.0123
in-store research company™ -- Dedicated to the
relentless pursuit of WHY?