By Kevin Keeshan, News Director - KGO Television and Richard Hertz, Pollster - Hertz Research

There's an old journalistic cliche about not letting facts get in the way of a good news story. To be sure, Michael Stoll's article questioning the validity of our polling at KGO television wasn't guilty of that.

But the article did contain numerous misstatements, mischaracterizations and misquotes about our polls. However, its biggest failing was its complete omission of facts or analysis regarding the accuracy of our online polls. It's interesting that these facts are missing, because the author was aware of them. In an e-mail he specifically asked us to provide him with proof regarding our polls' accuracy. And although, he could have found all of our poll findings and the election results easily accessible on the web and verified them himself, we went ahead and did the math for him.

Perhaps this information wasn't included in the article because it didn't fit the author's conclusions. But the fact is that for both of the elections we've covered with our online polling, the poll results and news stories developed from them accurately reflected the election outcomes, or the trends going on in the races at the time the polls were conducted. In our opinion, excluding these facts from an article questioning the legitimacy of our work violates the most basic standards of journalism. This is a serious failing for any publication, but in particular for one that grades the fairness and accuracy of other people's work.

Is our online polling perfect? No. Have we learned things in the early stages of this type of polling that will help us improve the accuracy of future polls? You bet. We shared the full breadth of our online polling experiences with Mr. Stoll. Yet he focused his article almost exclusively on our use of the words "scientific" or "statistically valid" in describing our polls and in the belief by some experts that all polls conducted online are junk science.

But, as any field of science evolves, the knowledge base about what may be possible expands. And there is always controversy about whether a new way of doings things is really science.

This is not the first time there have been significant changes in how polls are conducted. Before telephone polling evolved into what most experts considered to be scientific research, most surveys were conducted using lengthy in-person interviews at people's homes. For years there was a long and necessary debate about when telephone polling became a legitimate way to conduct "scientific" research. That debate centered largely around how many households had to have telephones before polls conducted this way could be accurate and representative. Once the percentage of households having phones reached about three quarters of the population, most considered it an acceptable way to conduct scientific research.

Today we find ourselves in a strikingly similar situation. Although telephone polling is still a viable way to conduct accurate research, there are a growing number of problems associated with it that make them more difficult and expensive to conduct. These problems include technical barriers such as caller ID and the increasing dependence on cellular phones. But there are non-technical issues as well, such as the ever-increasing non-cooperation rate due to the proliferation of telemarketing and today's generally busier lifestyles. And although, pollsters are legally exempt from the national do-not-call list, that will probably be of subtle distinction to people on the list, further adding to the non-cooperation rate.

Meanwhile, a growing majority of the population now has access to the Internet. In the Bay Area, one of the most wired places in the country , it is estimated that three quarters or more of the population already has web access. While critics of Internet polling decry it as "convenience sampling", they are right on one level: it is more convenient to be surveyed online and given a choice, most people prefer taking polls online.

Given those preferences and the continuing trend in communications toward e-mail and instant messaging, we feel we have an obligation to communicate with poll respondents in the manner they prefer and is

GTN response

We welcome the rebuttal from Kevin Keeshan and Richard Hertz and agree with several of their comments.

To their credit, Messrs. Keeshan and Hertz concede that claiming the ABC7 Listens poll is "statistically valid" is inappropriate. We trust they will end the deceptive practice of calling them so -- and reporting margins of sampling error which apply only to probability samples -- on broadcasts and the KGO Web site.

Messrs. Keeshan and Hertz make three claims we feel require response:

• The Grade the News article contained "numerous misstatements, mischaracterizations and misquotes about our polls."
• Voluntary polling on the Internet is part of a march of progress in the science of polling from face-to-face queries to telephone interviews and now the Internet.
• KGO's polls have accurately reflected election outcomes in the nine-county Bay Area.

We take claims of inaccuracy very seriously. KGO's rebuttal specifies only one misstatement in our article -- that KGO's sample is "limited to visitors to the KGO Web site." However, as Mr. Keeshan confirmed in a follow-up email exchange on Dec. 16, the statement is true. One has to visit the KGO Web site to register. We made no claim about how the poll was advertised.

Mr. Keeshan believes the original article also should have said that volunteers for its poll are solicited more widely than at the KGO Web site.

As for Internet polling representing the next step in the science of public opinion polling, Messrs. Keeshan and Hertz miss one key point. The personal and later, telephone, interviews he describes were both based on random samples of an entire population, not on a self-selected sample of interested people.

The science of polling is based on mathematics, specifically probability theory. That requires that each member of the population whose characteristics you want to describe have an equal chance to be included. When KGO abandons probability theory, it exits the science of polling, rather than taking the next step forward.

While the ABC7 Listens poll isn't scientific, it is systematic. That brings us to the third claim -- the KGO's polls have predicted elections.

As Mr. Keeshan's suggested, we compared KGO's poll with a probability poll taken by the Field Organization prior to the November 2002 election for governor and the October 2003 recall election.

KGO's poll was comparably accurate in both. Mr. Keeshan is entirely correct in saying KGO's results were as good as probability pollsters' and reasonably accurate predictors of the actual vote.

Does that make KGO's methodology trustworthy?

It's hard to project from the results of two elections whether the ABC7 Listens Poll will perform as well in future elections. It is also another thing to claim that the results of surveys on other issues are equally accurate.

Our in-house polling expert, Stanford Prof. Jon Krosnick, advised that even if Internet surveys predicted most elections, that could be because both the survey and the vote exhibit a similar divergence from public opinion as a whole.

People who vote (less than half of those eligible in most elections) tend to be motivated and interested in public policy. So, it could be argued, do volunteer participants in on-line polls. That's what's called a "self-correcting bias." In that sense, the electorate itself is unrepresentative of the population at large. "Dynamics that help for election results hurt for other issues," Mr. Krosnick said.

In fairness to KGO, we tried to find ABC7 Listens polls addressing the same questions at the same time as probability polls. Poll results are notoriously sensitive to small changes in question wording and the time span when those questions were asked. We did not find matches for Bay Area residents close enough to provide a reasonable test.
_______________________________________

most convenient for them. So we designed a system that uses the speed and reach of the Internet, but also employs practices and safeguards that aren't normally used in most Internet polls. These practices include giving respondents several days to complete the poll and not releasing any results as we go. In addition, before they can participate in our polls, respondents are required to register and provide us with some brief background demographic information. We also employ stringent security measures to prevent respondents from participating more than once and from using bogus e-mail accounts.

The biggest question we had when we started this effort was who would sign up to participate and how representative of the Bay Area population this group would be. Would they look more like the Bay Area population as a whole, or more like the registered voter pool which is much older and less ethnically diverse. But what if they didn't look like anything coherent or usable at all, then what?

To our considerable relief, a diverse group of people have signed up and participate in our polls. In most respects, they look more like the Bay Area population as a whole rather than the registered voter population. They are most representative in terms of their geographic distribution and age. The diverse age range of our respondents is a welcome change from most telephone samples which usually have an older age bias because of the difficulty in reaching or getting younger people to participate in surveys. Their political makeup is very representative of the party registration of the Bay Area. While more ethnically diverse than the voter population, the respondents are not as diverse as the Bay Area population as a whole.

Pollsters use a technique called statistical weighting to deal with demographic differences between those they interviewed and the population where the survey was taken. If, for example, we interview too many people of one gender or political party, statistical weighting adjusts the results so that the views of each gender or party are represented in their true proportion of the population. These procedures are widely employed in all types of polling.

Mr. Stoll asked us how much statistical weighting we used for our online polls. That was a good question because it gets to the point of the representativeness of our poll respondents. But again, the answer did not fit the headline and was omitted from the story.

So far, we have used comparable or somewhat lesser amounts of statistical weighting in our online polls compared to our telephone polls. Why is that so important? Although statistical weighting can be employed to iron out demographic differences, you don't want to go too far with it. One wants the sample of who you poll to demographically resemble, as close as possible, the area where you're polling. That way, little statistical weighting is needed. Excluding our experiences with this aspect (that Mr. Stoll asked for and we provided), in our view was just another example of "the author tailoring his story to fit his conclusion".

We believe that no poll today, whether conducted online or over the phone, is perfectly random. None meet the definitions of scientific polling outlined in the article since they all exclude some potential respondents from participating. Telephone polls do not meet this standard for a number of reasons. First, not all people have land line phones. Even though this is a relatively small percentage of people, they do exist and, more importantly, a growing segment of the population use cell phones as their exclusive or primary means of communication. In addition, a much larger group of people who have caller identification, will not take calls from numbers they don't recognize and are not exposed to potential surveys. So the standard for non-exclusion can not be met by even the most stringent telephone surveys.

Mr. Stoll incorrectly states in his article that our sample is "...limited to visitors to KGO's web site..." In reality, our poll respondents come from many other sources, such as people who watch ABC7 News, those who watch other ABC programming when promotional announcements about the polls are run, the audiences of affiliated media outlets and from people who heard about the polls from friends, colleagues or family members. Taking into account the audience or visitor census of these sources, we conservatively estimate the number of people exposed to the polls through one or more of these means to be well into the hundreds of thousands, if not higher.

We also believe we can expand our sampling frame to expose even more Bay Area residents to the existence our polls by increasing our affiliations with other media outlets and our presence on other high-traffic Web sites And since the Internet is probably the most metered place on Earth, it is not difficult to find sites that have high visitor counts of people in demographic categories for which we are seeking more respondents.

The question then becomes: are the people who are exposed to the existence of the polls, reasonably representative of the entire Bay Area population? So far, they appear to be, at least for the most part. What makes us feel that way? One reason is that the demographics of KGO's news audience are largely reflective of the Bay Area population, and those who sign up for the polls have similar demographics. This is critical for the methodology we are employing. For example, we don't believe this approach would work solely by soliciting the readers of a newspaper because the demographics of newspaper readers are distinctly different than those of the population as a whole.

One might assume that people who sign up to be polled might be more informed or opinionated than the population as a whole. This is probably true to some degree, as it is also likely true of people who participate in phone surveys. People are probably a little more thoughtful in online polls because they don't have the pressure of having to take the poll right then and can think about their answers a little more without the presence of the interviewer. And to whatever degree interviewer bias influences respondents, that bias has been eliminated.

Like other surveys, respondents to our online polls change their opinions as circumstances change and as they learn more about issues and candidates. An excellent example of this is how public opinion in the Bay Area about the war in Iraq has shifted as events have unfolded. The people who sign up for our polls are not public opinion junkies or policy wonks. We can see it demographically and by the responses and comments they make to our poll questions. You can check it out for yourself by looking at our poll results web page at: http://abclocal.go.com/kgo/news/polling_center/index.html.

Critics of our polls suggest that there are fundamental differences between people who agree to participate in a phone survey and those who sign up to be polled online. However, the evidence we've seen so far suggests that, if anything, the demographics of the Internet samples more closely resemble those of the Bay Area population as a whole, than do the samples of most phone surveys. And in parallel polls we've conducted by phone and online, the poll results have turned out similar.

If all you were interested in was conducting surveys of likely voters, and cost was not an issue, one could make a stronger case about the balance of factors still tipping toward using telephone polling. Although some of our polls are about election contests, many are about other issues like the war in Iraq, how people are doing economically and how people feel about social issues such as gay and lesbian rights. For polls on these types of subjects, we are interested in the views of all Bay Area residents, not just likely voters.

We recognize that anytime you try something new or different, you are opening yourself to criticism from proponents of older methods. But as circumstances change, some practices might not work as well as they used to do, indicating that adjustments need to be made or new ways of doing things found. This doesn't imply that the old ways are invalid, only that consideration of alternative approaches should not be excluded.

With the increased use of electronic communication, and online being used for a growing amount of all survey research, the question shouldn't be telephone versus online, but instead, what are the best ways to conduct polling online and how can we combine the best qualities of telephone and online research to make polling better in general. This is not the beginning of the end of scientific polling. Rather it is the beginning of a new era that will allow us to do things with polling that couldn't be done before, like having respondents evaluate debates and other video content, work with much larger samples allowing more detailed analysis of the data, and make polling more affordable to do and do more often.

We and others engaged in serious online polling have learned and will continue to learn a great deal as this field of research develops. We assume that all but the most hardened minds would acknowledge that, if current trends toward both increased internet usage and declining phone survey participation continue, at some point the advantages of polling online will outweigh the advantages of polling over the phone. In certain situations like here in the Bay Area, we think the balance already tips in that direction. Others may need to see higher internet access and telephone refusal rates before they'll reach that conclusion.

But that's fine - a healthy debate about the circumstances in which online polling is appropriate and welcomed. But Mr. Stoll's one sided portrait, replete with its factual errors, omission of important facts regarding our poll's accuracy and its failure to differentiate between the standards, practices and safeguards we employ compared with Internet polls that aren't designed to be serious, was not a step forward in that debate.

Let's concede one point that Mr. Stoll makes in his article. Perhaps our use of the words "statistically valid" is inappropriate because that term usually refers to pure probability rules that neither online polls nor those conducted by telephone can achieve, if that standard is applied equally.

But we will not concede the use of the terms "scientific" or "polls" solely to those conducting survey research in a certain way. Earlier this year, it was hard to believe when a news network tried to legally claim the words "fair and balanced" as their sole domain. Yet in Mr. Stoll's article we see much the same thing happening. We'll leave it to our viewers and other Bay Area residents to examine the depth and quality of our research and then draw their own conclusions.