Nutrition Report
Card
Who Provides the Bay Area’s Most Nutritious News?
Here’s how we graded the news.
Each story was rated on three characteristics:
· Topic, the subject of the story;
· Episodic vs. thematic reporting¾ was the focus a single event or an issue or theme;
· Impact, how many people were likely to be affected in a non-trivial and lasting way by what was reported.
Stories were coded by two experienced journalists, one a former newspaper reporter, the other a former Bay Area television news director. A sub-sample of stories was coded by both with overall agreement of 84 percent on those questions requiring some judgment. (Scott’s Pi, a chance-corrected measure of agreement ranged from 1.0 to .56.) Differences of 1/3 grade between news providers may not be significant due to sampling error--the chance that our sample happened to catch news providers on better or worse days than their true average work.
Scoring
Stories on core topics each received two points. Core topics include just about everything but celebrity news, minor fires and accidents, sports, promotions¾such as Channel 5 “Survivor” stories and Channel 7 “Millionaire” stories, and stories primarily focused on human interest¾e.g., Casey the sewer pipe-loving dog. Non-core stories each received one point.
Stories were also coded based on whether they were covered episodically¾as unconnected events¾or thematically¾as connected events or dealt with issues. Stories in which half or more of the content was thematic or about an issue received an additional 2 points for the explanatory value of treating the story within a broader context. Simple event stories, received no additional points. For example, a story about gun safety, or patterns of violence, or an attempt one city was making to reduce it, would score 2 more points while a story limited to a particular shooting, would gain 0.
Finally, stories likely to affect a significant number of people more than momentarily (6 months or more) received an additional 2 points, while stories likely to affect relatively few gained no additional points.
A story about a fire, for example, has a direct and lasting effect on the people who used or owned the building and may have some impact on those who live within a few blocks, but not on many others. That’s a few thousand people in a region of 6 million. Likewise, a shooting in San Jose is unlikely to make a lot of difference in the lives of people in Walnut Creek or Santa Rosa. But a story about state-mandated high school graduation tests affects a huge number from Gilroy to Guerneville. Stories affecting 10,000 or more people earned the 2 points.
All told, stories could earn from 1 to 6 points. In the analysis, each story was weighted by its size. It wouldn’t be fair to equate a 3-minute story with a 30 second “brief.”
A system that equalizes print and television
Grades were based on the percent of total newscast time or newspaper space that was spent on high quality journalism. We could compare print and broadcast equally, because absolute volume didn’t matter. To keep the playing field even, we only examined top stories¾those on the front page and local news front page of newspapers and in the first 30 minutes¾the front half¾of premier evening newscasts. (Channel 5 only runs half-hour newscasts, so we analyzed the entire show.)
We used a gentler scoring system than in school, however, because it may not be realistic to think journalists can always make what’s important interesting. Thus a station or newspaper could indulge almost a quarter of its news time or space in cotton candy stories and still earn an “A.” (Each story was multiplied by its newsworthiness score and added into a grand total for the station or paper. That amount was divided by a perfect score--the sum if all stories rated a 6. Overall scores of 80 percent or higher received A’s; scores below 50 percent failed.)
Stories were randomly selected over 11-months based on a system guaranteeing equal numbers of each day of the week. That balances traditionally lean news days with fuller ones. In almost all cases, stations and newspapers were compared on the same news-gathering cycle¾so all had access to the same set of events. In other words, all stations were sampled on the same evening and all papers on the following morning.
Subjectivity
Even though two independent coders graded the stories approximately the same way, the scoring conventions¾such as point values and separation of core and non-core stories¾are subjective. Just as a teacher makes judgments about what level of achievement rates an “A” and what’s “C” work, we have tried to measure journalistic performance.
Further, we’ve based our yardstick on a consensus of journalism’s codes of ethics. Most of these are derived from a “social responsibility” theory of journalism’s purpose. This is the idea that journalism receives special privileges from government¾such as low postal rates, freedom from content regulation, exemptions from anti-trust and child labor laws, and free access to air waves¾which are unavailable to other businesses, in exchange for doing its best to build an informed citizenry.
Problems
As with much social science, there were occasional problems with data gathering. Equipment sometimes failed, stations would change broadcast schedules at the last minute (usually for sports playoff games), and paper deliveries were missed. Sometimes we just plain forgot to tape. For the most part, however, the method used was a systematic sample¾every 10 days¾with a random start. We believe this gave us a generally accurate picture of newsworthiness.
One of the categories¾whether the story was episodic or thematic¾only reached 78 percent agreement between coders. When chance agreements are statistically subtracted, reliability, as measured by Scott’s Pi, fell to .56. We will work on a clearer distinction in our coding manual. Ideally, the reliability analysis would also have been randomly sampled. Unfortunately the timing of our funding did not permit this.
We believe in transparency and welcome criticism as a means of becoming better. For further details about the grading system and its assumptions, click here. To see the complete coding manual we used to evaluate stories, click here. To comment, click here.
Editors at all seven news providers were sent a pre-posting copy of the analysis. None contested its conclusions.
--John McManus
Next up? Which topics got the most attention, and which the least?