Seven basic yardsticks of journalism quality

Posted Feb. 1, 2005
-Back to main story-

Local relevance
Civic contribution

Sometimes journalists ask, "Who's Grade the News to tell me what's good and bad reporting?" We answer that we attempt to use the standards to which professional journalism organizations themselves aspire publicly.

Grade the News created seven measures, or "indices," of quality journalism derived from the codes of ethics of various journalism organizations. Most of the "rules" of journalism warn against certain unethical behaviors. Only a few command positive action. So our guidelines are interpretations based on the codes' stated core principles.

For example, the Society of Professional Journalists' Code of Ethics enjoins journalists to "diligently seek out subjects of news stories to give them the opportunity to respond to allegations of wrongdoing." This is the basis of our "fairness" index.

The preamble of the SPJ Code proclaims that "public enlightenment is the forerunner of justice and the foundation of democracy." Our concept of "newsworthiness" comes from this assumption that journalism's purpose is to advance self-government by highlighting the most important public issues.

News "should not oversimplify or highlight incidents out of context." That is the basis of our "explanation" index. Stories that focus on events to the exclusion of trends -- miss the forest for the trees -- do news consumers a disservice.

For further details, refer to our 2003-04 coding manual.



NEWSWORTHINESS is based on two factors: 1) whether the story topic is "core" or "peripheral," and 2) whether the story is likely to have a direct and lasting (six months or more) informational impact on a wide audience (at least 10,000 people).

For example, on Feb. 5, 2004, KRON Channel 4's Greg Lyon produced a thoughtful and practical story on the threat of mad cow-tainted beef in the Bay Area. Public health is a core topic and tainted beef is a concern affecting millions. The story got a perfect score on newsworthiness. Later in the broadcast, though, we learned of the "Precarious Pooch Perch," about a dog stuck on a San Francisco cliff, complete with video from the dramatic rescue. It was a fun story but less newsworthy on our scale.

Core topics include just about everything but fender-benders, Hollywood gossip, sports and heartstring-tugging purely "human interest" pieces.

Core topics:   Peripheral topics:
Government action
Natural disaster
Other important social issues
Major fires, accidents
Consumer reporting


Human interest
Minor fires, accidents
Other stories (including promotional)

Journalism's purpose, according to ethical codes like that of the Society of Professional Journalists, is to help readers and viewers make sense of pressing issues and events that affect them.

With occasional exceptions, core topics advance the purpose of news more than peripheral topics. Even if many are interested, who ends up on "American Idol" is less likely to affect the quality of our lives over the long run than the state budget crisis, the course of the war in Iraq or even a school board's decision to cut teachers and enlarge class sizes.

Similarly, stories with the potential to affect large numbers in a significant and non-transient way are usually more newsworthy than stories with only brief impact or consequence for just a few.

Newsworthiness index


Core topics rated 2 points, while the peripheral topics got 1 point.

Stories with wide impact rated 3 points, narrow impact 1 point. With seven million residents, the odds are good that tragedy will strike someone in the Bay Area every day. While that accident, shooting, fire, rape or abduction will have incalculable impact on the family and friends, perhaps even the entire neighborhood of the victim, for most of us the effect is likely only to be sympathy or fear.

We set the threshold for wide impact at 10,000 people or more. That may sound like a lot. But it's less than a fifth of 1% of local residents -- about the population of the smallest Bay Area city. We chose 10,000 to give full credit for any story affecting an entire municipality or even a moderate-sized school district.

Grades are computed for each story by multiplying topic by impact rating. We weighted the stories by size when we figured the news outlet's score. It would not have been fair to count a 30-second "reader " TV story as much as a three-minute "package."

We recognize that news is a business and stories that are merely interesting may deserve prominent display to sell the newscast or newspaper. So we set grades to allow a station or newspaper to spend up to 15% of its most valuable time or space -- the front page or first segment -- on the lowest-scoring stories and still earn an A.

Letter grades for each news department are as follows: A perfect score would require all top stories to be core and have wide impact. But 90% or more space or time devoted to such stories still merits an A; 85-89% receives a B+; 80-84% rates a B; etc.

Because this index underlies all of the rest, we count it twice in computing the overall grade. No matter how fully sourced, local, fair, well-explained or enterprising a story might be, if the topic is peripheral and few people are affected in a significant way, the story is unlikely to maximize public understanding of current issues and events.

Changes from the last report card:

For this sample period we stopped coding the final sports and weather packages in each TV broadcast, on the assumption that they were analogous to the sports or weather pages in newspapers, which we do not grade. Excluding routine weather reports boosted scores for TV because they are usually graded as having little long-lasting impact; excluding sports likewise boosted scores because it is not a core topic.

At the request of area news directors, this report card includes the entire newscast, rather than just the first 30 minutes of hourlong newscasts that we analyzed in the past.

Another change from last year: As news media built the celebrity of Laci Peterson and her now-convicted killer, husband Scott, that began to drive coverage more than the criminal aspect of the case. So we moved Peterson stories during this sample period from crime to the celebrity category. News outlets that displaced other news from display pages or the scarce time available in a newscast to follow every twist in the Peterson trial may have lowered their newsworthiness grades.

Finally, we increased the value of the "impact" rating to 3 points from 2 in our last survey. We did this because we felt the overall importance of each story was not adequately reflected in the grades. This change did not affect the grades of any newspapers, but did affect the grades of some TV stations.



CONTEXT measures the number of sources, and independent expert sources, in the day's top stories. An average of four regular sources or two experts merits an A.

Everyone has different perceptions of reality. What becomes news is often complex; it describes life. And reporters are rarely there to record it firsthand, so they must rely on sources to tell stories.

Issues as complicated as the California ballot propositions, the war in Iraq and the appropriate staffing of local firehouses cannot be reduced to a single point of view. Yet local stations routinely did so. In our sample, TV stories with one or no source at all comprised 32% of airtime -- about the same as in the last sample. In comparison, newspapers with fewer than two sources took up only 6% of measured column inches, one-third less than in the last sample.

We counted only identified sources, or those who were expressly offered anonymity to protect them from reprisal. When reporters don't identify sources, the reader or viewer is left in the dark about how much to believe in the comment. On Dec. 29, 2003, the San Jose Mercury News, ran a half-page 1A story titled, "Sacramento's new look: cigars, designer suits are in ..." It was one of the least substantive stories we'd seen about the change in administration from Gov. Gray Davis to Arnold Schwarzenegger. And it showed up on the sourcing: We counted only one on-the-record source in the whole story. The rest was unattributed hubbub, such as "In: Esquire Bar & Grill. Out: turkey sandwiches."

Context index


Each identified source rates 20 points. Independent expert sources, who can lift a story above "he said, she said" confusion, merit an additional 20 points, for 40 points each. To level the playing field between print and broadcast (which has less time than print has space) we capped the number of expert sources at three and regular sources at five. In newscasts, but not newspapers, we also counted "sidebar" stories as part of the main story. Since stories were weighted by size, that improved broadcast scores.

Sources could be persons or documents. They could be quoted directly or paraphrased, on camera or off. Sources who refused comment were counted just the same as those who spoke.

In setting our standards, we presumed that stories important enough to merit display pages or inclusion in a newscast aired in prime viewing hours should average more than two sources.

To merit an A, a TV station's stories, weighted by size, needed to average 80 points or more. We assigned a B+ for averages between 75 and 79 points, a B between 70 and 74 points, etc. Because newspapers have space for more sources, we set a slightly higher standard for them. An A required an average of 100 points or better. A B+ required 95-99 points, etc.

Changes from the last report card:

Newscast context scores improved when we stopped coding the final sports and weather packages in each TV broadcast, because those stories contain few sources. (We made the change on the assumption that they were analogous to the sports or weather pages in newspapers, which we do not grade).

For the first time, newspapers had to average one source more than TV for the same grade.



EXPLANATION means big-picture reporting (about issues and thematic treatment of events) as opposed to episodic reporting (micro-view news of isolated events that focus only on the event itself).

Many stories accurately record the "what," but ignore the "why" and "so what." For example, on Nov. 13, 2003, KNTV Channel 11 reported an incident of road rage involving a shooting. The reporter unsuccessfully tried to suggest there was a broader lesson to be learned in the dramatic incident, saying: "Crime tape and paint marked the spot where the bullet casings landed. It reminded drivers today how dangerous road rage can be."

On Aug. 25, 2003, KPIX Channel 5 painted a more useful picture. A weekend of violence had wracked Oakland -- seven incidents in three days. While the report certainly played the sensational angle -- complete with a widow weeping on-camera -- the reporter also broadened it to talk about larger themes. It included a map of the city and an interview with Mayor Jerry Brown about his views about how improvements at the probation department could prevent future violence.

It's cheaper and easier to fill a newspaper or newscast with reports of seemingly random violence, fires, parades, reunions or even fisticuffs between politicians than to treat a problem as an issue. All the sources necessary to harvest such a story are at the scene. A reporter can complete the story in several hours. And the drama or spectacle will draw readers or viewers across the region.

But such reporting leaves us mostly afraid, sad or perhaps feeling lucky to have avoided harm -- not informed of causes, effects and possible solutions. Reporting violence episodically cultivates a sense that nothing can be done.

Researchers have learned that news viewers and readers gain much more insight from issue and thematic reporting than from an episodic approach.

Explanation index


We classified a story as issue-oriented if at least half of it concerned an idea, such as a government or school policy, or about causes, effects or patterns of events -- essentially anything other than an event reported for its own sake. Since some simple events merit top billing, we set the grading standard loosely: If 70% or more of the stories were about issues or events treated thematically, an A was awarded. A score of 65-69% earned a B+; 60-65% earned a B, etc.

Local relevance

LOCAL RELEVANCE is a measure of the percentage of news that takes place in or directly concerns the San Francisco Bay Area (the customary nine counties surrounding the Bay plus Santa Cruz).

Proximity is a core news value. Generally, the nearer some event or issue strikes to home, the more impact it has on our lives.

For example, on Nov. 26, 2003, KNTV Channel 11 reported that NBC's Rhode Island affiliate's helicopter experienced a "hard landing" while filming a promo for the news station. Why did the NBC station in San Jose deem this crucial for its viewers to know?

By contrast, on May 7, 2004, KTVU Channel 2 reported on Secretary of Defense Donald Rumsfeld's apology for the Abu Ghraib scandal. Rather than merely rely on the satellite news feed from Washington, the station sought extensive reaction from local folks.

And yet, the wars in Iraq and Afghanistan constitute obvious exceptions to the value of proximity; our soldiers and tax dollars are on the line. What happens around the world can affect us very directly, as we learned so tragically on Sept. 11, 2001. The Bay Area's large immigrant population also makes news from abroad more important locally than it might be in the American heartland.

The value of proximity is also diminished by the global economy. Corporations are shifting jobs and investment dollars across national borders with powerful effects on employment and standards of living.

So we counted not just events taking place in the Bay Area, but those occurring elsewhere in which a third or more of the report made a local connection. We also counted stories about what the California Legislature is doing with our money and statewide elections.

Local relevance index


Three stations, KGO Channel 7, KPIX Channel 5 and KNTV Channel 11, broadcast network newscasts adjacent to their local offerings. They can count on the network to cover national and international news, leaving them free to follow local news. On the other hand, KRON Channel 4 and KTVU Channel 2, as well as the newspapers have an obligation to carry more than local news. So we created two standards, one for the newspapers and stations without a network program to provide national and international news, and one for those stations with such newscasts.

Even for stations with network news, we had previously increased allowances for non-local content from our 2000 survey, not just because of the war, but because this year we sampled early and late newscasts. Producers of 11 p.m. newscasts cannot assume viewers saw a network newscast hours earlier. It's also true that not everyone is home to see network newscasts, because most local stations schedule them before 6:30 p.m.


Civic contribution

CIVIC CONTRIBUTION measures how well reporters kept an eye on those in control of government at every level from school board to the White House.

Government and political stories need not be boring. The most successful stories that hold elected leaders to account reach down to the level of the taxpayer, the MediCal recipient, the prisoner or the commuter, to ask how government decisions affect them.

For example, on Feb. 18, 2004, KTVU Channel 2 interviewed a young man who spent six years as a ward of the California Youth Authority. He told a harrowing tale of being locked in tiny isolation cells, stripped naked, and of guards instigating fights among wards. His account was broadly substantiated by an expert evaluation of the system. A mother whose son slit his wrists during his detention spoke of the "barbarity" of the system. The report held government to account without a single foot of videotape recorded at a meeting.

Journalism ethics asks for constant scrutiny of how our government is behaving. So such stories ought to be a mainstay of coverage. However, they need not consume the majority of news space or time even for the day's top stories.

Civic contribution index


We measured reporting on the actions of those at the controls of our government at all levels, including discussion or protest concerning those people or their policies.

To allow reporters to file stories from the places where policies meet people's lives, we counted the percentage of time or space taken by all stories in which one-quarter or more of the story described politics, actions (related to public business) or deliberations of government supervisory or regulatory or lawmaking boards or bodies, elected officials or the heads of any agencies of the U.S., state or local governments. We included studies (conducted by government or outsiders) of effectiveness or problems of arms of domestic government and their policies. We excluded actions of governments outside the United States.

The grading standard is as follows: 40% or more time or space spent on such stories earned an A; 35-39%, a B+; 30-34% for a B; etc.



ENTERPRISE measures whether reporters passively react to press releases and events learned from listening to the emergency scanner radio, or actively seek answers to the community's basic questions, including investigating the exercise of power.

We measured the proactivity of the newsroom, its willingness to seek out answers to the public's questions, rather than simply react to events or others' agendas.

Issue-based stories initiated by journalists themselves, either "enterprise" reporting or the more time-intensive genre of investigative reporting, help readers and viewers make sense of the world by presenting a story that's more than just the snapshot of the events of the last 24 hours.

On Feb. 18, 2004, for example, Mercury News reporter Paul Rogers produced an investigative report on the possible dangers from the shipment of dangerous chemicals. He found that the Bay Area "remains at risk, some shipping experts say, because chemical ships are not required to have tugboat escorts" like oil tankers are. There are no safeguards that would protect against an accident, like the one that nearly happened when a ship containing pressurized anhydrous ammonia lost its steering while passing through the Golden Gate. It's unlikely that Mr. Rogers found this information on a press release.

Enterprise index


For this index, we analyzed only stories produced by local news organizations, passing over stories provided by networks or wire services. We divided stories into categories based on how much initiative the newsroom showed.

At the passive end, we classified information typically learned from press releases or listening to the emergency "scanner" radio. In newsrooms this is called "spot" news -- coverage of events occurring during the last 24 hours that are typically terse and focused on the basic elements of an event. While valuable, such reporting is often episodic and disconnected. Letting press releases set the news agenda turns over power to outsiders with deep enough pockets to arrange and publicize events.

At the active end, we identified enterprise reports -- coverage decided upon in the newsroom. It usually takes a longer view than the past news cycle and often offers perspectives that are unique in the region.

No type of enterprise reporting demands more newsroom resources than the investigative story. Because the watchdog function of journalism is so important, time or space spent on investigative reports is weighted by a factor of 4. In other words, a three-minute investigative story is treated as if it lasted 12 minutes.

Even a news department well funded enough to develop enterprise stories must cover the top breaking news of the day. For that reason, enterprise rarely constitutes more than half of the top stories. The grading standard allows plenty of room for breaking news, while rewarding the big-picture reporting that journalists originate to tie events into comprehensible patterns.

Grading follows the same standard as the civic contribution index: 40% or more rates an A; 35-39% merits a B+; 30-34% earns a B, etc.



FAIRNESS measures whether reporters get at least one other side in controversies or reports of wrong-doing.

This was perhaps the area in which newsrooms collectively improved the most. On television, stories in which one side in a controversy or someone accused of malfeasance or neglect was not given a chance to respond took up 17% of airtime. That was a marked improvement from the last study, in which the figure was 31%. Compare that with newspapers: Only 4% were graded as blatantly unfair, again an improvement from the last study, which reported 8% unfairness.

However, our study still turned up several of examples of unfair or one-sided reporting. That includes naming criminal suspects who were given no chance to respond to the charges against them. Some reports strongly suggested the suspect of a crime was guilty, but still failed to get that person's response. It matters because charges are sometimes dropped, the accused is acquitted and investigators accuse the wrong person or mischaracterize the misdeed.

This is especially important in political reporting, in which politicians routinely take one another to task in public. That happened on July 30, 2003, when KTVU Channel 2 political reporter Randy Shandobil talked with then-San Francisco Mayor Willie Brown, who had harsh words for then-Gov Gray Davis' campaign to avoid recall by the voters. Rather than let the charges go unanswered, Mr. Shandobil tracked down the governor at an unrelated press conference for a response, demonstrating a willingness to go out of his way to be fair.

The Society of Professional Journalists' Code of Ethics enjoins journalists to "diligently seek out subjects of news stories to give them the opportunity to respond to allegations of wrongdoing."

We asked our advisory board of local journalists whether it was practical to seek comment from those recently arrested. Eight of nine said the attempt should be made. As Raul Ramirez, news director at KQED explained: "The fact is that journalists who make an effort to get 'the other side' under those circumstances often get a curt 'no comment.' But, occasionally, they are rewarded with insights and angles that make their stories richer and, certainly, better balanced."

We saw and read several instances of blatant unfairness to the accused. Sometimes, though, unfairness can occur through innuendo. On Nov. 13, 2003, an anchor on KGO Channel 7 told viewers: "Barry Bonds, Sammy Sosa -- big stars in Major League Baseball who deny taking steroids. But today drug tests confirm that some in the sport are." To us, this reads like a backhanded accusation that Mr. Bonds and Mr. Sosa tested positive. But no evidence was presented that either used the drugs. And the report goes on to say that only 5% of anonymous tests came back positive. Neither player was asked to respond to the news.

Fairness index


We added up the percentage of time or space consumed by controversial stories in which more than one side was given the opportunity to make its case (even if that opportunity was rejected).

Stories without controversy or allegations of wrongdoing were excluded from the analysis, as were those produced outside the local newsroom. Also excluded were stories in which an opposing view would not be available, e.g., a not-yet-apprehended criminal suspect. Finally, opinions offered by columnists about an event or issue were not counted; journalism ethics codes permit them wider berth.

The fairest stories offer competing sides the same opportunities for comment. If one side is quoted directly, so are the others if they choose (and can be reached for comment). If one side is on air, so are the others (unless there is some obvious reason why they cannot be). Neither time nor space must be equal, however. (We seek objectivity of method, but not of result.)

Less fair are stories in which all sides are represented, but not given equal opportunity to respond. One side may be on camera or directly quoted while a reporter paraphrases the other side's argument. Unfair stories don't give all obvious sides a chance to comment.

Fairness grades are based on the percentage of news time or space in stories judged completely fair plus one-half the time or space in stories judged partly fair. Grades range from A for 85% or above; B+ for 80-84%; B for 75-79%, … to F, for less than 55%.