2/24/01

Content Analysis Standards

 

Creating a valid and reliable measure of the quality of a product as complex and varied as a newspaper or newscast is very difficult. The news is as varied as life.

 

Evaluation can’t be so complicated and subjective that two judges would grade the same content differently. And it can’t be so simple that it lumps good with bad. Into the chasm between these two poles, we thrust the following plan.

 

I. General Rules

 

Sampling

 

To provide the fairest representation of news, we will sample randomly, but set up the survey so that each day of the week—Sunday through Saturday has an equal chance of selection. That way, certain weekdays when city councils and county boards of supervisors meet will be included equally with weekends when some news organizations may attempt longer stories with greater perspective.

 

            We will sample all news organizations on the same day, so the raw material of events available is the same. The sample will be drawn approximately one day out of ten during the first year.

 

Readership and viewership studies show that the front page and section fronts and the first segments of newscasts attract the largest audience. Stories appearing in these places are the most important to analyze since they are being offered to the largest audience. Extremely brief stories generally do not make as great an impression on readers and viewers as full-size ones. In addition, most newsrooms allocate more space or time to stories they judge more important.

 

So we will include all stories in premier evening newscast’s first half-hour running at least 20 seconds and all stories measuring at least 10 square inches on the front and local/metro front pages of the newspaper. (Extremely brief stories are also excluded so as not to artificially lower measures of quality such as number of sources.) Stations will be invited to nominate which newscast they would prefer sampled. 

 

Stories will be weighted by time or space, respectively. It wouldn’t be fair to average as equals a story that runs 5 minutes with a short one only 30 seconds long.

 

Sidebars—contiguous stories so closely related to one another that they could have been included in the main story—will be treated as part of one larger story. This avoids lower context and fairness scores that might result from judging stories individually.

 

We will only count news time or space. So banter among anchors, advertisements and promotions will not be considered. All indices will be based on the percentage of news time or space used a certain way. Absolute volume of news will not be considered so as to be fair to both newspapers and television.

 

Newspaper photographs that are related to a particular story on the relevant pages shall be included as part of that story, both in cutline information and space. Photos or graphics unrelated to a particular story on the target pages will be treated as separate stories. Informational graphics will be included with the relevant story.

 

Grading Criteria

 

We will evaluate news providers on seven basic criteria: newsworthiness, context, local relevance, accuracy, fairness, civic contribution, and enterprise. All count equally except the first, which counts double because of its fundamental importance.  All fractions and ties are rounded upward, in favor of the news organizations. The seven evaluative criteria are inferred from journalism’s principal codes of ethics (SPJ, APME, ASNE, RTNDA).

 

What counts as “A” work is necessarily subjective. We take into account that all seven news sources are parts of large corporations with great resources. All operate in one of the nation’s largest and wealthiest media markets. Finally, we are only grading top stories—where we can expect the best work.

 

II. Newsworthiness Index (all stories)

 

This measure will examine the selection priorities of news providers. It will be judged against the primary ethical mandate of journalism: To help the community served make sense of current issues and events. Events and issues that shape the quality of life are most important. The more people for whom the information has this quality, the greater its value. (Events exciting short-term emotional response, say over a sports team winning or losing a game, or even a championship, are important, but less so.)

 

Each story will be judged on three criteria: whether its topic is core news or marginal; the level of the reporting—issue-oriented (thematic) or event-oriented (episodic); and whether it affects few or many people. Core topics rate 2 points; non-core 1 point. Episodic stories gain no additional points while thematic stories gain 2 more points, to reflect their greater usefulness to citizens. Stories affecting many gain 2 more points; those affecting just a few gain no additional points. Thus each story can have any point value between 1 and 6. 

 

Putting a premium on thematic “big picture” reporting that affects many people in the top stories of the day is appropriate for media striving to serve an area as large as the nine-county Bay Area, just part of which constitutes the nation’s fifth largest media market. The size and extraordinary wealth of the market also provides the revenue needed for reporting broadly rather than chronicling isolated incidents that affect few. 

 

 

 

Episodes vs. themes

 

The essential distinction here is between reporting primarily focused on an event for its own sake¾a fire, crash, shooting, parade, protest¾or on an issue, an idea. Is the reporting episodic (event-oriented)¾primarily concerned with a particular event, or perhaps a summary of several unconnected events¾or is it thematic (issue/idea oriented)¾primarily directed at an issue, or a pattern in multiple events over a period of time, or looking at a single event from multiple perspectives across the Bay Area?

 

Episodic stories: More than half of the story describes a particular discrete action, or perhaps several actions unconnected, except by category, e.g. a round-up or summary of fires, athletic events, traffic jams, police actions, criminal acts, etc. Most of the reporting in an episodic story focuses on a single event, generally occurring at a particular place and time. It emphasizes the “what.” Examples include stories about a particular shooting, rape, crash, fire, trial, protest, earthquake or battle.

 

Thematic stories: Half or more of the story concerns issues¾ideas, policies, or areas of concern. If the story is about events, it must examine a pattern or trend of events, or systematically examine the impact of a single event on multiple levels and places across a region. Such stories usually involve a variety of people in different locations coping with a common concern. Reports usually focus more on “why,” “who” or “how” and causes or solutions than “what.” Examples include trend stories about educational practice, or transportation or housing conditions, or systematic reporting on the criminal justice system, the environment or economic conditions.

 

A story, or series of sidebars, about the extent or impact of a heat wave, power shortage, or natural disaster would qualify as thematic if it systematically gathered information from multiple communities about the effects of the event. Even a weather report providing temperatures from across the Bay Area would qualify as thematic, so long as it is comprehensive, summarizing the entire region. Election results would, of course, fit in this category.

 

Stories focusing on specific problems here or there, such as traffic jams or accidents, DO NOT count as thematic. However, a systematic survey showing traffic across the region¾jams and flows¾would qualify, as would trend stories about traffic.

 

Note that a story can be presented in an abbreviated, shallow or superficial manner and still be considered thematic if it concerns an issue. For example, a 20-second story in which the mayor delivers a slogan for gun control would count. However, if s/he spent more than 10 seconds (more than half the time) talking about a specific incident of such violence, the story would be considered episodic.

 

Issues within events: Some stories are about issues, but occur within a discrete event, perhaps a government meeting or a corporate press conference, or a demonstration. If half or more of the reporting concerns the issues discussed, we mark it as thematic. It would be episodic only if the event¾the fact that certain people/groups met, or an action was taken, such as a vote, or people demonstrated or clashed with police or rode bikes to raise funds for charity¾predominates. An analytical report, audit, survey, or plan would almost always be considered an issue.

 

Size of impact

 

Does the issue or event¾as reported¾exhibit the reality or potential to affect a significant number of persons in a direct and lasting way? A significant number of people in an area that’s home to 6 million would be a minimum of 10,000 persons. (The 10,000 need not be local.) A direct effect is one that makes a difference in people’s lives¾it may improve or diminish quality of life. A lasting effect is one that endures for at least six months.

 

By “potential” we mean one could make a convincing argument that the effect was likely, not that it was merely “possible.” The reporting should make the size of the impact clear, unless it could be implied by any reasonable person. E.g., it’s obvious that the appointment of a new school superintendent or police chief or a political nomination portends the possibility of wide and significant effects on the community, but a policy change might need elaboration to earn a “2.”

 

All three criteria have to be met. A particular weather report or storm may have a direct effect on millions of people’s lives, but short of a natural disaster, for most the effect will be fleeting. Sporting events are similar. Tens of thousands of avid fans may be greatly elated or disappointed by a game’s outcome, particularly if a championship is at stake, but for most the effects will have dissipated six months later.

 

Note that the distinction between micro and big picture stories is determined by their breadth of focus, not their popularity or length. The trials of O.J. Simpson, though very popular and frequently lengthy, were micro stories¾event-oriented with direct and lasting effects on very few. A 90-second story about why spousal abuse happens, however, was a big picture story.

 

Core topics

 

1. Politics, including any news about candidates, their positions, political issues, poll results and ballot initiatives or fund-raising or the political system¾U.S. or foreign. Also includes political careers and leaders’ obituaries and memorials for social and political leaders, e.g., Martin Luther King Jr. Profiles of leaders of political, or reform, or social movements go here. Any news about the process of people taking political power, losing it, protesting it, or the process of transition fits in this category. However, if the focus of half or more of the story is on violence, we mark it either 6 if police or crime are involved; or 13 if the military or para-military are involved, or the action is politically-inspired terrorism. We consider action of incumbent politicians in this category only if they are clearly political in nature. Otherwise, place them in category #2.

 

Unless stories bear on fitness for office or are related to public policy, articles about personal lives are excluded (other than obits).

 

2. Government (executive and legislative branches) new government policies, reports, actions, deliberation and decision-making—from the level of city councils and mayors to county boards of supervisors to state legislatures, directors of state bureaucracies and governors to the president, directors of federal agencies and Congress, and other nation's governments. But if half or more of a story is about some issue listed below, we choose that topic. If more than half is about the governmental decision-making, it stays here. Thus a city council discussion about police misconduct would go here, but were half or more of the story about the misconduct itself, and non-council discussion, it would move to the criminal justice category.

 

Deliberations and decision-making of school superintendents and school boards and supervisors or commissions for police or other public services that are below the level of city council, go in the more specific categories below.

 

      Profiles of government leaders that relate to their fitness for office or actions in office go here, but exclude stories that concern only their private lives.

 

If a proposal by politicians is introduced as legislation, it goes here. But if it’s merely a political proposal, not yet submitted to become law, we place it in #1.

 

3. Natural disasters, earthquakes, floods, landslides, avalanches, etc. To qualify, it must include severe damage--destruction of life, homes or major buildings.

 

4. Education, private or public, from pre-school to high school to college to post-graduate. Also corporate training, recruiting, retaining teachers, fund-raising, building/stocking public libraries, museums, zoos, etc. Stories about sports as part of educational or personal development belong here, but not accounts of varsity games.

 

5. Economic events and issues, including business conditions¾local, national or international¾how a company is doing (except health care), labor, employment, stock markets, trade agreements and disputes, civil suits in which one business sues another (criminal and government regulatory business suits, however, go in the next category), etc. Promotional stories¾those about a single company that are written from a positive, rather than objective point of view, and contain no independent expert sources¾go in category 19, “Other Stories.”

 

Stories reported primarily from a consumer’s point of view¾e.g., about prices¾go in #14. Those about government spending, budgets, etc., go in #2.

 

6. Crimes, criminal justice system (police, prosecution/defense, grand juries, courts, penal institutions, parole), criminal and civil trials involving individuals, police actions (including FBI, ATF, INS, IRS and Coast Guard--but not rescues unless criminal action occurs also). Also investigations of wrongdoing including grand juries.  Aftermath of crimes and courts including funerals, memorials, hospital reports, investigations, imprisonment, parole, cash settlements, etc.

 

Supreme Court, and other court decisions go here. So do regulatory trials, e.g. Microsoft.

 

7. Medical/health/fitness events and issues, including scientific research and discoveries in these areas and threats to public health or physical safety. Includes worker safety issues. Also includes fund-raising for medicine--such as AIDS bike rides. Also medical ethics. Includes the business of medicine/health/ nutrition and assessments of medical performance (other than lawsuits). If the threat to public health is primarily environmental¾arising from polluted air, water, etc.¾we mark as #8.

 

8. Environmental events and issues. Includes government or private actions to clean toxic waste sites, limit or increase housing density, protect open space or environment, limit urban sprawl, reduce pollution, etc. Also natural phenomena such as eclipses, global warming, El Nino, etc. Also profiles of environmental leaders past and present.

 

9. Other important social events, issues or trends, e.g., race-, sexual orientation-, ethnic-, disability-, class- relations or tensions or programs; transportation/ commuting, electric power shortages/supply; agriculture, housing, parenting styles; divorce or marriage rates; lifestyle changes, religion, historical info. Also educational cultural events (but not general celebrations like Cinco de Mayo, Fourth of July, etc.) and live plays. If a story fits as well in another, more specific category¾e.g., environment or technology, it goes there rather than here.

 

10. Technology/science issues and events. Non-medical scientific discovery--physical and social science, e.g. space exploration. History of technology. New social theories. Business aspects of technology companies/services go in category 5.

 

11. Major fires, accidents--auto, air, rail and otherwise--or other mishaps, their investigation or their repair, e.g. sewer line rupture, etc.—that involve more than $25 million in damage, 5,000 or more acres burned, or the loss of two or more lives of Bay Area residents, or five or more lives elsewhere. Also the results of such incidents—hospital reports, funerals, damage reports, investigations, inspections, etc. (Accidents involving criminal charges, however, go in category 6.) If there is no damage estimate, we assume it’s under $25 million unless obviously over.

 

      We are careful with round-up stories summarizing a series of unrelated events, such as accidents, fires, crimes or investigations. If reported as a single story, we do not aggregate the injuries or damage to decide between this category and #18. Instead, we evaluate the most serious incident in the summary and use that alone as a placement guide.

 

12. Weather (short of natural disasters) and direct effects of weather, record heat or cold or wetness, heavy surf. However, if effects include content that might be placed in another topic category, we place the story in the category in which the majority of the time or space is spent. Thus a minor brush fire caused by dry weather would go under category 18 if half or more of the time or space was spent describing the fire or attempts to control it or damage estimates, etc.

 

13. Military actions, deployments, attacks, threats, exercises, accidents, military peace-keeping efforts and terrorist acts with a likely political motivation. Also military policies, treaties concerned with weapons or aggression, studies of readiness, budgets and weapons. Includes US and foreign.

 

14. Consumer reporting about commercial goods and services (but excluding medicine/fitness/ health). Generally, consumer reporting must include explicit comparisons of products, price, service or selection with some reasonable standard of quality, or at least one other retailer or manufacturer, or with an average. The tone must be objective, or critical, rather than promotional. Promotional stories go in category 19, “Other Stories.”

 

Stories about what average consumers pay for gasoline or housing in a region, or over time, also go here. However, if the emphasis is not on consumer prices, but on attributes of corporations such as profit performance, stock prices, or other economic data not directly tied to the average consumer, then we choose category 5.

 

Non-core topics

15. Human interest, seasonal celebrations/festivals (e.g., Christmas, July 4th) and entertainment stories, including stories about TV programs, movies, and popular culture. Also fashion trends, style, auto and boat shows, and fads. Stories in this category often focus primarily on amusement, or other emotions: curiosity, surprise, adventure, joy, love, loyalty, sorrow, sexiness, anger, disgust, rivalry--e.g. cute animal or other stories, re-unions, what people are doing on a holiday,  features on children, tales of oddities, etc.. These are often light stories meant to bring a smile or tug a heartstring, but can also be sad or serious reminiscences about tragedy or the human condition.

 

16. Information about celebrities and other widely known persons both famous and notorious. Generally concerns the lives/deaths of actors, actresses, athletes, authors, etc. Excludes politicians unless story un-related to public's business.

 

17. Sports and sports-related (including recreation and hobbies), amateur and professional, including equipment and facilities. Stories in which more than half the content addresses the business of sports, however, go in category 5, Economics. Also excludes stories about sports facilities where half or more of the content focuses on the use of public money or a vote, or facilities in public schools or parks. Also excludes the role of sports in education. These go in categories for politics, government or education.

 

18. Minor fires, accidents, mishaps, power outages, traffic reports/ problems, BART/mass transit delays—those in which fewer than two Bay Area residents (fewer than five from elsewhere) lost their lives or less than $25 million in damage was done, or less than 5,000 acres burned. Results of such incidents, e.g., hospital reports, damage, funerals, etc. Misadventures—people hurt or missing or killed in non-criminal or uncertain circumstances--individual suicides.

 

19. Other stories. Including promotional journalism¾stories extolling the virtues of a particular business. Otherwise, we use this category only if the story can’t go elsewhere.

 

Grades

 

Grades are based on a weighted average. Depending on topic, level of reporting, and number affected, stories earn between 1 and 6 points. A score of 100% would indicate the top stories were all about core topics, treated thematically, and affecting many (>10,000). Stories with 5 points count 5/6ths what a 6 point story does, down to stories with 1 point counting 1/6th toward the station’s or paper’s newsworthiness average. Averages of 80% or more earn an A; 77-79 =A-; 74-76%=B+; 71-73%=B; 68-70%=B-; 65-67%=C+; 62-64%=C; 59-61%=C-; 56-58%=D+; 53-55%=D; 50-52%=D-; < 50%=F.

 

The grading scale is less stringent than an academic evaluation for several reasons. First,  some important breaking news will necessarily be episodic, thus earning 4 of 6 possible points. Second, while we believe that skilled journalists with sufficient resources can render the consequential compelling, we won’t penalize the judicious display of merely interesting stories. (Of course, stories in the second half of the newscast or inside pages and other sections of the newspaper may address whatever editors deem appropriate.)

 

III. Context Index (locally-originated stories or parts of stories only)

 

This measure is based on the number and quality of sources in a story. Up to 150 points are assigned as follows:

 

·        100 for a random sample of more than 100 persons conducted by the newspaper or station under normal social science requirements.

·        40 for each named independent expert source, up to a total of 3 sources.

·        20 for each specific source, up to a total of 5 sources.

·        10 per story for 1 or more un-named sources.

 

Sources may be persons or documents. They may be quoted directly or paraphrased. They need not be on air. Generalized sources, such as “police,” “educators,” or “critics” count as unnamed sources. A specific source must be a particular person, never plural. He or she may be anonymous only if expressly granted anonymity in the report, e.g., a source who spoke only on condition of anonymity. A document’s specific title is unnecessary, but the authoring institution, department or individual must be named, e.g., a report prepared by the Public Utilities Commission.

 

Because stories vary in how many sources they need, stories with more than 100 points can make up for those with less. We estimated that the top stories of the day should average more than two identified non-expert sources and one non-specified source. Simple event stories might appropriately have fewer and issue stories more. 

 

Grades range from A, 85 points or above to F, less than 55. Minus and plus grades are assigned at 3 point intervals, i.e., 82-84=A-; 79-81=B+, etc.

 

IV.  Local Relevance Index (all stories)

 

This measures the percentage of all stories that either take place in the nine-county Bay Area or have a direct relationship to the region’s citizens. Such directly related stories would include all stories affecting the entire state, especially those emanating from state government. Thus a story about a train wreck in Southern California would not be included (unless the story focused on stricken residents from the Bay Area), but a story about how Africanized bees are making their way north in the state, would. Weighted by space or time, stories would be sorted into Bay Area/Bay Area-related and outside stories.

 

While the Bay Area has large numbers of immigrants and foreign nationals and what happens outside the U.S. is important to them, they do not represent a majority of residents, nor is what happens abroad as important as often as what happens in their present locale.

 

The index would be graded differently for newscasts with (usually adjacent) national/international news supplements (network newscasts) than for stand-alone newscasts and newspapers. The former would be expected to concentrate more on local news than the latter. Channel 2 cannot rely on a network newscast to present news outside the Bay Area. Channels 4, 5 and 7, however, also carry network news.

 

The grading standard for Channel 2 and the three newspapers is at left; at right is the standard for Channels 4, 5 and 7:

 

            Non-supplemented                                                      Supplemented

 

 

 

V. Accuracy Index (locally-originated stories and parts of stories only)

 

This measure will be based on telephone interviews with identifiable and reachable Bay Area sources. Unlike other indices, this survey will occur infrequently (because it will consume so much time).  The unit of analysis also changes, from the story to the source. Accuracy assessment will be from the source’s perspective. Points are assigned as follows:

 

·        20 for correctly spelled name. (In TV, 10 for spelling and 10 for pronunciation.)

·        10 for correct title or identifier.

·        40 for accurate, or mostly accurate quotation; 20 for moderately accurate; 0 for substantially inaccurate.

·        30 for accurate or mostly accurate representation of source’s main point during interview; 15 points for moderately accurate; 0 points for substantially inaccurate.

 

However, if the interview was adversarial and the source reports substantial inaccuracy, we’ll deduct 15 points from this category and add it to the previous category. (In adversarial interviews, sources may have a self-interest in disparaging reporter’s accuracy, particularly about which parts of the interview were most newsworthy.)

 

Unlike other indices where the unit of analysis is the story, here scores are based on the average degree of accuracy per source. Grading standards are the same as on the context index.

 

Note that we have not yet been able to overcome logistical problems that block our assessment of accuracy on television.

 

VI. Fairness Index (locally-originated stories and parts of stories only)

 

This measures the percentage of time/space consumed by controversial stories in which more than one side is given the opportunity to make its case (even if that opportunity is rejected). Stories in which a single perspective is acceptable are excluded from the analysis. Such stories include uncontested decisions of boards and other groups, authoritative accounts of events such as fires, accidents, and the weather. Also excluded from analysis are stories in which an opposing view would not be available, e.g., a not-yet-apprehended criminal suspect. Finally, opinions offered by columnists are not counted.

 

The fairest stories offer competing sides the same opportunities for comment. If one side is quoted directly, so are the others if they choose (and can be reached for comment). If one side is on air, so are the others (unless there is some obvious reason why they cannot be.) Neither time nor space must be equal, however. (We seek objectivity of method, but not of result).

 

Less fair are stories in which all sides are represented, but not given equal opportunity to respond. Unfair stories don’t give all obvious sides a chance to comment.

 

Fairness grades are based on the percentage of news time or space in stories judged completely fair plus one-half the time or space in stories judged partly fair. Grades range from A, 85% or above to F, less than 55%. Minus and plus grades are assigned at 3 point intervals, i.e., 82-84=A-; 79-81=B+, etc.

 

VII. Civic Contribution Index (all stories)

 

This question seeks to measure reporting on the actions of those at the controls of our government at all levels, but not those who carry out their decisions. So we count the percentage of time or space taken by all stories in which 1/4 or more of the story describes politics, actions (related to public business) or deliberations of government supervisory or regulatory or lawmaking boards or bodies, elected officials or the heads of any agencies of the U.S., state or local governments. We include studies (conducted by government or outsiders) of effectiveness or problems of arms of domestic government and their policies. Likewise, we include court cases involving civil liberties, and government vs. corporate defendants (e.g. Microsoft), but not prosecutions of individuals unless they are government officials.

 

We exclude routine actions of police, schools, fire departments and other arms of government, unless expressly political. Also excluded are actions of governments outside the U.S. One exception: We include any investigative report about government or about a private company that touches on its relationship with government.

 

Journalism ethics asks for constant scrutiny of how our government is behaving. So such stories ought to be a mainstay of coverage. However, they need not consume the majority of news space or time.

 

The grading standard is as follows: 40% or more=A; 37-39=A-; 34-36=B+; 31-33=B; 28-30=B-; 25-27=C+; 22-24=C; 19-21=C-; 16-18=D+; 13-15=D; 10-12=D-; <10=F.

 

VIII. Enterprise Index (locally-originated stories and parts of stories only)

 

This measures the proactivity of the newsroom, its willingness to seek our answers to the public’s questions rather than simply react to events or other’s agendas. The most demanding of such stories on newsroom resources is the investigative story.

 

An enterprise story differs from spot news in time and focus. It may begin with a recent event--although not usually something that happened that day--but the view is more informational and broader.  It generally answers a question the public may have about some issue or event. Enterprise stories are originated by journalists usually trying to explain or contextualize important happenings or issues.

 

            Investigative stories are enterprise stories that focus on mis- or malfeasance usually by government, but sometimes also by private industry or others. The investigation is conducted by the journalist, not law enforcement authorities.

 

Because they are more demanding, time/space spent on investigative reports is weighted by a factor of 4. In other words, a 3-minute investigative story is treated as if it lasted 12 minutes.

 

Grading follows the same standard as the Civic Contribution Index: 40% or more=A; 37-39=A-; 34-36=B+; 31-33=B; 28-30=B-; 25-27=C+; 22-24=C; 19-21=C-; 16-18=D+; 13-15=D; 10-12=D-; <10=F.

 

 

IX. Overall Grade

 

This is a mean, computed as if the constituent letter grades were part of an academic grade: An A is worth 4.0, A- is 3.7, B+ is 3.3, B is 3.0, etc. The first index, newsworthiness, is counted twice.  The grade assigned is whichever lies closest to the mean grade with ties going to the news organization, e.g., a 2.5 average lies equidistant between 2.3, a C+ and 2.7, a B-; we would assign a B-.

 

Standards revised on 4/24/01