Ethan Zuckerman’s online home, since 2003

Metrics for civic impacts of journalism

How do news organizations measure impact?

That’s the question I found myself talking with Phil Bronstein of the San Francisco Chronicle earlier this week. He’d gotten in touch to talk about what tools are available to help newspaper editors track audience and reach for their stories, hoping that I’d have some insights on “cutting edge” techniques to track the reach and impact of news stories posted online. We talked a bit about the challenges of social media tracking after the demise of Technorati, the possible benefits of bit.ly-type analytics, questions of influence and reach raised by Klout and similar systems. All well and good, but measuring how many people read a story is something any web administrator should be able to do. Audience doesn’t necessarily equal impact.

There may have been a day in the rosy past of newspapers when a wall between the publisher and the editor meant that newsrooms published only what was most newsworthy and civically important, without consideration of a given story’s appeal to their audience. In an age where editors can know instantly whether a story on a school council meeting is playing better than a story about a labor action, it’s hard to believe that access to analytics doesn’t shape coverage decisions. Some outlets, like the Huffington Post, have embraced this new world to the point where they are poster children for analytics-driven coverage, using feedback from Google Analytics to inform most if not all decisions about story placement and emphasis. This willingness to respond rapidly to market feedback has likely helped HuffPo’s rapid audience and market growth – whether or not AOL’s acquisition of the site was a wise move, most newspaper publishers would welcome ten-figure interest in their properties.

The danger of traffic-based analytics driving journalism is that you may end up with newspapers that look more like Demand Media-style content farms and less like the civic guardians we want and need them to be. It’s certainly fair to observe that newspapers have been audience driven, at least in part, since inception and that some of the shortcomings of contemporary papers, as well as local newscasts, derive from a focus on driving readership and viewership. But adding an analytics into the newsroom puts the question “Is this story reaching a broad audience?” front and center in a way that’s hard to ignore or avoid.

If an ideal editor is making decisions based on what’s newsworthy, and a realistic editor is civic and audience concerns, how do editors determine whether they’re successfully serving both masters? What are appropriate analytics for civic impact?

As Phil and I talked, I found myself thinking about the LA Times’s coverage of obscenely high government salaries in the city of Bell, CA. In depth, investigative reporting by Ruben Vives and Jeff Gottlieb led to fraud trials, a turnover of the city government, and ultimately to a Pulitzer for the pair of reporters. The reporting on Bell, CA suggests two ways newspapers might measure civic impact: the arrest of bad guys, and the praise of one’s peers and professional societies. But these aren’t exactly quick metrics, and not every worthwhile piece of civic journalism has this magnitude of impact.

Traffic doesn’t seem to be the right measure of civic impact. A story that gets lots of page views or is widely shared might be civically relevant, but might also be salacious – amusing and popular as much of the Anthony Weiner coverage has been, I’m not sure it’s been a positive contributor to our civic involvement. Phil suggested that comments aren’t an adequate metric either. Stories that garner long comment threads could suggest broad involvement, but also may suggest partisan controversy.

I mentioned an idea that I’ve been trying to pitch for a while: in an age of participatory media, news demands participation. Or to quote Benjamin Barber, “People are apathetic because they are powerless, not powerless because they are apathetic.” For people to pay attention to an important story, it’s possible that we need to work to make it possible for people to have an impact on the outcome of the story.

Ideally, we can find better ways to do this than turning our Twitter icons green in solidarity with Iranian activists. Reporting on local civic issues offers the possibility of connecting people to opportunities for action in their own communities. And if newspaper web sites start trying to broker these connections, we gain another possible metric – the efficacy of a story in connecting people to community organizations, volunteering opportunities, and other forms of civic engagement.

That’s not a comfortable role for newspapers to take, Phil reminds me – it smacks of advocacy journalism. But the Bell, CA story is another form of activist journalism: by relentlessly shining a light on political malfeasance, Vives and Gottlieb were demanding that someone take action against these corrupt officials. Eventually, both citizens and prosecutors did. The difference between what I’m proposing and what the Pulitzer winning reporters did is that I’m suggesting newspapers link to possible solutions and measure how effective at driving engagement they are.

This would be far from a perfect metric. It wouldn’t tell you how many people read a story on homelessness, and then sought out community organizations on their own to volunteer with… though adding a feedback cycle where local organizations could communicate changes in community involvement to newspapers might. And it wouldn’t track one of the most critical functions of investigate journalism: the fear it generates in politicians and corporate actors that they could end up on the front page of a newspaper if they break the law. Clay Shirky is worried that losing this deterrence effect is one of the dangers of losing “accountability journalism” in the transition from broadcast, offline models of journalism to participatory, digital ones.

My point is not that I’ve got good metrics for civic engagement for newspaper journalism… or any journalism. It’s that we need to be thinking about finding and developing them. What we measure, we become. If we measure only how many people view, like or tweet, but not how many people learn more, act or engage, we run the risk of serving only the market and forsaking our civic responsibilities, whether we’re editing a newspaper or writing a blog.

10 Responses to “Metrics for civic impacts of journalism”

  1. Kate Fink says:

    Hi–this is something I’ve been really interested in (first as a reporter and now as a student), so I’d love to know more about ideas for how to measure this. This could be done through a social audit-type approach… few news organizations do social audits now, and most audits don’t cover this kind of thing.

  2. Ethan says:

    Would love to hear more about social audits, Kate – how would you suggest a paper go about doing that? Probably isn’t going to address my hope for near-realtime metrics, but sounds like a really worthwhile addition to what newspapers should be doing to measure impact.

  3. Nice vet of the challenge, Ethan. I was really hoping you had solved this dilemma! In public broadcasting, where social return on investment (SROI) has been eluding us for decades, we continue to satisfy ourselves that audience = public service. (And the more loyal, the more they support the effort.) But a) this borrows from the commercial ratings marketplace, b) it doesn’t give us that better target of journalistic impact to aim for, and c) we can get big ratings without any local news effort (just play NPR national programming) which begs the question of why local stations should invest in local news (or exist in the future)? I recently made a list of some 15 quantifiable outcomes of our work… none resembling the better bulls-eye… but maybe we need *a host of metrics* to get at something as difficult as SROI.

  4. Kate Fink says:

    Building on the Guardian’s social audit model is one option–they address editorial but could do so much more to evaluate impact (still, the fact that they address editorial at all seems to put them miles ahead of most news organizations).
    I would like to see newsrooms adopt a strategic planning process–set goals that align with their missions, formulate concrete plans to help them achieve those goals, and periodically evaluate their progress. If they don’t take the time to figure out how to measure impact, they’ll just fall back to what’s easy to measure–like traffic.

  5. Hi, Ethan:

    I’ve been pursuing these questions for a few years now with different intellectual partners—Tracy Van Slyke with the impact summits we did last spring; a set of documentary filmmakers and funders to think about how to assess impact of longer term projects, and with Knight, which released an impact assessment guide for community info projects earlier this year. There are many juicy questions still to be answered, which I’d love to talk to you about more if you’re interested.

  6. Andrew W says:

    Ethan, for the sake of completing the logical inference, is there evidence you’d cite proving that newspapers under the editor/publisher-wall model were of a consistently better quality? How was success measured in the 20th century? In circulation? Revenues? Pulitzers?

  7. Ethan says:

    Jessica, what I’d really like is for you to come to Civic at some point this fall and talk with all of us about what you’re doing and how we might work together…

  8. Ben says:

    Isn’t that what oneline polls are for?

  9. Sarah Stonbely says:

    From another perspective, news outlets have been trying since the 1980s to pander to what they PERCEIVE to be audience tastes, with the result that news has become filled with crime, scandal, and fluff. However, I would argue that, despite many focus groups and ratings services, editors and journalists still usually only guessed at what they thought would interest their most prominent demographic. I have great optimism that most people, even (or especially) young people, want real, good news (i.e. local politics, relevant culture), and that perhaps these new metrics will lead to a turn away from what has come to stand in for good journalism.

  10. Stephen Lippman says:

    Ehtan,

    Which groups organize,host, and moderate in person community meetings to discuss civic issues raised by media? Newspapers could tie-in to such meetings. Radio and tv talk shows allow some listener contributions but depend mostly on experts or story authors. Perhaps newspapers can put links to organizations on their websites that try to help address the problems that appear in the articles. I may start a related business soon. Keep me posted.

Trackbacks/Pingbacks

  1. Joho the Blog » Berkman Buzz - [...] Ethan Zuckerman considers metrics for the civic impacts of journalism: link [...]
  2. links for 2011-07-02 | Beyond the Echo Chamber - [...] …My heart’s in Accra » Metrics for civic impacts of journalism "Traffic doesn’t seem to be the right measure …
  3. Hvordan måle journalistikkens gennemslagskraft? « Ny Journalistik - [...] Et interessant spørgsmål, der ikke er noget let svar på, som det også påpeges af Ethan Zuckermann, nyudnævnt direktør …
  4. Tensegrities » Blog Archive » How do we measure impact? - [...] intrigued by this reflection of how journalists and news media might measure their impact. Is it simply by traffic? …
  5. Hudin » The future of journalism for now - [...] is information overload, part of it is laziness, and part of it is metrics-driven production which no one really …
  6. » Quantifying impact: A better metric for measuring journalism The Linchpen - [...] Metrics for civic impacts of journalism by Ethan [...]
  7. Papo On - Diário do Nordeste - [...] conteúdo. Lamentavelmente, boa parte das métricas são medidas unidimensionais, que não revelam impacto ao jornalismo. As métricas seriam, infelizmente, …
  8. The billion-dollar question: What is journalism for? — Tech News and Analysis - [...] because there is no easy or reliable way to measure the impact that a printed newspaper has on the …
  9. The billion-dollar question: What is journalism for? | Electronic Staff - [...] don’t, since there is no easy or arguable approach to magnitude a impact that a printed journal has on …
  10. Nuove metriche per misurare l’ impatto del giornalismo | LSDI - [...] come Ethan Zuckerman dice nel suo intelligente post su metriche e impatto pubblico, “misurare quante persone leggono un articolo …
  11. Measuring the impact of journalism « intelligent measurement - [...] Ethan Zuckerman writes about how to measure the civic impact of journalism and one conclusion is: [...]
  12. What Journalists Can Learn from Apple’s Map Mishap « Groundswell - [...] is work intensive, but it is critical. In a post on measuring civic impact for journalists Ethan Zuckerman points …
  13. Come misurare l’ impatto del giornalismo? | LSDI - [...] come Ethan Zuckerman spiega in un suo interessante post su metriche e impatto civile, “misurare quante persone leggono un …
  14. Which News Organizations Influence Wikipedia? | Tow Center for Digital Journalism - […] who are incorporating impact audits into their operations. Before that, in 2011, Ethan Zuckerman asked what the metrics should …
  15. {learning} The Art of Evaluation_#ICM820 | Goals Go Global - […] Metrics for Civic Impact of Journalism […]

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

 

Powered by WordPress | Designed by Elegant Themes