… My heart’s in Accra Ethan Zuckerman’s online home, since 2003

June 30, 2011

Metrics for civic impacts of journalism

Filed under: CFCM,ideas,Media — Ethan @ 1:25 pm

How do news organizations measure impact?

That’s the question I found myself talking with Phil Bronstein of the San Francisco Chronicle earlier this week. He’d gotten in touch to talk about what tools are available to help newspaper editors track audience and reach for their stories, hoping that I’d have some insights on “cutting edge” techniques to track the reach and impact of news stories posted online. We talked a bit about the challenges of social media tracking after the demise of Technorati, the possible benefits of bit.ly-type analytics, questions of influence and reach raised by Klout and similar systems. All well and good, but measuring how many people read a story is something any web administrator should be able to do. Audience doesn’t necessarily equal impact.

There may have been a day in the rosy past of newspapers when a wall between the publisher and the editor meant that newsrooms published only what was most newsworthy and civically important, without consideration of a given story’s appeal to their audience. In an age where editors can know instantly whether a story on a school council meeting is playing better than a story about a labor action, it’s hard to believe that access to analytics doesn’t shape coverage decisions. Some outlets, like the Huffington Post, have embraced this new world to the point where they are poster children for analytics-driven coverage, using feedback from Google Analytics to inform most if not all decisions about story placement and emphasis. This willingness to respond rapidly to market feedback has likely helped HuffPo’s rapid audience and market growth – whether or not AOL’s acquisition of the site was a wise move, most newspaper publishers would welcome ten-figure interest in their properties.

The danger of traffic-based analytics driving journalism is that you may end up with newspapers that look more like Demand Media-style content farms and less like the civic guardians we want and need them to be. It’s certainly fair to observe that newspapers have been audience driven, at least in part, since inception and that some of the shortcomings of contemporary papers, as well as local newscasts, derive from a focus on driving readership and viewership. But adding an analytics into the newsroom puts the question “Is this story reaching a broad audience?” front and center in a way that’s hard to ignore or avoid.

If an ideal editor is making decisions based on what’s newsworthy, and a realistic editor is civic and audience concerns, how do editors determine whether they’re successfully serving both masters? What are appropriate analytics for civic impact?

As Phil and I talked, I found myself thinking about the LA Times’s coverage of obscenely high government salaries in the city of Bell, CA. In depth, investigative reporting by Ruben Vives and Jeff Gottlieb led to fraud trials, a turnover of the city government, and ultimately to a Pulitzer for the pair of reporters. The reporting on Bell, CA suggests two ways newspapers might measure civic impact: the arrest of bad guys, and the praise of one’s peers and professional societies. But these aren’t exactly quick metrics, and not every worthwhile piece of civic journalism has this magnitude of impact.

Traffic doesn’t seem to be the right measure of civic impact. A story that gets lots of page views or is widely shared might be civically relevant, but might also be salacious – amusing and popular as much of the Anthony Weiner coverage has been, I’m not sure it’s been a positive contributor to our civic involvement. Phil suggested that comments aren’t an adequate metric either. Stories that garner long comment threads could suggest broad involvement, but also may suggest partisan controversy.

I mentioned an idea that I’ve been trying to pitch for a while: in an age of participatory media, news demands participation. Or to quote Benjamin Barber, “People are apathetic because they are powerless, not powerless because they are apathetic.” For people to pay attention to an important story, it’s possible that we need to work to make it possible for people to have an impact on the outcome of the story.

Ideally, we can find better ways to do this than turning our Twitter icons green in solidarity with Iranian activists. Reporting on local civic issues offers the possibility of connecting people to opportunities for action in their own communities. And if newspaper web sites start trying to broker these connections, we gain another possible metric – the efficacy of a story in connecting people to community organizations, volunteering opportunities, and other forms of civic engagement.

That’s not a comfortable role for newspapers to take, Phil reminds me – it smacks of advocacy journalism. But the Bell, CA story is another form of activist journalism: by relentlessly shining a light on political malfeasance, Vives and Gottlieb were demanding that someone take action against these corrupt officials. Eventually, both citizens and prosecutors did. The difference between what I’m proposing and what the Pulitzer winning reporters did is that I’m suggesting newspapers link to possible solutions and measure how effective at driving engagement they are.

This would be far from a perfect metric. It wouldn’t tell you how many people read a story on homelessness, and then sought out community organizations on their own to volunteer with… though adding a feedback cycle where local organizations could communicate changes in community involvement to newspapers might. And it wouldn’t track one of the most critical functions of investigate journalism: the fear it generates in politicians and corporate actors that they could end up on the front page of a newspaper if they break the law. Clay Shirky is worried that losing this deterrence effect is one of the dangers of losing “accountability journalism” in the transition from broadcast, offline models of journalism to participatory, digital ones.

My point is not that I’ve got good metrics for civic engagement for newspaper journalism… or any journalism. It’s that we need to be thinking about finding and developing them. What we measure, we become. If we measure only how many people view, like or tweet, but not how many people learn more, act or engage, we run the risk of serving only the market and forsaking our civic responsibilities, whether we’re editing a newspaper or writing a blog.

25 Comments

  1. Hi–this is something I’ve been really interested in (first as a reporter and now as a student), so I’d love to know more about ideas for how to measure this. This could be done through a social audit-type approach… few news organizations do social audits now, and most audits don’t cover this kind of thing.

    Comment by Kate Fink — June 30, 2011 @ 2:01 pm

  2. Would love to hear more about social audits, Kate – how would you suggest a paper go about doing that? Probably isn’t going to address my hope for near-realtime metrics, but sounds like a really worthwhile addition to what newspapers should be doing to measure impact.

    Comment by Ethan — June 30, 2011 @ 3:00 pm

  3. Nice vet of the challenge, Ethan. I was really hoping you had solved this dilemma! In public broadcasting, where social return on investment (SROI) has been eluding us for decades, we continue to satisfy ourselves that audience = public service. (And the more loyal, the more they support the effort.) But a) this borrows from the commercial ratings marketplace, b) it doesn’t give us that better target of journalistic impact to aim for, and c) we can get big ratings without any local news effort (just play NPR national programming) which begs the question of why local stations should invest in local news (or exist in the future)? I recently made a list of some 15 quantifiable outcomes of our work… none resembling the better bulls-eye… but maybe we need *a host of metrics* to get at something as difficult as SROI.

    Comment by Michael Marcotte — June 30, 2011 @ 8:35 pm

  4. Building on the Guardian’s social audit model is one option–they address editorial but could do so much more to evaluate impact (still, the fact that they address editorial at all seems to put them miles ahead of most news organizations).
    I would like to see newsrooms adopt a strategic planning process–set goals that align with their missions, formulate concrete plans to help them achieve those goals, and periodically evaluate their progress. If they don’t take the time to figure out how to measure impact, they’ll just fall back to what’s easy to measure–like traffic.

    Comment by Kate Fink — July 1, 2011 @ 9:53 am

  5. Hi, Ethan:

    I’ve been pursuing these questions for a few years now with different intellectual partners—Tracy Van Slyke with the impact summits we did last spring; a set of documentary filmmakers and funders to think about how to assess impact of longer term projects, and with Knight, which released an impact assessment guide for community info projects earlier this year. There are many juicy questions still to be answered, which I’d love to talk to you about more if you’re interested.

    Comment by Jessica Clark — July 1, 2011 @ 12:58 pm

  6. Ethan, for the sake of completing the logical inference, is there evidence you’d cite proving that newspapers under the editor/publisher-wall model were of a consistently better quality? How was success measured in the 20th century? In circulation? Revenues? Pulitzers?

    Comment by Andrew W — July 1, 2011 @ 5:39 pm

  7. […] Ethan Zuckerman considers metrics for the civic impacts of journalism: link […]

    Pingback by Joho the Blog » Berkman Buzz — July 1, 2011 @ 6:38 pm

  8. Jessica, what I’d really like is for you to come to Civic at some point this fall and talk with all of us about what you’re doing and how we might work together…

    Comment by Ethan — July 1, 2011 @ 6:38 pm

  9. […] …My heart’s in Accra » Metrics for civic impacts of journalism "Traffic doesn’t seem to be the right measure of civic impact. A story that gets lots of page views or is widely shared might be civically relevant, but might also be salacious – amusing and popular as much of the Anthony Weiner coverage has been, I’m not sure it’s been a positive contributor to our civic involvement. Phil suggested that comments aren’t an adequate metric either. Stories that garner long comment threads could suggest broad involvement, but also may suggest partisan controversy. I mentioned an idea that I’ve been trying to pitch for a while: in an age of participatory media, news demands participation. Or to quote Benjamin Barber, “People are apathetic because they are powerless, not powerless because they are apathetic.” For people to pay attention to an important story, it’s possible that we need to work to make it possible for people to have an impact on the outcome of the story." […]

    Pingback by links for 2011-07-02 | Beyond the Echo Chamber — July 2, 2011 @ 10:02 am

  10. Isn’t that what oneline polls are for?

    Comment by Ben — July 2, 2011 @ 11:26 am

  11. From another perspective, news outlets have been trying since the 1980s to pander to what they PERCEIVE to be audience tastes, with the result that news has become filled with crime, scandal, and fluff. However, I would argue that, despite many focus groups and ratings services, editors and journalists still usually only guessed at what they thought would interest their most prominent demographic. I have great optimism that most people, even (or especially) young people, want real, good news (i.e. local politics, relevant culture), and that perhaps these new metrics will lead to a turn away from what has come to stand in for good journalism.

    Comment by Sarah Stonbely — July 5, 2011 @ 11:34 am

  12. Ehtan,

    Which groups organize,host, and moderate in person community meetings to discuss civic issues raised by media? Newspapers could tie-in to such meetings. Radio and tv talk shows allow some listener contributions but depend mostly on experts or story authors. Perhaps newspapers can put links to organizations on their websites that try to help address the problems that appear in the articles. I may start a related business soon. Keep me posted.

    Comment by Stephen Lippman — July 6, 2011 @ 7:35 am

  13. […] Et interessant spørgsmål, der ikke er noget let svar på, som det også påpeges af Ethan Zuckermann, nyudnævnt direktør for Center for Civic Media i dette blog-indlæg. […]

    Pingback by Hvordan måle journalistikkens gennemslagskraft? « Ny Journalistik — July 8, 2011 @ 3:23 am

  14. […] intrigued by this reflection of how journalists and news media might measure their impact. Is it simply by traffic? Or might […]

    Pingback by Tensegrities » Blog Archive » How do we measure impact? — July 9, 2011 @ 1:10 pm

  15. […] is information overload, part of it is laziness, and part of it is metrics-driven production which no one really understands […]

    Pingback by Hudin » The future of journalism for now — July 22, 2011 @ 3:40 pm

  16. […] Metrics for civic impacts of journalism by Ethan […]

    Pingback by » Quantifying impact: A better metric for measuring journalism The Linchpen — January 29, 2012 @ 6:03 am

  17. […] conteúdo. Lamentavelmente, boa parte das métricas são medidas unidimensionais, que não revelam impacto ao jornalismo. As métricas seriam, infelizmente, voltadas para o que os anunciantes […]

    Pingback by Papo On - Diário do Nordeste — May 3, 2012 @ 1:22 pm

  18. […] because there is no easy or reliable way to measure the impact that a printed newspaper has on the society around it, apart from pure sales. As Stray puts it: “Evaluating the impact of journalism is a […]

    Pingback by The billion-dollar question: What is journalism for? — Tech News and Analysis — August 21, 2012 @ 12:47 pm

  19. […] don’t, since there is no easy or arguable approach to magnitude a impact that a printed journal has on a multitude around it, detached from pristine sales. As Stray puts it: “Evaluating a impact of broadcasting is a […]

    Pingback by The billion-dollar question: What is journalism for? | Electronic Staff — August 22, 2012 @ 2:15 am

  20. […] come Ethan Zuckerman dice nel suo intelligente post su metriche e impatto pubblico, “misurare quante persone leggono un articolo è qualcosa che ogni amministratore di siti web […]

    Pingback by Nuove metriche per misurare l’ impatto del giornalismo | LSDI — August 24, 2012 @ 3:56 am

  21. […] Ethan Zuckerman writes about how to measure the civic impact of journalism and one conclusion is: […]

    Pingback by Measuring the impact of journalism « intelligent measurement — September 3, 2012 @ 2:48 pm

  22. […] is work intensive, but it is critical. In a post on measuring civic impact for journalists Ethan Zuckerman points out, “What we measure, we become. If we measure only how many people view, […]

    Pingback by What Journalists Can Learn from Apple’s Map Mishap « Groundswell — September 28, 2012 @ 9:13 am

  23. […] come Ethan Zuckerman spiega in un suo interessante post su metriche e impatto civile, “misurare quante persone leggono un articolo è qualcosa che ogni amministratore di rete […]

    Pingback by Come misurare l’ impatto del giornalismo? | LSDI — January 4, 2013 @ 3:51 am

  24. […] who are incorporating impact audits into their operations. Before that, in 2011, Ethan Zuckerman asked what the metrics should be for civic journalism. Jonathon Stray’s 2012 article in The Neiman Lab asked what the right […]

    Pingback by Which News Organizations Influence Wikipedia? | Tow Center for Digital Journalism — January 17, 2014 @ 4:01 pm

  25. […] Metrics for Civic Impact of Journalism […]

    Pingback by {learning} The Art of Evaluation_#ICM820 | Goals Go Global — October 20, 2014 @ 9:52 am

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

Powered by WordPress