Ethan Zuckerman’s online home, since 2003

You go to war with the data you have

Academic research, at its best, is about asking interesting questions and then designing experiments to answer them. One of the toughest challenges is designing experiments that are both possible and capable of answering the question at hand. The gap between the set of possible experiments and convincing experiments can sometimes be a vast one.

Just like the former US secretary of defense, you go to publication with the data you’ve got, not the data you’d like to have. And different data can lead to some very different conclusions, which helps explain why two competent groups of researchers can answer a question very differently.

I’m interested in questions of whether the internet, and specifically the rise of the read/write web, is broadening or narrowing the perspective of users. On the one hand, the rise of pervasive read/write media means that lots and lots of people are creating content – tens of millions by some counts – which should make a great wealth of perspectives and viewpoints available to the internet user. On the other hand, the structure of digital media means that it’s very easy to select just the media you’re interested in… a scenario Nicholas Negroponte termed “the Daily Me“.

Negroponte thought the Daily Me was an exciting thing. Cass Sunstein thought it was worrisome and wrote a book (twice!) about the reasons it troubled him. An academic debate continues to rage, pitting Sunstein the skeptic against a long list of cyberenthusiasts, including Yochai Benkler and Henry Jenkins.

This is the sort of question social scientists love to explore: thorny, complicated, but ultimately testable. All we’ve got to do is look at some people who spend a lot of time online and others who spend little and see who’s got a broader view of the world. Piece of cake. (The < SARCASM > tag may not be rendering properly in your browser.)

In October 2004, John Horrigan, Paul Resnick and Kelly Garrett released a paper called “The internet and democratic debate” as part of the Pew Internet and American Life project. They didn’t lack for confidence in their conclusions, subtitling the paper, “Wired Americans hear more points of view about candidates and key issues than other citizens. They are not using the internet to screen out ideas with which they disagree.”

The first sentence of the subtitle is experimentally true – that’s what their research demonstrates. The University of Michigan School of Information and Pew IALP conducted a study which surveyed 1500 Americans, some who used the internet to follow the news, some who didn’t. They asked participants whether they’d heard certain political arguments, in favor or against particular political candidates or sensitive issues. Out of a set of of eight arguments (four for a candidate or an issue, four against), broadband internet users had heard 5.5 arguments, while respondents as a whole had heard 5.2 and non-internet users had heard 5.0.

The tougher question is what significance we should attach to this result. The study sees a slightly larger gap in knowledge between people with strong political opinions and the average respondent (5.6 for strong Kerry supporters, 5.7 for strong Bush supporters) – perhaps people with strong political opinions get broadband so they can track politics more closely? Maybe rich people have more leisure time to hear political arguments and more money to buy broadband connectivity? It’s difficult to determine which variables are correlated without doing regression analysis, which attempts to isolate the inpact of each variable.

The Pew report promises that regression analysis has been done and that “Internet use did have an indepedent and positive effect on the number of statements people heard about the candidates.” Unfortunately, it doesn’t include the analysis, and it’s not clear whether internet use is a major or small factor, or whether other factors are better predictors of having heard a large set of arguments.

More difficult for me to swallow is the connection between awareness of political arguments and diversity of viewpoint. I don’t consume a lot of right-leaning US media (to my detrminent – I’d likely be a better informed citizen if I did), and I’d heard all the arguments on the left and right of the issues presented. I’d be willing to guess that I’d heard many of the “right” arguments from “left” media, sometimes in posts that began, “The right is trying to sell argument A – here’s why it’s wrong.” I’m tempted to conclude that Horrigan et al. found a correlation between internet usage and a more-informed citizenry – a correlation that may not be causal, as one can imagine people with a strong interest in being well-informed might seek out broadband internet, or might be wealthier and likelier to have broadband.

But concluding that internet users “are not using the internet to screen out ideas with which they disagree” seems like it’s blurring an important distinction between knowing the other side’s arguments, and having listened to someone make a persuasive case for them. But the distinction quickly points to the difficulty of determining how one would measure a broadened perspective. Is it knowledge of other opinions? A softening of one’s political stances to acommodate other positions? Are we looking for knowledge, for sympathy, or for some sort of change to demonstrate an ideological diversification that comes from online media?

Henry Farrell, Eric Lawrence and John Sides are intrigued by the same question explored by Horrigan and friends. They approach the topic in a paper titled “Self-Segregation or Deliberation? Blog Readership, Participation in American Politics“. The title invokes a related set of political science arguments – Habermas and others have argued that societies depend on a healthy public sphere where issues can be freely deliberated. Sunstein’s recent research on this topic focuses specifically on deliberation, where he makes the argument that deliberation with like-minded fellows can lead to increased political polarization. (My objections to the Pew study might be summarized by saying “knowing the other side’s key arguments isn’t the same thing as good-faith deliberation.”)

Habermas believes that deliberation is an essential ingredient in a healthy society. Sunstein worries that polarized deliberation can make society worse, and fears that the internet enables this sort of polarization. Farrell abd friends point to another complication – citing Diana Mutz, they see evidence that deliberation with people who hold other opinions can lead to increased tolerance but to decreased political participation.

To test whether internet users are more polarized than non-users and how this might affect their participation, Farrell and friends look at a different set of data, the 2006 Cooperative Congressional Election Study. This is a much larger sample set than the Pew data, covering 36,501 participants, but since the researchers didn’t control the questions asked (they were proposed by a large set of researchers) they’re forced to mine the data they have. 5,481 respondents reported reading blogs, and 3,948 listed one or more blog they read. The researchers coded the blogs and discovered that 2,312 respondents reported reading one of 476 political blogs.

Using this data, Farrell and crew classified blog readers as “carnivores” or “omnivores”. Omnivores consumed both left and right leaning blogs, while carnivores consumed only right or left. There are a lot of carnivores – 94%, according to their analysis. Unsurprisingly, folks who read a lot of left-leaning blogs tend to be politically left, and vice versa across the ideological divide. The researchers see a similar pattern in television news consumption – viewers of Fox News tend to be from the right, while viewers of other networks skew left.

This polarization in consumption appears to be correlated to political polarization. The carnivores are not only more polarized than the average citizen, but roughly as polarized as US senators, folks whose political success often depends on their strong, steady party alleigance. (The CCES political opinion data is designed to be comparable to the NOMINATE data collected on senatorial votes.)

The small group of omnivores – 6% of those who report reading political blogs – don’t seem to face the demobilization Mutz warns of. All the blog readers participate at a higher level than the average citizen, other than right-wing carnivores, who participate at roughly the same level as the average respondent.

Farrell and his collaborators conclude with some confidence that blog readers are significantly more politically involved and polarized than the average citizen, a result that might seem to be at odds with the Pew results. After all, Pew’s study concludes that wired Americans “are not using the internet to screen out ideas with which they disagree.” But if we ignore Pew’s interpretation and just focus on their data, the studies are more compatible. Both Pew and Farrell see internet users as better informed than average citizens. Farrell would likely tell you that some of these well-informed internet users are carnivorous blog readers, who are knowledgeable about political arguments through their voracious consumption of ideologically-compatible blogs.

My interest in these experiments has less to do with questions of political polarization and more to do with interest in international news. Are internet readers more inclined to look for information about other countries, since they’ve got such a wealth of information at their fingertips? Or are they more inclined towards information on their home countries, since they can easily choose to avoid international news. Extrapolating from Pew’s data suggests that wired readers might consult more sources and perhaps consume a more diverse diet; Farrell’s research points to a strong homophily effect, which suggests the possibility of geographic cocooning.

Guess I’ll need to design my own experiments using whatever data I can as a proxy to indirectly answer the question… and hope other researchers find other data and other methods to challenge my assumptions.

3 Responses to “You go to war with the data you have”

  1. Tino says:

    Great post!! Does the internet forge a larger consumption of international news stories? Good question. But as you point out, the causal relationship between interest and available sources has never been proven. What Horrigan and Farrell have shown is merely that internet users access more news sources, even if it’s not necessarily a broader political perspective they gain.

    As Wu (e.g. keeps showing us, the other factor is what international news stories you _can_ get, even from a plethora of sources: if LAT and NYT and CNN give the exact same story on a Chadian rebel group (because they all use the same wire AP report), users will probably be less inclined to use more than one source for international news. I think in the Pew study you can see that Americans still prefer the big networks and portals for news, and it’s still very commonplace for (left-leaning) blogs to use the Times and other biggies to source their information.

    So, as international stories go, how much of a diverse story _can_ you get? (yes, GV is the beautiful solution, but unfortunately just not that popular yet…)

    I’m working on a similar research design with South African students, focusing on their mobile media consumption (the cell phone is the most widely used internet technology by this point). We’re doing a mix between quantitative surveys and experimental, almost ethnographic methods. I’ll send you the results.

  2. Thanks for the insightful post. As one of the authors of the Pew study, and as a researcher actively studying the questions you discuss here, I want to offer a couple additional thoughts.
    First, I wholeheartedly agree with your assertion that the Pew results and the results of Farrell and colleagues are compatible. I think the similarity may be a bit deeper than you have suggested. In their claims about how people seek political information online, both groups of researchers suggest that people prefer opinion reinforcement to opinion challenge. Where our interpretations differ is in how to explain this preference. One common explanation is that people seek opinion reinforcement while actively avoiding opinion challenges. For instance, this appears to be a key element of Sunstein’s “echo chamber.” The problem with this explanation is that there is little evidence that people want to avoid other viewpoints. Yes, people prefer information sources they agree with, but the Pew study (and others before it) suggests that people don’t put much effort into avoiding contact with other perspectives. For example, if online news users really want to avoid other viewpoints, they should abandon the comparatively balanced mainstream media for more partisan alternatives, but as the Pew report indicates, that’s not what people are doing.
    In your post, you also point out that the Pew report doesn’t provide much detail about the regression analyses. If you’re interested, you can see a more complete analysis in this paper: The linked paper also describes results of an experiment examining how online news readers choose among diverse news stories about contentious political issues. The bottom line is that individuals’ issue positions do influence their decisions about what they look at, but the effect is small and people are drawn to opinion reinforcement more than they are repelled by opinion challenges.
    You also point out that the Pew results don’t tell us about the context in which people encounter arguments that differ from their own. Perhaps they learned about the other side on a partisan blog favoring their position. Even when this is the case, these results are important. One concern stemming from Sunstein’s echo chambers is that people will become less aware of other viewpoints and that this will lead them to be more extreme, less accepting of others views, and potentially more violent. People who are familiar with both sides of a debate are more tolerant than those who only know arguments favoring their viewpoint. So exposure to criticism of your political beliefs is an important element of effective democracy. To be sure, you are correct when you observe that hearing the other side’s arguments is not the same as giving them your unbiased consideration, but both activities are important.

  3. Cos says:

    When I first heard and read about Robert Putnam’s Bowling Alone, I felt there was something wrong with his thesis that social capital in America was declining. I remember hearing him interviewed on the radio, talking about declining participation in neighborhood groups, bowling leagues, and so on, and thinking “but that’s not how my social capital works.”

    How many times this week, he asked, have you held a dinner party, attended a civic group meeting… well, I actually do attend monthly meetings of my neighborhood group sometimes, but my thought was, “he didn’t ask how many times a *day* I’ve commented on the LiveJournal of someone I know in person and see regularly – let alone how many times I’ve exchanged comments with one of *their* friends”.

    A lot of this clicked together for me when I read Richard Florida’s The Rise of the Creative Class a couple of years ago. Florida argues, persuasively, that the kinds of social capital Putnam measured are in fact *barriers* in today’s world. Those older forms of social capital are stable and tightly-knit, and make it harder for people to move around and integrate into new places and groups of people. A more powerful form of social capital today, he argues, as a more networked kind of social capital full of shifting masses of loose connections – just like LiveJournal, I thought! Cities & regions that have a more networked style of social capital are more welcoming to newcomers and diversity, according to Florida, and better able to attract & integrate talent, and therefore outperform cities & regions where the old kind of tightly-knit, stable social capital still dominates.

    The reason I write all this here is that I suspect we’re liable to make a similar mistake when looking at whether the Internet is broadening or narrowing people’s perspectives, whether homophily or diversity of diet are stronger influences. We may be tempted to look at the broad vs. narrow divide we think we recognize, and in doing so, ignore ways in which breadth & narrowness of perspective is *different* in the Internet age, rather than more or less towards one side of the existing continuum. We may, as Robert Putnam did, be asking the wrong questions.


  1. links for 2008-08-22 « andrew golis - [...] …My heart’s in Accra » You go to war with the data you have Ethan Zuckerman on homophily/polarization research. …
  2. Science Blogs » Blog Archive » Obligatory Readings of the Day: Science Outreach and Online Behavior - [...] You go to war with the data you have: My interest in these experiments has less to do with …
  3. …My heart’s in Accra » The Partisan Internet and the Wider World - [...] evidence that internet users had broader exposure to political arguments than non-users. (I offer my critique of the Horrigan, …
  4. …My heart’s in Accra » You go to war with the data you have | Lessons from the Net | - [...] A Marketplace of Echoes?   "The Partisan Internet and the Wider World."  [...]

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Powered by WordPress | Designed by Elegant Themes