Ethan Zuckerman’s online home, since 2003

How a Pakistani ISP briefly shut down YouTube

One of the inspirations for Global Voices Advocacy, our online free speech project, was the good work being done by Don’t Block the Blog, a project led by two Pakistani bloggers to protest that nation’s block of Google’s Blogger platform. Don’t Block the Blog has been a real innovator in the anti-censorship space, and has done some excellent work reaching across borders, including the border with India, where they provided scripts to allow India’s bloggers – who were briefly blocked by their government – to reach their blogs. This sort of cross-border collaboration is what Global Voices is all about, and we hope to encourage more collaboration when some of these anti-censorship activists meet face to face in Budapest this June.

Today I’m leaning heavily on the Don’t Block the Blog guys to understand Pakistan’s attempts to block YouTube, and the widespread consequences of that decision. Sami ben Gharbia has a copy of the block order posted at GV Advocacy. It orders Pakistani ISPs to block a single URL – http://www.youtube.com/watch?v=o3s8jtvvg00 – as well as three IP addresses. The URL now leads to a YouTube page that declares “This video has been removed due to terms of use violation.” But many Pakistani users are discovering that YouTube, as a whole, is inaccessible as their ISPs are blocking access to the IP addresses associated with the site.

There are very few Pakistani ISPs sophisticated enough to implement the block of a specific URL – the Don’t Block the Blog guys note that ISPs using the TWA-1 cable rather than the FLAG cable to connect to the global internet can access all of YouTube, just not the specific URL ordered to be blocked. Most ISPs followed the block order much more closely, blocking the IP addresses and preventing Pakistani users from accessing any content on YouTube, not just the offending video. (There’s speculation that some ISPs chose to block widely because there are new videos on YouTube alleging fraud in recent parliamentary elections.)

While it’s always a drag when countries overblock, taking offline whole services instead of offending content, it’s become increasingly common. Sami’s Access Denied Map shows blocks – current or past – of YouTube in Turkey, China, UAE, Iran and other nations. What made the net sit up and take notice was that a Pakistani ISP managed to take YouTube off the net as a whole for a brief period of time.

Here’s how it happened, more or less:

There are at least three ways to block internet traffic to the customers of an ISP. The most exact way is to block specific URLs, like the URL specified by the Pakistan Telecommunications Authority. There’s two problems with this, from the perspective of the blocking ISP. It’s expensive – every request that goes over your network needs to be examined, not just at the header level, but within the packet to check for the offending URL. And it’s imperfect – tell the system to block “http://www.youtube.com/watch?v=o3s8jtvvg00″ and it will fail to block requests for “http://youtube.com/?v=o3s8jtvvg00″, which will get you the same video.

A second way to block traffic is wicked cheap, but extremely ineffective – change the Domain Name System (DNS) records. People using that DNS suddenly discover that youtube.com no longer resolves to 208.65.153.238 but to an IP address controlled by the ISP – you can then put up a page saying, “YouTube is blocked by order of the Pakistan Telecom Authority.” The problem with this method is that it’s trivially easy to defeat – users can simply type in the correct IP address for the site, or switch to using an “unpolluted” DNS server like OpenDNS.

A third way involves blocking IP addresses. It’s far cheaper than blocking URLs, as the IP address is in the header of every packet, and all routers can be easily programmed to block IPs (it’s a routine way to fight denial of service attacks.) It’s effective – users need to use a proxy to get around it – though the major downside is that it’s overbroad. If you’re trying to block a single offensive page on Tripod, block the IP and you block all the millions of pages hosted there.

PieNet, a Pakistani ISP decided to block YouTube using the IP block method. (It’s unclear to me from these accounts if PieNet is part of Pakistan Telecom or not – I’d appreciate insights from my readers if you know.) They also wanted to implement a blockpage. Rather than tell their routers to block all traffic to 208.65.153.238 and redirect any requests to a blockpage, they decided to tell their users that they controlled all IP addresses that began 208.65.153.* – when a user requested YouTube, the DNS server would tell them that the site was at 208.65.153.238. Their browser would request that page, and they’d be redirected to a PieNet server, which would tell them they were blocked.

So far, so good. But PieNet’s rewiring of the net was propogated by their upstream internet service provider. PieNet accomplished its rewiring using Border Gateway Protocol, the internet’s core routing protocol. BGP is designed to allow large network operators (called autonomous systems, or AS’s) to announce paths to certain destinations on the internet. BGP is designed so that router prefer the most specific route possible. Google had announced a (correct) route that covered a fairly large block of IP addresses, including YouTube’s – PieNet announced a much more specific route, covering only 256 addresses. So routers that “trust” AS17557 – Pakistan Telecom – began telling their neighbors about a great new way to reach YouTube… which was now hosted by PieNet.

As a substantial portion of the world’s YouTube traffic attempted to make requests on PieNet’s servers, they went offline. In the meantime, Google released the correct routing information with more specific routing info, and most routers were looking at the correct address within two or three hours. To prevent Pakistan Telecom from propogating the bad address, its upstream provider, the Hong Kong based PCCW, removed Pakistan Telecom from the internet until they were able to fix its servers.

This sort of misrouting isn’t uncommon on the net – old net hands have been citing the AS7007 disaster, where a misconfigured router advertised that all Internet routes could be reached through it… and a good chunk of the Internet believed it and attempted to route itself through a single, overwhelmed router.

There’s at least two, and possibly three, things that went wrong in this case:

- Modifying routing records via BGP is probably a bad way to censor the net. (I’ll wait for confirmation from smarter people like Nart Villeneuve before removing that probably.) There are lots better ways to block IPs than to claim IPs that aren’t yours via BGP.

- Pakistan Telecom should not have advertised this route to other routers – they should have checked that PieNet was authorized to route this block of IP addresses before passing it on to the rest of the network. And PCCW should not have accepted Pakistan Telecom’s new route, as Pakistan Telecom didn’t have authority over these nunbers.

- Depending on who you ask, BGP is too trusting a network protocol. Back when the net was used mostly by academics, lots of systems simply trusted each other – you could send mail to another mailserver and ask it to deliver it for you. That doesn’t work so well in the age of commercial spam, and so network operators don’t do it anymore. The idea that one network can simply tell another that it’s got a route to certain IPs scares some operators, who see this as a system that would be very vulnerable to attack. There’s discussion on some blogs, mailing lists and message boards that this incident will get network operators to start using more secure forms of BGP that check carefully to make sure that another network is who it says it is, and that it has authority over the addresses it wants to route, before accepting its instructions.

Interesting to me has been the tone of discussions around the incident. Many network operators believed the incident was an accident “on the part of probably stressed NOC staff at 17557.” Others concluded that “it was almost certainly an intentional hijacking designed to make a political statement. A bogus BGP advertisement is a very loud and rude way to make such a statement.”

NANOG, the North American Network Operators Group, has a long and intriguing thread, which includes technical discussion on fixing the problem, speculation on how to prevent such accidents/attacks in the future… and political arguments about the Islamic world. At least one network operator threatened to block all routing changes from 17557 – Pakistan Telecom – while others pointed out that, even on “a sleepy Sunday”, it took less than two hours to fix this problem. In other words, even if a network operator wanted to “attack” the rest of the network by propogating bad IPs, they’d likely be shut down pretty quickly.

I’m always interested when realworld and network politics intersect. While issues like this take some effort to understand, the solutions offered and adopted on lists like NANOG can have profound implications for how the Internet works for everyone. It bothers me a great deal that there’s routine discussions on some forums on how to block entire nations from accessing a website. I’d be very disappointed to discover that some network operators began blocking routing information from Pakistan based on what, I suspect, was a mistake… but I wouldn’t be surprised.

At its heart, the Internet is based on trust. That trust has been badly abused over the past decade, and the net as we know it today is a patchwork of trust and verification. I don’t know who you are, but you’re able to read this blog entry from (pretty much) any IP address in the world. But when you go to comment, I try to verify that you’re a human being using a CAPTCHA… and Akismet tries to verify that you’re not a spammer by checking your comments against other comments identified as spam. This doesn’t always work – I end up blocking more comments than I mean to, but with over a million attempts to post spam on this blog in the last 18 months, running comments without verification would be disastrous.

What worries me is when we move from verification strategies to strategies that simply deny access based on preconceptions about national identity and character. Yes, it sucks that there’s a lot of spam and fraud originating from Nigeria. Yes, it’s a drag that Pakistan decided to censor YouTube. But it would be truly, truly stupid for network operators to stop accepting certain types of traffic from these countries due to laziness and prejudice.


Lots more information on the situation and the tech behind it at Data Center Knowledge and at Renesys. Apologies in advance for anything I’ve gotten wrong in trying to offer a readable gloss on what’s a very complex technical issue.

14 Responses to “How a Pakistani ISP briefly shut down YouTube”

  1. Thanks for the elaborate explanations Ethan. I’m still surprised though, that a site as huge and global as YouTube backed by the mighty Google, can suffer down time inflicted by some distant ISP. I guess mighty Google isn’t as mighty as we thought it is.

  2. Ethan says:

    Drima, it’s not realy Google that’s vulnerable, but any website, no matter how big or small. That’s what’s a bit scary about this attack. It won’t work for very long against a big and sophisticated site like YouTube, and it will take your routers down in the process… but it might be an effective attack against a small and unsophisticated site… though only if you have control of an autonomous system (AS).

  3. Cd123456789 says:

    This went beyond my technical expertise. When I first heard the story, my reaction was ee gad, what stops our own government from being able to censor information to US?

  4. tom says:

    Ethan, Thanks for bringing attention to this. I’m in Turkey, and we have been frustrated here by the ban on YouTube (twice in a year)because of offensive material published *outside* of Turkey which Google/YouTube refused to remove. The offending material was removed.

    WordPress has been banned here since August 2007 because of the abuse by two litigious users hurling insults at each other. It seems that people are figuring out work-arounds for that though, as I’m finding more blogs coming back from the dead (and they always showed up in my feed reader anyway).

  5. tom says:

    He he, I just noticed that you’re using WordPress and you got through! The ban now works mostly against urls that have the string “wordpress” in the url.

  6. billso says:

    Ethan, thanks for the mention in your article! As I pointed out in my blog, protecting against BGP errors and attacks can be expensive. The router upgrade costs would be passed along to companies, users, and customers.

  7. Why do I get the sinister feeling that this is just censorship?

  8. John Smith says:

    I’m using VPN tunneling by http://strongvpn.com This service really helps me in that idiotic government policy… It gives the American IP without any laggs

  9. Slarty says:

    The sad and enraging point is that securing bgp against this type of attacks is quite simpel. Simply don’t accept unchecked new routes from your customer ISPs. I’m practicing this since my company is providing IP transit for customers – it’s working and definitly worth the little extra effort. As with ddos attacks and spoofing, this has to be prevented at the source.

  10. IP says:

    Blocking such websites now are trend in internet world. Many countries block dangerous sites.

  11. aamir says:

    so you think youtube is dangerous website as well??? lol

  12. Kelyn says:

    How do you reverse other peoples’ blocks?

  13. EpiQ says:

    “Reverse”? as in your government blocking you and then you turning it around and unblocking yourself while simultaneously blocking your government?…

    Sorry if i was way off…

  14. Jeff says:

    Censorship will always remain so long as the government who controls or influence the media to be censored fear or repulses the “offending” content.

    On the same note, truth will always prevail so long as the one in search of it risk himself/herself to find that truth among the lies and deceit.

Trackbacks/Pingbacks

  1. netzpolitik.org: » Wie Pakistan Youtube ausschaltete » Aktuelle Berichterstattung rund um die politischen Themen der Informationsgesellschaft. - [...] in seinem Blog, wie die Youtube-Sperre in Pakistan dafür sorgte, dass Youtube global ausfiel: How a Pakistani ISP briefly …
  2. How Pakistan Hijacked YouTube : Glen Bowes - [...] http://www.ethanzuckerman.com/blog/2008/02/25/how-a-pakistani-isp-briefly-shut-down-youtube [...]
  3. Bits und so #78 (Das Enterprise-Paradox) | Bits und so - [...] Youtube kaputt - Pakistan schuld - Hintergrundinfos [...]
  4. …My heart’s in Accra » The Cute Cat Theory Talk at ETech - [...] research topic. One answer is that it’s surprisingly difficult to censor the web well. (Pakistan’s recent shutdown of YouTube …
  5. Hacktivism: A Path to Exploitation | Issues in Cyberspace - [...] Detailed information about blocking internet traffic: (http://www.ethanzuckerman.com/blog/2008/02/25/how-a-pakistani-isp-briefly-shut-down-youtube/) [...]

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

 

Powered by WordPress | Designed by Elegant Themes