See my previous post from November 2008. Oh, and if you plan to write your own letter to somebody about this matter, feel free to use my references, but please write your own content. I doubt it’s effective to spam busy politicians with duplicates that waste their time.

Firstly, I resent Stephen Conroy’s repeated assertion that those who oppose his plan to filter the internet are interested in “opting in to child porn” [1]. There have been many other occasions where Stephen Conroy has made such remarks. For example: “If people equate freedom of speech with watching child pornography, then the Rudd-Labor Government is going to disagree.” [2]. Nobody is trying to suggest that child pornography is acceptable, these kind of statements are inflammatory, and framing the debate in such a way is offensive and unhelpful.

It is also disturbing to see him outright mislead the public in so many respects, just one example was claiming that most ISPs support his filter plan [3], both iiNet[4] and Telstra[5] have been on record in the past as saying that the filter plan is a bad idea, so this was an outright falsehood that he definitely should have been aware of the facts before making such a statement. This is particularly irritating because he on so many occasions accuses others of misleading the public.

For the record, the rest of the industry is also against the plan, including Google, Yahoo, the US government [6] and even the “Save the Children” organisation [7]. There are good reasons why there is such widespread disapproval, which I will now cover.

The filter plan seems to have no concrete goals:

• Clearly it cannot protect those children who are actually suffering child abuse. Hiding the problem does not make it go away.
• Will it protect ordinary law abiding citizens browsing the internet at home? Considering the goal is to have a blacklist of less than 10,000 of the 1,000,000,000,000+ websites on the internet, I think it’s easy to see that it cannot possibly hope to make much of a difference to the chance of you coming across a website that may offend you. If Australians were interested in such things, they already have the option of signing up to an ISP like WebShield that offers this service at a small premium to anybody who can get ADSL through the Telstra wholesale network (that’s everybody who can get ADSL)
• It will not stop people trading this content via email, peer 2 peer file sharing networks, instant messaging chat software and private encrypted websites, or using virtual private networks. This means it will be trivial for anybody who desires to bypass the filter. The report that Stephen Conroy published on the filter testing acknowledged this.

So the filter will not help abused children, protect casual internet users, or reduce criminal activity. What exactly will it do? Here are some negatives that it will do:

• Add cost to ISPs, that will be passed on to all users
• Become an extra point of failure that will make our internet connections are less reliable [8]
• Noticeably slow down access to high traffic websites if it is used to block any pages on them
• Provide a false sense of security to the Australian public, due to Stephen Conroy advertising that it will protect children online
• Introduces a sinister secret censorship blacklist that is not subject to review by the Australian public

All these negatives, all this time and money spent on it, and no positives to show? All it allows is for the government to say “We tried!”. If your best try will have no positive effect and many negatives, then perhaps a better option would be to do nothing. After all, is there really a huge outcry in the public for the government to protect us from the big bad internet? For those who are interested in such protection, options do exist, such as WebShield at the ISP level and countless more downloadable packages.

Thanks for reading :)

Hulu is a website that offers commercial-supported streaming video of TV shows and movies from NBC, Fox and many other networks and studios. Currently Hulu is only available from within the United States of America.

If you want to be notified of updates to this post, subscribe to the comments feed.

This is kind of lame. One of the fundamental principles of the internet is global access. That’s why we have the world wide web, not the America-web.

So, in that spirit, here’s how to make Hulu fully functional on your local network and accessible from any ordinary web browser.

Requirements

• A machine in the US that you can run programs and serve web pages from.
• A Linux machine at home that shares your net connection

For my friends

You can skip the USA proxy configuration and just use 72.232.203.84 everywhere you would put that server’s IP.
Make sure you email me and I’ll add your local IP address to the allow list for the RTMP proxy.

USA Proxy Configuration

First we need to set up your US server to forward some selected HTTP sites as well as the Flash RTMP video streams. Do these steps on the server in the US:

• Create a file, hulu_proxy.ini with these contents:

  [proxy]
  mode = proxy
  listen_port = 9997
  
  [allowed]
  host1 = YOUR_LOCAL_IP_OR_DOMAIN
  

• Download proxy.py and run this command to start forwarding requests on port 9997 from the allowed hosts.

  proxy.py -d hulu_proxy.ini

• Create an HTTP virtual server for these domains: releasegeo.hulu.com
• Download proxy.rb and path.cgi to these virtual hosts.
• Edit path.cgi to end with

  proxyTo "http://" + ENV["HTTP_HOST"], False

• Now set up a rewrite rule to forward / to /path.cgi for each of these virtual hosts.

Local network

If you have a Linux router you can configure it to forward all packets destined for Hulu from inside your network to your US server.
Follow these steps on your LAN’s router or your on any Linux computer:

• Add this to your /etc/hosts file (these addresses may be out of date, see the last section in this blog):

  XXX.YYY.ZZZ.AAA releasegeo.hulu.com
  205.241.224.55 cp41752.edgefcs.net # HULU
  205.241.224.45 cp39465.edgefcs.net # HULU
  205.241.224.158 cp51756.edgefcs.net # HULU
  205.241.224.37 cp47346.edgefcs.net # HULU
  

  where XXX.YYY.ZZZ.AAA is the IP address of your US server.

• If you’re using dnsmasq for your LAN’s DNS server ensure that it reads these addresses from your and resolves them for hosts on your network.
• Add this to your firewall config for Linux:

  grep 'HULU$' /etc/hosts | awk '{print $1;}' | while read huluip; do
      iptables -t nat -A PREROUTING -i eth0 -p tcp \
          --destination "$huluip"  --dport 1935 -j REDIRECT --to-ports 9997
      iptables -t nat -A OUTPUT -p tcp \
          --destination "$huluip" --dport 1935 -j REDIRECT --to-ports 9997
  done
  

• Add this to your firewall config for OSX:

  grep 'HULU$' /etc/hosts | awk '{print $1;}' | while read huluip; do
      ipfw add 50000 fwd 127.0.0.1,9997 \
          tcp from any to "$huluip" dst-port 1935
  done
  

• Create a file, hulu_interceptor.ini with these contents:

  [proxy]
  mode = interceptor
  listen_port = 9997
  host = XXX.YYY.ZZZ.XXX
  port = 9997
  

• Download proxy.py and run this command to start capturing and forwarding the Flash RTMP port to your US server.

  proxy.py -d hulu_interceptor.ini

Some ISPs in Australia (Internode, iiNet and others) override the DNS entries for Akamai servers. This is why the modification to /etc/hosts is needed. Note that using the rules above, only traffic to specific Akamai Flash RTMP servers will go through your US server. So unmetered ABC iView content will remain unmetered. However, if Hulu adds new servers to this list, or if I’ve missed some, then some videos may not work and the hosts file will need updating.

If you are having difficulty viewing videos and you suspect that this is the issue, try running this tcpdump command to see whether your connections are being sent to a server within your ISPs address range:

# tcpdump -i INTERFACE port 1935

You’ll then need to find the corresponding domain name and the ‘correct’ IP to add to /etc/hosts

Automatic /etc/hosts updating

The /etc/hosts file is used to know which IP addresses to route over the tunnel to USA. My USA server resolves the Hulu video hosts that I know about and makes them available at this URL: http://delx.net.au/files/huluhosts.txt. This is automatically updated every hour.

I use a daily cronjob on my client machines in order to keep their hosts file up to date. Adapt this to your needs.

#!/bin/bash

cp /etc/hosts /etc/hosts.hulubak &&
grep -v HULU /etc/hosts > /etc/_hosts.new &&
curl -s http://delx.net.au/files/huluhosts.txt >> /etc/_hosts.new &&
chmod 0644 /etc/_hosts.new &&
mv /etc/_hosts.new /etc/hosts &&

# Ensure you clear the previous Hulu firewall rules and restart your
# DNS server (if you use one)
/root/hulutables
/etc/init.d/dnsmasq restart > /dev/null

Background

You may want to skip this section initially and refer back to it if you come across a concept that you do not fully understand later in the document.

I assume that you have a working understanding of networking. This means you must know what an IP address is, what TCP is, and what the relationship of these to application level protocols HTTP and SSH is. IPv6 is not covered here but probably will be in the future.

QoS, or Quality of Service, is any mechanism for guaranteeing a particular amount of throughput or latency for some type of traffic.

VoIP is basically about making phone calls over your internet connection. NodePhone uses the SIP standard for this. It supports incoming and outgoing calls to the PSTN by giving you an extra phone number. Internode recommends getting a QoS equipped router, but I use a Linux PC as my router and was unwilling to switch for various reasons. Using VoIP without QoS is much like talking on a mobile phone while going through a subway. The call is jittery with frequent voice dropouts.

IP packets have a Type-Of-Service (TOS) field in them. Applications can set this field to values such as Minimise-Delay, or Maximise-Throughput. This allows applications such as BitTorrent to signal that they are more interested in throughput than latency, while an interactive SSH session (think typing at a shell) could signal the opposite. The default is for normal service.

Shaping is when we prioritise outgoing traffic. It is possible to do this really well because our router is in full control of what data is sent out onto the internet. It is possible to specify what rate packets should be sent at and in what order. Eg, VoIP packets before BitTorrent is a sensible rule.

Policing is attempting to enforce rules upon incoming data. This is not possible because we cannot directly influence how much data other computers will send our way. Instead what we do is drop packets that are coming in too fast in the hope that the sender will slow down. Thankfully this is exactly what TCP is designed to do, so this strategy works reasonably well. The aim here is to make your router into a bottleneck that is slightly slower than your internet connection so you influence over the rate at which different classes of packets come in.

Goal

I needed to set up QoS to prioritise the VoIP traffic on my network. The existing documentation is not exactly plentiful and while it was invaluable, I found it difficult to understand. There don’t seem to be any turnkey solutions that do QoS for Linux in the way that I wanted.

I have four categories of traffic.

1. VoIP traffic – guaranteed 64Kbit, highest priority
2. High priority – interactive SSH sessions (not SFTP)
3. Normal priority – the default
4. Low priority – bulk data transfers like SFTP or BitTorrent

If I’m on a phone call, that traffic absolutely needs priority over everything else. The packets from the VoIP phone call should never be dropped or delayed.

Immediately after that comes other high priority traffic. This is anything with the Minimise-Delay TOS bit set in the IP header. SSH sets Minimise-Delay for interactive shells, but not for SFTP/SCP. Very convenient. Be aware that if you use ControlMaster to piggyback SFTP on an existing SSH connection that the TOS bits are set per TCP connection. I also put ACK packets in here; one of these is sent for every packet that you download. If you’re doing a large upload then these tiny packets can get delayed, causing your download to slow down. I find it works well to give them a high priority.

Next priority is general traffic, this is the default bucket. By default all traffic goes in here. This includes web traffic, instant messaging, email, etc

Finally we have the data that we don’t care about at all. I’m a little nasty and dump any traffic with the Maximise-Throughput TOS bit set in here. That includes my BitTorrents (using rtorrent) and SFTP/SCP traffic. This means that the traffic doesn’t really get it’s throughput maximised at all, but it works well for my purposes.

Linux Traffic Control

This is all done using the Linux Traffic Control system. It’s made up of a tree of queues, each with a specific algorithm for dequeuing packets.

Have a look at the shaper script. All of the rates in this file are specified in Kilobits/sec. You may need to read this text more than once.

I’m using the Hierarchical Token Bucket (HTB) for each of the four categories mentioned above. Each bucket has an associated rate, maximum rate and priority. The available outgoing bandwidth is measured and divided up amongst the different categories of traffic. The bucket gets a minimum throughput specified by ‘rate’, and a maximum specified by ‘ceil’. This maximum could be reached if one of the other buckets is not using its allowance. For example, while there is no VoIP traffic other buckets can use that 64Kbit allowance. Packets are dequeued from the buckets in order of the priority that each is given. Lower priorities first.

Next we add a Stochastical Fairness Queue (SFQ) to each bucket. The SFQ organises all packets it receives into sessions, like TCP connections, using a hashing algorithm. It dequeues packets from these sessions in a round-robin fashion. This means that if you have two connections they each receive an equal share of the available resources.

Packets are assigned to a particular queue/bucket by inspecting their header for source/dest addresses/ports as well as the TOS field.

Finally, some basic policing is applied to incoming traffic. Incoming VoIP traffic is never policed, but all other incoming traffic is policed at slightly less than the link’s maximum throughput. This forces the router to be the bottleneck and encourages the sender of any traffic that has been dropped to slow down.

QoS on incoming traffic really needs to be handled by your ISP, and while Internode claims that they do this, I found that I needed these policing rules for VoIP to work while large downloads were occurring. I suspect this is due to Internode not being able to provide proper QoS for my ADSL1 connection on a 1.5Mbit Telstra port.

References

1. Shaper script – latest version
2. Shaper script – original version
3. Linux Advanced Routing & Traffic Control HOWTO
4. RFC1349 – Type of Service in the Internet Protocol Suite

This illegal material is initially to be defined as child pornography, but once a system such as this has come into place you can bet that this definition will be stretched. Senator Nick Xenophon is already pushing to block online gambling sites. Others want to block access to information about euthanasia, abortion and other sensitive topics. Additionally any material that is Refused Classification by the OFLC is illegal to distribute in Australia.

In government trials the filter slowed internet access by up to 80%, while still having false positives and false negatives up 1-3%. That’s 1 in every 100 web pages you visit blocked for no reason. As you would expect, the filters with higher blocking accuracy were slower. It’s a trade-off that the government will make on your behalf.

Finally, this just won’t work! The task that the government is attempting to tackle is an impossible one. Doomed to failure for technical and social reasons.

Firstly, and most importantly, Australia does not need this. Parents that wish to rely on filtering software to protect their children already have the option to acquire, free of charge, software that runs on their computer to filter undesirable websites. See NetAlert. The lack of public interest in this scheme should provide some clue to the government about Australia’s lack of interest in web censorship.

Secondly, attempting to place ratings on the billions of web pages that already exist today would seriously tax the resources of any government department. The only reliable way of censoring all the “bad” web pages would be to have an explicit whitelist of “good” pages. This is undesirable for too many reasons to list. This leaves the option of blacklisting pages that are considered “inappropriate”. The government is left with the impossible task of finding all these pages and adding them to a blacklist. Artificial intelligence has not advanced to the point where this can be automated without resulting in a substantial number of false positives. When a website does find itself on the blacklist it can simply move to another host and wait for itself to be blacklisted again.

Finally, from a technical point of view, this scheme completely goes against the spirit of the Internet, which is why it cannot work. China, a government which has also decided against the will of its citizens to implement censorship of the web, fails to do so. What makes the Australian government think that it will do any better? Will this filtering scheme stop anybody who can type http://www.anonymizer.com into a web browser? No, but it will cost a great deal of money and inconvenience all Australians who use the web. What about websites like YouTube that have mixtures of content? What about Google, it might index illegal content, does it get blocked too? Does this plan even consider the non-web parts of the Internet? Illegal content will still be available over encrypted P2P like BitTorrent. This is impossible to stop unless you plan to disable all connectivity on the Internet except for web browsing. And even then there are (inefficient) work-arounds. Does the government perhaps think that Australians (criminal or not) are not intelligent enough to work around this filter if we want to? Or is this scheme some misguided attempt to protect the population of Australia from Internet “nasties” that they, as adults, are incapable of dealing with alone?

In summary, this “clean feed” is not needed to protect children because parents already have sufficient options to do so. As anybody familiar with networking would say, it is technically impossible to implement reliably and *trivial* for people with minimal technical background to work around. In the best case it would cause minimal inconvenience and cost a great deal of money. In the worst case it would cost a great deal more money and render the Internet useless and Australia’s government the laughing-stock of the technical world, while still only causing minimal inconvenience.

References:

• Australian Government Censorship ‘Worse Than Iran’
• Sydney Morning Herald – Filtering out the fury: how government tried to gag web censor critics
• Internet Gambling May be Censored in Australia
• EFA’s No Clean Feed