My approach is different to what most people do. The goal is to access Hulu and Pandora from outside USA, so obviously we have to use a proxy of some kind. However I don’t want to proxy all my traffic through the USA, as then I would not be able to access Australian specific content. So what I have is a few hacks on my router that tell it to forward only Hulu and Pandora traffic through my USA server.

Note for people who followed the old guide, this one requires no CGI, lighttpd or apache configuration. I’ve completely eliminated that part of it.

Requirements

• A machine in the US that you can run programs and serve web pages from.
• A Linux machine at home that shares your net connection

For my friends

You can skip the USA proxy configuration and just use 72.232.203.84 everywhere you would put that server’s IP.
Make sure you email me and I’ll add your local IP address to the allow list for the proxy.

USA Proxy Configuration

First we need to set up your US server to forward some selected HTTP sites as well as the Flash RTMP video streams. Do these steps on the server in the US:

• Create a file, usa_proxy.ini with these contents:

  [proxy]
  mode = proxy
  listen_port = 9997
  
  [allowed]
  host1 = YOUR_LOCAL_IP_OR_DOMAIN
  

• Download proxy.py and run this command to start forwarding requests on port 9997 from the allowed hosts.

  proxy.py -d usa_proxy.ini

Local network

If you have a Linux router you can configure it to forward all packets destined for Hulu from inside your network to your US server. This setup is most convenient as it gives all computers on your LAN access with further configuration, regardless of their operating system. Otherwise you must set this up on each Linux client separately.

Follow these steps on your Linux router or any other Linux computer:

• Download huluhosts.txt and append it to your /etc/hosts file.
• If you’re using dnsmasq for your LAN’s DNS server ensure that it reads these addresses from your and resolves them for hosts on your network.
• For Hulu, add this to your firewall config:

  grep 'HULU$' /etc/hosts | awk '{print $1;}' | while read huluip; do
      iptables -t nat -A PREROUTING -i eth0 -p tcp \
          --destination "$huluip" -j REDIRECT --to-ports 9997
      iptables -t nat -A OUTPUT -p tcp \
          --destination "$huluip" -j REDIRECT --to-ports 9997
  done
  

• For Pandora, add this to your firewall config:

  iptables -t nat -A PREROUTING -i eth0 -p tcp \
      --destination 208.85.40.0/21 -j REDIRECT --to-ports 9997
  iptables -t nat -A OUTPUT -p tcp \
      --destination 208.85.40.0/21 -j REDIRECT --to-ports 9997
  

• Depending on your other iptables rules you may need to add this somewhere to make the interceptor port allowed:

  iptables -A INPUT -i eth0 -p tcp --dport 9997 -j ACCEPT
  

• Create a file, proxy_interceptor.ini with these contents:

  [proxy]
  mode = interceptor
  listen_port = 9997
  host = XXX.YYY.ZZZ.XXX
  port = 9997
  

• Download proxy.py and run this command to start capturing and forwarding the Hulu content through your US server.

  proxy.py -d proxy_interceptor.ini

Some ISPs in Australia (Internode, iiNet and others) override the DNS entries for Akamai servers. This is why the modification to /etc/hosts is needed. Note that using the rules above, only traffic to specific Akamai Hulu servers will go through your US server. So unmetered ABC iView content will remain unmetered. However, if Hulu adds new servers to this list, or if I’ve missed some, then things may not work and the hosts file will need updating. See the last section of the blog on how to automate this with a cron job.

If you are having difficulty viewing videos and you suspect that this is the issue, try running your dnsmasq with the log-queries option enabled. Then have a look at /var/log/syslog to see what names are being looked up. Try resolving them from within USA and add the result to /etc/hosts

Automatic /etc/hosts updating

The /etc/hosts file is used to know which IP addresses to route over the tunnel to USA. My USA server resolves the Hulu video hosts that I know about and makes them available at this URL: http://delx.net.au/files/huluhosts.txt. This is automatically updated every hour.

I use a daily cronjob on my client machines in order to keep their hosts file up to date. Adapt this to your needs.

#!/bin/bash

cp /etc/hosts /etc/hosts.hulubak &&
grep -v HULU /etc/hosts > /etc/_hosts.new &&
curl -s http://delx.net.au/files/huluhosts.txt >> /etc/_hosts.new &&
chmod 0644 /etc/_hosts.new &&
mv /etc/_hosts.new /etc/hosts &&

# Ensure you clear the previous Hulu firewall rules and restart your
# DNS server (if you use one)
/root/hulutables
/etc/init.d/dnsmasq restart > /dev/null

Here’s the script to generate the huluhosts.txt file. This should be run from a machine in the USA so that you get the correct IP addresses.

#!/bin/bash

HOSTS="hulu.com www.hulu.com releasegeo.hulu.com urlcheck.hulu.com p.hulu.com s.hulu.com r.hulu.com assets.hulu.com"
OUTFILE="/var/www/huluhosts.txt"

echo -n > "$OUTFILE"
for host in $HOSTS; do
        echo "$(dig +short $host | tail -n 1) $host # HULU" >> "$OUTFILE"
done

This has caused trouble for a lot of people with incorrectly tagged libraries, and I have a work-around for manually correcting the Music Player database.

The problem is that if you have a compilation album with tracks from several artists, each artist will show up as a separate albums. For MP3s this can be fixed easily by giving all your compilation album tracks an ‘album artist’ tag. They will then be grouped correctly.

However this doesn’t seem to work for AAC/M4A music files, such as those purchased from the iTunes Store. As far as I can tell the Music Player just ignores the ‘aART’ atom which is where the album artist field is stored. I’ve reported this to Nokia and I’m waiting for their response.

Music Player conveniently stores all its gathered data in an sqlite database which can be edited. As far as I’m aware if you stuff up this process you can just delete the folder mentioned in step (2) and Music Player will automatically rescan your library from scratch. Here’s what to do:

1. Connect your phone in USB Mass Storage mode
2. Navigate to the Private/10281e17/ folder, the music database should be in a file named like [101ffc31]mpxv3_2.db, probably best to backup this file now.
3. On Mac OS X or Linux you can edit this file using this command:
   $ sqlite3 [101ffc31]mpxv3_2.db
4. You should get an sqlite> prompt where you can enter SQL commands to modify the database.
5. Copy/paste the following SQL commands below to identify the duplicate albums and merge them into one
6. Press CTRL-D to exit, and you’re all done! You’ll probably have to do this again each time you refresh the music library.

This technique works well for me, though I wish Nokia would just fix Music Player to understand ‘aART’ on AAC files. Also, it’s worth checking out LCG Jukebox. It’s a great player that works purely on folders and playlists, so you can use whatever organisation scheme you want. It also plays FLAC & OGG, and has automatic bookmarking which is very useful for podcasts and audiobooks.

SQL to Merge Albums


create temporary table dupalbum(KeepId INTEGER, DuplicateId INTEGER PRIMARY KEY);
insert into dupalbum(KeepId, DuplicateId) select a2.UniqueId as KeepId, a1.UniqueId as UniqueId from Album a1 join (select Name, UniqueId from Album group by Name having count(*) > 1) a2 on a1.Name = a2.Name where a2.UniqueId != a1.UniqueId;
update Music set Album=(select d.KeepId from dupalbum d where d.DuplicateId = Album) where Album in (select d.DuplicateId from dupalbum d);
delete from Album where Album.UniqueId in (select DuplicateId from dupalbum);
drop table dupalbum;


SQL to Remove Tracks outside the \Sounds\Music\ folder


delete from Music where not Location like '\Sounds\Music\%';
delete from Album where UniqueId not in (select Album from Music);
delete from Artist where UniqueId not in (select Artist from Music);


The files I’m converting at the moment are mono, 22.05kHz, 32kbps mp3 audio, so I don’t mind retranscoding them. As I mentioned, they can be very long so I didn’t want to tie up any of my computers for the entire time. So I will rip these to wav files in a VNC session on my Ubuntu Linux machine. PulseAudio will capture the output from wine and route it to a dummy output. This lets the machine be used for other things, including audio, while the rip is in progress.

1. $ aptitude install wine tightvncserver
2. Start vncserver and connect
3. $ padsp winecfg – configure audio output to OSS
4. Go to System->Prefs->Sound->Applications and check you can see wine in there. Turn the volume down a little bit so that it won’t clip.
5. Install Audible Manager on Linux using wine then add the audio files, no special hacks here
6. Sometimes the play button doesn’t start, in that case try using the fast forward and rewind buttons as well to make it play.
7. Run my pulse_rip script to read the audio and create an mp3

Now the Netgear factory firmware actually is a heavily stripped down old version of OpenWRT. After reading a few of forum posts and wiki pages I decided that I should go for a bleeding edge snapshot version of OpenWRT (r27153). It’s all working very well now :)

Installation

First thing, I grabbed openwrt-ar71xx-generic-wndr3700-squashfs-factory.img from the website and flashed it onto the router using the stock firmware’s upgrade page. This process went smoothly and rebooted the router. I then used telnet to connect and was prompted to set a password. That disabled telnetd and enables the dropbear ssh server. I also put an ssh public key into /etc/dropbear/authorized_keys. So far so good.

OpenWRT uses a squashfs as the base read-only filesystem, with jffs2 set up as an overlay filesystem so you can write to anywhere. You can use the builtin opkg package manager to install a bunch of useful software beyond what is built in. Currently I have 2.3M used with 4.2M free :)

I didn’t bother installing a web interface, instead preferring to do all the configuration using the UCI config files in /etc/config. See the UCI docs for a description of what to put in all these files. Whenever you start a service, eg dnsmasq using the /etc/init.d/dnsmasq script, the appropriate UCI files are read, a temporary native config file for the service is created (if appropriate) and any necessary command line args are generated to start the service.

Syslog

If you’re trying to debug something, you can view the syslog on the device using the logread command. The logger command will echo its arguments to syslog.

Wireless Setup

Most of the /etc/config/wireless settings are automatically detected. Just set the encryption to psk2 (WPA2 AES) and put in your preferred passphrase and ssid.

The radios on this device support multiple SSIDs. Each of these shows up as a separate network interface in Linux. I chose ‘guest’ as the name for this interface.

config wifi-iface
  option device radio0
  option network guest
  option mode ap
  option wmm 0
  option encryption none
  option ssid 'your-guest-ssid'

Quick note, I discovered that leaving wmm (Wireless MultiMedia extensions?) enabled (the default) made SIP VoIP calls from my Nokia wifi clients unusable. It’s easy to disable as seen above.

Firewall (iptables)

Now we have our guest wifi network interface, lets set up some firewall rules to isolate it from the main lan.

I’m not using the WAN port on my WNDR3700. There are three interfaces that are relevant. The ethernet switch and my main wifi are both bridged as ‘lan’. This network is served by DHCP and DNS from my Debian router. It is 192.168.1.0/24. I also have the ‘guest’ network, which is not bridged, and gets DHCP and DNS from dnsmasq on the WNDR3700, it uses 192.168.2.0/24. dnsmasq is set to forward DNS requests onto my main router and to explicitly ignore the ‘lan’ interface.

The firewall policy is to disallow everything on the guest network except:

• ICMP
• DHCP/DNS requests to dnsmasq on the WNDR3700
• Packets from 192.168.2.0/24 not addressed to 192.168.0.0/16

That last rule ensures that anything coming from the guest network must be using one of the expected addresses, and also that the guest network cannot send packets to hosts on my main lan. By default OpenWRT allows all related,established packets using Linux iptables’ conntrack module.

Here’s the complete firewall config I’m using: /etc/config/firewall.

Routing

My Debian router also needs to know how to route back to the guest wlan subnet.

# route add -net 192.168.2.0 netmask 255.255.255.0 gw 192.168.1.3

This can be put into /etc/network/interfaces as a post-up line. It tells the Debian router that whenever it needs to forward traffic from the internet back to the guest wlan it should do so through 192.168.1.3. This is the address of my WNDR3700.

Cron and Button events

I wanted to be able to turn the guest network on and off with the push of a button. I also wanted it to automatically disable itself each night at 3am if I forgot to turn it off.

First thing was to write a simple /root/guest-wifi script to enable/disable the guest wifi network from the command line. Then I’d hook this up to the button hotplug system and cron.

Cron is installed but not enabled by default:

cat > /etc/crontabs/root <<EOT
# m h  dom mon dow   command
  0 3   *   *   *     /root/guest-wifi disable
EOT
ln -s /etc/crontabs/root /etc/crontab
/etc/init.d/cron enable
/etc/init.d/cron start

Finally, set up the push button:

mkdir /etc/hotplug.d/button
cat > /etc/hotplug.d/button/guest-wifi-toggle <<EOT
#!/bin/sh
if [ "$BUTTON" = "BTN_2" -a "$ACTION" = "pressed" ]; then
  /root/guest-wifi toggle
fi
EOT
chmod +x /etc/hotplug.d/button/guest-wifi-toggle

System upgrades

You can update and install new packages with opkg, however at some point you may want to upgrade the kernel. Also if you’re using the squashfs images, anything you upgrade is taking up valuable space on the squashfs and the jffs2 overlay.

OpenWRT has a neat solution: sysupgrade. Always make sure to add any files that you want to keep across upgrades to the /etc/sysupgrade.conf file. Any files or directories listed here will be preserved by sysupgrade.

Update 2011/06/14 – Add C++ and Java solutions and added new numbers for J2 after improvements

The problem

Read decimal digits from a file, separated by newline chars, into a 64bit long.
Generate a sample file like this:

#!/usr/bin/env python
SIZE=200 * 1024 * 1024
DIGITS=1

MIN=10**(DIGITS-1)
MAX=(10**DIGITS)-1
import random
random.seed(0)
f = open("sample%d" % DIGITS, "w")
for x in xrange(SIZE / (DIGITS+1)):
    f.write("%d\n" % random.randint(MIN, MAX))
f.close()

Results:

Run your program against the input more than once and ignore the first results until you get the whole input file in the disk cache. Here are the results of running a few tests piped to /dev/null on my Intel Core 2 Quad Q8400. The sample code was compiled with gcc 4.4.3 from Ubuntu 10.04 using -O3. Times are measured in seconds and are the result of averaging three runs.

Discussion

The goal is to run in less than 2sec, which means we’re processing ~100MB/sec, which is about how fast a hard disk can feed data to the CPU.

X1, X2 and X3 are what I considered to be obvious and reasonably optimised solutions in C, Java and C++. It was surprising to me that none of these were able to keep up with the disk IO.

J1 and J2 are two more optimised solutions I’ve written that I’ll post about later with more discussion. Good luck!

Sample solutions

X1 – stdcatoi.c

#include <stdio.h>
#include <stdlib.h>

int main() {
  char line[32]; // 64bit integers have < 20 digits
  long value;
  while(fgets(line, sizeof(line), stdin)) {
    value = atoll(line);
    fwrite(&value, sizeof(value), 1, stdout);
  }
  return 0;
}

X2 – jatoi.java

import java.io.BufferedReader;
import java.io.BufferedOutputStream;
import java.io.DataOutputStream;
import java.io.InputStreamReader;

public class jatoi {
  public static void main(String[] args) throws Exception {
    DataOutputStream out = new DataOutputStream(new BufferedOutputStream(System.out));
    BufferedReader in = new BufferedReader(new InputStreamReader(System.in));
    String line;
    while((line = in.readLine()) != null) {
      out.writeLong(Long.parseLong(line));
    }
  }
}

X3 – cppatoi.cpp

#include <iostream>
using namespace std;

int main() {
  long x;
  while(cin >> x) {
    cout.write((char*)&x, 8);
  }
  return 0;
}

It has a simple command line interface and requires that you already have rtmpdump on your path. Get the sbs-downloader script here.

Usage example:

$ sbs-downloader
...
10) Documentary
...
41) mY generation
0) Back
Choose> 10
1) Designer People Ep 11 - Young Baek Min
2) Designer People Ep 12
3) Destination Australia - A Family Divided Full Ep
4) Empire of the Seas Ep 4
5) Gaddafi - Our Best Enemy Full Ep
....
0) Back
Choose> 3 5
RTMPDump v2.3
(c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
Connecting ...
INFO: Connected...
Starting download at: 0.000 kB
....


The bolded parts are what you type. Note that you can go back on any screen by typing “0″. Also at the list of episodes you can download a single episode by typing one number, or multiple episodes by typing several numbers separated by spaces.

It’s a simple CGI script with a form to submit a YouTube URL. It then streams the video download to the user. If you run this on a machine in the USA then it provides an easy way to bypass YouTube’s region checking.

Ask me if you want the URL for an installed copy.

USB

USB

MicroUSB

MiniUSB

Everybody will have seen used a USB cable. These cables are very multi-purpose:

• Simple charging; many new phones now charge over MicroUSB or MiniUSB. They are also often used as chargers for many other gadgets like cheap remote control helicopters. Note that due to protocol issues not all chargers will work with all devices.
• File storage; mobile phones, cameras, memory card readers, thumbdrives, external hard drives all usually make their contents accessible using the standard USB mass-storage protocol.
• Input devices; all modern keyboards, mice, gamepads, etc work over standard USB protocols so they should work on any computer.
• Many other things, such as 3g modems, mobile phone synchronisation, etc

Audio & Video Cables

3.5mm Audio

RCA-Audio

A 3.5mm audio cable is the standard audio jack that you use for earphones and many analogue audio signals. Most mp3 players, stereo systems and even many cars have a 3.5mm audio socket which you can connect to headphones or speakers.

RCA cables are fairly multi-purpose, in the picture it is a 3.5mm audio cable on one end and two RCA cables on the other. Many TVs and some speakers take two RCA cables for a stereo audio signal so this adaptor is useful.

RCA Video/Audio

Composite Video/Audio (RCA)

Component Video/Audio (RCA)

You’ll notice that these are the some plugs as the RCA cables pictured for audio above. That’s because they’re exactly the same. A composite RCA cable has two audio cables, for left & right stereo, and one for the entire video signal. This is a fairly low quality, but very ubiquitous analogue video cable.

A component RCA cable is a much higher quality analogue cable. Once again it has two audio cables, but it has three video cables, one for the red, green and blue signals respectively. Note that the colours here are only a guide, each plug is exactly the same as the other, so if you swap the red with blue on both the source and destination everything will be fine.

TV Aerial, Radio Frequency (RF)

RF

These used to be the only cable plugged into the back of a TV (besides power). It’s most commonly used to connect a TV receiver or tuner of some kind to an aerial that picks up a TV signal. This can be a digital or analogue TV signal and carries more than one channel of audio & video all mixed together on different frequencies so you need a tuner to process it. Most TVs have a tuner built in.

Digital Video/Audio

HDMI

HDMI is fast becoming the new standard for digital video & audio to connect TVs to DVD players, game consoles, etc. It carries a very high quality signal.

Computer Video

HDMI-DVI

VGA

These are used to connect computers to a monitor. VGA is an analogue signal, but still usually decent quality. DVI has two forms, DVI-A (analogue) and DVI-D (digital). DVI-A is the same signal as VGA, so you can get a simple cheap adaptor between the two. DVI-D is the same signal as HDMI, so the same thing applies. A DVI-HDMI cable is pictured above. There’s also DVI-I that carries both signals at once.

Networking

RJ12

RJ45

RJ12 cables are usually phone cables. They’re used to connect a plain old analogue telephone to the wall, or an ADSL modem to the wall. RJ45 connectors usually appear on ethernet cables, which are used for computer networking.

eSATA

eSATA

eSATA is a neat way to connect hard drives to your computer. It makes an external hard drive just as fast as an internal one, unlike USB hard drives which are much slower.

Firewire

Firewire

Firewire, more formally known as ‘IEEE 1394′ is less common than USB, but serves many of the same purposes. File storage, video cameras, audio devices. It has higher data capacity, but is less common and so usually more expensive.

See the NotiPod project page

See my previous post from November 2008. Oh, and if you plan to write your own letter to somebody about this matter, feel free to use my references, but please write your own content. I doubt it’s effective to spam busy politicians with duplicates that waste their time.

Firstly, I resent Stephen Conroy’s repeated assertion that those who oppose his plan to filter the internet are interested in “opting in to child porn” [1]. There have been many other occasions where Stephen Conroy has made such remarks. For example: “If people equate freedom of speech with watching child pornography, then the Rudd-Labor Government is going to disagree.” [2]. Nobody is trying to suggest that child pornography is acceptable, these kind of statements are inflammatory, and framing the debate in such a way is offensive and unhelpful.

It is also disturbing to see him outright mislead the public in so many respects, just one example was claiming that most ISPs support his filter plan [3], both iiNet[4] and Telstra[5] have been on record in the past as saying that the filter plan is a bad idea, so this was an outright falsehood that he definitely should have been aware of the facts before making such a statement. This is particularly irritating because he on so many occasions accuses others of misleading the public.

For the record, the rest of the industry is also against the plan, including Google, Yahoo, the US government [6] and even the “Save the Children” organisation [7]. There are good reasons why there is such widespread disapproval, which I will now cover.

The filter plan seems to have no concrete goals:

• Clearly it cannot protect those children who are actually suffering child abuse. Hiding the problem does not make it go away.
• Will it protect ordinary law abiding citizens browsing the internet at home? Considering the goal is to have a blacklist of less than 10,000 of the 1,000,000,000,000+ websites on the internet, I think it’s easy to see that it cannot possibly hope to make much of a difference to the chance of you coming across a website that may offend you. If Australians were interested in such things, they already have the option of signing up to an ISP like WebShield that offers this service at a small premium to anybody who can get ADSL through the Telstra wholesale network (that’s everybody who can get ADSL)
• It will not stop people trading this content via email, peer 2 peer file sharing networks, instant messaging chat software and private encrypted websites, or using virtual private networks. This means it will be trivial for anybody who desires to bypass the filter. The report that Stephen Conroy published on the filter testing acknowledged this.

So the filter will not help abused children, protect casual internet users, or reduce criminal activity. What exactly will it do? Here are some negatives that it will do:

• Add cost to ISPs, that will be passed on to all users
• Become an extra point of failure that will make our internet connections are less reliable [8]
• Noticeably slow down access to high traffic websites if it is used to block any pages on them
• Provide a false sense of security to the Australian public, due to Stephen Conroy advertising that it will protect children online
• Introduces a sinister secret censorship blacklist that is not subject to review by the Australian public

All these negatives, all this time and money spent on it, and no positives to show? All it allows is for the government to say “We tried!”. If your best try will have no positive effect and many negatives, then perhaps a better option would be to do nothing. After all, is there really a huge outcry in the public for the government to protect us from the big bad internet? For those who are interested in such protection, options do exist, such as WebShield at the ISP level and countless more downloadable packages.

Thanks for reading :)