See my previous post from November 2008. Oh, and if you plan to write your own letter to somebody about this matter, feel free to use my references, but please write your own content. I doubt it’s effective to spam busy politicians with duplicates that waste their time.

Firstly, I resent Stephen Conroy’s repeated assertion that those who oppose his plan to filter the internet are interested in “opting in to child porn” [1]. There have been many other occasions where Stephen Conroy has made such remarks. For example: “If people equate freedom of speech with watching child pornography, then the Rudd-Labor Government is going to disagree.” [2]. Nobody is trying to suggest that child pornography is acceptable, these kind of statements are inflammatory, and framing the debate in such a way is offensive and unhelpful.

It is also disturbing to see him outright mislead the public in so many respects, just one example was claiming that most ISPs support his filter plan [3], both iiNet[4] and Telstra[5] have been on record in the past as saying that the filter plan is a bad idea, so this was an outright falsehood that he definitely should have been aware of the facts before making such a statement. This is particularly irritating because he on so many occasions accuses others of misleading the public.

For the record, the rest of the industry is also against the plan, including Google, Yahoo, the US government [6] and even the “Save the Children” organisation [7]. There are good reasons why there is such widespread disapproval, which I will now cover.

The filter plan seems to have no concrete goals:

• Clearly it cannot protect those children who are actually suffering child abuse. Hiding the problem does not make it go away.
• Will it protect ordinary law abiding citizens browsing the internet at home? Considering the goal is to have a blacklist of less than 10,000 of the 1,000,000,000,000+ websites on the internet, I think it’s easy to see that it cannot possibly hope to make much of a difference to the chance of you coming across a website that may offend you. If Australians were interested in such things, they already have the option of signing up to an ISP like WebShield that offers this service at a small premium to anybody who can get ADSL through the Telstra wholesale network (that’s everybody who can get ADSL)
• It will not stop people trading this content via email, peer 2 peer file sharing networks, instant messaging chat software and private encrypted websites, or using virtual private networks. This means it will be trivial for anybody who desires to bypass the filter. The report that Stephen Conroy published on the filter testing acknowledged this.

So the filter will not help abused children, protect casual internet users, or reduce criminal activity. What exactly will it do? Here are some negatives that it will do:

• Add cost to ISPs, that will be passed on to all users
• Become an extra point of failure that will make our internet connections are less reliable [8]
• Noticeably slow down access to high traffic websites if it is used to block any pages on them
• Provide a false sense of security to the Australian public, due to Stephen Conroy advertising that it will protect children online
• Introduces a sinister secret censorship blacklist that is not subject to review by the Australian public

All these negatives, all this time and money spent on it, and no positives to show? All it allows is for the government to say “We tried!”. If your best try will have no positive effect and many negatives, then perhaps a better option would be to do nothing. After all, is there really a huge outcry in the public for the government to protect us from the big bad internet? For those who are interested in such protection, options do exist, such as WebShield at the ISP level and countless more downloadable packages.

Thanks for reading :)

After hacking the Nokia 5800 phoneplugin and turning off contact sync in iSync I got older firmware versions (V10, V11 and V12) to sync my calendars. This was the most important thing for me, but still very annoying.

First download Nokia-N97.phoneplugin.zip. Extract to ~/Library/PhonePlugins/ as usual. Now pair your phone with bluetooth and select the option to enable iSync. Open iSync and turn off contacts sync. It should now sync calendars properly. This is where I’d been up to V20.

To get contacts to sync you need to reset the contacts database. Insert a MicroSD card into your phone and do a backup of your contacts from the phone’s “File Mgr” app. Now immediately restore just the contacts, the phone will reboot. Turn on contacts sync in iSync and it should work great =)

Enjoy!

Package installation

Install the required packages:

# aptitude install e2fsprogs cryptsetup

Partitioning

Use your favourite partitioning tool, for example fdisk or cfdisk to set aside a whole partition with enough space to store your data.

Erase

This step is optional. For complete security you want the contents of the disk to be random before you start using it. Use the badblocks tool to do this.

# badblocks -c 10240 -s -w -t random -v /dev/sdX9

Format

The luksformat tool will format a partition to be used with LUKS (Linux Unified Key Setup) and then create a filesystem on it.

# luksformat -t ext3 /dev/sdX9

crypttab and fstab

Now all that remains is to a line in each of these config files.

Add this line to /etc/crypttab:

myname /dev/sdX9 none luks

The crypttab file is examined by the system during boot. Each line maps a real encrypted device file (/dev/sdX9) to a virtual decrypted device file (/dev/mapper/myname). Once you’ve added done this run the following command to actually set up the mapping:

# /etc/init.d/cryptdisks restart

Now you can set up that virtual device file to be mounted like any other. For example, the following command would mount your filesystem:

# mount /dev/mapper/myname /mnt

Note that you should probably use partition UUIDs (UUID=XXXXX) in place of device file names (/dev/sdX9) in your crypttab for a more robust system. The easiest way to find these is by running:

# ls -l /dev/disks/by-uuid

Add a line like this to /etc/fstab

/dev/mapper/myname /path/to/mountpoint ext3 defaults 0 2

The Fink project has two components. First, the port of Debian’s apt-get tools to Mac OS X combined with a binary package repository for the stable distribution.

Second, the ‘fink’ command line tool. It generates .deb packages from a source package description (located in /sw/fink/dists). Once generated you can move these packages between machines and install them with apt-get or dpkg as you would in Debian. This can save rebuilding from source.

Next the infamous stable vs unstable distributions. Unstable in Fink has the same meaning as it does with Debian. Don’t be too scared of using unstable. It won’t make your system crash. What it will do is have up to date packages (in most cases). It will get more frequent updates, meaning you’ll have more to recompile more often. Because of this, if you’re using the unstable distribution there tends not to be any prebuilt binaries so installation can take longer than in Debian. Also note these more recent packages may have had less testing, so sometimes they may not install.

One other thing you should know about is a neat package called ‘debfoster’. It’ll help you clean out unused packages and libraries. One of the consequences of Fink unstable building from source is that you often get lots of development packages installed that you don’t need anymore.

Just install debfoster and run it as root, then answer either y(es), n(o) or p(rune) to keep a package and its deps, not keep, or remove a package and all it’s deps. It’s quite clever. Give it a shot.

Finally, remember Fink is a volunteer project that you can contribute to. If you find a missing or out of date package, feel free to dig in to the .info file and update it. This is often as easy as just changing the version number. See the Fink website for a good packaging tutorial and reference.

This post describes how to fix the function keys and swap command (windows or ‘super’) keys with the alt (or option) keys.

This command will fix the function keys, it saves you pressing fn-F1 whenever you want F1. The first command is for older kernels, the second is for version 2.6.28 or later.

# echo 2 > /sys/module/hid/parameters/pb_fnmode
# echo 2 > /sys/module/hid_apple/parameters/fnmode

Then add that line to your /etc/rc.local file, somewhere before the exit 0 at the end, so that it gets run on startup.

Next to swap the Command/Alt keys using xmodmap. I’m aware you can do this from the Gnome Keyboard Settings panel, but I’ve found this method works better. Particularly when combined with synergy.

Create a file called ~/.xmodmaprc with this inside:

clear mod1
keycode 115 = Alt_L
keycode 116 = Alt_R
keycode 64 = Super_L
keycode 113 = Super_R
add mod1 = Alt_L Alt_R

On another computer I’ve found this worked:

clear mod1
keycode 133 = Alt_L
keycode 134 = Alt_R
keycode 64 = Super_L
keycode 108 = Super_R
add mod1 = Alt_L Alt_R

Now run to activate the new keys, run:

$ xmodmap ~/.xmodmaprc

Don’t forget to add it to your list of startup programs. If you’re using Gnome, look at System->Preferences->Startup Applications
Otherwise you can just add it to ~/.xsession

To find the keycodes above I used the xev program. Try running it from a console. It shows you all X11 events that the xev window receives, including key presses/releases.

Hulu is a website that offers commercial-supported streaming video of TV shows and movies from NBC, Fox and many other networks and studios. Currently Hulu is only available from within the United States of America.

If you want to be notified of updates to this post, subscribe to the comments feed.

This is kind of lame. One of the fundamental principles of the internet is global access. That’s why we have the world wide web, not the America-web.

So, in that spirit, here’s how to make Hulu fully functional on your local network and accessible from any ordinary web browser.

Requirements

• A machine in the US that you can run programs and serve web pages from.
• A Linux machine at home that shares your net connection

For my friends

You can skip the USA proxy configuration and just use 72.232.203.84 everywhere you would put that server’s IP.
Make sure you email me and I’ll add your local IP address to the allow list for the RTMP proxy.

USA Proxy Configuration

First we need to set up your US server to forward some selected HTTP sites as well as the Flash RTMP video streams. Do these steps on the server in the US:

• Create a file, hulu_proxy.ini with these contents:

  [proxy]
  mode = proxy
  listen_port = 9997
  
  [allowed]
  host1 = YOUR_LOCAL_IP_OR_DOMAIN
  

• Download proxy.py and run this command to start forwarding requests on port 9997 from the allowed hosts.

  proxy.py -d hulu_proxy.ini

• Create an HTTP virtual server for these domains: releasegeo.hulu.com
• Download proxy.rb and path.cgi to these virtual hosts.
• Edit path.cgi to end with

  proxyTo "http://" + ENV["HTTP_HOST"], False

• Now set up a rewrite rule to forward / to /path.cgi for each of these virtual hosts.

Local network

If you have a Linux router you can configure it to forward all packets destined for Hulu from inside your network to your US server.
Follow these steps on your LAN’s router or your on any Linux computer:

• Add this to your /etc/hosts file (these addresses may be out of date, see the last section in this blog):

  XXX.YYY.ZZZ.AAA releasegeo.hulu.com
  205.241.224.55 cp41752.edgefcs.net # HULU
  205.241.224.45 cp39465.edgefcs.net # HULU
  205.241.224.158 cp51756.edgefcs.net # HULU
  205.241.224.37 cp47346.edgefcs.net # HULU
  

  where XXX.YYY.ZZZ.AAA is the IP address of your US server.

• If you’re using dnsmasq for your LAN’s DNS server ensure that it reads these addresses from your and resolves them for hosts on your network.
• Add this to your firewall config for Linux:

  grep 'HULU$' /etc/hosts | awk '{print $1;}' | while read huluip; do
      iptables -t nat -A PREROUTING -i eth0 -p tcp \
          --destination "$huluip"  --dport 1935 -j REDIRECT --to-ports 9997
      iptables -t nat -A OUTPUT -p tcp \
          --destination "$huluip" --dport 1935 -j REDIRECT --to-ports 9997
  done
  

• Add this to your firewall config for OSX:

  grep 'HULU$' /etc/hosts | awk '{print $1;}' | while read huluip; do
      ipfw add 50000 fwd 127.0.0.1,9997 \
          tcp from any to "$huluip" dst-port 1935
  done
  

• Create a file, hulu_interceptor.ini with these contents:

  [proxy]
  mode = interceptor
  listen_port = 9997
  host = XXX.YYY.ZZZ.XXX
  port = 9997
  

• Download proxy.py and run this command to start capturing and forwarding the Flash RTMP port to your US server.

  proxy.py -d hulu_interceptor.ini

Some ISPs in Australia (Internode, iiNet and others) override the DNS entries for Akamai servers. This is why the modification to /etc/hosts is needed. Note that using the rules above, only traffic to specific Akamai Flash RTMP servers will go through your US server. So unmetered ABC iView content will remain unmetered. However, if Hulu adds new servers to this list, or if I’ve missed some, then some videos may not work and the hosts file will need updating.

If you are having difficulty viewing videos and you suspect that this is the issue, try running this tcpdump command to see whether your connections are being sent to a server within your ISPs address range:

# tcpdump -i INTERFACE port 1935

You’ll then need to find the corresponding domain name and the ‘correct’ IP to add to /etc/hosts

Automatic /etc/hosts updating

The /etc/hosts file is used to know which IP addresses to route over the tunnel to USA. My USA server resolves the Hulu video hosts that I know about and makes them available at this URL: http://delx.net.au/files/huluhosts.txt. This is automatically updated every hour.

I use a daily cronjob on my client machines in order to keep their hosts file up to date. Adapt this to your needs.

#!/bin/bash

cp /etc/hosts /etc/hosts.hulubak &&
grep -v HULU /etc/hosts > /etc/_hosts.new &&
curl -s http://delx.net.au/files/huluhosts.txt >> /etc/_hosts.new &&
chmod 0644 /etc/_hosts.new &&
mv /etc/_hosts.new /etc/hosts &&

# Ensure you clear the previous Hulu firewall rules and restart your
# DNS server (if you use one)
/root/hulutables
/etc/init.d/dnsmasq restart > /dev/null

I was aware you could set a default and default-push repository in .hg/hgrc, in fact whenever you clone a repository a default remote path is created for you.

You can actually put arbitrary names in there. For example, I have a repository “chatbots” that I cloned from Katie’s repository: http://katharos.id.au/hg/chatbots". I want to be able to push and pull changes from my repository as well as pull from her’s without typing the full path out at any time.


[paths]
default = ssh://hg@delx.net.au/chatbots
katie = http://katharos.id.au/hg/chatbots


The above snippet goes in the repositories .hg/hgrc, and I can now run commands like this:

$ hg pull katie
$ hg push

This pulls any new changes in from Katie’s repository and pushes them to the default, mine.

There’s a Git repository with the patch included here:
http://delx.net.au/git/offlineimap

I followed the plan John Goerzen suggested in offlineimap ticket 18. Here’s an overview of the changes.

• Use imaplib2 as it implements the IDLE command and a few other nice things. This is written by Piers Lauder, the author Python’s standard imaplib.
• Some small changes to the OfflineIMAP code were needed for it to work with imaplib2.
• Added a config parameter ‘idlefolders’ to specify a list of mailboxes to monitor. This parameter forces holdconnectionopen, keepalive and maxconnections to be sane values.
• Hijack the keepalive thread. Use the available connections for IDLE, one on each of the given mailboxes. If there are leftover connections we send NOOP as before.
• Added documentation to the sample offlineimap.conf

Background

You may want to skip this section initially and refer back to it if you come across a concept that you do not fully understand later in the document.

I assume that you have a working understanding of networking. This means you must know what an IP address is, what TCP is, and what the relationship of these to application level protocols HTTP and SSH is. IPv6 is not covered here but probably will be in the future.

QoS, or Quality of Service, is any mechanism for guaranteeing a particular amount of throughput or latency for some type of traffic.

VoIP is basically about making phone calls over your internet connection. NodePhone uses the SIP standard for this. It supports incoming and outgoing calls to the PSTN by giving you an extra phone number. Internode recommends getting a QoS equipped router, but I use a Linux PC as my router and was unwilling to switch for various reasons. Using VoIP without QoS is much like talking on a mobile phone while going through a subway. The call is jittery with frequent voice dropouts.

IP packets have a Type-Of-Service (TOS) field in them. Applications can set this field to values such as Minimise-Delay, or Maximise-Throughput. This allows applications such as BitTorrent to signal that they are more interested in throughput than latency, while an interactive SSH session (think typing at a shell) could signal the opposite. The default is for normal service.

Shaping is when we prioritise outgoing traffic. It is possible to do this really well because our router is in full control of what data is sent out onto the internet. It is possible to specify what rate packets should be sent at and in what order. Eg, VoIP packets before BitTorrent is a sensible rule.

Policing is attempting to enforce rules upon incoming data. This is not possible because we cannot directly influence how much data other computers will send our way. Instead what we do is drop packets that are coming in too fast in the hope that the sender will slow down. Thankfully this is exactly what TCP is designed to do, so this strategy works reasonably well. The aim here is to make your router into a bottleneck that is slightly slower than your internet connection so you influence over the rate at which different classes of packets come in.

Goal

I needed to set up QoS to prioritise the VoIP traffic on my network. The existing documentation is not exactly plentiful and while it was invaluable, I found it difficult to understand. There don’t seem to be any turnkey solutions that do QoS for Linux in the way that I wanted.

I have four categories of traffic.

1. VoIP traffic – guaranteed 64Kbit, highest priority
2. High priority – interactive SSH sessions (not SFTP)
3. Normal priority – the default
4. Low priority – bulk data transfers like SFTP or BitTorrent

If I’m on a phone call, that traffic absolutely needs priority over everything else. The packets from the VoIP phone call should never be dropped or delayed.

Immediately after that comes other high priority traffic. This is anything with the Minimise-Delay TOS bit set in the IP header. SSH sets Minimise-Delay for interactive shells, but not for SFTP/SCP. Very convenient. Be aware that if you use ControlMaster to piggyback SFTP on an existing SSH connection that the TOS bits are set per TCP connection. I also put ACK packets in here; one of these is sent for every packet that you download. If you’re doing a large upload then these tiny packets can get delayed, causing your download to slow down. I find it works well to give them a high priority.

Next priority is general traffic, this is the default bucket. By default all traffic goes in here. This includes web traffic, instant messaging, email, etc

Finally we have the data that we don’t care about at all. I’m a little nasty and dump any traffic with the Maximise-Throughput TOS bit set in here. That includes my BitTorrents (using rtorrent) and SFTP/SCP traffic. This means that the traffic doesn’t really get it’s throughput maximised at all, but it works well for my purposes.

Linux Traffic Control

This is all done using the Linux Traffic Control system. It’s made up of a tree of queues, each with a specific algorithm for dequeuing packets.

Have a look at the shaper script. All of the rates in this file are specified in Kilobits/sec. You may need to read this text more than once.

I’m using the Hierarchical Token Bucket (HTB) for each of the four categories mentioned above. Each bucket has an associated rate, maximum rate and priority. The available outgoing bandwidth is measured and divided up amongst the different categories of traffic. The bucket gets a minimum throughput specified by ‘rate’, and a maximum specified by ‘ceil’. This maximum could be reached if one of the other buckets is not using its allowance. For example, while there is no VoIP traffic other buckets can use that 64Kbit allowance. Packets are dequeued from the buckets in order of the priority that each is given. Lower priorities first.

Next we add a Stochastical Fairness Queue (SFQ) to each bucket. The SFQ organises all packets it receives into sessions, like TCP connections, using a hashing algorithm. It dequeues packets from these sessions in a round-robin fashion. This means that if you have two connections they each receive an equal share of the available resources.

Packets are assigned to a particular queue/bucket by inspecting their header for source/dest addresses/ports as well as the TOS field.

Finally, some basic policing is applied to incoming traffic. Incoming VoIP traffic is never policed, but all other incoming traffic is policed at slightly less than the link’s maximum throughput. This forces the router to be the bottleneck and encourages the sender of any traffic that has been dropped to slow down.

QoS on incoming traffic really needs to be handled by your ISP, and while Internode claims that they do this, I found that I needed these policing rules for VoIP to work while large downloads were occurring. I suspect this is due to Internode not being able to provide proper QoS for my ADSL1 connection on a 1.5Mbit Telstra port.

References

1. Shaper script – latest version
2. Shaper script – original version
3. Linux Advanced Routing & Traffic Control HOWTO
4. RFC1349 – Type of Service in the Internet Protocol Suite

So I was having a discussion with Greg yesterday about file systems and I claimed the above. I have actually made this claim, that OS X automatically defragments files, many times in the past. However this time Greg decided to call me on it. He told me: Citation needed.

I knew I’d read this somewhere, but couldn’t track my original source. So I did a little bit of research and found there wasn’t an easily accessible authoritative source anywhere.

Apple Technote HT1375 was one of the first pages I found. However it only talks about the Hot-File-Adaptive-Clustering, which is a completely different technique.

A little more searching and I found this Ars Technica article, a review of Mac OS X 10.3. This actually contains some solid details on the auto defragmentation feature. The reference for this information is only a newsgroup post on comp.macosx.general, not exactly definitive.

So I decided to go to the source. Apple publishes the source code for large components of Mac OS X in the Darwin Source Code repository. In this case we needed to look at the HFS+ driver, which is part of XNU – the OS X kernel.
Incidentally, thanks go to Apple for making this code available to everybody for random research like this.

Before long I’d narrowed it down to this file: xnu-1228.7.58/bsd/hfs/hfs_vnops.c. The function that we’re interested in is hfs_vnop_open(), which is called whenever the system needs to open a file or directory on disk for reading or writing. The last two comments in that function explain the conditions under which defragmentation occurs. Nicely written and well commented code.

Since you need a free Apple developer account to view the above link, here’s the relevant section of the source code:

/*
 * Open a file/directory.
 */
static int
hfs_vnop_open(struct vnop_open_args *ap)
{
  struct vnode *vp = ap->a_vp;
  struct filefork *fp;
  struct timeval tv;
  int error;

  /*
   * Files marked append-only must be opened for appending.
   */
  if ((VTOC(vp)->c_flags & APPEND) && !vnode_isdir(vp) &&
      (ap->a_mode & (FWRITE | O_APPEND)) == FWRITE)
    return (EPERM);

  if (vnode_isreg(vp) && !UBCINFOEXISTS(vp))
    return (EBUSY);  /* file is in use by the kernel */

  /* Don't allow journal file to be opened externally. */
  if (VTOC(vp)->c_fileid == VTOHFS(vp)->hfs_jnlfileid)
    return (EPERM);
  /*
   * On the first (non-busy) open of a fragmented
   * file attempt to de-frag it (if its less than 20MB).
   */
  if ((VTOHFS(vp)->hfs_flags & HFS_READ_ONLY) ||
      (VTOHFS(vp)->jnl == NULL) ||
#if NAMEDSTREAMS
      !vnode_isreg(vp) || vnode_isinuse(vp, 0) ||
      vnode_isnamedstream(vp)) {
#else
      !vnode_isreg(vp) || vnode_isinuse(vp, 0)) {
#endif
    return (0);
  }

  if ((error = hfs_lock(VTOC(vp), HFS_EXCLUSIVE_LOCK)))
    return (error);
  fp = VTOF(vp);
  if (fp->ff_blocks &&
      fp->ff_extents[7].blockCount != 0 &&
      fp->ff_size <= (20 * 1024 * 1024)) {
    struct timeval now;
    struct cnode *cp = VTOC(vp);
    /*
     * Wait until system bootup is done (3 min).
     * And don't relocate a file that's been modified
     * within the past minute -- this can lead to
     * system thrashing.
     */
    microuptime(&tv);
    microtime(&now);
    if (tv.tv_sec > (60 * 3) &&
       ((now.tv_sec - cp->c_mtime) > 60)) {
      (void) hfs_relocate(vp, VTOVCB(vp)->nextAllocation + 4096,
                          vfs_context_ucred(ap->a_context),
                          vfs_context_proc(ap->a_context));
    }
  }
  hfs_unlock(VTOC(vp));

  return (0);
}