Dutch log

We had two days in Dublin while Matt and Kat were at work, so we did a bit of exploring.

The first day we went to Phoenix Park, a park created by King Charles Ii’s man-on-the-ground as a hunting park in the late 1600s. The park is still the home of a herd of wild Fallow deer. We saw droppings, keep-off-the-deer signs, but no deer. The visitor centre suggested we go stand on the Papal Cross (a hilltop memorial to the fact that Pope John Paul Ii once gave mass there - to one million people!), which lead us finally to our prize.

We then went to the Guinness Storehouse, and did the big tour. I tried some Guinness (to which Greig said “Wow… Just… WOW!” in a later text) - it didn’t seem to taste like whatever it is in beer that makes me sick, but it still wasn’t enough to make me want to drink the free pint that we were given in the cafe at the top of the building. Want to like the stuff, just can’t.

The next day we caught the DART regional train around the city. We went first to the Bree end of the line, where there’s some nice beaches and a little coastal village feel. We then went back into central Dublin and had lunch with Matt at Google. Of course my non-disclosure agreement prevents me from telling you anything (read: I still don’t know what he does), but I will mention that it was Mexican day, I ate too much, the ice-cream was fantastic, and I’d be twice my weight in a month if I got a job there.

The port in the other direction is Howth - more fishing and sailing here than beachfront, but still some fantastic views. I think this is the smallest dog I’ve ever seen outside Paris Hilton’s handbag! The Irish president lives in Phoenix Park, but I hear Bono has a house at Howth.

We had an 8am flight so arranged a taxi for 6:30am - of course, the taxi driver decides to pass someone on the left and pops a tyre on the way. We were not impressed.

I’d like to thank Matt and Kat for opening their apartment to us and being incredibly generous and hospitable during our visit. I’m sorry we didn’t bring any Pineapple Lumps!

Here’s the 30-second commercial we produced for Alliance Atlantis for the show, which airs this Friday night at 7pm PT/10pm PT from our new studios.

The music you hear in the promo is from Derek Miller… check him out at http://www.penmachine.com

A year ago, I ranted about the fact that Windows will map a newly added USB drive on the first available drive letter, even if there’s a subst’d or mapped network drive on that letter.

Kyle pointed me to USBDLM, the USB Drive Letter Manager. This is a piece of software that, as well as working around this bug, will allow you to ensure that bad USB devices are always mapped to the same drive letter.  It’s free for personal or educational use, but costs for commercial use.  Not that it’s really a solution - Microsoft, this is a simple bug that you could fix today.  It is an exercise for the reader to get Raymond Chen to tell me why it’s not as simple as I think.

Another great way that the Internet gets what it wants is with the Greasemonkey extension to Firefox.  It lets you automatically add to the Javascript on pages of your choice.

I wrote a script to suffix “open new” onto RequestTracker searches, as version 3.6’s quick search box seems to show you all the tickets, even the ones that have been resolved for three years.

Today, a workmate asked for my Expert-Sex-Change login, to get around the ROT13 and blurring they now put on their answers if you’re not signed up.  (Cheatin’ bastards want Google juice for it, be prepared to show it to everyone.) “I bet someone’s written a Greasemonkey script for this”, I exclaimed, and lo, they had.  I reproduce it here with a nice little click-to-install as the author’s Wordpress eats the quotes.

By the way, if you do any web development at all (and I don’t do very much - just tidying things up), install the Firebug extension.  Right now.  It’s fantastic.  We do everything through this. Everything.

(You may have to rename these scripts to .user.js.)

Got VHS tapes? Borrow a friend’s DVD recorder and copy them to digital. Unless, of course, you borrow said DVD recorder, and find it dead.. then find it has magically come back to life a week later.

14 VHS tapes in the cupboard… net stuff worth saving: 10 minutes worth.

Interesting notes: programs I used to watch on TV before we got the DivX Channel: The Sopranos, Boston Public & Ed

Here’s the 7 minute piece that was worth keeping for posterity: an interview David Strassman did with Tim Finn.

(Americans don’t get quite as much Neil and Tim as we do down here, so they like when people like me put things like this online…)

I finished my 390.  I got an A+ for it.  Rock on.

Now I’m doing a 520 (honours project) in a similar area.  I got a Playstation 3 for it’s multi-core capabilities, and so I’ve been tinkering with that a bit.

Mostly, however, I’ve been working on getting automatic parallelisation working at the theoretical level.  I now have a program which I can give some sequential code to, and it turns it into N different programs which, when executed concurrently, are semantically equivalent.  I’ll post some screenshots of the super-high-tech visualisation GUI I came up with to display some of the intermediate output when I’m at uni next.

I started deploying puppet at work, after installing it a couple of weeks ago and playing with it. I’ve got it doing some pretty basic management so far:

• SSH key management
• Timezone and Locale
• NTP
• Minimum required packages
• /etc/resolv.conf

None of this is overly complicated, but it is very useful for me.

SSH key management

This is actually really trivial at this stage - I have an authorized_keys file which I push out to all hosts, thus allowing anyone with a key matching one in that file to log in as root. I’ll eventually change this to having user profiles managed by puppet, but for now this is good enough.

Timezone and Locale

For some reason, these two things get missed an awful lot when configuring new servers quickly, particularly if they are VMs (when debootstrapping a server, you don’t get prompted for these things). I manually symlink /etc/localtime to the right timezone file, and I use debconf-set-selections to pick the right locale then install localeconf

NTP

Once the timezone is set correctly, I install and configure ntpd. I give it a simple config using a few of the nz.pool.ntp.orNTPg servers and an internal one, and call it done. We seem to get a lot of calls from customers about time/date being wrong on their email/reports/something, so this will fix that.

Minimum required packages

Other than my preferred editor, there are a large number of packages that all hosts should have installed, no questions asked. less. tcpdump. mtr. strace. nmap. host. dig. And the rest. Puppet installs these for me now, rather than having to wait till I need the tool to install it. Puppet will happily install packages from a range of sources, and autodetects which one to use based on your OS version. Of course, package names will probably change (ssh vs openssh-server, etc), but that can be dealt with on a per-OS override.

/etc/resolv.conf

This one caught me out quite badly. I figured I was being really smart by pushing out a resolv.conf to all sites, until I then realised I was overwriting references to any local DNS servers. Up until now I had one node definition - the default one. Everything else was being autodetected at run time.

I could depart from this model and have a per-node definition which included the searchpath and the local resolvers to use, but that means creating a node definition for every new node I add. That’s not too much work, granted, but it’s more than I want to have to do for something as basic as /etc/resolv.conf. I could also create a per-site class and have every node include that class, which would save some work, but still require a per-node definition.

It looks like it’s currently impossible to have a wildcard definition of a node, eg:

So I signed up to the puppet mailing list and asked how to do this, and predictably I worked out how to do it about 30 seconds later. Puppet uses the ruby ‘facter’ library, which lets you get various facts about the system you are running on. One of these facts is the domain name.

Now I have a couple of switch statements in my resolvconf class which specify the searchpath and nameservers based on the domain name of the node being configured, which are then used to fill out a template. Magic!

I implemented QOS for inter-office phone calls for a client today using tc and diffserv. The phones and phone systems were configured by the supplier to set “Diffserv 46″, as their technician called it, which is also known as the EF PHB, or Expedited Forwarding Per-Hop Behaviour. This was made slightly trickier by having to re apply the DSCP on outbound packets due to tunnel traversal. In the end I decided it was easier to use iptables to do this, rather than trying to get tc to do it via dsmark:
/sbin/iptables -t mangle -A OUTPUT -d a.b.c.d -j DSCP –set-dscp-class EF
Actually applying the shaping is relatively straight forward using dsmark and tcindex:
#!/bin/sh
# Create root DiffServ qdisc, attach to proper network interface
# This also uses any existing DSCP flags within the packet as the tcindex
tc qdisc add dev eth2 handle 1:0 root dsmark indices 64 set_tc_index
tc filter add dev eth2 parent 1:0 protocol ip prio 1 tcindex mask 0xfc shift 2
#
# Create class-based queuing discipline to hold the two classes
tc qdisc add dev eth2 parent 1:0 handle 2:0 cbq bandwidth 10Mbit cell 8 avpkt 1000 mpu 64
#
#Create EF class, create queuing discpline for EF, create filters
tc class add dev eth2 parent 2:0 classid 2:1 cbq bandwidth 10Mbit rate 5Mbit avpkt 40000 prio 1 bounded isolated allot 1514 weight 1 maxburst 30
tc qdisc add dev eth2 parent 2:1 tbf rate 5Mbit burst 2Mbit limit 5Mbit
tc filter add dev eth2 parent 2:0 protocol ip prio 1 handle 0×2e tcindex classid 2:1 pass_on
#
# Create BE class, create queuing discipline for BE, create filters
tc class add dev eth2 parent 2:0 classid 2:2 cbq bandwidth 10Mbit rate 3Mbit avpkt 1000 prio 7 allot 1514 weight 1 maxburst 21 borrow split 2:0 defmap 0xffff
tc qdisc add dev eth2 parent 2:2 red limit 50Kbit min 10Kbit max 30Kbit burst 20 avpkt 1000 bandwidth 3Mbit probability 0.4
tc filter add dev eth2 parent 2:0 protocol ip prio 2 handle 0 tcindex mask 0 classid 2:2 pass_on
I then decided I needed a way to monitor whether this was actually working. A quick google search unveiled http://www.docum.org/docum.org/monitor/, which had a couple of different tc monitors. The author states he is no longer working on them, but they work well enough, and the iproute2+tc suite hasn’t exactly changed much lately anyway
./monitor_tc_top_bis.pl
18:52:18 up 30 min, 3 users, load average: 0.10, 0.08, 0.08
Interval Monitor Monitor Total
Dev Classid Priority Speed Bytes Speed Bytes Comment
——————————————————————————–
eth2 2: N/A 64.18Kbps 2.47MB 36.69Kbps N/A
eth2 2:1 1 6.03Kbps 86.27KB 1.25Kbps N/A
eth2 2:2 7 64.18Kbps 2.47MB 36.69Kbps N/A

I have been upgrading our MTA infrastructure at work from qmail and vpopmail, to a more robust system built using exim4, cyrus, and openldap for authentication and configuration data. I’ve been running a similar setup for ages on meta.net.nz, so I took the opportunity to do some work on the codebase as well.
SOAP API
The backend has had a SOAP API for a while now but it was pretty basic. I just used Nusoap and PHP to create the SOAP server, and didn’t bother with WSDL. I decided it would be a good idea to get Nusoap to provide decent WSDL so I could do introversion at the client end, and in doing so realised just how much work Nusoap does for you if you let it. It’ll automatically marshall PHP arrays into the right things so they appear as you expect at the other side.
I’m still having problems getting some complex types working, but otherwise it’s going well. As well as a bunch of single-purpose python scripts (using SOAPpy this time), I have an API wrapper script which lets you call any of the functions exposed by the API from the command line. With WSDL providing function arguments, return values and function descriptions, it even provides useful help. I forsee this being more use for debugging or quick modifications, or maybe used inside a wrapper to do more complicated tasks, but it’s probably better in that case to call the SOAP functions directly.
Secure Replicated LDAP
I’ve been doing replicated LDAP inside a XEN multiple-virtual-server network, but I decided with a mail infrastructure it is worth using SSL to secure the replication between hosts. I set up a CA for this purpose. Replication over SSL is no harder than normal replication, which I’ve done often enough now that it’s pretty easy to handle. Having this infrastructure in place means I can host a backup MX offsite and export my entire configuration to it via LDAP, so it can be as efficient as my onsite MXes as possible.
Spam / Malware scanning

I’m also taking the opportunity to work out some “best practices” for SA and so on. Greylisting is something that comes up fairly often, so I’m trying to find a decent greylisting implementation that will scale between multiple hosts, potentially offsite hosts, and will work sensibly within exim. A lot of them seem fairly immature, or rely on exim talking directly to a database. This latter point might not be a huge concern, but I’d rather have a system I can submit an email via, or better still a greylist tuple, and have it return a succeed or fail. There are a large number of implementations however, so this bit is taking a while to work through. Tools like AWGL (don’t have a link handy) or IMMDT.pm (Perry’s original concept for AWGL) are interesting too.
Exim
And of course, I get to go over my exim configuration, which started out as an exim 3.3 config and has been upgraded throughout the years to a 4.6 config, and pull out any quirks, and add in all the new features people are using. Even fairly trivial things such as recipient verification callout (checking with the destination server, possibly local, if the username exists - if it doesn’t, reject the email at SMTP time) have made a huge difference already.
So far
My new server is currently only running as 2MX for a couple of domains, and during that time the primary hasn’t gone offline at all. This means that approximately all of the mail it is seeing is spam. It’s dropped about 70% so far, and about another 10% of the email it has processed has been locally generated from various things happening on the system. That figure is quite high, so I’ll have to look at it and work out why it’s being sent (and where it’s going, as it seems it’s not ending up in the public folder I thought it would).
Still to go
I have to rebuild the IMAP/POP infrastructure somewhat, and that’s the worst bit of the job as it involves changing passwords for the hundred or so clients who connect directly to our server. My overall infrastructure will end up with a couple of inbound MX servers, a POP/IMAP server, possibly a separate server for spam/virus scanning (although I might look at having these services local to each MX and maintaining configs/databases between them), and an outbound MTA. This will hopefully alleviate some issues we’ve had where a lot of outbound email has effectively stopped inbound email due to loading on the MTA.
Links

Exim CheatSheet
SPAM Filtering HOWTO

Work has finally calmed down enough that I’m able to write a post about this. Last week I flew up to San Jose to attend the first XenSource University.  This was a two day event, the first of which had a series of business and technical presentation from XenSource and some of their strategic partners (Intel, Entisys etc). The second was a split between business/strategy one-on-one meetings, or a full day technical training course on the Xen Enterprise product, culminating in an exam for accreditation.
The technical components of the event were towards the lighter side of a technical forum, but I there were a wide range of people there, from those who have been using Xen directly for long time (like myself), to those who are comfortable with installing and managing VMWare, but have never touched linux, to those who are merely in the sales/demo teams of their companies. There was a good presentation from Intel discussion the VT enhancements and the future of VT, and the XenSource roadmap covered some aspects in quite a bit of detail.   There were a couple of interesting announcements too, which will be coming out later in the month.
I think the two most important aspects of this event for me were the networking with other users of Xen and XenSource products, as well as meeting more of the XenSource team; and discovering some limitations that are inherent in Xen itself. Xen apparently doesn’t support more than 4 NICs on the host, which is of major concern to anyone used to deploying VMWare ESX / Enterprise, which apparently needs about 8 gige nics just to operate. This will be ameliorated somewhat by the better performance you can expect to see under linux/Xen, however there are still enough situations in which you might want more than 4 NICs. XS doesn’t support bonding or VLANS just yet either, although both linux and Xen do - it’s just not in the UI. This will be fixed later.
We also identified an efficiency problem within the bridging system. You should be able to send data between Xen guests at relatively high speeds with an internal-only bridge, however we didn’t have much luck making this happen. This could be related to memory bandwidth issues, as the boxes we were using were not overly flash systems. Performance dropped almost linearly with an increase in MTU as well. These bugs might be in Xen, or might be in the linux bridging code, or might be in the PV ethernet driver being used.
Patrick Naubert from Xelerance, the custodians of the OpenS/WAN project, also pointed out that entropy is basically non-existant inside a Xen guest. This is a problem for anyone wanting to do crypto, of course. This shouldn’t be hard to fix if you are running Xen-aware kernels, so hopefully we’ll see a fix to this soon.

Ruby is horrible. Truly. Having been required to use Ruby in non-trivial situations for COMP313, I feel that I am now in a position to make comments about what I feel are some major flaws in the language design. It’s a pity, really, because it looks like it had the makings of a really good language. A good object system (without the difficulties inherent with Java’s primitives, for example), a large and active user community and a nice clean implementation of the Uniform Access Principle (i.e. properties are just methods with arity 0).
However, despite these positive things, Ruby is the least pleasant language I’ve ever had to use. This mostly comes down to a matter of syntax. Here is some Ruby code I grabbed from the ‘Programming Ruby’ book:

class Song
@@plays = 0
def initialize(name, artist, duration)
@name = name
@artist = artist
@duration = duration
@plays = 0
end
def play
@plays += 1
@@plays += 1
“This song: #@plays plays. Total #@@plays plays.”
end
end

Now, here is the same code snippet, but this time I have taken the liberty of replacing various Ruby syntactic constructs with images of fish:

class Song
plays = 0
def initialize(name, artist, duration)
name = name
artist = artist
duration = duration
plays = 0
end
def play
plays += 1
plays += 1
“This song: plays plays. Total plays plays.”
end
end

Strangely enough, this has actually done nothing to reduce the readability of Ruby code! I’m not just being facetious here (well, not entirely) - if we were to replace these exact same constructs with fish in Java, C++, Python or any other language, we would _significantly_ reduce the readability of the code. Ruby does not have this problem, because all of its bizarre syntactic constructs have no syntactic meaning to begin with! @, @@, $, #, %, ! all suffer from this. Let’s look at another example:
foo = “Hello, World”
foo.chomp!
bar = foo.chomp
Versus:
foo = “Hello, World”
foo.chomp
bar = foo.chomp
The latter has just as much meaning as the first.  In fact, I find that having ‘!’ littered all over the place reduces readability more than fish.
I believe that Ruby syntax is flawed for this reason.  A few meaningless delimiters are fine because they aid experienced programmers (at the expense of new or uninitiated programmers), but Ruby code is meaningless and unreadable if you do not know Ruby.  I cannot think of any other language in common use today which suffers from this problem (or at least not to this extent).
The second problem I have with Ruby is the type system.  This problem is not unique to Ruby, but it is particularly pronounced.  Dynamic typing can work fine, but Ruby’s strongly and dynamically typed approach is horrible.  If you’re going to do dynamic typing, at least build in some useful conversion and coercion.  Why on earth should I have to explicitly do type checking on parameters (think: raise “This argument is of the wrong type!” if !x.kind_of?(SomeType)), and have to explicitly call to_s methods (toString for Java-natives)?  It makes no sense!  I am being forced to a lot of extra work as a programmer.  To me, this is the sign of a poorly designed programming language.  Let’s think about this one.  I’m manually writing argument type-checking, and yet operands like ‘+’ are very picky about their parameter types.  INFER THE GODDAMNED TYPE INFORMATION, OR PUT IN EXPLICIT STATIC TYPE CHECKING!  It’s not that hard.  Either way, the current implementation leads to programmers doing more work than they should have to.  I personally favour a militant type system (SML!).  If you look at your average block of ML code, there is very little type information which cannot be inferred:
fun foobar (x) = x + 37
In case the compiler can’t work it out (or I don’t want it to work it out), I can add additional explicit type annotations (and these type annotations can go anywhere - x is a string, foobar returns a float, 37 is an Andalusian dog - that sort of thing).  The static type checker goes through and makes sure that it all makes sense.  Your type annotations must agree with both what is written (37 is plainly not an Andalusian dog), and with the know types for operators (’+’ requires homogenous arguments, and the operator is only defined for floats and integers.  Because 37 is an integer, x must be an integer too!).  Because the type of the body of the function is known, we know the return type of the function too!  The above has absolutely no explicit type information, and yet an SML compiler can tell me at compile time if I’m trying to pass incorrect parameter types to a function.  We can go even further with static assertion checking!
Ruby has none of this.
Finally (there were a lot more things I thought of today, but I can’t remember half of them - I’ll add them as they come to me!) - why is ++ not defined for integers?  I mean, Ruby has every piece of syntactic sugar under the sun, and no increment operator?  What the hell?

I had a couple of albums of mp3s that were encoded with really low gain. Rather than re-encode the mp3s (which wouldn’t have taken too long), I had a look for tools that would let me normalise these tools. I didn’t have a lot of luck, and then yesterday I saw a link to mp3gain pop up in my aggregator.
Mp3gain works by analysing the mp3s passed to it, then tweaking the mp3 metadata to adjust the gain. It doesn’t re-encode the mp3s. It can normalise the gain on a single mp3, or work out the ‘normalised’ gain on an entire album (or repository) and tweak each mp3 to bring it in line with the others.  To be honest, I didn’t even know mp3s had metadata you could tweak to do this, so it didn’t occur to me this was an option.
I ran it  across the albums in question, and it decided they were consistent within themselves. Rather than run it across the entire repository, I increased the gain by a set 3dB, and then after listening to the resulting output, another 3dB. Maybe one day I’ll back up my mp3s and run it across the entire repository. This works pretty well for now.

XenSource have announced a couple of new commercial offerings to go along with their Xen Enterprise release. While Xen itself is opensource, XenSource have decided to make commercial packages offering a GUI management console, more advanced management APIs, and perhaps most importantly, PV drivers for Windows guests.
The full suite of products now looks like:

Xen Enterprise. Unlimited guests, multi OS. Pricing starts at $498 US for a dual-socket system
Xen Server. 8 Windows guests, 8 GB of ram. $99 US annual subscription, dual-socket system only
Xen Express. 4 guests, multi OS, 4 GB of ram. Free.

All of these products, including the free version, have the PV drivers for windows. There is also a seamless upgrade path between the products, so you can do a test deployment with Xen Express, then purchase Xen Server or Xen Enterprise as you need.
There are some differences other than those listed above. Xen Express will not allow multi-host management. The other two products will - this means you can log into multiple servers from the same console at the same time, and get holistic view of your virtualised servers. Also, while XenSource has yet to release any HA/DR, Live migration or integrated backup plugins, it is unlikely that these will be able to run on Xen Express.
http://www.xensource.com/