A story of woe, a Belkin wireless router and IPv6.

Brief History

I was looking to replace my able but rather old 3com access point. The 3com was an enterprise grade access point (bought when I had more money than sense) and had served very well, but was just too old to support the modern N standard.

So I looked at web reviews, and made my selection. The Belkin Double N+ wireless router (F6D6230-4 v1000) received some mixed reviews, but claims had been made that this was a good performer. It is a high-end consumer level product, Belkin’s flagship but also one of few to support the 5 Ghz N band, which I particularly wanted. As it is a consumer product, you don’t get VLANs, RADIUS mac authentication etc. But it does support multiple SSIDs. I can live without the VLANs and RADIUS.

On checking the manual on line, I was also pleased to see that it supported an “Access Point” mode, where by all the routing functions are disabled, and it acts purely as an access point. This suited me exactly, as I have never been too happy with the router/access point model. If you change your internet connection (cable – adsl) then you have to replace a more expensive box than if you have a dedicated piece of kit for both. Also if your router is compromised, then the cracker has access, both to your LAN and to your internet connection. Far better for security to have separate devices on very different parts of network.

IPv6, the way forward…?

A couple of months before, I had configured my network to work with IPv6. The way the internet will work in the future. I am using Hurricane Electric’s tunnel broker service (http://tunnelbroker.net/) which is generally excellent. The only problem I have with it is that they don’t protect your account password with https. It was all working very well. (debian systems actually default to IPv6 in preference, if it is available).

IPv6 Broken

However, shortly after installing by Belkin Double N+ router, although it took me a month or two to realise the connection, the IPv6 has been intermittent, or downright broken.

The reason I did not suspect the Belkin router, was that no where in it documentation, nor any review that I had read did it mention anything about IPv6.

Noone supports IPv6

To me, this lack of IPv6 is a bizarre state of affairs. IPv6 is an established standard, Microsoft, Apple, unix/linux, cisco, juniper and HP have all supported IPv6 for ages, yet in the UK, no major ISP (that I know of) natively supports IPv6, and it is almost impossible to buy a router or an access point which supports IPv6. IPv6 is great for the consumer too. You truly could give every electrical item in you house a unique internet routable address and have billions of billions of addresses free in your allocated subnet. In addition, it is very easy to set up an ad-hoc network with IPv6. You don’t need any DHCP or similar as all IPv6 routers can provide clients with addresses.

Investigation

I do not profess to be an IPv6 expert, so the following may not be totally accurate.

IPv6: Billions and billions of addresses

A quick summary on the format of IPv6 addresses and how to write them:
IPv6 has 128 bits which represent the IP address. These are written as 8 16 bit hexadecimal numbers separated with colons:
2001:0db8:0001:AC10:0000:0000:0000:0001
However, as this is long winded, and 0 is common in IPv6 addresses, the example above can be written with a short hand:
2001:db8:1:AC10::1
Here, leading zeros are omitted, and the sequential run of 3 0000s can be replaces with a double colon. You can only have one double colon in an address though. Interestingly, though not really relevant to this discussion if you need to specify a port on an IPv6 host, you use square brackets around the IP address. This prevents the colon to specify the port being interpreted as part of the address:
[2001:db8:1:AC10::1]:80

Wikipedia has a useful article on IPv6.

What no DHCP?

IPv6, on your LAN works very differently from IPv4. Especially how they learn about the local network configuration. A DHCP client uses a broadcast packet to locate a DHCP server. The server then responds with information on what IP address, what name server, default route etc. IPv6 doesn’t work like this. Instead of broadcasts, IPv6 uses multicasts to do this. As well as responding to multi cast packets, the client generates itself a unique “local” address within a particular range of the form fe80::/64 This address is generated from the MAC address of the host, but is only visible within the LAN. (there are mechanisms to check that this is a unique address, before it uses it)

eg:
mac address: 00:22:75:34:e2:77
link address: fe80::222:75ff:fe34:e277

The strong numbers are taken directly from the MAC address.

These addresses will ping, but they will not route. On an automatically configured IPv6 network, the route is advertised as one of these addresses.

This is not a tutorial on how to configure an IPv6 LAN. There are several available: Here is a good one for debian.

What went wrong

When my network doesn’t have the Belkin router connected:

On a typical IPv6 client:

# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:16:3e:a0:38:67
inet addr:172.30.0.67  Bcast:172.30.0.255  Mask:255.255.255.0
inet6 addr: 2001:XXXX:XXXX:XXXX:216:3eff:fea0:3867/64 Scope:Global
inet6 addr: fe80::216:3eff:fea0:3867/64 Scope:Link
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:317 errors:0 dropped:0 overruns:0 frame:0
TX packets:135 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:28381 (27.7 KiB)  TX bytes:22706 (22.1 KiB)

and

# route -6
Kernel IPv6 routing table
Destination                    Next Hop                   Flag Met Ref Use If
2001:XXXX:XXXX:XXXX::43/128    ::                         U    256 0     0 eth0
2001:XXXX:XXXX:XXXX::/64       ::                         UAe  256 0     1 eth0
fe80::/64                      ::                         U    256 0     0 eth0
::/0                           fe80::216:3eff:fe19:4d15   UGDAe 1024 0     1 eth0
::/0                           ::                         !n   -1  1 34700 lo
::1/128                        ::                         Un   0   1   269 lo
2001:XXXX:XXXX:XXXX::43/128    ::                         Un   0   1     0 lo
fe80::216:3eff:fea0:3867/128   ::                         Un   0   1     3 lo
ff00::/8                       ::                         U    256 0     0 eth0
::/0                           ::                         !n   -1  1 34700 lo

The ::/0 is the default route (except where a ! appears in the flags). And finally:

# ping6 -c 4 ipv6.google.com
PING ipv6.google.com(2a00:1450:8002::67) 56 data bytes
64 bytes from 2a00:1450:8002::67: icmp_seq=1 ttl=56 time=44.0 ms
64 bytes from 2a00:1450:8002::67: icmp_seq=2 ttl=56 time=36.0 ms
64 bytes from 2a00:1450:8002::67: icmp_seq=3 ttl=56 time=36.0 ms
64 bytes from 2a00:1450:8002::67: icmp_seq=4 ttl=56 time=32.0 ms
^C
--- ipv6.google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3012ms
rtt min/avg/max/mdev = 32.000/37.000/44.000/4.358 ms

Everything working as it should.

When I add the access point: (The following have been abridged)

# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:16:3e:a0:38:67 
 inet addr:172.30.0.67  Bcast:172.30.0.255  Mask:255.255.255.0
 inet6 addr: 2001:db8:1:0:216:3eff:fea0:3867/64 Scope:Global
 inet6 addr: 2001:XXXX:XXXX:XXXX:216:3eff:fea0:3867/64 Scope:Global
 inet6 addr: fe80::216:3eff:fea0:3867/64 Scope:Link
...

The interface now has an additional address (shown in strong), and it also has an additional route:

# route -6
...
::/0                           fe80::216:3eff:fe19:4d15   UGDAe 1024 0    13 eth0
::/0                           fe80::222:75ff:fe34:e277   UGDAe 1024 0     0 eth0
...

The new route is shown in strong. That IPv6 address is a Link scope address which matches the MAC address of my Belkin router.  So how is the ping looking now?

# ping6 -c 4 ipv6.google.com
PING ipv6.google.com(2a00:1450:8002::63) 56 data bytes
From fe80::222:75ff:fe34:e277 icmp_seq=1 Destination unreachable: No route
From fe80::222:75ff:fe34:e277 icmp_seq=2 Destination unreachable: No route
From fe80::222:75ff:fe34:e277 icmp_seq=3 Destination unreachable: No route
From fe80::222:75ff:fe34:e277 icmp_seq=4 Destination unreachable: No route

--- ipv6.google.com ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3004ms

Not so good. So for a device that “doesn’t support” IPv6, my belkin router is having a big effect on my IPv6 configuration. Finally, if it responds to an IPv6 ping, I can be pretty sure that it is talking IPv6:

# ping6 -c 4 -I eth0 fe80::222:75ff:fe34:e277
PING fe80::222:75ff:fe34:e277(fe80::222:75ff:fe34:e277) from fe80::216:3eff:fea0:3867 eth0: 56 data bytes
64 bytes from fe80::222:75ff:fe34:e277: icmp_seq=1 ttl=64 time=0.000 ms
64 bytes from fe80::222:75ff:fe34:e277: icmp_seq=2 ttl=64 time=0.000 ms
64 bytes from fe80::222:75ff:fe34:e277: icmp_seq=3 ttl=64 time=0.000 ms
64 bytes from fe80::222:75ff:fe34:e277: icmp_seq=4 ttl=64 time=0.000 ms

--- fe80::222:75ff:fe34:e277 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.000/0.000/0.000/0.000 ms

Yes. It seems it does. But hold on, what is this address that the router has given me?

# whois 2001:db8:1:0:216:3eff:fea0:3867/64
% [whois.apnic.net node-4]
% Whois data copyright terms    http://www.apnic.net/db/dbcopyright.html

%WARNING:905: fixed lookup key
%
% The key "2001:DB8:1:0:216:3EFF:FEA0:3867/64" has been changed to "2001:db8:1::/64" for lookup.

inet6num:       2001:0DB8::/32
netname:        IPV6-DOC-AP
descr:          IPv6 prefix for documentation purpose
country:        AP
admin-c:        HM20-AP
tech-c:         HM20-AP
status:         ALLOCATED PORTABLE
remarks:        This address range is to be used for documentation
remarks:        purpose only. For more information please see
remarks:        http://www.apnic.net/info/faq/ipv6-documentation-prefix-faq.html
...

It seems that the Belkin has given me an IPv6 address in a range that should only ever be used as an example in documentation. And if we look at that URL:

… it has been decided to set aside a range of addresses that operators will know should never be routed to the public Internet. The documentation prefix is the IPv6 address range that has been set aside for this purpose.

The 2001:0db8::/32 range should never be allocated to a client! It will never route properly.

Proof!

radvdump is a program that queries IPv6 routers and prints out their configuration. This is the output of radvdump when run on my LAN. You can see the Belkin router’s Link scope IPv6 address in the comments at the beginning, and the bogus 2001:db8::/32 subnet in the prefix section below.

# radvdump -e
#
# radvd configuration generated by radvdump 1.1
# based on Router Advertisement from fe80::222:75ff:fe34:e277
# received by interface eth0
#

interface eth0
{
    AdvSendAdvert on;
    # Note: {Min,Max}RtrAdvInterval cannot be obtained with radvdump
    AdvOtherConfigFlag on;
    AdvDefaultPreference low;

    prefix 2001:db8:1::/64
    {
    }; # End of prefix definition

}; # End of interface definition

Unfortunately in the “shout loudest” stakes, the Belkin router wins over my proper IPv6 router. For every response from my legitimate IPv6 router, the Belkin sent five. The real router seems to send them less frequently the longer it has run too, where as the Belkin sends about one every two seconds.

Conclusion

Therefore, if you are using, or are ever planning to use IPv6 on your LAN, I strongly recommend that you do not buy a Belkin Double N+ router. At best, it will interfere with your IPv6 configuration, at worst, it will totally break it. Belkin do not understand IPv6, and have created a product (possibly a range or products) that will never be useful on an IPv6 network. As it is part of the core network of a LAN, it is impossible to work round it with firewalls or similar.

The specifications of the device makes no mention IPv6 at all, so it should not. Worse, as I personally am using it as an Access point, it should be a pure layer 2 device and not be sending or receiving any layer 3 traffic at all, except for management. There is no work round, no way of disabling this broken IPv6 functionallity.

Belkin’s support

After a few days of exchanging messages with Belkin’s technical support and a few “It doesn’t support IPv6″ denials, I did get a confirmation that there is some IPv6 functionality in the router:

The router does not support IPv6 for WAN connections. However, it does support IPv6 for internal LAN connectivity. There is no option to block this protocol for LAN connectivity.

So that confirms that there is no way to disable this broken implementation. Looks like I shall be getting myself a new access point.

You can manage each machine separately. This soon gets tedious, and prone to errors or inconsistency.

For linux/unix you have NIS or LDAP. For the windows side of things, there are several cludgy hacks you can use (yes Home Server, I include you in this category) or the official way is to run a server with a domain. This can either be an expensive microsoft product, or a free samba implementation of a domain server. Either way you get the benefits of central management across a number of machines.

I don’t include kerberos in the linux/unix list because kerberos is purely an authentication system. You still need some mechanism if identifying user ID numbers with user login details. Basically this means LDAP or NIS.

NIS is very simple, it does basic management, it provides simple redundancy and is light weight. My one big complaint about NIS is that it seems to give away all the user’s password hashes to anyone that asks. This is such a backward step in security. Since linux moved to having the password in /etc/shadow, it has become impossible for anyone except the superuser to read the password hashes.  Someone with a password hash and access to a moderate amount of processing power, or rainbow tables or something can easily nobble several passwords, if not many.

OpenLDAP configuration

The debian default openldap package is a fine starting point. It is configured correctly to serve authentication information. The only thing that isn’t configured is the SSL part. There are many tutorials on how to do this, so I won’t cover this here.

Some things to note:

• Use the debian dpkg configuration to set up the base and administrator account for you LDAP directory.
• Ensure that the nis.schema line is included in /etc/ldap/slapd.conf.
• For large numbers of users, make sure you tune for performance, with indexes etc.
• Check the access lines for your particular configuration.

You need to choose a few things about how your ldap server is to work.

• Is your directory going to be used for other things as well as user authentication? If so, do you want to limit your user accounts to a particular area of the tree?
• Are you going to have different classes of user? There are two ways of achieving this: Separating two different branches of the tree or use of attributes. You can even use a combination of both.

Create any containers that you need for your user accounts:

$ ldapadd -x -D cn=admin,dc=mydomain,dc=com -W
Enter LDAP Password: <enter the administration password here>
dn: ou=users,dc=mydomain,dc=com
ou: users
objectClass: top
objectClass: organizationalUnit

adding new entry "ou=users,dc=mydomain,dc=com"

Create your first user account using ldapadd:

$ ldapadd -x -D cn=admin,dc=mydomain,dc=com -W
Enter LDAP Password: <enter the administration password here>
dn: uid=taskme,ou=users,dc=mydomain,dc=com
uid: taskme
objectClass: top
objectClass: person
objectClass: posixAccount
objectClass: shadowAccount
objectClass: inetOrgPerson
givenName: Task
sn: Me
loginShell: /bin/bash
uidNumber: 1201
gidNumber: 100
shadowMin: -1
shadowMax: 999999
shadowWarning: 7
shadowInactive: -1
shadowExpire: -1
shadowFlag: 0
cn: Task Me
homeDirectory: /home/taskme

adding new entry "uid=taskme,ou=users,dc=mydomain,dc=com"

As you can see, the entry has all the important parts that you can see in /etc/passwd and /etc/shadow:
uid, uidNumber, gidNumber, loginShell, homeDirectory, and cn which matches to the comment field in the password file.

Now we have a suitable user account, that we can configure to appear as an account on our system.

NSS and PAM

There are two parts that need to be configured correctly. The Name Switching Service provides the fields in the passwd file, the home directory, shell, uid number, default gid number etc. The Pluggable Authentication Module layer provides verification of a users identification, ie matching a username and a password.

Install the two ldap packages, libpam-ldap and libnss-ldap. These two are both configured with configuration files, which clearly had the same source. The files (on debian) are /etc/libnss-ldap.conf and /etc/pam_ldap.conf. It may be possible to configure both with the same configuration file, but sometimes it helps to have differences.

For instance, the NSS information is generally requred to be published and there is not much need to encrypt NSS data, as it is easy to determine. Unless of course, you intend to manage it using the rootbinddn parameter, in which case you will be sending the cn=admin password in the clear.

The PAM information should always be kept secret, so all PAM information should be encrypted using ssl.

File configuration parameters

The dpkg configuration basically sets the base and uri parameters in both /etc/libnss-ldap.conf and /etc/pam_ldap.conf. These are not normally enough to configure a working system. As each file has the same parameters and settings, here are some pointers on what to configure:

OpenLDAP by default can be accessed using anonymous binds, so there is no need to set binddn and bindpw.

If you would like the root user to be able to set and reset passwords, set the rootbinddn parameter appropriately. You should only do this on machines that are well trusted, like the server. Anyone that gains access to these credentials can compromise your entire authentication system. So worth considering where to do this carefully. Also, as mentioned above, you should consider SSL if you intend to use this system across your network.

ssl should be set to on if desired, or start_tls if you have configured this.

tls_checkpeer is set to yes by default. This ensures that the client validates the certification authority for the server. Using the ssl authentication mechanism, this ensures that noone can easily impersonate your server.

tls_cacertdir specifies where your certificate authority certificate resides. If you leave this as the default on your debian system, (/etc/ssl/certs), then you may find a lot of certificates in this directory. This makes searching for the correct CA slow, so I would recommend changing this to another directory with only the relevant CA file(s) in.

You may configure nss_base_passwd, nss_base_shadow and nss_base_group to be more specific to reduce search times. These paramaters take the form of an ldap filter.

Finally, pam_password specifies how a new password is encrypted into the ldap directory. I would recommend md5 or similar, this is equivalent to the has used in /etc/shadow.

Next step, is to configure /etc/nsswitch.conf.

There are two basic ways to do this, the simpliest is to add “ldap”  to passwd, group and shadow parameters.

Alternatively you can use the compat mechanism. Simply add three lines:

passwd_compat:  ldap
shadow_compat:  ldap
group_compat:   ldap

This allows the use of + and – in the /etc/passwd, /etc/shadow and /etc/group files.

To add your ldap users you need to put a single ‘+’ in a line on it’s own in each file (no quotes).

For more flexibility, you can have

+taskme, which would just add one of the ldap users to this machine.

Or you can have a ‘+’ line and a subsequent line ‘-taskme’ to remove that particular user from this machine.

Or you can even have lines like this:

+taskme:::::/home/temporary/taskme:

Which would override the value of the home directory from the ldap directory setting to what is specified in the passwd file.

Now, you should be at the stage where you can see your ldap users in the list of accounts.

$ getent passwd
...
taskme:x:1201:100:Task Me:/home/taskme:/bin/bash

All that remains is to configure the PAM side, which is certainly more complicated.

PAM is a wonderful system, which allows you to write arbitrary authentication systems, to allow pretty much any mechanism to authenticate users. This could be passwd/shadow files, or voice recognition, or fingerprint scanners or anything.

PAM also supports resetting passwords.

There are two issues that I see:

1. Messing with the PAM configuration is potentially nasty. You can make it impossible to log in to your machine, or possible for anyone to log in without a password. Any changes you make should be thoroughly tested.
2. Installing random PAM modules is installing some software that you must have complete trust in. If there is any flaw in that particular module, your system may be vulnerable.

On your debian system, you need to change three files in /etc/pam.d, common-auth, common-password and common-account. Before you change these files, make sure you know what you are doing. The settings that result in PAM should allow you to manage both local and ldap accounts without worrying about what kind it is.

common-account:

account sufficient      pam_unix.so
account sufficient      pam_ldap.so

common-auth:

auth    sufficient      pam_unix.so nullok_secure
auth    required      pam_ldap.so use_first_pass

common-password:

password   sufficient   pam_ldap.so
password   sufficient   pam_unix.so nullok obscure md5

So, with these changes you should be able to authenticate your user:

Except that your LDAP user doesn’t have a password yet.

If you have configured the rootbinddn parameter, and it is set correctly, you need merely type passwd taskme as root on the appropriate system.

If you have not done so, you can set the password like so:

$ ldapmodify -x -D cn=admin,dc=mydomain,dc=com -W
Enter LDAP Password: <Enter your admin password here>
dn: uid=taskme,ou=users,dc=mydomain,dc=com
changetype: modify
replace: userPassword
userPassword: wibble

modifying entry "uid=taskme,ou=users,dc=mydomain,dc=com"

Of course, this puts a cleartext password in your database. This is probably a bad idea. Alternatively you can use the slappasswd utility to create a hash of the password, then put this into the ldap directory.

Interestingly, ldap supports multiple values for userPassword. So you could have more than one valid working password for your account. If you are prepared to manage it at the ldap layer.

To confirm it all works:

ssh localhost -l taskme

Advanced Configuration

I like to configure any process I run so that it doesn’t allow anyone outside of my LAN to get any data from it. So even if my firewall fails, then noone can abuse my server from the internet.

So, to the slapd.conf file, I changed the default access line to the following:

access to *
        by dn="cn=admin,dc=my,dc=com" write
        by peername.ip=192.168.0.0%255.255.255.0 read
        by peername.ip=127.0.0.1 read

This ensures that anyone that connects via the internet will not be able to retrieve any data.

However, as a normal user, I was not seeing any of the ldap accounts. Only the root user could see them.

This seemed very peculiar to me. I was unable to determine why this was the case.

This was until I looked in the /etc/hosts file:

127.0.0.1       localhost
127.0.1.1       myhost.mydomain.com        myhost

Debian configures the host file so that the name of the machine has IP address 127.0.1.1, which is fine as all 127.* address connect to the localhost.

Except that my access list is only expecting connection from 127.0.0.1, so refused to serve the user data.

To fix this, I simply had to change the last line of my access list to:

        by peername.ip=127.0.0.0%255.0.0.0 read

And all was as expected.

Congratulations. You have now configured your system to authenticate your ldap users as local users.

One word, migration.

Xen has this capability of being able to move a virtual image from one machine to another. Usually this is so quick that users of the virtual image do not even notice that anything has changed.

There are a few caveats:

• The two machines must be on the same LAN segment.
• The two host machines must be the same architecture.
  I am not sure exactly how far this extends, but my experience is that transitions from intel based hardware to AMD hardware doesn’t work. As well as the more obvious transition between 32 and 64 architectures. This is a shame because many other virtualisation systems can do this. And, being a home user, I have a mixture of ages, types and machine architectures. See here too.
• The two host machines must both be running xend.
• The target machine must have enough physical resources (RAM) to run the image.

Another other pain is that the XEN dom0 patch is incompatible with the nvidia hardware 3d acceleration driver. This is an issue because my newest and fastest machines both run the nvidia driver. They also have more memory than my servers. It would be nice, if I needed to run a long job on a virtual machine, to pop it onto my fastest machine, and let it burble away there for a while. But this is not possible if you are using the nvidia driver.

In order for XEN to migrate, the domU has to be able to access it’s disk drives from any of the servers it might be migrated to. This can be achieved with fancy SAN based infrastructure, such as fiber channel, or similar. Or by using the network. There are three ways the network can be used:

1. Root on nfs
2. network block devices
3. iscsi

Because root on NFS has drawbacks in performance and speed, and nfs mounted filesystems are unsuitable for certain tasks, my choice was between NBD and iscsi. NBD have been around for ages, and although mature, they are specific to linux, and have pretty much been superseeded by iscsi.

Iscsi is fairly new, but the latest debian has support for both initiators (the equivalent of a SCSI card) and targets (the equivalent of a SCSI disk) ISCSI brings the kind of flexibility of fiber channel systems to any system with a network card, though not the performance.

These instructions are based on the debian lenny distribution, with extensive references to here and here. They assume that the XEN server will also be the ISCSI target server. If this is not the case, and extra step will need to be inserted to copy the raw disk image onto the ISCSI target. The Lenny installer does not currently support installation directly onto an ISCSI target. However, I am sure this will come soon. The stock kernel supports ISCSI fully, it would just need some tweaks in the installer.

On the dom0 server, create an LVM object big enough to represent the whole disk space needed by your virtual machine. This will be the virtual disk for the domU.

lvcreate -L10240 -n iscsitest system

Download the xm-debian.cfg into /etc/xen. Copy it to a new name, say xm-iscistestinstall.cfg and edit it for your needs, particularly:

memory=128
name="iscistest"
vif=['mac=00:16:3e:1a:2a:2a']

(If you’re using dhcp to provide IP addresses to your hosts, fixing the MAC addresses means that it is more likely that your virtual server will receive the same IP address each time it boots. The first half of the MAC address: 00:16:3e  identifies the address as belonging to a Xen virtual machine. You can put whatever you like in here, so long is it doesn’t clash with any other MAC addresses on your LAN. This example assumes you are running dhcp)

disk = ['phy:/dev/mapper/system-iscsitest,xvda,w']

Identify your logical volume as the main disk for your new machine. We will ultimately remove this line, but for the install, it is needed.

Now run xm create with the installer to create a bootable xen virtual server:

xm create -c xm-iscsitestinstall.cfg install=true
   install-mirror=http://ftp.uk.debian.org/debian install-extra="clocksource=jiffies
   priority=low" install-suite=lenny

See the debian install guide for how to install lenny.

Some notes to bear in mind when installing:

• When you get to partitioning the disks, I would recommend using a single LVM partition. This is because we will change the underlying physical device name of the disk. LVM will identify the volume group from either the iscsi target, or the xvda drive, meaning that we don’t have to modify /etc/fstab each time we flip between the two devices. The same behaviour could be achieved using volume labels, if you are not so inclined to use LVM.
• Make sure you select a kernel image with “xen” in the name. And choose the “generic” option for initramfs building.
• When given the tasksel option, you can choose any configuration you want. If you select the minimum (ie no groups selected) then it will work, but you will need to add some specific packages to get everything to work.  I will identify any special packages you need during the process.
• There is no benefit in installing a boot loader (grub or lilo). So choose the “do without a bootloader” option.
• You will need to add some additional packages by hand, before you finish the installation:

Important: Before you “Finish the installation”, choose Execute a shell: You must remove the network-manager package and anything that depends on it, as it tries to manage the network connection for you, which is a bad idea when your root filesystem is only available when the network is up.

chroot /target
aptitude install openssh-client rsync open-iscsi libc6-xen
aptitude remove --purge network-manager
cd /boot

Copy the kernel and ram disk onto the xen dom0 server. Otherwise it is not possible to boot the virtual machine.

rsync vmlinuz-2.6.26-2-xen-686 server:/etc/xen

(replace “vmlinuz-….” with the kernel name and “server” with the name of your xen server)

rsync initrd.img-2.6.26-2-xen-686 server:/etc/xen/

Now exit the shell

exit
exit

And choose “Finish the installation”

Now, copy the xm-iscsitestinstall.cfg file to a new name, say xm-iscsitest.cfg and make the following edits:

Remove all the setup sections, pretty much everything before “memory = 128″ and everything including and after “# Debian Installer specific variables” is redundant.

Add the following lines:

kernel="/etc/xen/vmlinuz-2.6.26-2-xen-686"
ramdisk="/etc/xen/initrd.img-2.6.26-2-xen-686"
extra="root=/dev/mapper/itsystem-root ro console=hvc0 clocksource=jiffies"

The “clocksource=jiffies” is very important. I will explain it later. Leave everything else as is.

The following should modify your virtual machine so that it will run with iscsi as the filesystem.

On the xen dom0/iscsi server, install the iscsi target modules and management binary packages:

aptitiude install iscsitarget-modules-2.6.686 iscsitarget

(Remembering to replace the iscsitarget-modules with one relevant for your kernel)

Edit the file /etc/ietd.conf, add the following lines:

Target iqn.2001-04.com.example:storage.lun1
 Lun 0 Path=/dev/mapper/system-iscsitest,Type=fileio
 Alias LUN1

And restart the iscsi target daemon:

/etc/init.d/iscsitarget restart

Run up the virtual machine:

xm create -c xm-iscsitest.cfg

Now your virtual machine is running, but it is still using the /dev/xvda device for its drive.

iscsiadm -m discovery -t st -p <ip address of server>

And connect to the device:

iscsiadm -m node --targetname "iqn.2001-04.com.example:storage.lun1"
  --portal "<ip address of server>:3260" --login

ls /dev

Should show a /dev/sda and /dev/sda1 … devices. Don’t try to access them, however, as you already have them open via the xvda device, and changing them in any way could cause data corruption.

Copy the file:

cp /usr/share/initramfs-tools/scripts/local-top/lvm2 /etc/initramfs-tools/scripts/local-top/lvm2

(The /etc version of this fill will not be overwritten by updates to lvm, whereas the /usr version will)

Edit /etc/initramfs-tools/scripts/local-top/lvm2

Add a line, towards the end of the file, between the lines as shown:

...
modprobe -q dm-mirror

sleep 5 # added by taskme lvm needs to settle before mounting volumes on iscsi

activate_vg "$ROOT"
...

Modify/etc/init.d/open-iscsi

The line in the stop() function stoptargets, comment it out:

...
stop() {
 # stoptargets     # commented out by taskme see bug #501580
 log_daemon_msg "Stopping iSCSI initiator service"
...

This is referred to on debian bug #501580

Now, create a file /etc/iscsi/iscsi.initramfs

With contents:

ISCSI_TARGET_NAME=iqn.2001-04.com.example:storage.lun1
ISCSI_TARGET_IP=<server ip address>

These lines tell the kernel to include the iscsi target code for a root file system into the initial ram disk.

Now, with this setting added, recreate the initial ram disk. This will also add our modification to /etc/initramfs-tools/scripts/local-top/lvm2

dpkg-reconfigure linux-image-2.6.26-2-xen-686

Copy the new initial ram disk onto the server

rsync /boot/initrd.img-2.6.26-2-xen-686 server:/etc/xen/

Finally, the virtual machine now configures the interface from the kernel. Any settings in /etc/network/interfaces seem to muck up the network conectivity of the machine, so comment out all lines in this file that refer to eth0:

# The primary network interface
#allow-hotplug eth0
#iface eth0 inet dhcp

Shutdown the virtual machine:

shutdown -h now

If your dom0 server and iscsi target server are different machines, now is the time to copy the /dev/mapper/system-iscsitest onto the iscsi target, ensuring it is the correct size etc. In theory it would be possible to have the iscsi target made available to the virtual machine using an iscsi initiator on the dom0 machine. An exercise for another day, perhaps.

On the dom0 server, modify /etc/xen/xm-iscsitest.cfg, comment out the disk line:

#disk = ['phy:/dev/mapper/system-iscsitest,xvda,w']

And restart the virtual machine:

xm create -c xm-iscsitest.cfg

And hopefully, your server will be running. Look in /dev to ensure that xvda does not exist.

To allow your server to migrate, it is necessary to modify /etc/xen/xend-config.sxp on the dom0 server. Uncomment the following line:

(xend-relocation-server yes)

On both the main dom0 server and the dom0 server on which you wish to migrate the machine to. But be careful, as this may potentially open up a security hole on your servers.

Restart xend on both machines.

/etc/init.d/xend restart

Now, you can migrate your virtual machine. The second server has to be on the same LAN segment as the master server.

xm migrate iscsitest <ip address or hostname of second server>

Users of the virtual machine should notice a very brief pause as the server migrates (basically the time it takes to copy the virtual RAM of the server across your network). The -l options should eliminate any noticable pause, though I found this didn’t work as reliably.

The reason behind clocksource=jiffies

This setting is vitally important on debian systems running with root filesystems on iscsi. This is because the TCP sequence numbers are determined by the system clock. The bug in the lenny xen implementation means that the domU clocks can get out of sync with the dom0 clock and go backwards. This means that your TCP sequence numbers are invalid, and the TCP connection fails. You therefore loose your root filesystem. This setting de-couples the virtual machine system clock from the dom0 sufficiently that the clock does not run backwards, and your TCP connections keep working.

This strikes me as a very elegant solution. The firewall would be almost un-hackable, as anyone attacking from the internet would only be able to attack at layer 3, but as the firewall is essentially invisible at layer 3, it would be difficult to compromise it, without first compromising a machine on the section of ethernet. The machine doesn’t even have to have an IP address, and it should still work.

I am aware that a few people tout such a system as a good idea, but I have been unable to find much detail on any solutions. Still, fewer people are using virtual hosts for firewalling.

But, would it even be possible to run a layer two machine within a virtual environment? Enter the twisty turny world of ethernet bridging, virtual interfaces and arp packet mangling.

Would it be possible to do NAT? This would be ideal as I can have a clean break across the firewall, with most of my LAN hosts unaware that they have internet routable addresses. We shall see.

This example is based on a xen virtual server, but the details apply equally to any machine with two network interfaces (be they physical or virtual)

Perform a basic debian install. Only install the base packages plus ebtables and arptables. I also installed tshark for diagnostics and less for my own sanity.

Modify /etc/network/interfaces.

The debian mechanism for managing the network interfaces has always worked very well for me. It is a constant pain that although the underlying tools are the same, ie. ifconfig(8), and now ip(8), each linux distribution uses a completely  different method of setting permanent values for the network settings. The redhat distributions use a script in /etc/sysconfig/network-scripts for each of the permanent interfaces. Slackware just has an init script with a load of ifconfig lines, although slackware is always about 10 linux years behind everything else. Even solaris is so different from anything else, trying to move between the different systems is a headache. Debian’s system although it is unique to debian, (and presumably debian based distributions) seems to sit at a nice point on the complexity/usability curve where other systems end up too far one way or the other. I particularly like how everything is in a single file. And how, you can fix an interface to be a particular MAC address even if the interface isn’t brought up automatically by the system. Very useful for virtual interfaces such as used by openvpn. Whenever I have thought that interfaces(5) wouldn’t be able to achieve what I need, and I would have to start messing with rc scripts and hard coded calls to /sbin/ifconfig, it has always pleasantly surprised me. Today was no exception.

After the install, the stanza for the active interface will probably look like:

auto eth0
iface eth0 inet static
 address XXX.XXX.XXX.65
 netmask 255.255.255.0
 broadcast XXX.XXX.XXX.255
 gateway XXX.XXX.XXX.1

or

allow-hotplug eth0
iface eth0 inet dhcp

Replace the static or dhcp with “manual”. Now, manual doesn’t mean “ignore the interface during the boot sequence, I will bring it up by hand”, as you might think, but it means that you will supply suitable “up” directives to configure the interface “manually”. The former explanation would be redundant within the context of the interfaces mechanism.

For the two interfaces apply the following settings:

auto eth0
iface eth0 inet manual
 up /sbin/ip link set eth0 up
 up /sbin/ip link set eth0 arp off
 down /sbin/ip link set eth0 down

auto eth1
iface eth1 inet manual
 up /sbin/ip link set eth1 up
 up /sbin/ip link set eth1 arp off
 down /sbin/ip link set eth1 down

The interface in an up state will allow traffic to cross it, even if the interface doesn’t have an IP address. Remember that ethernet is just a layer two protocol, and that IPX, decnet and other layer three protocols can all share the same network with IP.

ip(8), is the new all singing all dancing version of ifconfig/route. It is supposed to be the way forward in configuring your linux interfaces. It is very different from ifconfig, but is supposed to be a single executable that can be used to configure network interfaces, addresses, masks, routes etc. It is certainly more precise than ifconfig, which tends to leave interfaces in unexpected states (ifconfig down not work for anyone else?). However, everyone is supposed to start using it as one day ifconfig might just top working!

In addition, add a bridge interface. This causes the bonded ethernet interfaces to work like two ports on an ethernet switch, but of course, since the etch release of debian, you can apply iptables based rules to the traffic.

auto br0
iface br0 inet manual
 pre-up /usr/sbin/brctl addbr br0
 pre-up /usr/sbin/brctl addif br0 eth0
 pre-up /usr/sbin/brctl addif br0 eth1
 up /sbin/ip link set br0 up
 down /sbin/ip link set br0 down
 post-down /usr/sbin/brctl delif br0 eth1
 post-down /usr/sbin/brctl delif br0 eth0
 post-down /usr/sbin/brctl delbr br0

With this configuration, the virtual machine successfully routes ethernet packets. And I can use iptables to block or permit certain host/port combinations. But I am unsure of the next step:

Now, although I have successfully used a bridging firewall, I have no idea whether a bridging firewall is capable of performing NAT. In theory it should be possible, but theory and practice are often far removed.

I perceive two steps:

1. The layer three packet manipulation, which is using the SNAT and DNAT rules in iptables -t nat.
2. The ARP manipulation. ARP is an interesting protocol, as it sits somewhere between layers two and three. In fact, I have seem some call it layer two, and others call it layer three. In truth, it is neither, and both. It sits between the two layers, and mediates information between both the layer three level and layer two.

The layer thee issue is fairly well established. I have used NAT for a number of years. The ARP layer, I am going to have to learn!

As I understand ARP, it allows devices on an LAN to correlate layer three addresses (IP addresses) with layer two addresses (MAC addresses). Each device maintains a look up table of IP/MAC addresses, and each address stays valid on that table until about 30 seconds after the last packet was exchanged with that host.

There are two basic type of arp traffic, the “Who has” packet, and the “is at”.

The “Who has” packet is target at the layer two broadcast address. It basically says. I have IP address X.X.X.X and am looking for a host with IP address Y.Y.Y.Y.

The host with address Y.Y.Y.Y then responds with a directed response to say Y.Y.Y.Y is at MM:MM:MM:MM:MM:MM (layer two address). An ARP exchange as recorded by tshark:

163045.629003 4a:5a:6a:1a:2a:3a -> Broadcast    ARP Who has Y.Y.Y.Y?  Tell X.X.X.X
163045.629003 4b:5b:6b:1b:2b:3b ->  4a:5a:6a:1a:2a:3a ARP Y.Y.Y.Y is at 4b:5b:6b:1b:2b:3b

Crossing a bridging/NATting firewall, the layer three addresses are different on one side to the other. Therefore, the Y.Y.Y.Y and X.X.X.X need to be translated when they cross the firewall.

This, it turns out, is fairly straight forward.

arptables -A FORWARD --source-ip IX.IX.IX.1 --destination-ip IX.IX.IX.2
    -j mangle --mangle-ip-s NX.NX.NX.254 --mangle-ip-d NX.NX.NX.1
arptables -A FORWARD --source-ip NX.NX.NX.1 --destination-ip NX.NX.NX.254
    -j mangle --mangle-ip-s IX.IX.IX.2 --mangle-ip-d IX.IX.IX.1

Using this, I successfully see incoming ARP requests being correctly translated. The router’s arp table contains the correct MAC address from the host, inside the firewall.

This, however, is where it goes wrong.

The Layer three part of netfilter seems not to be able to translate the address correctly. The following rule is based on a rule that I have used for many years to modify an incoming packet to make it appear to be destined for a private address.

iptables -t nat -A PREROUTING -d IX.IX.IX.1 -i eth1 -j DNAT --to-destination 1.2.3.4

As the packet crosses eth1, the external interface, it should be translated to appear as 1.2.3.4. Unfortunately this does not happen, and you see pakets on the LAN within the firewall with the external to address:

163045.629003 IY.IY.IY.IY -> IX.IX.IX.1 TCP 42417 > ssh [SYN] Seq=0 Win=5840
    Len=0 MSS=1380 TSV=75548755 TSER=0 WS=7

Of course, there are no machines with that IP address on that LAN segment, so the target machine doesn’t respond. One clue might be the chain name “PREROUTING”. The kernel may do the address translation as part of the routing stack. Being a layer two firewall, it is not responsible for routing.

To find out what is wrong, and if it can be fixed, I will have to look at the kernel source and talk to the netfilter people.

The problem with my current server has basically been that it has been added to, layers upon layers of software, java, stuff for viewing and processing digital photos, mplayer, and pretty much anything that I felt was a good idea at the time. There is a whole load of stuff on there, that will probably never again be used, but is occupying disk space, presenting security vulnerabilities, and generally making a messy working environment.

Ideally I would want a series of servers that I could leave well alone, save for security updates and a few tweaks. One server that manages the user accounts, one that has all the user files. An internet facing server, that can run email and web, and just that. A separate firewall. These machines would not provide shell access. Then in contrast, one or more general servers, that I can process digital pictures on, add software on to evaluate, or batch process a load of mp3s or whatever. The general purpose server, can almost be throwaway, in that it could be re-created every so often, maybe even run from a LVM snapshot. The throwaway servers , I am less worried about keeping clean and secure. They may not even be running most of the time, so would not present a security hazard.

However, the overhead of running a whole lot of machines for each of these tasks would take up a lot of space, use a lot of electricity and be expensive to buy.

Step in Xen.

I have wanted to try XEN out in a “production” situation for a while. But my 24/7 server had a paltry 640Mbytes of memory, and the one thing you really need for virtualisation is a shed load of memory.

On acquiring a new server with 3 Gibibytes of memory,  the vast possibilities of all that memory opened out seemingly endless vistas of virtual servers. I could create nearly 4 virtual machines with the equivalent of my old server plus one with 512M.

Fortunately, some kind debian kernel developer managed to port the XEN hypervisor code to the 2.6.26 debian kernel in lenny. It was a late addition to the code base. Xen only officially support 2.6.18 or 19, which is old by any standards. Except for some issues with system clocks, the port has worked well for me. Well done debian.

But most of the virtual images that I am likely to run would fit into significantly less memory than the 640 Mibibytes of my old server. On my old home server the largest executable running is a mere 72 Mibibytes. That is squid, and it is only using that much memory because I told it it could.

Even apache v2′s various threads are only using a total of 130 Mibibytes, and, again, that is only because I said it could.

The more heavyweight users, that are less easy to deal with are:

• clamd 68 Mibibytes (ouch)
• Gnome-terminal uses 11 Mibibytes, compared with xterm’s 2.6 (!)
• Iceweasel (firefox for debian users) weighs in at 26 Mibibytes
• etc.

In short, if I were to be selective about what to run on my virtual machines, I could probably get away with much less than the 640 Mibibytes that my old server uses. Even if I made a bad call, a huge advantage that Xen has over other similar virtualisation solutions is that you can dynamically re-assign memory, and even processors. The virtualised servers will hardly notice!

Although this description does concern the building of a firewall virtual server, I have left most of the firewall configuration detail for the next part of the series. Most of the instruction and information concerns what would need to be done to build any virtual server. All that would need to change is the config file, and the virtual disk.

The new server has two onboard network interfaces. One Gigabit, sutiable for connecting to the LAN, which at least partially runs at gigabit speed. The second is a Fast Ethernet interface, which is sutiable for the WAN connection.

When Xen starts, on the dom0 host, the one with the phyical ethernet cards, the default behaviour, is to take the default ethernet interface, eth0, and rename it to peth0. (the P standing for physical). Then it creates a virtual interface vif0, and creates a bridge consisting of peth0 and vif0, which it calls eth0. This is done for a few reasons:

1. The user doesn’t see the default network interface name change.
2. Any firewall rules that apply to eth0, do not effect the virtual machine.
3. If the user is not terribly familiar with ethernet networking, bridging etc. it is a path of least resistance. Most people will not come unstuck unless they know what they are doing.

This article on the xen.org website has a lot of helpful information on how networking works with Xen. In fact, it seems to suggest that you can create just about any kind of virtual network within a set of xen virtual machines that you like. And it will just work. This gave me hope that what I was trying to do would work.

At this point, I feel I should point out that this is not a high performance configuration. If you are working with an enterprise, running a Class-A subnet from multiple gigabit internet connections, the number of layers and bottle knecks will almost certainly cause all you expensive bandwidth to ebb away. For a home or small business configuration with a fairly small 8 Megabit ADSL connection, the extra delays and processing effort is negligible. I would say, that if you are trying to serve more than one broadcast domain (and cisco say a layer 2 broadcast domain should be smaller than 500 machines, I say, probably smaller that that), then this solution isn’t for you.

Working from a minimal debian install, with plenty of available disk space, install the xen-linux-system package for your architecture. This pulls in the hypervisor and the relevant linux image. and the xen-utils package. It is also worth getting the libc6-xen version of glibc, as it is optimised to work better on a xen system. In the file /etc/xen/xend-config.sxp, add the following line (somewhare around the network configuration part):

(network-script network-bridge)

Reboot, and your system should restart with the xen hypervisor lurking in the background.

First up, create a logical volume on your storage. I have a single volume group, generally called “system”. Perhaps I am odd. I put everything except /boot into a logical volume. It means you can almost completely forget about partitions, and using some LVM jiggery pokery, resize any partition if you find you made some bad calls early on.

Because this machine will be the firewall is basically a very small debian install, I have allocated it 2 Gigabytes of disk space:

#lvcreate -L2048M -n firewall system

This should be more than enough for the base install, and a few firewalling applications.

Building the virtual server.

A really handy guide to doing this is the debian xen wiki. The debian installer now supports installing into a virtual server, and it is pretty much as easy as installing onto a standalone machine.

You basically download an example xm-debian.cfg file, edit it a bit, to identify block devices, networks, memory etc. and run it with a install options.

Of interest, the changes made to the .cfg file:

memory = 128
name = "firewall"
vcpus=1
vif = ['mac=00:16:3e:42:39:18, bridge=eth1', 'mac=00:16:3e:42:39:19, bridge=eth0']
disk = ['phy:/dev/mapper/system-firewall,xvda,w']

I also modified the bootloader line to have the full path to pygrub, otherwise the system doesn’t properly find pygrub, and it is impossible to boot your new instance.

bootloader="/usr/lib/xen-3.2-1/bin/pygrub"

Then, simply invoke the xm create command with the appropriate switches:

xm create -c xm-debian.cfg install=true install-mirror=http://ftp.uk.debian.org/debian install-extra="clocksource=jiffies \
  priority=low" install-suite=lenny

Of importance are the “clocksource” line, which is fed to the kernel. Without this you may get the dreaded “clocksource/0: Time went backwards” error. Although the debian wiki says that you need to add some other bits, I didn’t find them necessary. In fact, independent_wallclock means that if you reboot the dom0, or pause your domU it will come back with a slow system clock. Most unsatisfactory. If the error exceeds an hour, then ntp will not correct the clock. The clocksource jiffies setting seemed to solve the problem for me, and it can even be applied at run time.

Priority=low is the expert install mode. You may not wish to use this.

The standard debian installer starts, and you can configure your debian in the usual way.

If you are not familiar with installing debian, have a look at the reams of documentation available on the debian.org website.

If you use LVM in the domU, make sure that you use a different volume group name than that in use by the dom0 (I used fwsystem). This allows you to perform low level maintenance on the domU filesystem from dom0. Maybe more on how to do this in a future different post.

Make sure you install grub. Although you can’t actually boot from the virtual machine using grub, pygrub uses the menu.lst file that grub creates within the domU filesystem. This is elegant, as kernel updates can be applied within the virtual server, and not need manual intervention to copy kernel images and initial ram disk images onto dom0.

For this installation, you have, of course, to allow the firewall access to the interweb. This is obtained by simply configuring one of the interfaces and allowing it to retrieve the necessary install files.

In addion to the base install, I installed tshark, bridge-utils, ebtables and arptables. No other optional packages were installed. Once the installation has completed. The xen instance will stop. It can be restarted, in normal mode with the following command:

xm create -c xm-debian.cfg

You can, of course, trim the file to suit your needs. There is no need for the install cruft that is in there to stay there. However it will work on a day to day basis.

You should see the boot process and end up with a console prompt.

Add the text “clocksource=jiffies” to the default kernel options in /boot/grub/menu.list, and either reboot or:

echo "jiffies"> /sys/devices/system/clocksource/clocksource0/current_clocksource

Debian, by default, will start any virtual machines that were running when the dom0 was shutdown. It saves a state in /var/lib/xen/save (make sure your /var partition is big enough for all your virtual machine’s memory images) and will restart any images that it finds in there.

To ensure that the virtual machine starts every time the dom0 boots, even if the dom0 was shutdown ungracefully, copy or (more properly) link the .cfg file into /etc/xen/auto/

You can of course reboot the dom0 before many network connections are dropped from the domU instances. For example a top(1) process can be running in a terminal window, via a ssh session. The display output will pause while the dom0 is rebooted, and then resume exactly where it left off. Very cool.

Next time, part 3: Configuring a layer two internet firewall.

Mrtg has two major drawbacks:

1. Vanilla mrtg can only monitor integer values
2. It is designed to work with two values only – in and out.

cacti can do all that mrtg can, and much more. It is an absolute pig to configure, lots of non-intuitive settings, little logic, poor defaults. But it can make some nice graphs, with floating point values, and with colours, with a very high level of customisation.

With lm-sensors and apcupsd, not only can you monitor your network, but you can also monitor voltages, temperatures and much more.

Cacti can be extended by using your own scripts. If you can write a script for it, you can monitor it.

Eg:

#!/usr/bin/perl
# Display UPS data
#

@collect=("LINEV","LOADPCT","BCHARGE","TIMELEFT","MAXLINEV","MINLINEV",
	  "OUTPUTV","ITEMP","BATTV","LINEFREQ","LOTRANS","HITRANS");

foreach $_ (`/sbin/apcaccess status`) {
  ($line, $value) = split(/:/);
  chomp($value);
  foreach $val (@collect) {
    if ( index($line,$val) == 0) {
      @number = split / +/,$value;
      print "$val:$number[1] ";
    }
  }
}
print "\n";

Typical output is:

LINEV:240.5 LOADPCT:18.7 BCHARGE:100.0 TIMELEFT:73.0 MAXLINEV:241.8 MINLINEV:239.2 OUTPUTV:240.5
   LOTRANS:196.0 HITRANS:253.0 ITEMP:37.8 BATTV:55.6 LINEFREQ:50.0

which is basically a load of name value pairs. Cacti calls this script every five minutes, and extracts the values from the string, and stores them in a round-robin database.

The different graphs are generally interesting over different time periods. Temperatures show daily and yearly fluctuations. Network usage seems pretty much random, although you can identify big downloads months afterwards.

Some interesting graphs are included below…

For example, free space on my /home drive:

home space usage

As you can see, I keep my home space small. This allows me to back it up relatively easily, although, it is quite difficult to keep it so low. About a month ago, I gave up and added the rest of the available space on my MD RAID partition. It seems to bob along about 5 Gigabytes free, just enough space to download a knoppix DVD image.

Another interesting one is the mains voltage.

Mains voltage over two years

Daily fluctuations in mains

The first graph is moderately interesting because of the step at the end of January. The second is less interesting but  you can see how much it varies in a day.

For a few years, my UPS kept tripping out in the winter with “over voltage”, after some research I discovered that the electricity board have to provide electricty at 230v + 10% or -6%, the over voltage switch over for my UPS was at the same point that the incoming electricity became illegally high – 253 volts. (the top red line on the graph)

It may have been that my UPS was a bit over sensitive. The graphs only show average voltage for the time, not peak voltage, which is why they don’t appear to cross the upper red line limit.

Although the UPS was protecting my IT hardware, and some other bits and bobs, I felt that the constant tripping of my UPS would be reducing its life, as it is going onto battery several times a day. Any equipment not protected by the UPS would also be vulnerable to over voltages, so I contacted the electricity company. They installed a line monitor for a week, and confirmed that the voltage had gone over 252v twice in that period. So my UPS, despite complaining 4-6 times a day, did have a valid reason for complaint.

The downward step was caused when the electricity company dropped the local voltage my moving the supply tap one loop on the sub-station transformer, thus reducing the local voltage. Although I thought electricity in the UK had to be supplied at 230v, most places, it seems, are still configured to run at the traditional 240v, despite the official change being made over 15 years ago.

As you can see, the voltage dropped by nearly 8 volts, and my UPS stopped complaining, so a positive result, and proof that you can get things changed for the better. Complaining works!

From my CPU fan speed monitor, you can see when I clean out the case.

Fan speed over two years

December, a year and a half ago the fan was so choked up, it was starting to fail. I was expecting to have to replace it but the clean out revitalised its fortunes. Next time, the following October, you can see a step as the fan turned more easily with less dust in it. I try to clean out the machine at least once a year.

The final graph is the 12 v graph. It is quite boringly flat. I find it impressive that that power supply that must be over 8 years old, in use 24/7 has managed to supply a such a consistent voltage for at least 2 years.

Constantly boring

I have had a home server since before 1998. This particular one has survived, more or less unmodified since 2001, when it upgraded a Pentium 90. My home servers always run a version of Debian. With each passing Debian version, Woody, Sarge, Etch, each time a new stable arrives it is upgraded. I expect the root file system is a very old version of ext3. The hardware was pretty much End of Life when it was bought, so I think I got my moneys worth. Its main responsibilities are:

1. Internet firewall and transparent squid proxy.
2. DHCP, DNS services for the LAN.
3. SMTP/IMAP email – IMAP is great because you get the same view of your email from any machine running any operating system.
4. NFS and SMB for file serving. Again, it is really useful that you can access the same files from any machine and any operating system.
5. Apache for miscellaneous files at work, and the fantastic squirrelmail
6. Recently, openvpn, for secure access from my laptop.
7. Any generic shell/processing that needs to be done.

As it has seemed to be slower and slower, I have decided to upgrade it. Retrofitting LVM and addional memory has, to a certain extent, extended it’s life, but certain things seem to run so slowly on it, particularly virus/spam scanning, and it’s memory, being the most obscure RIMM memory, means that it is beginning to seem short of it. Backing RIMM was intel’s biggest gaff of the late 1990s.

Specification:

1. Early 1.4 GHz P4 (PGA423)
2. Asus P4T motherboard
3. Memory – 2 x 64Mbyte RIMM plus 2 x 256 RIMM (total 640 Mbytes)
4. Disks – 1 x 160Gbyte IDE, 1 x 160Gbyte SATA, 2 x 250 Gbyte SATA (SATA drives on an IDE controller)
5. Video Nvidia TNT 2 with S-Video output, particularly useful as it means I can plug it into the TV rather than dragging a monitor around.

With my budget of £0 I had to rely on alternative means to obtain an updated system. As a member of a local linux user group I recently acquired a couple of old servers. These are 1U supermicro Xeon based systems, about 5 years old. Not the latest and greatest but certainly a reasonable upgrade from before. Specification:

1. 2 x 2.4 GHz hyperthreading Xeons
2. Supermicro P4DPL motherboard
3. Memory 6 x 512 ECC SDRAM – total 3GBytes (wow, so much memory)
4. Drives 180 GByte IDE (but will also get the current server’s drives)
5. Onboard RAGE XL Video, not so convenient as a card with a S-Video output. But with monitors being smaller and easier to lug around, I think I can live with it.

Unfortunately, being 1U and without dynamic cooling management, these boys are loud. Think along the lines of a 747 cranking up. Although they would fit right at home in a server room, they are not well suited to home use. 1st job, therefore, to find a quieter solution.

This means, an alternative case, and replacement CPU coolers (the stock ones are simple finned heat syncs and centrifugal fans – that look a bit like a snail. They are stuck in front of the CPUs, separate from the motherboard, to keep the height within the confines of a 1U case but they are noisy!)

Being servers, the motherboards are E-ATX, and are nearly 13″ wide. This is an issue for most midi cases, as the CPU coolers would not fit behind the drive bays. However, another member of my local linux group was giving away an old full tower case, which, also supermicro, had enough clearance to fit the motherboard in.  And a quick visit to ebay found me some PGA603 coolers for £10 each. Ok so that is £20 over budget, but a small sacrifice.

A spare 450W ATX PSU and we’re in business. And considerably quieter to boot.

Next time: Part 2, XEN and the art of virtualization.