Tomasz Korwel
programmer, administrator, engineer – my everyday fights with reality

Today when I woke up my MacBook greeted me with very unpleasant alert:

The network backup disk does not support the required AFP features.

It turnes out that during the upgrade to Lion Apple used their newest set of commands that were not compatible with stable netatalk package in Ubuntu.
To get my network storage to work again I had to upgrade to the latest netatalk 2.2.beta4.

First download all needed files from Ubuntu’s repositories

wget http://launchpadlibrarian.net/73070555/netatalk_2.2~beta4-1_amd64.deb
wget http://launchpadlibrarian.net/74978789/libgcrypt11_1.5.0-1_amd64.deb
wget http://launchpadlibrarian.net/75629511/multiarch-support_2.13-9ubuntu3_amd64.deb
wget http://launchpadlibrarian.net/72120162/libgpg-error0_1.10-0.3ubuntu1_amd64.deb
wget http://launchpadlibrarian.net/74358655/libdb5.1_5.1.25-11_amd64.deb


Then install it:


dpkg -i multiarch-support_2.13-9ubuntu3_amd64.deb
dpkg -i libgpg-error0_1.10-0.3ubuntu1_amd64.deb
dpkg -i libgcrypt11_1.5.0-1_amd64.deb
dpkg -i libdb5.1_5.1.25-11_amd64.deb
dpkg -i netatalk_2.2~beta4-1_amd64.deb


I’ve chosen to keep old version of configuration files, the only notable difference was that now afpd.conf lists -tcp -noddp options as defaults. I added them to my list. Quick restart of netatalk daemon and voila – Time Machine works again.

I am a fan of Apple’s Time Machine backup system since it’s introduction. And I’ve always wanted to implement something similar on my server. Since I have that nice spacious NAS disk right now space stopped to be a problem (at least for a little while). Little googling shows that rsync has a special option allowing me to implement Time Machine’s method of incremental backups using rsync. In that method rsync uses existing backup as additional source for comparing files and if file did not change since last backup has been done a hard link is being created to this file instead of copying. This way I should end up with daily directories of files but only new/changed files will be eating up disk space.

Let’s give it a try. My backup script looks like this:

#!/bin/bash
date=`date “+%Y-%m-%d”`

#Mysql dump
for i in /var/lib/mysql/*/; do
dbname=`basename $i`
/usr/bin/mysqldump -u root -pyourpasshere $dbname | gzip -c > /home/mysql/$dbname-$date.sql.gz
done

#Rsync
rsync -arpvogDtSWz \
-e “ssh ” \
–delete \
–link-dest=../mysql-current \
./mysql/ \
root@nas:/path/to/backups/mysql-$date

#Moving links up one day
ssh root@nas \
“cd /path/to/backups/ && rm -f mysql-current && ln -s mysql-$date mysql-current”

The script dumps all dbs to a directory on hard drive, then rsyncs this directory with dated folder on a nas server using mysql-current (which is a symlink to latest daily backup) as additional sorce of files (–link-dest parameter).

I’ve run the script and then checked out the results:

root@nas# du -sh mysql-*
92M mysql-2011-05-16
19M mysql-2011-05-17
512 mysql-current

Looks like indeed the disk usage of the newer directory reflects only today’s copies of dbs. To confirm that hard links work as they should I deleted one file in the oldest directory. And now:

root@nas# du -sh mysql-*
86M mysql-2011-05-16
25M mysql-2011-05-17
512 mysql-current

the older directory size dropped but the newer directory size went up as the same file was hard linked there too and now that directory represents the only physical copy of the file on disk.

That’s exactly what I wanted to achieve.
Now to fully implement Time Machine mechanisms I need to make a script that will calculate the amount of free space needed on backup drive and delete the oldest backups accordingly until there is enough space available (as you see above simply deleting one file may not free up any hard drive space) to mve on with the transfer.

All parts are here, time to do some work.

My main goal was small initial cost as well as small footprint if at all possible. I got myself old Dell GX280 desktop with P4 2.8 GHz processor and 1 GB RAM. It has “relatively” new technology onboard including one PCI Express port in which I was able to insert controller card.

The enclosure itself is highly apprised all aver the Internet Sans Digital Tower Raid that can house up to 5 drives. As you can see both are rather small and do not take too much space on/under my desk:

Now onto the setup.

As my server is supposed to serve multiple purposes, including driving my hvac system, I quickly dropped the idea of using one of dedicated NAS distributions (FreeNAS namely) and decided to stay with old mighty Ubuntu 10.04LTS. The installation went smooth and within minutes I had the server running.

But there was nowhere to find my drive inserted into external enclosure. It turns out that the controller provided by Sans Digital is built on HighPoint Technologies’ 622 chip which itself is not supported by Linux Kernel. Thankfully Sans Digital provides source codes for drivers in both RAID and passthrough versions. I’ve chosen the passthrough version as I plan to skip hardware raid provided by the enclosure and use zfs.

First some additional software needed to be installed:

sudo apt-get install dkms

Then we’ll get some sources:

wget http://dl.sansdigital.com/images/downloads/TR8M-BP_TR8MP/Non_RAID_Driver_Linux-SourceCode-v1.0-100421-1320.tar.zip
unzip Non_RAID_Driver_Linux-SourceCode-v1.0-100421-1320.tar.zip
tar -xzf rr62x-linuxla-src-v1.0-100421-1320.tar.gz
cd rr62x-linuxla-src-v1.0/

We are going to use dkms so dkms.conf file needs to be created:

vi dkms.conf

MAKE=”make -C product/rr62x/linuxla/”
CLEAN=”make -C product/rr62x/linuxla/ clean”
BUILT_MODULE_NAME=rr62x
DEST_MODULE_LOCATION=/kernel/drivers/scsi/
BUILT_MODULE_LOCATION=product/rr62x/linuxla/
PACKAGE_NAME=rr62xla
PACKAGE_VERSION=1.O
AUTOINSTALL=yes
REMAKE_INITRD=yes

Now the source neds to be moved into /usr/src folder:

sudo cp -R . /usr/src/rr62xla-1.0

And some dkms magic applied:

sudo dkms add -m rr62xla -v 1.0
sudo dkms build -m rr62xla -v 1.0
sudo dkms install -m rr62xla -v 1.0

Mere seconds later I was able to load the module:

modprobe rr62x

and see this in syslog:

[ 260.187351] rr62x: module license ‘Proprietary’ taints kernel.
[ 260.187357] Disabling lock debugging due to kernel taint
[ 260.190622] rr62x:RocketRAID 62x SATA controller driver Non-RAID v1.0 (Feb 20 2011 11:26:43)
[ 260.190678] pci 0000:01:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[ 260.190689] pci 0000:01:00.0: setting latency timer to 64
[ 260.190727] rr62x:adapter at PCI 1:0:0, IRQ 16
[ 260.196263] rr62x:[0 0 ] start port.
[ 260.196263] rr62x:[0 0 ] start port hard reset (probe 1).
[ 260.196263] rr62x:[0 1 ] start port.
[ 260.196263] rr62x:[0 1 ] start port hard reset (probe 1).
[ 263.808262] rr62x:[0 1 ] start port soft reset (probe 1).
[ 264.456614] rr62x:[0 1 ] pmp attached: vendor 1095 device 3726.
[ 268.104271] rr62x:[0 0 ] failed to hard reset.
[ 268.104286] rr62x:[0 0 ] failed to perform port hard reset.
[ 269.709009] rr62x:[0 1 4] start device soft reset.
[ 270.360323] rr62x:[0 1 ] port started successfully.
[ 270.360323] rr62x:[0 1 4] device probed successfully.
[ 270.396411] scsi4 : rr62x
[ 270.399912] scsi 4:0:9:0: Direct-Access ATA SAMSUNG HD103SI 1AG0 PQ: 0 ANSI: 5
[ 270.401625] sd 4:0:9:0: Attached scsi generic sg2 type 0
[ 270.402425] sd 4:0:9:0: [sdb] 1953525168 512-byte logical blocks: (1.00 TB/931 GiB)
[ 270.402513] sd 4:0:9:0: [sdb] Write Protect is off
[ 270.402520] sd 4:0:9:0: [sdb] Mode Sense: 2f 00 00 00
[ 270.402565] sd 4:0:9:0: [sdb] Write cache: disabled, read cache: enabled, doesn’t support DPO or FUA
[ 270.402907] sdb: sdb1
[ 270.419594] sd 4:0:9:0: [sdb] Attached SCSI disk

So far so good. All pieces of hardware accounted for and work well.

Now we just have to make sure it will load the module at boot time:

sudo echo rr62x >> /etc/initramfs-tools/modules
sudo update-initramfs -u

And we are set for now.

Referrals:
1. Ubuntu forums.

Recently our collection of storage hard drives started to look a little bit too big. Yet we had several drives with a lot of free space fragmented between them but neither of them has enough free space itself to make you me comfortable. Trying to do something with it I decided to start looking for a solution that would allow me to aggregate those drives and their space providing me with one big logical partition. Oh, and having some kind of security would be nice too.

Browsing the internet I have found nice solution called Drobo. That’s exactly what I want to have. Nice, compact solution providing scalable space at the same time protecting data in RAID like manner. There is only one problem with Drobo (and it’s not the price). Drobo uses proprietary software and from several cases described over the internet it seems like when it goes bad, it goes really bad really quick, usually loosing most of the info stored. And then the only chance you have is to send the box with all drives to the company itself and pray they can restore something for you. That effectively deletes this device from my list.

But browsing further I stumbled upon ZFS. This seems to be ‘the best file system and last you’ll ever need’. It provides ability to create software RAID systems which are supposed to be secure and fast. So home brewing my own NFS server seems to be the best option. Even more appealing as I’ll be able to use the same machine to run our HVAC system.

First step was to get all needed hardware.

1. Server. As I do not need performance at any cost I’ve decided to use old Dell GX280 machine lying around and gathering dust. It has 1GB of RAM and P4 2.8GHz processor which should be plenty enough for my needs. Cost: $0
2. Hard drive enclosure. I got myself Sans Data Tower Raid enclosure with port replicator that is able to host 5 drives and connects to server via one eSATA cable. Cost $173
3. Hard drives. I dissected external 1TB Iomega USB drive which turned out to house Samsung’s HD103SI drive. Not a speed hog but for now will suffice. Cost $0

So far total spent: $173. Not bad I would say.