One of the issues I’ve found with having lots of data is the fact that I’m worried that a hard drive will fail, and I’ll lose something important. Since I did have that happen earlier this year, I am now determined to not let it happen again. I’ve been focusing a lot on resiliency – making it so that there are no single points of failure. Since I have been collecting a lot of hard drives, I decided to put them to good use, and set up some data replication onto multiple drives.

You might ask, why not just set up a RAID5? Or, alternatively, why not buy a Drobo and be done with it? I hear the first one a lot, especially since I deal with RAID day in and day out for my day job. The main issue I don’t do either of these things is that I’m cheap! I want to be able to use all of the hardware I have without tossing out old stuff. On my server, I have the following drives:

• disk1: 120 GB ATA
• disk2: 200 GB ATA
• disk3: 300 GB USB
• disk4: 400 GB Firewire
• disk5: 600 GB SATA
• disk6: 1000 GB USB
• disk7: 1500 GB USB

Try to set up a RAID with all of that! The problem with this is the disparate drive sizes (RAID requires disks to be the same size, and will use the lowest common denominator if not.) I’ve been ‘collecting’ hard drives for quite a while, and I like the fact that I can just go to the local electronics store and pick up another drive when I run out of space. This configuration does have a downside though, and that is, how do you decide where to put everything? I decided to sit down and figure out how much space I actually need for each of my data types, and came up with this list (taking approximate growth rates into consideration):

• Photos: 100 GB
• Music: 300 GB
• Movies: 1 TB
• TV: 400 GB
• Software: 150 GB

I then tried to map things out onto the drives I have:

• disk1: Photos
• disk2: Software
• disk3: Music
• disk4: TV
• disk6: Movies

This leaves two disks, disk5 (600 GB) and disk7 (1500 GB) left to be able to mirror to. Since I can’t use standard mirroring software, I’m not able to have a real-time mirror. This is OK for my purposes, as I don’t intend on having them be hot-swappable, etc. — I just want another copy of my data out there, without having to worry about making backups all the time.

I set up my server to only export the data disks (I don’t want to be able to write to the backups from my desktop, as the changes will be overwritten by the backups) I also set up my filesystem in a convenient manner — everything is a directory off of /data:

• /data/Photos
• /data/Video
• /data/Software
• /data/Audio

This makes it very easy to get to any of my data (try that in Windows!)

One inherent advantage of having an asynchronous mirror is the fact that it can serve as a temporary backup if you accidentally delete a file/directory, whereas with a normal mirror, you’d still have to back up to tape or some other drive to be able to get this data back.

To do the actual mirroring, I opted to create a simple rsync script that will copy the data nightly from cron. Nothing overly complex, but something that makes it easy to add new filesystems via a config file:

#!/usr/bin/perl
# Mirror
# Asynchronously mirror filesystems on a local machine
# by Ed Salisbury (ed@edsalisbury.net)
# http://www.edsalisbury.net
# (c)2009 Ed Salisbury, Some Rights Reserved
#
# External Utilities Required:
# * rsync
#
# License:
# Except where otherwise noted, this work is licensed under Creative Commons
#   Attribution ShareAlike 3.0.
#
# You are free:
#   * to Share — to copy, distribute and transmit the work
#   * to Remix — to adapt the work
#
# Under the following conditions:
#   * Attribution. You must attribute the work in the manner specified by the
#     author or licensor (but not in any way that suggests that they endorse
#     you or your use of the work).
#   * Share Alike. If you alter, transform, or build upon this work, you may
#     distribute the resulting work only under the same, similar or a
#     compatible license.
#   * For any reuse or distribution, you must make clear to others the license
#     terms of this work. The best way to do this is with a link to the
#     license's web page (http://creativecommons.org/licenses/by-sa/3.0/)
#   * Any of the above conditions can be waived if you get permission from the
#     copyright holder.
#   * Nothing in this license impairs or restricts the author's moral rights.

use warnings;
use strict;

# Configuration file for mirrors
# Example Config:
# /data/Photos/       /backup/Backup/Photos/
# /data/Software/     /backup/Backup/Software/
# /data/Audio/        /backup/Backup/Audio/
# /data/Video/        /backup/Backup/Video/
# /data/Misc/         /backup/Backup/Misc/
my $CONFIG = "/usr/local/etc/mirror.cfg";

open(CFG, $CONFIG);
my @lines = <CFG>;
close(CFG);

foreach my $line (@lines)
{
    my ($src, $dest) = split(/s+/, $line);
    system("/usr/bin/rsync -av --delete $src $dest");
}

I put this script into /usr/local/bin, and then created a config file /usr/local/etc/mirror.conf similar to this:

/data/Photos/       /backup/Backup/Photos/
/data/Software/     /backup/Backup/Software/
/data/Audio/        /backup/Backup/Audio/
/data/Video/        /backup/Backup/Video/
/data/Misc/         /backup/Backup/Misc/

I then added a cronjob to have it run nightly:

0 0 * * * /bin/backup > /dev/null 2>&1

A word of caution: Make *sure* you have the directories right! rsync is set up to *DELETE* files in the destination if the file isn’t in the source directory, so if you have it rsync to two different directories, it will delete files as needed in the destination to sync them up. You have been warned! I take no responsibility for you deleting your data!

Please let me know if you found this guide/script useful.

After my hard drive crash a few months ago, I’ve been a bit paranoid about losing data.   By “a bit” I mean I’ve been sticking copies of my data on different machines to make sure I have various copies in case another hard drive dies, or a computer gets fried, melting everything inside, or my house burns down… Oh wait, I don’t have that scenario covered….  Time to fix that!  What I want is the ability to copy my data to an offsite location, and keep the backup up to date.  The main thing I care about is my family photos.   I was talking to my cousin Dave, and he had the same issue.  He wanted access to my photos, as well as have an offsite backup copy of his family’s photos.  I thought of a solution.  What if we could set up a two-way mirror between his machine and mine, and keep it updated nightly?  That way if he sticks something in the directory on his machine and vice versa, keeping a live backup.  I began doing some research, and I found the tool — Unison.

One slight wrinkle is the fact that he runs Mac OS X and I run Ubuntu Linux.  Unison will run on both, so it should be fine.

A couple of things to consider:

Mirroring over the internet is SLOW — if you want to keep a large quantity of files mirrored, it’s best to “seed” the mirror by doing a local copy to the drive.  Dave sent me his hard drive via snailmail, so this worked out.

This process will involve poking holes in your firewall, and setting up SSH keys with no password.  These are security risks (which can be minimized).  You have been warned.

To make things easy, I’ll specify the hostnames:

Mac Host: eve

Linux Host: twiki

One thing needs to be decided before we go on is which host will be initiating the transfer.   You can initiate from both sides, but to make it easy, I’ll just be initiating from my computer, twiki.

The first step is to verify connectivity between the two machines.  Since I’m initiating the transfer, I’ll be adding a hosts entry for eve.  I first have Dave go to www.whatismyip.com and have him tell me his IP address.  I put this into /etc/hosts on my machine:

<hismachinesipaddress>        eve

I try to first ping the address.  Most likely it will fail.  Dave will need to poke a hole in the firewall (set up port forwarding) – How to do this is outside the scope of this article, but www.portforward.com has a lot of info on how to do this.  Dave forwards port 22 (SSH) to his local IP address and saves settings.

Ping still doesn’t work, but SSH should.  I try sshing to the box, and voila, I get a password prompt.

The next step is to have Dave create an account for me.  To make it easy, I have him call it “ed” (same as on twiki)

After my account has been created, I’m able to login via ssh.  The next step is to set up an SSH key to make it so that I can ssh in without having to give a password.  On twiki, I run:

twiki% ssh-keygen -t dsa -f ~/.ssh/unison

(Press enter when it asks for a passphrase)

Add the contents of ~/.ssh/unison.pub to ~/.ssh/authorized_keys2 on eve.  Make sure it’s all on one line (no linebreaks)

To test it out, I ssh from twiki to eve, specifying the unison key:

twiki% ssh -i ~/.ssh/unison eve

It shouldn’t ask me for a password, so I’m good to go for this step.  If it does, make sure the permissions on the files in ~/.ssh are set to 600.

Next, we’ll install Unison on both machines:

On Eve:

• Get and install MacPorts
• Use MacPorts to install Unison:

eve% sudo /opt/local/bin/port install unison
--->  Fetching ocaml
--->  Attempting to fetch ocaml-3.11.0.tar.bz2 from http://distfiles.macports.org/ocaml
--->  Verifying checksum(s) for ocaml
--->  Extracting ocaml
--->  Configuring ocaml
--->  Building ocaml
--->  Staging ocaml into destroot
--->  Installing ocaml @3.11.0_0
--->  Activating ocaml @3.11.0_0
--->  Cleaning ocaml
--->  Fetching unison
--->  Attempting to fetch unison-2.27.57.tar.gz from http://distfiles.macports.org/unison
--->  Verifying checksum(s) for unison
--->  Extracting unison
--->  Applying patches to unison
--->  Configuring unison
--->  Building unison
--->  Staging unison into destroot
--->  Installing unison @2.27.57_0
--->  Activating unison @2.27.57_0
--->  Cleaning unison

On Twiki:

twiki% sudo apt-get install unison
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
unison
0 upgraded, 1 newly installed, 0 to remove and 11 not upgraded.
Need to get 0B/529kB of archives.
After this operation, 1262kB of additional disk space will be used.
(Reading database ... 150151 files and directories currently installed.)
Unpacking unison (from .../unison_2.27.57-1ubuntu1_i386.deb) ...
Processing triggers for man-db ...
Processing triggers for doc-base ...
Processing 1 added doc-base file(s)...
Registering documents with scrollkeeper...
Setting up unison (2.27.57-1ubuntu1) ...

Next, set up the profile on twiki:

twiki% mkdir ~/.unison
twiki% vi ~/.unison/default.prf

Put the following into default.prf (changing as appropriate):

root = /data/EdsPhotos
root = ssh://eve//DavesPhotos
rshargs = -C
sshargs = -i /home/ed/.ssh/unison
servercmd = /opt/local/bin/unison
ignore = Name _*
ignore = Name .*
batch = true
log = true

For security purposes, add the following to ~/.ssh/authorized_keys2 on eve.  This will only allow this ssh key to be used to run unison:

command="/opt/local/bin/unison",no-port-forwarding,no-X11-forwarding,no-agent-forwarding

On twiki, run unison:

twiki% unison

It will spit out a lot of data, but should start scanning and eventually copying data back and forth, depending on how much data there is.   In order to run nightly, add the following to your crontab on twiki:

0 0 * * * nohup /usr/local/bin/unison > /dev/null 2>&1

That should be it.  Any feedback would be appreciated.