Encrypted Remote Backups

EVERYTHING HERE IS DEPRECATED, JUST USE TARSNAP


Tarsnap is the best solution to this problem I know, and better than everything I ever came up with.



I've been toying around with my offsite backup system for Quite Some
Time Now (probably more than 5 years). I've settled into something
that really does seem to work.
My old method had
some problems. In particular, backups kept taking longer and longer
(presumably because of block fragmentation in the loop-aes FS) and
deleting lots of stuff increased backup time. Ick. It also had
the annoying side effect of the backups being too large for using
rsync.net to be viable; I'm using them with my
new method.

Much of the text and layout in this doc is stolen from
my old method.

Introduction


Introduction

I have a method of personal data backup that seems to be fairly novel, in that it has all of the following properties:

  • I back up multiple OSes on multiple machines, with incremental backups and a CGI interface and all sorts of good stuff like that
  • I synchronize it offsite (daily)
  • The offsite backup is encrypted, and only I have the key
  • I don't have to synch everything every time I backup
  • It deals with OS special files, like symlinks
  • It's reasonably fast; I'm at about 7GiB of data, and the daily synch takes about an hour.
  • It's free (except for the offsite account)


None of these things are especially exceptional, but the combination of all of
them is rare enough that in searching for a good solution I've found a number
of people also looking for this feature set (and largely failing to find it, by
the way).

My method is based around BackupPC as the
main backup system. The problem with
BackupPC, is that it likes hard links. A
lot. No non-UNIX file system understands these and, in particular, it means
Amazon's S3 is right out.
I'm not aware of any Linux multi-OS backup system that doesn't suck that also
doesn't have this problem; that is, such systems make few assumptions about the
file system they're backing up, but a lot of assumptions about the file system
they're backing up to. The big advantage to
BackupPC's hard link system is that it
minimizes the size of each new backup, whilst still retaining the
enterprise-level tiered backup system we've all come to know and, umm,
tolerate.

The Machines

  • Various and sundry home machines (a Windows XP box, a couple of Linux boxes).
  • The BackupPC server; we'll call it "LOCAL".
  • The remote server, which we'll call "REMOTE". I recommend rsync.net for this purpose.


The Software

  • rsync, and a REMOTE that fully supports the rsync algorithm when operating at the block-level. You really want rsync 3.0 or better here if possible; it speeds the transfer of large file trees immensely.
  • rsyncrypto, which performs rsync-friendly file encryption. It's a bit rough around the edges, but our method is such that we avoid the encrypted file name stuff, which is the stuff that (as of March 2008 anyways) still needs some work
  • BackupPC or a similar multi-OS backup system that runs on Linux as the LOCAL server


The Process

Note that this document doesn't discuss setting up
BackupPC at all.

Set Up rsyncrypto


You can just follow the man page tutorial to set up a key. In
addition, I gpg symmetrically encrypt both the public and private
key and send them with the backups. I use a very long key for
this (over a hundred characters). You can do that with just "gpg -c [file]".

The password to the key is used in my scripts, but if someone
compromises my machine to the extent required to see it, they won't
need to compromise my backups, since they can just copy all the live
data.

Extra Bits You Should Transfer


If you want to be able to decrypt fully without too much trouble,
you should probably copy the following as part of your backups.
Encrypt as appropriate.

  • BackupPC_zcat (for uncompressing)
  • FileZIO.pm (BackupPC_zcat needs it)
  • a restore script (make sure it doesn't have any passwords in it!)
  • the backuppc configs (encrypted!); without them you can't use backuppc for restoration and will have to do it by hand
  • the rsyncrypto keys (as I said, I gpg encrypt them; otherwise you need to keep secure copies elsewhere)

Set Up Data For Encryption


I have a few bits of miscellaneous data and so on that I won't
bother describing here, but some setup definately needs to be done
before we try to encrypt.

I actually use two forms of encryption: gpg symmetric as described
above and rsyncrypto. This is largely for historical reasons.

  1. Stop backuppc
  2. Create a tar of the backuppc "pc" directory; this is the directory with all the hardlinks in it. BackupPC now has a utility to generate a tar of this directory without too much trouble. Between this and the cpool directory, you'll have everything you need. BackupPC_tarPCCopy is the utility

The Real Work

  1. Encrypt the BackupPC_tarPCCopy output with rsyncrypto
  2. Encrypt the cpool directory with rsyncrypto
  3. rsync it all to your remote site

The Code


Here's my backup script (with passwords cropped and such): 

#!/bin/sh

export PATH=/usr/bin:/bin:/usr/sbin:/sbin

# After 12 hours, assume the other rsync is dead or something.
lockfile -l 43200 -r 1 /var/tmp/lock.backup_rc_full

if [ $? -ne 0 ]
then
        echo "Could not obtain lock; exiting."
        exit
fi

log() {
    echo "
=======================================================================
$(date): $1
=======================================================================
"
}

rm -f /tmp/backup_rc_full.log

exec 1>/tmp/backup_rc_full.log
exec 2>&1

log "Stopping BackupPC."
# Stop backuppc
/etc/init.d/backuppc stop

log "BackupPC stopped."

log "Doing basic setup."

# Basic setup
chmod og-rwx /backups/rcb/
chown root /backups/rcb/
cp /usr/share/backuppc/bin/BackupPC_zcat /backups/rcb/BackupPC_zcat
mkdir -p /backups/rcb/BackupPC
cp /usr/share/backuppc/lib/BackupPC/FileZIO.pm /backups/rcb/BackupPC
cp /home/zroot/bin/restore_rc_full /backups/rcb/

log "Saving backuppc configs."

# Put the backuppc configs in there.
rm /backups/rcb/etc_backuppc.tgz.gpg
tar -zc /etc/backuppc/ -f /backups/rcb/etc_backuppc.tgz
echo **ERASED*PASSWORD** | gpg -q --batch --passphrase-fd 0 --symmetric /backups/rcb/etc_backuppc.tgz
rm /backups/rcb/etc_backuppc.tgz

log "Saving keys."

# Put the backup keys in there
cp /home/zroot/backup.key.gpg /backups/rcb/
cp /home/zroot/backup.crt.gpg /backups/rcb/

chmod og-rwx /backups/rcb/*
chown root /backups/rcb/*

CERT=/backups/rcb_plain/backup.crt

# Make unencrypted crt for rsyncrypto to work with
echo **ERASED*PASSWORD** | \
    gpg --batch -q -d --passphrase-fd 0 /home/zroot/backup.crt.gpg >$CERT

log "Creating tar of the pc dir"
cd /backups/rcb_plain
sudo -u backuppc /usr/share/backuppc/bin/BackupPC_tarPCCopy /var/lib/backuppc/pc >pc.tar

log "Running rsyncrypto against the pc tarball"
/usr/local/bin/rsyncrypto -c -vv /backups/rcb_plain/pc.tar /backups/rcb/backuppc/pc.tar /backups/rcb_plain/pc.tar.key $CERT

log "Done with rsyncrypto"

log "Running rsyncrypto against the pool and such"
find /var/lib/backuppc/* -type d -prune | egrep -v '/(\.ssh|\.gnupg|ball|chain|pc|trash)$' | sed 's;$;/;' | /usr/local/bin/rsyncrypto --trim=2 --delete -c -vv --filelist - /backups/rcb/ /backups/rcb_plain/keys/ $CERT

log "Done with rsyncrypto"

# Fix ownership
chown root /backups/rcb/* /backups/rcb/*/*

log "Starting BackupPC."

# Start backuppc
/etc/init.d/backuppc start

log "BackupPC started."

log "Syncing to REMOTE"

# Note that everything is already compressed
/usr/bin/rsync --no-compress --delete -S --stats -av /backups/rcb/ REMOTE:/backups/rcb/

log "Done syncing to REMOTE"

rm $CERT

# Remove the lockfile
rm -f /var/tmp/lock.backup_rc_full


Here's the restore script: 

#!/bin/sh

export PATH=/usr/bin:/bin:/usr/sbin:/sbin

log() {
    echo "
=======================================================================
$(date): $1
=======================================================================
"
}

log "Doing basic setup."

mkdir -p /backups/rcb_restore/backuppc

CERT=/backups/rcb_restore/backup.key

# Make unencrypted crt for rsyncrypto to work with; asks for password
gpg -d /backups/rcb/backup.key.gpg >$CERT

log "Running rsyncrypto decrypt"
/usr/local/bin/rsyncrypto --trim=2 --delete -d -vv -r /backups/rcb/backuppc/ /backups/rcb_restore/ /backups/rcb_restore/keys/ $CERT

log "Done with rsyncrypto"

log "Expanding tar of the pc dir"
# the P is important; see the backuppc man page
mkdir -p /backups/rcb_restore/backuppc/pc
cd /backups/rcb_restore/backuppc/pc
tar -xvPf ../pc.tar

log "Done with rsyncrypto"

# Fix ownership
chown root /backups/rcb_restore/* /backups/rcb_restore/*/*

rm $CERT