Ubuntu/Debian - Encrypted incremental backups with duplicity on Amazon S3

An example on how to use duplicity to perform encrypted incremental backups on Amazon S3.

I'm using duplicity version 0.6.20 (released October 28, 2012).

Getting started

If you've never heard about duplicity before, you should check the documentation.

Install duplicity

First, you need to install duplicity, I always install it from source since the duplicity package is not often updated.

$ sudo apt-get install python-dev librsync-dev
$ cd /opt
$ sudo wget https://code.launchpad.net/duplicity/0.6-series/0.6.20/+download/duplicity-0.6.20.tar.gz
$ sudo tar xvzf duplicity-0.6.20.tar.gz
$ cd duplicity-0.6.20
$ python sudo setup.py install

But you can install it with apt-get

$ sudo apt-get install duplicity

Next you can also install s3cmd from S3 Tools, it's a command line tool for managing your S3 buckets, but it's not required.

$ sudo apt-get install s3cmd
$ s3cmd --configure

Encrypted Backups

Before backing up the data, you need to think about encryption, duplicity makes use of gpg and handles both private/public key pair (a gpg key) and symmetric encryption (a passphrase).

I use passsphrases since I'll never lose it and I don't have to backup a gpg key.

My backup script

Since you need to specify many args to perform the differents actions, I crafted a bash script that make working with duplicity easier, duptools.

Features

  • Backup multiple directories
  • Send email report on backup
  • Quickly list file and show bucket status
  • Restore your files easily

Duptools

Gist available here.

#!/bin/bash
export AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY
export PASSPHRASE=YOU_PASSHRASE

# directories, space separated
SOURCE="/home/thomas/backup /home/thomas/bin /home/thomas/documents"
BUCKET=s3+http://mybucket
LOGFILE=/home/thomas/tmp/duplicity.log
# set email to receive a backup report
EMAIL=""

backup() {
  INCLUDE=""
  for CDIR in $SOURCE
  do
    TMP=" --include  ${CDIR}"
    INCLUDE=${INCLUDE}${TMP}
  done
  # perform an incremental backup to root, include directories, exclude everything else, / as reference.
  duplicity --full-if-older-than 30D $INCLUDE --exclude '**' / $BUCKET > $LOGFILE
  if [ -n "$EMAIL" ]; then
    mail -s "backup report" $EMAIL < $LOGFILE
  fi
}

list() {
  duplicity list-current-files $BUCKET
}

restore() {
  if [ $# = 2 ]; then
    duplicity restore --file-to-restore $1 $BUCKET $2
  else
    duplicity restore --file-to-restore $1 --time $2 $BUCKET $3
  fi
}

status() {
  duplicity collection-status $BUCKET
}

if [ "$1" = "backup" ]; then
  backup
elif [ "$1" = "list" ]; then
  list
elif [ "$1" = "restore" ]; then
  if [ $# = 3 ]; then
    restore $2 $3
  else
    restore $2 $3 $4
  fi
elif [ "$1" = "status" ]; then
  status
else
  echo "
  duptools - manage duplicity backup

  USAGE:

  ./duptools.sh backup 
  ./duptools.sh list
  ./duptools.sh status
  ./duptools.sh restore file [time] dest
  "
fi

export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export PASSPHRASE=

Installation

Set up config vars at the top of the script and make the script executable.

Backup

$ ./duptools.sh backup

List/Status

$ ./duptools.sh list
$ ./duptools.sh status

Restore

Be careful while restoring not to preprend a slash to the path.

Restoring a single file to tmp

$ ./duptools.sh restore home/thomas/bin/setupscreen tmp/setupscreen

Restoring an older version of a directory to tmp (interval or full date)

$ ./duptools.sh  restore home/thomas/bin 1D3h5s tmp/bin
$ ./duptools.sh  restore home/thomas/bin 2012/7/5 tmp/bin

Bakthat, a lightweight solution

You may also want to check out bakthat, a python command-line tool I wrote, that allow you to compress, encrypt (symmetric encryption) and upload files directly to Amazon S3/Glacier in a single command.

And you ?

If you have any suggestions, feel free to share it !

You should follow me on Twitter

Share this article

Tip with Bitcoin

Tip me with Bitcoin and vote for this post!

1FKdaZ75Ck8Bfc3LgQ8cKA8W7B86fzZBe2

Leave a comment

© Thomas Sileo. Powered by Pelican and hosted by DigitalOcean.