Incremental snapshot backups via rsync and ssh

In follow-up to the previous post, I am compiling this as a separate post as this solution is been running very stable for a while with quite a few updates and changes...

I will be setting up a back-up of a remote web-host via rsync over ssh and creating the snapshot style backup on the local machine.

The backups are done incremental, only the files that have changed are backed up so there is very less bandwidth used during the backup and also does not cause any load on the server.

These are sliced backups, meaning that you get a full backup of the last 4 days, and the last 4 weeks. So data can be restored for upto a month of back date.

Below is an example listing of backups you would see.

Mar 11 - daily.0
Mar 10 - daily.1
Mar 9 - daily.2
Mar 8 - daily.3
Mar 5 - weekly.0
Feb 27 - weekly.1
Feb 20 - weekly.2
Feb 13 - weekly.3

Each of those is a full snapshot for the particular day/week. The files are all hard-linked and would only require 2 to 3 times the space used on the server. The backups should consist of web, database, email and some of the important server configuration files.

On the local backup machine:

1. Generate the private/public key pair for passwordless login via ssh to remote host.
   
   # ssh-keygen -t dsa -b 2048 -f ~/.ssh/rsync_key
2. Create an empty password by just hitting enter when prompted for password.
   
3. Protect the private key, which should only be readable by the owner.
   
   # chmod 600 ~/.ssh/rsync_key
4. Transfer the public key file to the remote host:
   
   # scp ~/.ssh/rsync_key.pub remote_host.tld:.
5. The backups are on separately partitioned drive and auto-mounted read-only.

   In "/etc/auto.master"
   

   /data/bak   /etc/auto.bak

   In "/etc/auto.bak"
   

   .snapshots      -fstype=ext3,ro,nosuid,noexec   :/dev/hdb2-vg00/lv-bak

6. Restart autofs:
   
   # service autofs restart
7. Create the rsync backup script to run via cron. The script also rotates out the backups creating a snapshot style backup:
   
   #!/bin/bash
# do_rsync.sh
# usage: ./do_rsync.sh {client}

RM=/bin/rm
TOUCH=/bin/touch
NICE=/bin/nice
CHOWN=/bin/chown
CHMOD=/bin/chmod
RSYNC=/usr/bin/rsync
SSH=/usr/bin/ssh
MOUNT=/bin/mount
UMOUNT=/bin/umount
KEY=/root/.ssh/rsync_keys

CLIENT=$1
EXPECTED_ARGS=1
NUM_ARGS=$#
CONFIG_PATH=/root/scripts/snapshots/config
CONFIG_FILE=${CONFIG_PATH}/${CLIENT}/conf.txt
EXCLUDE_FILE=${CONFIG_PATH}/${CLIENT}/exclude.txt
LPATH=/home/virtual/${CLIENT}.DOMAIN.TLD/home/${CLIENT}/bak/.snapshots
PID_FILE=/var/run/${CLIENT}_rsync.pid

# Check the number of args
badargs() {
  E_BADARGS=65
  if [ "$NUM_ARGS" -ne "$EXPECTED_ARGS" ]
  then
    echo "Usage: `basename $0` {client}"
    exit $E_BADARGS
  fi
}

# Include conf
include_conf() {
  if [ -f "${CONFIG_FILE}" ]; then
    # source the config file
    . ${CONFIG_FILE}
  else
    echo "ERROR: ${CONFIG_FILE}" does not exist.
    exit 1
  fi
}

# Mount for writing
mount_write() {
  $MOUNT -o remount,rw,nosuid,noexec,nodev $LPATH
  if (( e = $? )); then
    echo "ERROR: $e, Could not remount in ReadWrite mode. Exiting..."
    exit $e
  fi
}

# rotate backups
rotate_backups() {
  if [ -f daily.0/.rsync_bak_complete ]
  then
    if [ "`date +%w`" -eq 0 ]
    then
      [ -d weekly.3 ] && rm -rf weekly.3
      [ -d weekly.2 ] && mv weekly.2 weekly.3
      [ -d weekly.1 ] && mv weekly.1 weekly.2
      [ -d weekly.0 ] && mv weekly.0 weekly.1
      [ -d daily.3 ] && mv daily.3 weekly.0
    fi
    [ -d daily.3 ] && rm -rf daily.3
    [ -d daily.2 ] && mv daily.2 daily.3
    [ -d daily.1 ] && mv daily.1 daily.2
    [ -d daily.0 ] && mv daily.0 daily.1
  fi
  if [ ! -d daily.0 ]; then
    mkdir daily.0
    # Adjust permissions
    $CHMOD 050 daily.0
    $CHOWN root:${GID} daily.0
  fi
}


# run the rsync
run_rsync() {
  $NICE -n 12 $RSYNC -azR -e "$SSH -i $KEY -p $PORT" --exclude-from=${EXCLUDE_FILE} --delete --timeout=1800 --link-dest=../daily.1 $RUSER@$RHOST:"$RPATH" $LPATH/daily.0

  # error 23 = partial transfer, probably temp files
  # error 24 = file vanished
  if (( e = $? )); then
    if [ "$e" -ne 23 ] && [ "$e" -ne 24 ]; then
      echo "ERROR: $e, Rsync failed. Exiting..."
      post_sync $e
    fi
  fi

  # Confirm backup is complete so files are rotated on the backup side
  $RSYNC -a -e "$SSH -i $KEY -p $PORT" --timeout=10 $RUSER@$RHOST:/.rsync_bak_complete $LPATH/daily.0

  if (( e = $? )); then
    if [ "$e" -ne 23 ] && [ "$e" -ne 24 ]; then
      echo "ERROR: $e, Rsync CONFIRMATION failed."
      echo "Check backup completion. Exiting..."
      post_sync $e
    fi
  fi
}


# Remount Read Only
mount_read() {
  $MOUNT -o remount,ro,nosuid,noexec $LPATH
  if (( e = $? )); then
    echo "ERROR: $e, Could not remount in ReadOnly mode."
    echo "Check your mounts. Exiting..."
    exit $e
  fi
}

# Unmount
unmount() {
  $UMOUNT $LPATH
  if (( e = $? )); then
    echo "ERROR: $e, Could not unmount ${LPATH}."
    echo "Check your mounts. Exiting..."
    exit $e
  fi
}

# Check if the PID is running
check_pid() {
  if [ -f "$PID_FILE" ]
  then
    MYPID=`head -n 1 "$PID_FILE"`
    TEST_RUNNING=`ps -p ${MYPID} | grep ${MYPID}`
    if [ -z "${TEST_RUNNING}" ]
    then
      echo "PID file exists but PID [$MYPID] is not running... creating new PID file [$PID_FILE]"
      echo $$ > "$PID_FILE"
    else
      echo "`basename $0` is already running [${MYPID}]... quitting"
      exit -1
    fi
  else
    echo "`basename $0` not running... creating new PID file [$PID_FILE]"
    echo $$ >> "$PID_FILE"
  fi
}

# Post Sync
post_sync() {
  popd
  sleep 10
  unmount
  exit $1
}

# Pre Sync
pre_sync() {
  check_pid
  pushd $LPATH
  sleep 10
  mount_write;
  rotate_backups;
}

# Main
badargs;
include_conf;
pre_sync;
run_rsync;
post_sync 0;

exit 0

   There is a configuration and an exclude file, conf.txt and exclude.txt .

   "conf.txt" includes the details of the server that need to be backed up.
   

   PORT=22
RUSER=root
RHOST=DOMAIN.TLD
RPATH='/etc /var/lib/mysql /var/www /var/spool/mail /var/spool/squirrelmail'
GID=admin138

   "exclude.txt" file has the list of files to be excluded from being backed up.
   

   # exclude patterns (one per line).
#/var/www/vhosts/*/statistics/logs/*
/var/www/*/log

8. Change the mode to be executable on the script.
   
   # chmod 750 /usr/local/bin/rsync_backup.sh
9. Add the script to crontab for a daily incremental snapshot style backup.
   
   # Do rsync backup
07 06 * * * /usr/local/bin/rsync_backup.sh > /dev/null 2>&1
10. Also add the below line to "/etc/fstab" to auto fsck and recover journals on boot.
    /dev/hdb2-vg00/lv-bak  /mnt/lv-bak     ext3    noauto,nodev,noexec,nosuid        0 2

    This also allowed me to do quick read-write mounts when needed.

On the remote host:

1. Create script to validate the rsync command used.
   
   #!/bin/sh
# rsync_validate.sh

case "$SSH_ORIGINAL_COMMAND" in
*\&*)
echo "Rejected"
;;
*\(*)
echo "Rejected"
;;
*\{*)
echo "Rejected"
;;
*\;*)
echo "Rejected"
;;
*\<*)
echo "Rejected"
;;
*\`*)
echo "Rejected"
;;
rsync\ --server*)
$SSH_ORIGINAL_COMMAND
;;
*)
echo "Rejected"
;;
esac
2. Change the mode to be executable on the script.
   
   # chmod 750 /usr/local/bin/rsync_validate.sh
3. Modify the public key and prepend restricted host and command used, separated by just a comma and no spaces.
   
   from="backup_host.tld",command="/usr/local/bin/rsync_validate.sh" ssh-dss AAAAB3NzaC1kc3MAAA...
4. Append the public key to authorized_keys2 file.
   
   # cat ~/rsync-key.pub >> ~/.ssh/authorized_keys2
5. Modify "/etc/ssh/sshd_config" to accept forced-commands-only for root ssh login.
   
   PermitRootLogin forced-commands-only
6. Restart ssh:
   
   # service sshd restart

That should do it... the result will be an incremental backup of the last 4 days plus a weekly of the month.