This document describes a method for generating automatic rotating "snapshot"-style backups on a Unix-based system, with specific examples drawn from the author's GNU/Linux experience. Snapshot backups are a feature of some high-end industrial file servers; they create the illusion of multiple, full backups per day without the space or processing overhead. All of the snapshots are read-only, and are accessible directly by users as special system directories. It is often possible to store several hours, days, and even weeks' worth of snapshots with slightly more than 2x storage. This method, while not as space-efficient as some of the proprietary technologies (which, using special copy-on-write filesystems, can operate on slightly more than 1x storage), makes use of only standard file utilities and the common rsync program, which is installed by default on most Linux distributions. Properly configured, the method can also protect against hard disk failure, root compromises, or even back up a network of heterogeneous desktops automatically.
re:
Excellent. Thanks for sharing this informative articles. I found another good articles about linux manual. You can check it out at pdfph.com.
digitalcatalogue
regard, very usefull and thankyou ..
Incremental snapshot backups via rsync and ssh
I will be setting up a back-up of a remote web-host via rsync over ssh and creating the snapshot style backup as mentioned above on the local machine.
On the local backup machine:
Generate the private/public key pair for passwordless login via ssh to remote host.
Create an empty password by just hitting enter when prompted for password. Protect the private key, which should only be readable by the owner.
Transfer the public key file to the remote host: The backups are on separately partitioned drive and auto-mounted read-only.
In "/etc/auto.master"
In "/etc/auto.bak"
Restart autofs: Create the rsync backup script to run via cron. The script also rotates out the backups creating a snapshot style backup: Change the mode to be executable on the script. Add the script to crontab for a daily incremental snapshot style backup.On the remote host:
Create script to validate the rsync command used.
Change the mode to be executable on the script. Modify the public key and prepend restricted host and command used, separated by just a comma and no spaces. Append the public key to authorized_keys2 file. Modify "/etc/ssh/sshd_config" to accept forced-commands-only for root ssh login. Restart ssh:
That should do it... the result will be an incremental backup of the last 4 days plus a weekly of the month.
rsync backup shell script
I have since modified the rsync_backup.sh script and made it a bit modular with some amount of error checking:
#!/bin/bash
# do_rsync.sh
# usage: ./do_rsync.sh {client}
RM=/bin/rm
TOUCH=/bin/touch
NICE=/bin/nice
CHOWN=/bin/chown
CHMOD=/bin/chmod
RSYNC=/usr/bin/rsync
SSH=/usr/bin/ssh
MOUNT=/bin/mount
UMOUNT=/bin/umount
KEY=/root/.ssh/rsync_keys
CLIENT=$1
EXPECTED_ARGS=1
NUM_ARGS=$#
CONFIG_PATH=/root/scripts/snapshots/config
CONFIG_FILE=${CONFIG_PATH}/${CLIENT}/conf.txt
EXCLUDE_FILE=${CONFIG_PATH}/${CLIENT}/exclude.txt
LPATH=/home/virtual/${CLIENT}.DOMAIN.TLD/home/${CLIENT}/bak/.snapshots
# Check the number of args
badargs() {
E_BADARGS=65
if [ "$NUM_ARGS" -ne "$EXPECTED_ARGS" ]
then
echo "Usage: `basename $0` {client}"
exit $E_BADARGS
fi
}
# Include conf
include_conf() {
if [ -f "${CONFIG_FILE}" ]; then
# source the config file
. ${CONFIG_FILE}
else
echo "ERROR: ${CONFIG_FILE}" does not exist.
exit 1
fi
}
# Mount for writing
mount_write() {
$MOUNT -o remount,rw,nosuid,noexec,nodev $LPATH
if (( e = $? )); then
echo "ERROR: $e, Could not remount in ReadWrite mode. Exiting..."
exit $e
fi
}
# rotate backups
rotate_backups() {
if [ -f daily.0/.rsync_bak_complete ]
then
if [ "`date +%w`" -eq 0 ]
then
[ -d weekly.3 ] && rm -rf weekly.3
[ -d weekly.2 ] && mv weekly.2 weekly.3
[ -d weekly.1 ] && mv weekly.1 weekly.2
[ -d weekly.0 ] && mv weekly.0 weekly.1
[ -d daily.3 ] && mv daily.3 weekly.0
fi
[ -d daily.3 ] && rm -rf daily.3
[ -d daily.2 ] && mv daily.2 daily.3
[ -d daily.1 ] && mv daily.1 daily.2
[ -d daily.0 ] && mv daily.0 daily.1
fi
if [ ! -d daily.0 ]; then
mkdir daily.0
# Adjust permissions
$CHMOD 050 daily.0
$CHOWN root:${GID} daily.0
fi
}
# run the rsync
run_rsync() {
$NICE -n 12 $RSYNC -azR -e "$SSH -i $KEY -p $PORT" --exclude-from=${EXCLUDE_FILE} --delete --timeout=1800 --link-dest=../daily.1 $RUSER@$RHOST:"$RPATH" $LPATH/daily.0
# error 23 = partial transfer, probably temp files
# error 24 = file vanished
if (( e = $? )); then
if [ "$e" -ne 23 ] && [ "$e" -ne 24 ]; then
echo "ERROR: $e, Rsync failed. Exiting..."
post_sync $e
fi
fi
# Confirm backup is complete so files are rotated on the backup side
$RSYNC -a -e "$SSH -i $KEY -p $PORT" --timeout=10 $RUSER@$RHOST:/.rsync_bak_complete $LPATH/daily.0
if (( e = $? )); then
if [ "$e" -ne 23 ] && [ "$e" -ne 24 ]; then
echo "ERROR: $e, Rsync CONFIRMATION failed."
echo "Check backup completion. Exiting..."
post_sync $e
fi
fi
}
# Unmount
unmount() {
$UMOUNT $LPATH
if (( e = $? )); then
echo "ERROR: $e, Could not unmount ${LPATH}."
echo "Check your mounts. Exiting..."
exit $e
fi
}
# Post Sync
post_sync() {
$RM -rf .rsync_start
popd
sleep 10
unmount
exit $1
}
# Pre Sync
pre_sync() {
pushd $LPATH
sleep 10
if [ -f .rsync_start ]; then
echo "Rsync still running... Exiting!"
exit 1;
fi
mount_write;
rotate_backups;
$TOUCH .rsync_start
}
# Main
badargs;
include_conf;
pre_sync;
run_rsync;
post_sync 0;
There is a configuration and an exclude file, conf.txt and exclude.txt .
"conf.txt" includes the details of the server that need to be backed up.
PORT=22
RUSER=root
RHOST=DOMAIN.TLD
RPATH='/etc /var/lib/mysql /var/www /var/spool/mail /var/spool/squirrelmail'
GID=admin138
"exclude.txt" file has the list of files to be excluded from being backed up.
# exclude patterns (one per line).
#/var/www/vhosts/*/statistics/logs/*
/var/www/*/log
Problem with crashed mountpoints
When the server crashed and can back up, the mountpoints would fail to automount as the filesystem needed manual repair via fsck.
So, also added the below line to "/etc/fstab" to auto fsck and recover journals on boot.
/dev/hdb2-vg00/lv-bak /mnt/lv-bak ext3 noauto,nodev,noexec,nosuid 0 2
This also allowed me to do quick read-write mounts when needed.