Backups with rsnapshot and SSH

Posted on 17 July 2018 by AJ Ianozi

This is a simple schema on how I configure backups over SSH between my various servers. The howto on this stuff can be found all over the web, I’m mostly consolidating it here for my own records, in case I want to do it again later. Big thanks to a few guides on helping me put this together, if you’d like some further reading:

All devices in question run Gentoo, but this should work on anything with rsnapshot and openssh.

Naming Conventions

For the purpose of this guide, the system we’re backing things up to will be called NAS and the system we’re backing up will be called DESKTOP. Your naming conventions may very.

Step 1: Install needed software

On backup server, we need to install rsnapshot. In Gentoo this is:

aj@NAS ~ $ sudo emerge -av rsnapshot
These are the packages that would be merged, in order:

Calculating dependencies... done!

[ebuild  N     ] dev-perl/Lchown-1.10.0-r1::gentoo  4 KiB
[ebuild  N     ] app-backup/rsnapshot-1.4.2::gentoo  477 KiB

Total: 2 packages (2 new), Size of downloads: 481 KiB

Would you like to merge these packages? [Yes/No]

On the desktop, we need to install rsync. In Gentoo that’s sudo emerge -av rsync (I already have rsync).

Step 2: Create “backup” user on NAS and DESKTOP

This is the account that will be used to ssh into both of the computers.

aj@NAS ~ $ useradd -c "Backup user" backup
aj@DESKTOP ~ $ useradd -c "Backup user" backup

Step 3: Create rsync validation and wrapper scripts.

Big props to Arun Natarajan for coming up with this one. The purpose of this script/check is to only allow rsync as a possible command. This way, if anyone gets hold of the “backup” account on NAS, they can’t do anything (too) bad on my DESKTOP.

Authenticate as the “backup” user on DESKTOP, and create a new file called validate-rsync.sh:

aj@DESKTOP ~ $ # Change to the backup user
aj@DESKTOP ~ $ sudo su - backup
backup@DESKTOP ~ $ # Create an "sh" directory if it doesn't exist, we'll put it in there.
backup@DESKTOP ~ $ mkdir -p ~/sh
backup@DESKTOP ~ $ cd ~/sh
backup@DESKTOP ~/sh $ # Create script file
backup@DESKTOP ~/sh $ touch validate-rsync.sh
backup@DESKTOP ~/sh $ # Edit script file
backup@DESKTOP ~/sh $ vi validate-rsync.sh

Open “validate-rsync.sh” with your favorite text editor and paste the following:

#!/bin/sh
case "$SSH_ORIGINAL_COMMAND" in
  *\&*)
    echo "Rejected 1"
    ;;
  *;*)
    echo "Rejected 2"
    ;;
    *rsync*)
    $SSH_ORIGINAL_COMMAND
    ;;
  *true*)
    echo $SSH_ORIGINAL_COMMAND
    ;;
  *)
    echo "Rejected 3"
    ;;
esac

Once saved, we’ll want to make sure it’s executable and not readable by anyone else:

backup@DESKTOP ~/sh $ chmod 700 validate-rsync.sh

Now we do the same for the wrapper:

backup@DESKTOP ~/sh $ touch rsync-wrapper.sh
backup@DESKTOP ~/sh $ vi rsync-wrapper.sh

Paste the following into rsync-wrapper.sh:

#!/bin/sh
logger -t backup $@
/usr/bin/sudo /usr/bin/rsync "$@";

And make it executable:

backup@DESKTOP ~/sh $ chmod 755 /home/backup/sh/rsync_wrapper.sh

Step 4: Generate SSH keys for backup user authentication

The backup user on NAS will have to authenticate on DESKTOP. The most secure way of doing this is by creating a certificate. I choose to go with Ed25519.

aj@NAS ~ $ # Change to the backup user
aj@NAS ~ $ sudo su - backup
backup@NAS ~ $ # Generate the key
backup@NAS ~ $ ssh-keygen -t ed25519
Generating public/private ed25519 key pair.
Enter file in which to save the key (/home/backup/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
The key's randomart image is:
+--[ED25519 256]--+
|pretty image here|
+----[SHA256]-----+
backup@NAS ~ $ # Give the key a relevant name
backup@NAS ~ $ mv ~/.ssh/id_ed25519 ~/.ssh/id_ed25519_DESKTOP
backup@NAS ~ $ mv ~/.ssh/id_ed25519.pub ~/.ssh/id_ed25519_DESKTOP.pub

Step 5: Add the new certificate to DESKTOP

We’ll need to retrieve the public key from NAS and add it to the desktop’s authorized_keys file. Usually this is as simple as running ssh-copy-id, but remember, our backup system accounts don’t have passwords, so we won’t be able to authenticate. If you’re not like me, and actually created a password for both backup accounts, then you can simply run:

aj@NAS ~ $ # Change to backup user
aj@NAS ~ $ sudo su - backup
backup@NAS ~ $ cat ~/.ssh/id_ed25519_DESKTOP.pub
ssh-ed25519 backup@NAS

Copy that whole thing, then, on DESKTOP:

aj@DESKTOP ~ $ # Once again, we should be backup user
aj@DESKTOP ~ $ sudo su - backup
backup@DESKTOP ~ $ # Edit the "authorized_keys" file
backup@DESKTOP ~ $ vi .ssh/authorized_keys

Please use your editor of choice. On the last free line of the authorized_keys file, place:

command="/home/backup/sh/validate-rsync.sh"

Then, a space, and the entire contents of “id_ed25519_DESKTOP.pub”. For example, this is what it might look like:

command="/home/backup/sh/validate-rsync.sh" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIF3ZY+TF+/Whv4768hHxxgoPXWDnq7/iyiadSgE7DCq backup@NAS

After saving, make the file read-only:

backup@DESKTOP ~ $ chmod 600 .ssh/authorized_keys

Step 4.5: Configure SSHD

If you haven’t configured SSHD on DESKTOP yet, this is probably a good time to do it. It may be a good idea to lock it down completely, except for key-based authentication. The Gentoo Security Handbook has a pretty good writeup on securing SSH, so feel free to check that out. Warning: If your only access to DESKTOP is through SSH, you can lock yourself out of the system by doing this.

Step 5: Configure rsnapshot on NAS

We’re going to create a new configuration for rsnapshot. I’m just going to put it in /etc/, but you can have it in the /home/backup folder if you’d like. I haven’t tested it outside /etc/:

aj@NAS ~ $ sudo vi /etc/rsnapshot-DESKTOP.conf

Here’s my example config for rsnapshot; you’ll want to modify ssh_args, snapshot_root, and backup:

# Be sure to use TABS to separate values, not spaces!

# So the hard disk is not polluted in case the backup filesystem is not available
no_create_root		1

# Standard settings
cmd_cp		/bin/cp
cmd_rm		/bin/rm
cmd_rsync	/usr/bin/rsync
cmd_ssh		/usr/bin/ssh
link_dest	1

verbose		5
loglevel	5

rsync_long_args		-evaAX --rsync-path=/home/backup/sh/rsync-wrapper.sh
ssh_args		-i /home/backup/.ssh/id_ed25519_DESKTOP

# For convenience, so that mount points can be taken as backup starting points
one_fs		1

# Store all backups in one directory per machine
# A useful alternative may be to create a separate directory for each interval
snapshot_root	/backups/DESKTOP

# increments, which are kept
retain		daily			7
retain		weekly		4
retain		monthly		12

# Exclude pattern (refer to --exclude-from from rsync man page)
exclude		/dev
exclude		/proc
exclude		/sys
exclude		/run
exclude		/var/tmp
exclude		/var/run
exclude		/tmp
exclude		/lost+found
exclude		/mnt

# backup of remote.example.com server
backup		backup@DESKTOP:/	DESKTOP/
#backup		backup@remote.example.com:/otherdir/		remote.example.com/otherdir

NOTE: If /home or other things are on seperate partitions, you’ll have to add them to the bottom of the config, e.g.:

backup		backup@DESKTOP:/home		DESKTOP/home

WARNING: Remember, the config above requries tabs not spaces. If you need a copy of this config to edit in a plain text editor, you can retrive one here:

curl https://aj.immo/assets/scr/rsnapshot-DESKTOP.conf --output ~/rsnapshot-DESKTOP.conf

The file will be saved in your home folder to cat, vim, or cp.

Step 6: Configure sudo on DESKTOP

We’ll want to configure sudo to allow the backup user to do what it needs to do with rsync. To edit the sudo file I use the visudo command:

aj@DESKTOP ~ $ sudo visudo

And add this somewhere (probably near root ALL=(ALL) ALL):

## backup user
backup    ALL=(root) NOPASSWD: /usr/bin/rsync

Step 7: Cron Jobs

Before we go any further, let’s see if you have access to the ionice command. In your shell, just type ionice --help.

If you get command not found, I recommend consulting your distribution’s documentation to install it. For more information, check out the article Why aren’t you using ionice yet???

Now, onto the cron jobs!

Crontab vs cron.[time]

There are two ways to add a crontob; through crontab (usually by running the command crontab -e) or by adding the commands in /etc/cron.daily / .weekly / .monthly. On my server, I use crontab, as I want to specify the exact times that I want my backups to run; however, there’s nothing wrong with using the scripts in /etc/… I’ll cover both.

`/etc/cron.[time]`

Create scripts in /etc/cron.{daily,weekly,monthly}:

aj@NAS ~ $ sudo touch /etc/cron.{daily,weekly,monthly}/rsnapshot.sh

Put in /etc/cron.daily/rsnapshot.sh:

#!/bin/sh
ionice -c 3 nice -n +19 /usr/bin/rsnapshot -c /etc/rsnapshot-DEKSTOP.conf daily || echo "Daily backup for DESKTOP failed."

Put in /etc/cron.weekly/rsnapshot.sh:

#!/bin/sh
ionice -c 3 nice -n +19 /usr/bin/rsnapshot -c /etc/rsnapshot-DEKSTOP.conf weekly || echo "Weekly backup for DESKTOP failed."

Put in /etc/cron.monthly/rsnapshot.sh:

#!/bin/sh
ionice -c 3 nice -n +19 /usr/bin/rsnapshot -c /etc/rsnapshot-DEKSTOP.conf monthly || echo "Monthly backup for DESKTOP failed."

Note: If you don’t have ionice, remove ionice -c 3 nice -n +19 from the beginning of those commands.

Crontab

Run the command:

aj@DESKTOP ~ $ sudo crontab -e

And add somewhere in the crontab file:

# Backup DESKTOP
#12am every day
0 0 * * *    ionice -c 3 nice -n +19 /usr/bin/rsnapshot -c /etc/rsnapshot-DEKSTOP.conf daily || echo "Daily backup for DESKTOP failed."
#1am every week
0 1 * * 1    ionice -c 3 nice -n +19 /usr/bin/rsnapshot -c /etc/rsnapshot-DEKSTOP.conf weekly || echo "Weekly backup for DESKTOP failed."
#4am every month
0 4 2 * *    ionice -c 3 nice -n +19 /usr/bin/rsnapshot -c /etc/rsnapshot-DEKSTOP.conf monthly || echo "Monthly backup for DESKTOP failed."

Note: If you don’t have ionice, remove ionice -c 3 nice -n +19 from the beginning of those commands.

Wrapping up

If I forgot something or screwed up a command horribly, please let me know!