Administration Guide to Backups of the Entire Computer
Design Overview
The backup computer must be physically plugged into an ethernet network that is itself connected to the Internet. It will automatically configure itself via DHCP.
The backup computer is a stock [http://debian.org/ Debian] Linux computer. Backups are kept on an external USB 2.0 hard drive. Backups are daily rsync snapshots of all of papio's filesystems via a script invoked from /etc/cron.d/rsync_backup. There should be at least 100 of them, probably more, in directories named by timestamp. The oldest backups are removed as space runs out or the limit of 400 backups is reached. The filesystem is ext3. On the backup computer the external hard drive is mounted on /srv/backups/papio/.
The expectation is that to restore we would give the external hard drive to Duke technical support staff. We could restore individual files or even database backups over the network but anything close to a restore on bare metal we'd expect Duke to have primary responsibility.
Papio backs up the database to disk daily (by /etc/cron.daily/[http://papio.biology.duke.edu/repos/babase/doc/pgsql_examples/babase_postgres_backup.cron babase_postgres_backup.cron]) as standard pg_dump files. The output is in /srv/babase_database/postgres/. To restore the database use the the standard postgres restore tools, i.e. pg_restore. See the restore instructions in the backup script.
The VPN Tunnel
Because the backup server may not have a static IP, and to get around NAT issues on the network to which the backup server is connected, the backup server initiates an unencrypted (but authenticated) OpenVPN VPN tunnel to papio. All communication between papio and the backup server is through this tunnel.
Papio initiates all the backups. Papio ensures that the VPN tunnel to the backup server does not forward to the rest of the network. Other than the VPN connection there are no inbound connections to papio.
Hard links
The rsync command uses --hard-links so that no additional storage is allocated for those files that do not change between backups. The ext3 backup partition is created with (mke2fs -i 4096) 4096 bytes per inode to allocate the additional inodes such a scheme requires.
File ownership
Backups are stored with the numeric uid and gids used on papio. This may or may not, likely not, correspond to the uids and gids on the backup sever. Caution is required.
The backup script
The backup script is custom because at the time of this writing the rsnapshot program does not purge based on partition space available or is otherwise oriented around partitions.
Administrative Tasks
There are two administrative tasks to be performed daily.
The administrator of the backup computer receives daily emails reporting the status of the daily security updates. The first task of the administrator is to monitor these emails. Should the email report errors, an extremely unlikely occurence, a Linux administrator should be called in to examine the situation. Should the email report that manual intervention is required to install and security update the administrator should follow the procedure below to install the security update and to reboot the backup computer.
Should a daily backup, or some other automated operation, fail, the administrator will receive an email reporting this problem. The second task of the administrator is to refer these failures to a Linux administrator for resolution.
Aside from the backup itself, the only automated process specific to the backup system is a daily check that a minimum number of backups exist. Currently this minimum is set to 30. This number, as well as the maximum number of backups to keep, is set in /etc/cron.d/purge_backups. (The maximum should be set low enough to keep the filesystem from running out of inodes.)
Connecting to the backup computer
Logins are required to connect to the backup computer. They are handed out by Karl, or whomever has the root password.
Those with physical access can plug in a keyboard and screen, and even a mouse if a GUI is desired.
Connections may be made to the backup computer using ssh from the physical network (LAN) to which the backup computer is plugged in. The backup computer obtains it's IP address via DHCP. It is up to the LAN administrator to keep track of the IP address assigned to the backup computer.
The presumption is that NATting will prevent arbitrary hosts on the Internet from connecting to the backup server. If this is not the case it is up to the network administrator of the backup server's network to firewall the backup server if ssh from the Internet is not desired.
Connections from the Internet to the backup computer are made over ssh, via putty or some other ssh client. The user must first connect to papio and then connect over the VPN to the backup computer. This approach serves two purposes: it bypasses NATting and inbound connection firewalling; and it renders moot the occasional random changes most consumer ISPs make to assigned IP addresses which, in turn, randomly change the "location" of the backup server on the Internet.
Once logged in to papio the command to connect to the backup server is:
ssh -l username backup-server1
If the username on the backup server is the same as the username used on papio the command may be shortened to:
ssh backup-server1
Daily messages
There are 3 sorts of daily messages.
If no message at all is received this is a sign of trouble and should be investigated.
Nothing done
The typical daily message will indicate that nothing happened.
Subject: Cron <root@lethe> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) Date: 09/09/2009 06:25:33 AM From: Cron Daemon <root@lethe.meme.com> To: root@lethe.meme.com /etc/cron.daily/security_updates: No security updates to apply.
The administrator need do nothing.
Security updates performed