Differences between revisions 23 and 24
Revision 23 as of 2012-10-02 21:40:59
Size: 17002
Editor: KarlPinc
Comment: hold/unhold multiple packages at once.
Revision 24 as of 2012-10-03 14:00:20
Size: 17026
Editor: KarlPinc
Comment: Clarify how to unhold mutiple packages
Deletions are marked like this. Additions are marked like this.
Line 300: Line 300:
After identifying the package(s) which must be manually updated, ssh in to the backup server to perform the update and, again, reboot the computer. The following sequence of commands will update, e.g., the linux-image-2.6.26-2-486 package. If more than one package must be manually updated then unhold/hold all of the packages before/after doing {{{aptitude safe-upgrade}}}. (You may issue multiple {{{aptitude unhold ...}}} {{{aptitude hold ...}}} commands or simply list all the packages to be unheld or held on the command line following the {{{unhold}}}/{{{hold}}} keyword.) (Explanatory remarks beginning with ;# follow the commands.) After identifying the package(s) which must be manually updated, ssh in to the backup server to perform the update and, again, reboot the computer. The following sequence of commands will update, e.g., the linux-image-2.6.26-2-486 package. If more than one package must be manually updated then unhold/hold all of the packages before/after doing {{{aptitude safe-upgrade}}}. (You may issue multiple {{{aptitude unhold ...}}} {{{aptitude hold ...}}} commands or simply list all the packages, separated with spaces, to be unheld or held on the command line following the {{{unhold}}}/{{{hold}}} keyword.) (Explanatory remarks beginning with ;# follow the commands.)

Administration Guide to Backups of the Entire Computer

Design Overview

The backup computer must be physically plugged into an ethernet network that is itself connected to the Internet. It will automatically configure itself via DHCP.

The backup computer is a stock Debian Linux computer. Backups are kept on an external USB 2.0 hard drive. Backups are daily rsync snapshots, complete copies, of all of papio's filesystems via a script invoked from /etc/cron.d/rsync_backup. There should be at least 100 of them, probably more, in directories named by timestamp. The oldest backups are removed as space runs out or the limit of 400 backups is reached. The filesystem is ext3. On the backup computer the external hard drive is mounted on /srv/backups/papio/.

The expectation is that to restore we would give the external hard drive to Duke technical support staff. We could restore individual files or even database backups over the network but anything close to a restore on bare metal we'd expect Duke to have primary responsibility.

Papio backs up the database to disk daily (by /etc/cron.daily/babase_postgres_backup.cron) as standard pg_dump files. The output is in /srv/babase_database/postgres/. To restore the database use the the standard postgres restore tools, i.e. pg_restore. See the restore instructions in the backup script.

The VPN Tunnel

Because the backup server may not have a static IP, and to get around NAT issues on the network to which the backup server is connected, the backup server initiates a non-necessarily-encrypted (but defiantly authenticated) OpenVPN VPN tunnel to papio. All communication between papio and the backup server is through this tunnel.

Papio initiates all the backups. Papio ensures that the VPN tunnel to the backup server does not forward to the rest of the network. Other than the VPN connection there are no inbound connections to papio.

The OpenVPN tunnel used for backups listens on papio on port 1195, rather than the usual 1194. This is because the VPN used for backups uses certificate authentication whereas the regular VPN does not.

The rsync command uses --hard-links so that no additional storage is allocated for those files that do not change between backups. The ext3 backup partition is created with (mke2fs -i 4096) 4096 bytes per inode to allocate the additional inodes such a scheme requires.

File ownership

Backups are stored with the numeric uid and gids used on papio. This may or may not, likely not, correspond to the uids and gids on the backup sever. Caution is required.

The backup script

The backup script is custom because at the time of this writing the rsnapshot program does not purge based on partition space available or is otherwise oriented around partitions.

Backups of the backup server(s)

The backup server is itself backed up to papio using the same methods. Again, all connections are initiated by papio.

Administrative Tasks

There are two administrative tasks to be performed daily.

The administrator of the backup computer receives daily emails reporting the status of the daily security updates. The first task of the administrator is to monitor these emails. Should the email report errors, an extremely unlikely occurence, a Linux administrator should be called in to examine the situation. Should the email report that manual intervention is required to install a security update the administrator should follow the procedure below to install the security update and to reboot the backup computer.

Should a daily backup, or some other automated operation, fail, the administrator will receive an email reporting this problem. The second task of the administrator is to refer these failures to a Linux administrator for resolution.

Aside from the backup itself, the only automated process specific to the backup system is a daily check that a minimum number of backups exist. Currently this minimum is set to 30. Mail is sent to the administrator for possible action should the number of existent backups fall below the limit. This number, as well as the maximum number of backups to keep, is set in /etc/cron.d/purge_backups.

Connecting to the backup computer

Logins, usernames and passwords, are required to connect to the backup computer. They are handed out by Karl, or whomever has the root password.

Those with physical access can plug in a keyboard and screen, and even a mouse if a GUI is desired.

Connections may be made to the backup computer using ssh from the physical network (LAN) to which the backup computer is plugged in. The backup computer obtains it's IP address via DHCP. If connections via the LAN are to be made it is up to the LAN administrator to keep track of the IP address assigned to the backup computer.

The presumption is that NATting will prevent arbitrary hosts on the Internet from connecting to the backup server. If this is not the case it is up to the network administrator of the backup server's network to firewall the backup server to prevent ssh connections from the Internet.

Connections from the Internet to the backup computer are made over ssh, via putty or some other ssh client. The user must first connect to papio and then connect over the VPN to the backup computer. This approach serves two purposes: it bypasses NATting and inbound connection firewalling; and it renders moot the occasional random changes most consumer ISPs make to assigned IP addresses which, in turn, randomly change the "location" of the backup server on the Internet.

Once logged in to papio the command to connect to the backup server (where username is your assigned login name) is:

ssh -l username backup-server1

If the username on the backup server is the same as the username used on papio the command may be shortened to:

ssh backup-server1

Disconnecting from the backup computer

Those who use ssh to connect to the backup computer can disconnect by typing:

exit

This command will have to be typed twice if the su command is in effect, once to exit from the su command and cease being root and again to exit as a normal user.

The exit command can often be typed rapidly enough after the machine has been told to halt or reboot that the session will end before the backup computer finishes it's shutdown sequence. Typing exit in these circumstances has the advantage of taking effect immediately. It may otherwise take some time for the ssh session to realize that the backup computer has stopped.

Daily messages

There are 3 sorts of daily messages.

If no message at all is received this is a sign of trouble and should be investigated.

Typically if an email is not received it is because the DSL modem used to connect the backup server's network to the Internet has locked up. The typical solution is to physically unplug the power cord of the DSL modem, wait 30 seconds, plug the power back in, and wait 2 minutes before testing that the Internet is again available on the local network.

Nothing done

The typical daily message will indicate that nothing happened.

Subject:        Cron <root@foo> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
Date:   09/09/2009 06:25:33 AM
From:   Cron Daemon <root@foo.example.com>
To:     root@foo.example.com

/etc/cron.daily/security_updates:
No security updates to apply.

The administrator need do nothing upon receiving such a message.

Security updates performed

On occasion the system will automatically update itself. This example shows 2 packages being updated.

Subject:        Cron <root@foo> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
Date:   09/09/2009 06:25:33 AM
From:   Cron Daemon <root@foo.example.com>
To:     root@foo.example.com

/etc/cron.daily/security_updates:
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Reading task descriptions...
The following packages will be upgraded:
  libmysqlclient15off mysql-common
2 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 1920kB of archives. After unpacking 0B will be used.
Writing extended state information...
(Reading database ... 40166 files and directories currently i
nstalled.)
Preparing to replace mysql-common 5.0.51a-24+lenny1 (using .../mysql-common_5.0.51a-24+lenny2_all.deb) ...
Unpacking replacement mysql-common ...
Preparing to replace libmysqlclient15off 5.0.51a-24+lenny1 (using .../libmysqlclient15off_5.0.51a24+lenny2_i386.deb) ...
Unpacking replacement libmysqlclient15off ...
Setting up mysql-common (5.0.51a-24+lenny2) ...
Setting up libmysqlclient15off (5.0.51a-24+lenny2) ...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Reading task descriptions...

Note that the important part is the bottom where, unlike the next example, there is no indication that the administrator need manually intervene. There are times when the system automatically installs some updates but others require manual installation.

The administrator should scan such emails for words like "error" or "fail" in the unlikely event that something failed during the automatic security update.

Manual intervention required

Less frequently the administrator must manually make a security update and reboot the backup computer. Messages like the one that follows indicate this.

Subject:        Cron <root@foo> test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
Date:   09/09/2009 06:25:33 AM
From:   Cron Daemon <root@foo.example.com>
To:     root@foo.example.com

/etc/cron.daily/security_updates:
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Reading task descriptions...
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Need to get 0B of archives. After unpacking 0B will be used.
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Reading task descriptions...

Security (and other) updates that have been held back:
ihA linux-image-2.6.26-2-486        - Linux 2.6.26 image on x86

Note that the message may contain multiple lines that read: error: out of partition. This error is spurious and can be safely ignored so long as a line beginning with run-parts: executing /etc/kernel/postinst.d/zz-update-grub appears somewhere in the preceding text.

The important parts here are the last 2 lines. If the message contains the line:

Security (and other) updates that have been held back:

Then manual intervention is required.

The second line

ihA linux-image-2.6.26-2-486        - Linux 2.6.26 image on x86

reports which package must be manually upgraded. In this case the package is linux-image-2.6.26-2-486.

Before beginning note which package(s) must be manually updated.

Initial reboot of the backup computer

Although it is not required it is good practice to reboot the backup server before a manual upgrade. A reboot will trigger an automatic check of the disk content, if one is needed. Should a problem with the disk content ever be discovered it may require physical interaction with the backup computer's keyboard to fix. Rebooting before a manual upgrade allows for more accurate remote problem diagnosis should there be a problem bringing the backup computer back on-line after a manual update.

Note that should a reboot trigger a check of disk content (a "fsck") the check can take many hours to complete and the computer will not return to service until the check has finished.

To reboot ssh into the backup server and issue the following sequence of commands. (Explanatory remarks beginning with ;# follow the commands.)

su                                               ;# You will be prompted for the root password.
reboot                                           ;# Restart the computer, checking the disk if necessary.

Manually update packages

After identifying the package(s) which must be manually updated, ssh in to the backup server to perform the update and, again, reboot the computer. The following sequence of commands will update, e.g., the linux-image-2.6.26-2-486 package. If more than one package must be manually updated then unhold/hold all of the packages before/after doing aptitude safe-upgrade. (You may issue multiple aptitude unhold ... aptitude hold ... commands or simply list all the packages, separated with spaces, to be unheld or held on the command line following the unhold/hold keyword.) (Explanatory remarks beginning with ;# follow the commands.)

su                                               ;# You will be prompted for the root password.
aptitude update                                  ;# These next two commands perform 
aptitude safe-upgrade                            ;# non-security related updates.
aptitude unhold linux-image-2.6.26-2-486         ;# This begins the update of the desired package.
aptitude safe-upgrade                            ;# If prompted whether you really wish to replace
                                                 ;# currently running kernel, answer affirmatively.
aptitude hold linux-image-2.6.26-2-486           ;# Force manual updates in the future.
aptitude search linux-image-2.6.26-2-486         ;# Check the last step worked.
                                                 ;# The first 2 letters should be 'ih'.
reboot                                           ;# Restart the computer with the new kernel.

As the backup computer reboots your ssh session will be terminated and you will return to your login (ssh) session on papio. Check to ensure that the backup computer successfully reboots with the following command:

ping backup-server1

You know the backup server is responding when replies like the following are received:

64 bytes from foo.example.com (192.168.199.2): icmp_seq=1 ttl=254 time=1.59 ms

Terminate the ping command by holding down the Control key (usually labeled "Ctrl") and, while Ctrl is held down, pressing the "c" key.

Note that on occasion, typically at least every 120 days, the backup server will perform file system checks as part of the reboot. This can take quite some time, possibly hours, and the system will not respond to pings until it has finished. If the backup system does not come back up after reboot contact a Linux administrator for assistance.

After verification that the backup server is operational type:

exit

to end your ssh session with papio.

Note: At this time the only packages that require manual intervention are new versions of the kernel. This list should probably be expanded to include certain critical libraries like glibc.

Shutting down the computer

There are times when the computer must be dis-connected from power. If the backup computer is physically powered off without first being shutdown data loss is possible, although it is unlikely that anything significant will be lost unless a backup is running at the time of power loss and even then it is unlikely that database content will be lost.

To avoid any possibility of data loss shut down the computer manually before removing power. The easy way to do this is to press the power button on the front and wait for all the lights to go off. The computer will detect the button press and shut down cleanly.

An alternate approach, which may be necessary with older software (dumbo before it gets it's upgrade) is to first connect to the backup computer as described above. Then type:

su                                               ;# You will be prompted for the root password.
halt

After power is physically restored the power button on the front of the computer must be pressed to restart the machine.

OSBackups (last edited 2012-10-03 14:00:20 by KarlPinc)

Wiki content based upon work supported by the National Science Foundation under Grant Nos. 0323553 and 0323596. Any opinions, findings, conclusions or recommendations expressed in this material are those of the wiki contributor(s) and do not necessarily reflect the views of the National Science Foundation.