I describe essential backup procedures that we use.

Traditionally, we backup each night. We keep nightly backups for about one week; weekly backups for about one month; monthly backups for about three months; and quarterly backups for about two years. This is the main idea. The details depend on the backup system that we use each time.

The system that we use right now is duply, as described in my book, “Deploying Django on a single Debian or Ubuntu Server”. The backup space is on a server called thames. This is the main backup. However, we also have offline backups, for the reasons also explained in the book. What we do is occasionally connect a hard disk to thames (SATA using a SATA-to-USB adapter) and create an extra copy of its backup. We do this every month, recycling disks older than three months but not quarterly, and recycling the quarterly ones after a year. The disks are carried to someone’s home.

How to perform an extra backup

Plug a hard disk on to thames.

Note: You must have logged in at thames with the -A parameter so that you can logon to the other servers. In addition, you must start the procedure early enough to not interfere with daily backup that occurs at 06:25. Given that this procedure takes about fourteen hours, start it up to noon and you’re OK.

Run a terminal multiplexer:

tmux

From now on we will be referring to the device name (“sdc” or “sdd” or “sde” etc.) as $BACKUP_DISK.

Run fdisk:

fdisk /dev/$BACKUP_DISK

and make sure that all the disk is one partition, type 83.

Create an ext4 (older backups may be on XFS) filesystem on the disk, with journal disabled (faster), and mount it:

mkfs.ext4 -L `date +%Y-%m-%d` /dev/${BACKUP_DISK}1
tune2fs -O ^has_journal /dev/${BACKUP_DISK}1
mount -o noatime,nodiratime,nobarrier /dev/${BACKUP_DISK}1 /mnt

Copy the backup:

SERVER_LIST="
ilissos.openmeteo.org
megdobas.irrigation-management.eu
mornos.itia.ntua.gr
acheron.hydroscope.gr
"

for server in $SERVER_LIST; do
    echo $server
    mkdir /mnt/$server

    ssh -n root@$server \
       "awk '{ print \$2 }' /etc/duply/main/exclude >/tmp/exclude && \
        tar czf - --exclude-from=/tmp/exclude  /" | \
      split --bytes=200M - \
        /mnt/$server/$server-`date --iso-8601`.tar.gz. &
done

After it finishes:

umount /mnt
udisks --detach /dev/$BACKUP_DISK

Then disconnect the disk and stick a label to it that says “ITIA BACKUP YYYY-MM-DD“.

List of extra backups

Date	Was	Model	Serial	Location
2016-04-06	2015-09-08, 2014-12-15	WD75	5633	E
2017-05-11	2017-02-13, 2016-11-15	WD75	5980	E
2018-04-18	2017-09-11	T1		Off
2018-11-28	2018-02-21, 2015-10-19, 2015-02-10, 2011-12-22	WD75b	9664	Off
2019-01-14	2016-07-11, 2016-02-09, 2015-08-10	WD1	5264656	Off
2019-04-16	2017-01-09, 2015-01-19, 2012-04-10	WD1	5267182	Off
2019-05-13	2017-04-10, 2015-04-21, 2012-03-05	WD1	5325274	>C
2019-06-11	2019-02-12, 2016-10-10, 2014-10-15, 2013-03-20	WD1	5271585	>On
2019-07-24*	2019-03-18, 2018-05-15, 2016-01-12, 2015-06-17, 2013-06-18	WD1	5263615	Off
2019-09-23*	2017-07-10, 2015-07-21, 2013-07-18	WD1	5268593	Off
2019-10-14	2017-10-20, 2017-06-09, 2017-03-09, 2016-12-09	WD75	5741	>Off

*Acheloos was backed up on 2018-08-27, and its backup is incomplete (initially its connection was flaky, probably because of related works by NOC, and when we attempted again a month later at some point the connection broke).
2019-09-23 contains the last backup of acheloos.

Models: WD75=WD7500AACS-00D6B0 (serial starts with WD-WCAU5018), WD75b=WD7500AAKS-65RBA0 (serial starts with WD-WCAPT106), WD1=WD1002FAEX-00Z3A0 (serial starts with WD-WCATR), T1=Toshiba DT01ACA100 Rev AAA AA10/750 (serial 66D9YWWNS-WK7).

Locations: E=Has errors, On=On site, Off=Off site, C=Connected, >Off=To go off site, >On=To go to site, >C=To connect