3 2 1 Backup feast

Backups are important. Very important. Fortunately, the combined power of zfs, rsync, and a few shell scripts makes things a breeze.

I generally try my best to follow a 3 2 1 rule for backups. As it currently stands, I have a FreeBSD server running the latest -RELEASE channel (12.2-RELEASE as of this writing) with zfs on all filesystems. Zfs pool is divided into volumes for photos, documents, videos, etc.

TODO Offsite backups with ZFS

Jails

Backing up jails with zfs snapshots takes a toll on storage as, for example, my unifi controller jail takes up 30GB after only a few months due to the logs it collects. Log data is not important enough for me to keep as archive, so I'll have to be selective about what I'm backing up.

For jails, my strategy involves listing all configuration files I care about in a file in the root of the jail and having rsync collect all jail-related config files in a central location (a zfs volume somewhere) neatly beside all the other files I want to backup.

There are generally 3 pieces of information I need to have to be able to fully restore a jail:

  1. service-specific configuration files
  2. rc.conf
  3. pkgs

to get a list of manually installed (non-automatic), use pkg --chroot /path/to/jail/root query -e '%a = 0' %o (see pkg(8) and pkg-query(8)). In my case, I have rsyslog and ssmtp installed in all my jails with uniform configs, so I'll ignore those. I also don't use bitlbee anymore.

irc/bitlbee
ports-mgmt/dialog4ports
devel/libuv
ports-mgmt/pkg
sysutils/rsyslog8
mail/ssmtp
security/sudo
editors/vim-lite
irc/znc
shells/zsh

Now that we have all the data we need, it's a good idea to start thinking about how this data will be restored in case of catastrophic failure. I've opted to use ansible to automate restoration of not only the jail environment (including packages) but also configurations which can be shared or unique to each jail.

Restoring jails with ansible

I'll be using /mpool/scripts/backups/jail-deployment as the root of my ansible configuration files for jail deployment. I'm using ansible 2.9.7 on FreeBSD 12.2-RELEASE.

Setting up ansible

Ansible has support to run tasks in jails which we will use to setup the jails. You can also use ssh if your controller is remote, but I'll be using ansible on the host itself to keep things tidy. As jail connections use ansible connection plugins, a bit of setup is required.

configurations for jails goes into ansible.cfg. I had to specify python3 as ansible was only looking for specific python3 versions.

ansible.cfg

[defaults]
inventory = hosts
interpreter_python = python3

My hosts file looks as follows.

hosts

znc   ansible_connection=jail ansible_jail_host=/jails/znc   ansible_jail_user=root
db    ansible_connection=jail ansible_jail_host=/jails/db    ansible_jail_user=root
web   ansible_connection=jail ansible_jail_host=/jails/web   ansible_jail_user=root
...

site.yml

- import_playbook: znc/site.yml

znc/site.yml

- name: setup ZNC jail
  hosts: znc
  tasks:
    - name: test connection
      ping:

Make sure the jail is running. And test it.

ansible-playbook site.yml

And it fails! FreeBSD doesn't include python in its base install. To fix it, install python using pkg in the jail's root.

pkg --rootdir /jails/znc install python3

Why --rootdir as opposed to --chroot? --rootdir uses the host's package repository and simply installs the package when specified. I have setup a poudriere repository on my host system with custom port configurations which I want my jails to use, hence my use of --rootdir.

This is when I found pkg fails because bsdinstall had installed 10.2-RELEASE instead of 12.2-RELEASE in my basejail. So back to the beginning! After fixing that, all was well.

From here on, ansible can be used as usual to configure any of the jails. How to use ansible is left as an exercise to the reader.

Databases

postgresql

postgresql provides pg_basebackup which is very handy. Postgres also has pg_dumpall. See their respective documentation for further info. I ended up using pg_basebackup since it backs up the entire $PGDATA directory including configuration files and WAL files.

mkdir postgres_backup
pg_basebackup -D postgres_backup --format=tar --gzip --progress

To see what has been backed up, use tar.

tar -t -f base.tar.gz

To restore the database, untar base.tar.gz into $PGDATA (on FreeBSD it's /var/db/postgres/dataXX/ and untar pg_wal.tar.gz into $PGDATA/pg_wal.

mariadb

mariadb provides mysqldump which is equally as handy. my.cnf has to be backed up separately. I couldn't find an equivalent to pg_basebackup for mysql.

mysqldump -u root -p -x -A | gzip > mariadb_backup.gz

Be careful, if you want to backup from pre-10.4 server and restore in 10.4 or above, make sure to add --add-drop-database so the mysql database including users is removed before potentially being created.

The error in question is ERROR 1050 (42S01) at line 2027: Table 'user' already exists.

The issue is described here.

To restore, start mysql-server, and restore.

gunzip mariadb_backup.gz
mysql -u root < mariadb_backup
(C) 2015-2021 Armin Moradi. CC-BY-SA 4.0. Proudly built with Org-mode and Pandoc