Patching and migrating systems

Overview

Documentation on how to patch and replace or migrate systems in the sudo.is infrastructure.

Systems

Home Assistant

Downtime

Patching this system requires a brief peroid of downtime, as it is not possible to run Home Assistant in any sort of high availability configuration.

This is largely due to HA using the configuration directory to store data in .json files, separate from the data that is stored in the database.

HA needs to be shut down on the old host before a final rsync is done and, if needed, the MariaDB database is imported on the new host.

First start by standing up the new host and with inventory:

hass_container_state: stopped

Then converge the new host:

$ ansible-playbook common.yml --limit hass --diff
$ ansible-playbook hass.yml --diff

And copy over Home Assistant's home directory:

$ ansible-playbook rsync-paths.yml -e "srchost=${oldhost}" -e "dsthost=${newhost}" \
    -e "paths=/srv/hass/"

As a sanity check, validate that the dir is about the same size:

$ ansible hass -m command -a "du -sh /srv/hass/"

Before cutting over, take a backup of the MariaDB database, and rync both the .sql file and the HA home directory:

$ ansible $oldhost -m command -a "/usr/local/bin/mariadb_backup.sh"
$ ansible-playbook rsync-paths.yml -e "srchost=${oldhost}" -e "dsthost=${newhost}" \
    -e "paths=/srv/hass/"
$ ansible-playbook rsync-paths.yml -e "srchost=${oldhost}" -e "dsthost=${newhost}" \
    -e "paths=/srv/mariadb_dumps/hass.sql.gz"

Cutting over procdeure:

  1. Shut down HA on ${oldhost}.
  2. Create a MariaDB dump and import it on ${newhost}.
  3. Do a final rsync.
  4. Update DNS.
  5. Start HA on ${newhost}.

After cutting over, converge the new host again:

$ ansible-playbook hass.yml --diff
$ ansible-playbook common.yml --limit hass --diff

And then monitor for issues.