Ubuntu OS upgrade to 22.04

Last updated: 2023-06-02 15:56

As Ubuntu 18.04 has reached its End of (standard) Support on April 30, 2023, DC has to upgrade Ubuntu to the latest Long Term Support version: 22.04.

The upgrade process – if scheduled – runs at night (unless configured otherwise) after the nightly HD4DP 2.0 update service (hd-updater). It’s a set of services (hd-os-upgrader & docker_reinstaller) and a container that will upgrade both the OS and the required applications to run HD4DP 2.0.

The Ubuntu OS upgrade process at data providers is planned and tracked in this document.

For data providers, we do not schedule on Fridays.

How to run/enable the OS upgrade

To run the OS upgrade, the following must be set:

  • the "ubuntu_os_upgrader" role must be in the playbook of host (hd-devops repository), this in itself will not run the upgrade, it will just install the service that can run the upgrade
  • the "run_ubuntu_upgrade" variable must be set to true in the host_vars (hd-inventory) for the host to be upgraded, this variable defaults to false

Other optional settings

In addition to the above, some other settings can be set in host_vars:

  • "os_upgrade_oncalendar_timer": the scheduled time to run, by default set to run at 03:00 and 05:00 (Mon..Fri 3,5:30)
  • "arxus_dns": if set to true, the process will make sure the OS is configured to use Arxus' DNS servers, false by default
  • "is_data_provider": this will make sure some packages (related to lxd and snap) we know are unneeded at data providers are removed before the upgrade, set to true by default

The upgrade process

The upgrade process relies on the hd-updater image, but runs only the "ubuntu_os_upgrader" role.

The upgrade process creates service files in the /lib/systemd/system and in its own working directory: /opt/ubuntu_os_upgrader/

The OS upgrade process is as follows:

  1. installation of the "hd-os-upgrader" SystemD service and timer, the process stops here unless "run_ubuntu_upgrade" is set
  2. prechecks are run to make sure the OS upgrade can run successfully
  3. pre-update/upgrade tasks make sure the host is in an "upgradeable" state
  4. MongoDB and PostgreSQL backup services are run if they are present on the host
  5. the OS packages are updated to their latest version, this is a requirement of the Ubuntu OS upgrade process (see Canonical documentation)
  6. Ubuntu is upgraded to release 20.04 (focal)
  7. Ubuntu is upgraded to release 22.04 (jammy)
  8. the Docker reinstaller ("docker_reinstaller") service is installed, this is run by a separate service because the rest of the process runs in Docker, but we also have to reinstall Docker itself
  9. the docker_reinstaller, runs Ansible code locally to reinstall Docker and docker-compose (see next section)
  10. post-upgrade tasks are run to make sure the process will not run again

Docker reinstallation

Docker has to be reinstalled because after the OS upgrade old docker networking rules in iptables are kept ("iptables-legacy"). These will prevent normal operation. Reinstallation makes sure these old rules are purged.

First the Docker and docker-compose components are removed entirely. then, the reinstallation can run in two different ways, depending on the host:

  • if hd-docker-installer is present, the installation will be done with this service
  • if the above service is absent, the Ansible code will install the latest Docker available in Docker's repositories and docker-compose with pip

On completion - Validation

On completion of the OS upgrade some lock files should be present in the /opt/ubuntu_os_upgrader/ directory:

  • hd-os-docker_reinstall.lock
  • hd-os-upgrade-backup.lock
  • hd-os-upgrade.lock

If all three lock files are present, the OS upgrade was completed successfully and will not run again.

Check the Ubuntu release in the "/etc/os-release" file.

The status of the HD4DP 2.0 containers can be verified with the "docker-compose ps" command is the "/opt/hd-all" directory. The frontend can be reached at port 443 on the IP of the HD4DP 2.0 server, note that "Error loading organizations" errors may occur until all containers have loaded successfully (it may take a few minutes).

Troubleshooting

Check the status of the "hd-os-upgrader" and "docker_reinstaller" services.

OS upgrade logs are written to "/var/log/healthdata/hd-os-upgrader.log".