Standby hub
Weik.io 2026.2A standby hub on a second host restores and checks the primary's newest backup every six hours and takes over under the primary's names when you activate it.
On this page
For disaster recovery of a hub on one server, a standby hub keeps the primary’s newest backup archive restored on a second host, away from the primary’s location. Once you activate the standby and move the primary’s names to it in DNS, it serves under those names.
Only the standby’s published leaf address differs from the primary’s. Give the primary’s first init that address with --standby-leaf-url. Every agent’s bundle then lists it after the primary’s. After a takeover, the agents link to the standby with the bundles they already have.
Between refreshes the standby is down and keeps its leaf port closed. While the primary runs, no agent links to it.
The primary hub serves the hub's names, such as acme.example and nats.acme.example, and its backup service copies each archive to a bucket. The standby hub is a second host away from the primary's location, and it stays down between refreshes. Every 6 hours standby-refresh.sh downloads the newest archive and verifies it, removes what the restore replaces, restores the archive, stops the stack and writes one line to its log. An agent's leaf node tries the hub's leaf addresses in the order its bundle lists them, the primary's first and the standby's second, so it links to the primary while the primary runs. To take over, you remove the cron line, start the standby with docker compose up -d and point the primary's names at the standby in DNS, and the agents link to the standby.
What each refresh does#
init writes the script standby-refresh.sh beside the stack of a hub on one server. The script changes nothing on a stack whose .env has no WEIKIO_STANDBY_FROM, such as the primary’s, or while the stack’s backend runs. On a standby, the script refreshes the stack from the newest archive:
- Finds the newest archive at the address in
WEIKIO_STANDBY_FROM. It downloads that archive belowbackups/standby/and checks withrestore --verify-onlythat the standby’s master key opens every file. If the archive fails the check, the run deletes it and changes nothing else. - Stops the stack and removes the message server’s storage and the volumes of the gateway and the forwarder. It also removes every file of the keys directory except
operator.nk. - Restores the archive the way a whole restore does, in two runs with
initbetween them. The script givesinitthe standby’s leaf address. - Stops the stack again.
- Keeps the seven newest archives.
- Appends one line to
standby-refresh.login the home.
Each line in standby-refresh.log has the time and result of the run and the archive it used. For each hub name, the line also lists the certificate that name gets once you activate the standby and the days left on it.
For a hub name, the text about its certificate starts with Warning: when an uploaded certificate has under 30 days left or the renewal of a certificate Weik.io obtained is overdue. In an installation that obtains no certificates by itself, that text also starts with Warning: when the name would have only a temporary certificate.
If a step fails, the run writes its line to the log and ends with the stack down. The next run starts again from the bucket.
The script uses flock to keep two runs from overlapping.
Set up a standby#
Do each step once, on the standby’s host, after you set IMG as in Download and check the release.
-
Create the home and its keys directory as in the first step of a whole restore. Copy in the master key and the operator seed, and leave out the archive. When you change the primary’s master key with
weikio-backend secrets rotate, put the new key here too. -
Write the bucket’s address and the credentials of a key that may only read the bucket into a file only root can read, such as
/root/weikio-bucket.env. Give it one line each forRCLONE_S3_PROVIDER,RCLONE_S3_ENDPOINT,RCLONE_S3_ACCESS_KEY_IDandRCLONE_S3_SECRET_ACCESS_KEY. Then list the archives and download the newest one:Terminal$ docker run --rm --env-file /root/weikio-bucket.env --entrypoint rclone $IMG lsf --dirs-only :s3:weikio-backups/acme $ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio --env-file /root/weikio-bucket.env --entrypoint rclone $IMG \ copy :s3:weikio-backups/acme/<newest> /opt/weikio/backups/standby/<newest> -
Open the archive’s keys, then write the stack with
init. Unlike the primary’s firstinit, giveinitthe standby’s leaf address as--public-leaf-urland the primary’s as--standby-leaf-url:Terminal$ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio -e WEIKIO_HOME=/opt/weikio -e Weikio__Keys__Dir=/opt/weikio/keys \ -e WEIKIO_MASTER_KEY_FILE=/opt/weikio/master.key -e Weikio__Nats__Urls__0=nats://nats:4222 \ $IMG weikio-backend restore --from /opt/weikio/backups/standby/<newest> $ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio -e WEIKIO_HOME=/opt/weikio -e Weikio__Keys__Dir=/opt/weikio/keys \ -e WEIKIO_MASTER_KEY_FILE=/opt/weikio/master.key -e Weikio__Nats__Urls__0=nats://nats:4222 -e WEIKIO_BACKEND_IMAGE=$IMG \ $IMG weikio-backend init --public-leaf-url wss://nats.standby.acme.example:443 --standby-leaf-url wss://nats.acme.example:443As in a restore,
initstops with exit code 6. -
Set
.envas the primary has it, but with the lines of/root/weikio-bucket.envin place of the primary’sRCLONE_S3_*lines. Then add the address of the primary’s archives and the standby’s leaf address:.envWEIKIO_STANDBY_FROM=s3://weikio-backups/acme WEIKIO_STANDBY_LEAF_URL=wss://nats.standby.acme.example:443WEIKIO_STANDBY_FROMmay also be a folder on this host that has the archives, such as a mounted share. -
Delete
/root/weikio-bucket.env. -
Put no certificate on this host.
-
Run the first refresh. It prints its line:
Terminal$ sudo ./standby-refresh.sh -
Run the refresh every 6 hours with this line in
/etc/cron.d/weikio-standby-refresh:/etc/ cron.d/ weikio-standby-refresh 0 */6 * * * root /opt/weikio/deploy/compose/single/standby-refresh.sh > /var/log/weikio-standby-refresh.last 2>&1/var/log/weikio-standby-refresh.lasthas the latest run’s output. When a run fails, the output shows why.
While the standby is passive, each refresh writes a temporary certificate for each of the hub’s names.
Once you activate the standby, it serves the admin UI’s address with the primary’s certificate while that certificate is valid. It serves the gateway hostnames with their certificates from the archive. Weik.io obtains a certificate for the standby’s leaf address about a minute after the standby starts.
Take over with the standby#
- Remove the standby’s cron line.
- Start its stack with
docker compose up -din its stack folder. - In DNS, move the admin UI’s address and every gateway hostname from the primary to the standby.
Because the agents’ bundles already list the standby’s leaf address, you change nothing on their hosts.
The standby accepts callers from every address until it has applied the allowed networks the primary had when the archive was taken. /readyz/node returns 503 until then.
While the standby serves, a refresh changes nothing and writes the reason in its line. Keep the cron line out until the standby is passive again.
Match the primary’s release#
When you upgrade the primary, paste the same images.env lines into the standby’s .env. Its next refresh then writes the stack of the primary’s release. For the primary’s upgrade, see Upgrades and rollback.