Skip to content
Weik.ioDocs
2026.2
Try Weik.io

Standby hub

Weik.io 2026.2

A standby hub on a second host restores and checks the primary's newest backup every six hours and takes over under the primary's names when you activate it.

On this page

For disaster recovery of a hub on one server, a standby hub keeps the primary’s newest backup archive restored on a second host, away from the primary’s location. Once you activate the standby and move the primary’s names to it in DNS, it serves under those names.

Only the standby’s published leaf address differs from the primary’s. Give the primary’s first init that address with --standby-leaf-url. Every agent’s bundle then lists it after the primary’s. After a takeover, the agents link to the standby with the bundles they already have.

Between refreshes the standby is down and keeps its leaf port closed. While the primary runs, no agent links to it.

The primary hub serves the hub's names, such as acme.example and nats.acme.example, and its backup service copies each archive to a bucket. The standby hub is a second host away from the primary's location, and it stays down between refreshes. Every 6 hours standby-refresh.sh downloads the newest archive and verifies it, removes what the restore replaces, restores the archive, stops the stack and writes one line to its log. An agent's leaf node tries the hub's leaf addresses in the order its bundle lists them, the primary's first and the standby's second, so it links to the primary while the primary runs. To take over, you remove the cron line, start the standby with docker compose up -d and point the primary's names at the standby in DNS, and the agents link to the standby.

The primary copies its archives to a bucket. The standby restores the newest one every 6 hours and stays down until you activate it.

What each refresh does#

init writes the script standby-refresh.sh beside the stack of a hub on one server. The script changes nothing on a stack whose .env has no WEIKIO_STANDBY_FROM, such as the primary’s, or while the stack’s backend runs. On a standby, the script refreshes the stack from the newest archive:

  1. Finds the newest archive at the address in WEIKIO_STANDBY_FROM. It downloads that archive below backups/standby/ and checks with restore --verify-only that the standby’s master key opens every file. If the archive fails the check, the run deletes it and changes nothing else.
  2. Stops the stack and removes the message server’s storage and the volumes of the gateway and the forwarder. It also removes every file of the keys directory except operator.nk.
  3. Restores the archive the way a whole restore does, in two runs with init between them. The script gives init the standby’s leaf address.
  4. Stops the stack again.
  5. Keeps the seven newest archives.
  6. Appends one line to standby-refresh.log in the home.

Each line in standby-refresh.log has the time and result of the run and the archive it used. For each hub name, the line also lists the certificate that name gets once you activate the standby and the days left on it.

For a hub name, the text about its certificate starts with Warning: when an uploaded certificate has under 30 days left or the renewal of a certificate Weik.io obtained is overdue. In an installation that obtains no certificates by itself, that text also starts with Warning: when the name would have only a temporary certificate.

If a step fails, the run writes its line to the log and ends with the stack down. The next run starts again from the bucket.

The script uses flock to keep two runs from overlapping.

Set up a standby#

Do each step once, on the standby’s host, after you set IMG as in Download and check the release.

  1. Create the home and its keys directory as in the first step of a whole restore. Copy in the master key and the operator seed, and leave out the archive. When you change the primary’s master key with weikio-backend secrets rotate, put the new key here too.

  2. Write the bucket’s address and the credentials of a key that may only read the bucket into a file only root can read, such as /root/weikio-bucket.env. Give it one line each for RCLONE_S3_PROVIDER, RCLONE_S3_ENDPOINT, RCLONE_S3_ACCESS_KEY_ID and RCLONE_S3_SECRET_ACCESS_KEY. Then list the archives and download the newest one:

    Terminal
    $ docker run --rm --env-file /root/weikio-bucket.env --entrypoint rclone $IMG lsf --dirs-only :s3:weikio-backups/acme
    $ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio --env-file /root/weikio-bucket.env --entrypoint rclone $IMG \
      copy :s3:weikio-backups/acme/<newest> /opt/weikio/backups/standby/<newest>
  3. Open the archive’s keys, then write the stack with init. Unlike the primary’s first init, give init the standby’s leaf address as --public-leaf-url and the primary’s as --standby-leaf-url:

    Terminal
    $ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio -e WEIKIO_HOME=/opt/weikio -e Weikio__Keys__Dir=/opt/weikio/keys \
      -e WEIKIO_MASTER_KEY_FILE=/opt/weikio/master.key -e Weikio__Nats__Urls__0=nats://nats:4222 \
      $IMG weikio-backend restore --from /opt/weikio/backups/standby/<newest>
    $ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio -e WEIKIO_HOME=/opt/weikio -e Weikio__Keys__Dir=/opt/weikio/keys \
      -e WEIKIO_MASTER_KEY_FILE=/opt/weikio/master.key -e Weikio__Nats__Urls__0=nats://nats:4222 -e WEIKIO_BACKEND_IMAGE=$IMG \
      $IMG weikio-backend init --public-leaf-url wss://nats.standby.acme.example:443 --standby-leaf-url wss://nats.acme.example:443

    As in a restore, init stops with exit code 6.

  4. Set .env as the primary has it, but with the lines of /root/weikio-bucket.env in place of the primary’s RCLONE_S3_* lines. Then add the address of the primary’s archives and the standby’s leaf address:

    .env
    WEIKIO_STANDBY_FROM=s3://weikio-backups/acme
    WEIKIO_STANDBY_LEAF_URL=wss://nats.standby.acme.example:443

    WEIKIO_STANDBY_FROM may also be a folder on this host that has the archives, such as a mounted share.

  5. Delete /root/weikio-bucket.env.

  6. Put no certificate on this host.

  7. Run the first refresh. It prints its line:

    Terminal
    $ sudo ./standby-refresh.sh
  8. Run the refresh every 6 hours with this line in /etc/cron.d/weikio-standby-refresh:

    /etc/cron.d/weikio-standby-refresh
    0 */6 * * * root /opt/weikio/deploy/compose/single/standby-refresh.sh > /var/log/weikio-standby-refresh.last 2>&1

    /var/log/weikio-standby-refresh.last has the latest run’s output. When a run fails, the output shows why.

While the standby is passive, each refresh writes a temporary certificate for each of the hub’s names.

Once you activate the standby, it serves the admin UI’s address with the primary’s certificate while that certificate is valid. It serves the gateway hostnames with their certificates from the archive. Weik.io obtains a certificate for the standby’s leaf address about a minute after the standby starts.

Take over with the standby#

  1. Remove the standby’s cron line.
  2. Start its stack with docker compose up -d in its stack folder.
  3. In DNS, move the admin UI’s address and every gateway hostname from the primary to the standby.

Because the agents’ bundles already list the standby’s leaf address, you change nothing on their hosts.

The standby accepts callers from every address until it has applied the allowed networks the primary had when the archive was taken. /readyz/node returns 503 until then.

While the standby serves, a refresh changes nothing and writes the reason in its line. Keep the cron line out until the standby is passive again.

Match the primary’s release#

When you upgrade the primary, paste the same images.env lines into the standby’s .env. Its next refresh then writes the stack of the primary’s release. For the primary’s upgrade, see Upgrades and rollback.

Sign in to the live demo

Sign in with the shared viewer account. It can open every screen of the example projects but can't change anything.

E-mail
viewer@demo.weik.io
Password
try-weikio-live-demo
Open the live demo