Install Weik.io on several servers
Weik.io 2026.2Weik.io runs on as many servers as you need, in one data centre or across several, behind a TCP load balancer that sends callers and agents to the servers that are ready.
On this page
On several servers, your Weik.io installation keeps working when a server fails. You choose how many servers it has. Three or five of them are data servers, which each keep a copy of the hub’s storage. Each of the other servers is a standard server, which adds an APIM instance and room for more callers and agents.
Every server runs the same stack. The backend on any server can answer every request. A TCP load balancer in front removes a server from its pool within 5 seconds when the server stops being ready.
A hub with three data servers, one in each of three zones or data centres, is behind a TCP load balancer that checks /readyz/node on each of them. Every server runs a TLS terminator, the backend, a gateway and a message server. Each data server keeps a copy of the stored data, and any backend answers any request. Backup copies go to S3 storage with Object Lock, where no key can delete a copy early, and a standby hub keeps the newest backup restored and ready.
The tested shape#
The tested hub has three data servers (--servers 3 --replicas 3) of 4 virtual CPUs and 8 GB of memory each. Each has a disk of 100 GB under its message store. The hub was installed with --disk-budget 74GiB, which fits three environments and 26 projects of the default size besides the installation’s own stores. On servers with local solid-state disks this hub stored about 600 events a second, and about 70 to 100 a second on network-attached disks.
Size each server by its role:
| Role | Servers | CPU and memory | Disk |
|---|---|---|---|
| Data server | Nodes 1 to 3, or 1 to 5 | 4 virtual CPUs, 8 GB of memory | The disk budget plus a quarter of it for its copy of the storage, and about 2 GB for the images |
| Standard server | The nodes after the data servers | 4 virtual CPUs, 8 GB of memory | About 10 GB for the images and the build cache |
A standard server keeps no storage and passes every storage request to the data servers.
Node 1 also runs the three telemetry stores and the telemetry forwarder. The other servers reach them on node 1’s TCP ports 9428, 8428, 10428 and 14319. With their default caps the stores need up to about 75 GB, so put them on a disk of their own, or give node 1 a larger disk than the other data servers have.
You also need one TCP load balancer, set up as Set up the load balancer shows.
Before you start#
Prepare each server as Install Weik.io on one server says under “Before you start”:
- Install Ubuntu 24.04 LTS on x86-64.
- Encrypt the disks.
- Install Docker with the Compose plugin.
- Install the latest updates.
Put the servers on one private network, which can span data centres. Create the load balancer on that network with an empty pool, and point the backend’s name and the leaf address’s name at it.
Open the ports between the servers with your provider’s network firewall, and open them nowhere else:
| Port | From | To | Used for |
|---|---|---|---|
| TCP 6222 | Every server | Every other server | The links between their message servers |
| TCP 9428, 8428, 10428 and 14319 | The other servers | Node 1 | The stores and the forwarder |
An installation set up with --no-observability needs only 14319 to node 1.
Close each server’s port 443 to everything but the load balancer. Use the provider’s firewall for this, because Docker publishes a port past a host firewall such as ufw.
Write the servers’ directories#
On one host, create the home and the master key, and download and check the release, as on one server. Then run init with the number of servers:
$ install -d -o 1654 -g 1654 -m 0700 /opt/weikio
$ head -c 32 /dev/urandom > /opt/weikio/master.key && chown 1654:1654 /opt/weikio/master.key && chmod 0400 /opt/weikio/master.key
$ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio -e WEIKIO_HOME=/opt/weikio -e Weikio__Keys__Dir=/opt/weikio/keys \
-e WEIKIO_MASTER_KEY_FILE=/opt/weikio/master.key -e Weikio__Nats__Urls__0=nats://nats:4222 -e WEIKIO_BACKEND_IMAGE=$IMG \
$IMG weikio-backend init --servers 3 --replicas 3 --public-base-url https://acme.example --public-leaf-url wss://nats.acme.example:443 --disk-budget 74GiBinit writes one directory per server under /opt/weikio/deploy/compose/cluster, from node-1 to node-3, and prints each server’s role and zone. It also prints the installation’s operator seed, once. Save the seed and a copy of master.key away from the servers and apart from the backup archives.
You set the servers and the copies of the storage with flags of init:
| Flag | What it sets |
|---|---|
-- |
The number of servers, 3 or more. There’s no upper limit. |
-- |
How many of the servers keep a copy of the storage, 3 or 5. The default is 3. |
-- |
The zones the servers are in, given to the servers in turn. Give at least as many zones as copies. |
The storage keeps working while a majority of its copies is up. With 3 copies it keeps working when one data server is down, and with 5 copies when two are down. A hub on several servers keeps 3 copies or 5, because one server alone can’t form a majority. You set the number of data servers once, when you install Weik.io.
--zones keeps any two copies of a piece of storage out of the same zone. A zone can be an availability zone or a data centre. With servers at more than one cloud provider, it can also be a provider.
By default the images come from Docker Hub and the certificates from Let’s Encrypt. Both services operate from the United States. To pull the images from a registry of your own, mirror them into it with their upstream paths and give init the registry in --registry, as Air-gapped installations describes. To get the certificates from another ACME authority, give its directory in --certificate-authority and the email address the authority writes to in --certificate-contact. To have the platform’s own authority sign the installation’s names instead, give the first init the flag --hub-names-authority platform, as Certificates describes.
Start each server#
Copy the home to each server at the same path, keeping its owner. The copy leaves out the DNS token and any certificate of yours that init was given, because they stay on node 1:
$ tar -C / --exclude=opt/weikio/import --exclude=opt/weikio/keys/dns-connection.json --exclude=opt/weikio/keys/hub-certificates.json --exclude-tag-all=agent.creds -cpf - opt/weikio | ssh <server> 'tar -C / -xpf -'In each server’s node directory, /opt/weikio/deploy/compose/cluster/node-<n>, set the private addresses of every server and of the load balancer in .env. The lines are the same on every server:
WEIKIO_HUB_1_ADDRESS=10.0.0.11
WEIKIO_HUB_2_ADDRESS=10.0.0.12
WEIKIO_HUB_3_ADDRESS=10.0.0.13
WEIKIO_BALANCER_ADDRESSES=10.0.0.5Start the message server on every server first:
$ docker compose up -d natsOnce the message server runs on a majority of the data servers, start node 1’s stack and save the first platform-admin API key there, as on one server. Start every other server’s stack with the same commands. On those servers, init prints no key.
$ docker compose up -d
$ docker compose ps -a
$ docker compose logs init
$ docker compose rm -f init
$ docker compose run --rm infra weikio-backend doctorWhen the server has started, check the state of each service in docker compose ps -a:
natshealthyinfraandinitexited with 0backend,apim,splitter,terminatorandbackupup- On node 1 also
victoria-logs,victoria-metrics,victoria-traces,forwarderandtelemetry-doorup
docker compose logs init on node 1 shows the first platform-admin API key and the set-password link only once.
Delete /opt/weikio/import on node 1 once Weik.io has started, because Weik.io keeps its own encrypted copy of the DNS token and your certificates.
Set up the load balancer#
The load balancer’s settings are the same for any number of servers:
| Setting | Value |
|---|---|
| Pool | Every server, with equal weight and no stickiness |
| Listener | TCP 443 only, passed through to port 443 of each server |
| PROXY protocol | On |
| Health check | HTTPS GET / on port 443 of each server, every 5 s with a timeout of 4 s |
Server name and Host header of the check |
The backend’s name |
| Certificate check | Off |
| Healthy | 2xx |
| Out and back | Out of the pool after one failed check, back after one passed check |
The balancer needs no port other than 443, because an agent links over WebSocket on 443.
With the PROXY protocol on, each server’s terminator gets the caller’s address from the protocol’s header, so the audit trail records that address. The terminator accepts the header only on connections from the addresses in WEIKIO_BALANCER_ADDRESSES.
/readyz/node returns 200 while every part below is ready on the server, and 503 otherwise:
- The backend
- The APIM instance
- The message server’s listener for agent links
A server whose gateway or message server stops leaves the pool within 5 seconds. Its callers and agent links then go to the other servers.
When a server stops, as it does during upgrade.sh, it first returns 503 and keeps passing new connections for 10 more seconds. It leaves the pool before it stops accepting connections, so no call is closed unanswered.
Leave the certificate check off, because each server serves a temporary certificate until a name’s first certificate is issued. A check that verified the certificate would keep every server out of the pool. The certificate authority checks the name through the balancer, so the first certificate would then never be issued.
The hub checks the certificate that each server serves. The problem list shows a server that serves a temporary or an old certificate.
Also set up a check from outside, on https://acme.example/readyz/node through the balancer, as Health and problems describes.
Servers in different data centres#
A hub can run across data centres or cloud providers and stay one installation that your users and agents reach at one address. To keep each copy of the storage in a different data centre, add --zones dc1,dc2,dc3 to the init command above, one zone for each data centre.
With three data centres, the hub keeps working when one of them goes down. With --replicas 5 and five zones, it keeps working when two go down. When a data centre goes down, changes pause for a short time. The load balancer sends callers and agents to the servers that remain.
The network and the load balancer must reach across all of the data centres:
- The servers reach each other on the ports above, over a private network that spans the data centres or over a VPN between them.
- The load balancer reaches every server and keeps running when one data centre goes down. Use a balancer service that spans locations, or health-checked DNS in front of a balancer in each data centre.
A change is confirmed only once a second data centre has it, so a change takes about as long as the trip to the nearest other data centre.
Node 1 runs the telemetry stores. While its data centre is down, the rest of the hub keeps working, but those stores take in nothing until node 1 is back.
Add and remove servers#
Add standard servers when the hub needs more room. On the host where init first ran, run init again with a larger --servers, from the backend image of the release the hub runs:
$ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio -e WEIKIO_HOME=/opt/weikio -e Weikio__Keys__Dir=/opt/weikio/keys \
-e WEIKIO_MASTER_KEY_FILE=/opt/weikio/master.key -e Weikio__Nats__Urls__0=nats://nats:4222 -e WEIKIO_BACKEND_IMAGE=$IMG \
$IMG weikio-backend init --servers 5init writes node-4 and node-5 beside the others as standard servers, and changes no file of the servers the hub already has. Then bring up each new server:
- Copy the keys directory and the server’s directory to it.
- Set the address lines of every server in its
.env. - Start it with
docker compose up -d. - Add it to the balancer’s pool.
The other servers get the new server’s name from the data servers. Each server uses about 0.65 MB of memory for its four links to each other server.
To take a standard server out, remove it from the balancer’s pool and stop its stack with docker compose down. Then mark it retired on the host where init first ran:
$ docker run --rm -u 1654:1654 -v /opt/weikio:/opt/weikio -e WEIKIO_HOME=/opt/weikio -e Weikio__Keys__Dir=/opt/weikio/keys \
-e WEIKIO_MASTER_KEY_FILE=/opt/weikio/master.key $IMG weikio-backend init --remove-node 5Then copy the keys directory onto every other server, so that each one counts server 5 as retired and stops checking for an APIM instance on it. init never reuses a retired number for another server.
A server cut off#
When a server’s private network to the others is cut, the server keeps running but can store nothing. The load balancer still reaches it. You can’t tell beforehand which happens next:
- The balancer removes the server from its pool within about 15 seconds, because its backend returns 503 about 10 seconds after the cut. New calls and agent links go to the other servers.
- The server stays in the pool. Calls that land on it get the installation’s records as of the moment the cut began. A change sent through it fails with 503 and a
Retry-Afterheader. A subscription’s callers can make its full rate limit of calls through the server, on top of their calls through the other servers.
In both cases, about a minute after the cut, the problem list shows the server with the error hub-server-cut-off. The Health page lists its message server as offline. The check from outside passes either way, so watch the problem list. Run doctor on each server without waiting for its message server:
$ docker compose run --rm --no-deps infra weikio-backend doctorWhen one server is cut off, the servers row of doctor on every other server shows it, such as on 2 of 3: weikio-hub-1 not responding. When the network between all of them is cut, the row on each server shows only that server.
To be sure a cut-off server is out of the pool, and to close the connections it still has, stop its splitter in its stack directory:
$ docker compose stop splitterTake out at most one server this way. Once its network is back and docker compose ps shows nats healthy, start it again with docker compose up -d. The server is back when doctor shows on 3 of 3.
A data server also stops storing when its disk fills, and keeps its storage off until it restarts. A minute later the problem list shows it as message-storage-off. Free disk space on that server, then restart its message server while the others keep running:
$ docker compose restart natsUpgrades and rollback shows how to upgrade one server at a time. Storage and its budget shows how to raise each data server’s budget.