Skip to content
Weik.ioDocs
2026.2
Try Weik.io

Telemetry is waiting on a hub server

Weik.io 2026.2

A hub server keeps telemetry on its local disk because a store didn't accept it. Once the store answers, the server sends the telemetry with its original times.

On this page
Kind Severity Who sees it
telemetry-waiting Warning, and an error while the server keeps no new telemetry Platform administrators, in a list that isn’t narrowed to an environment or a project

Its row in the problem list has a figure such as 1.4 GiB waiting, oldest 2 h 05 min.

What it means#

A store didn’t accept some telemetry, so a hub server keeps it on its local disk until the store accepts it.

What raises it#

The problem shows when the oldest waiting batch has waited a minute. It shows at once as an error while the server keeps no new telemetry. The server stops keeping new telemetry when its space for waiting telemetry is full or its disk is down to a tenth free. It also stops when its data volume doesn’t accept writes. Each sender then gets an answer that the telemetry wasn’t stored and has to send it again. The senders are the hub’s services and each site’s agent and gateway. The problem clears when nothing waits and the server keeps new telemetry again.

The problem shows for each hub server that has telemetry waiting on its disk. It shows even while the server that runs the stores is offline.

What to check#

See the stores and what waits for each:

Terminal
$ weikio platform health

On a hub server, in the stack’s folder, run weikio-backend doctor:

Terminal
$ docker compose run --rm infra weikio-backend doctor

The output of doctor has a line for what waits for each store.

How to fix it#

Start the store that doesn’t answer. On the server that runs the stores, in the stack’s folder, run:

Terminal
$ docker compose up -d victoria-logs victoria-metrics victoria-traces

Sign in to the live demo

Sign in with the shared viewer account. It can open every screen of the example projects but can't change anything.

E-mail
viewer@demo.weik.io
Password
try-weikio-live-demo
Open the live demo