Telemetry forwarding
Weik.io 2026.2Forward Weik.io's logs, metrics, traces and audit records to Azure Monitor, Splunk or any system that accepts OTLP.
On this page
To send Weik.io’s telemetry and audit records on to your systems, configure the telemetry forwarder, an OpenTelemetry collector that runs in your Weik.io installation. You write its configuration in the collector’s format, inside a telemetry-forwarding document that you keep in Git beside your other documents. Only platform administrators apply the document.
The forwarder receives the telemetry your configuration selects and gets the audit records from the audit trail. An audit record stays in the trail and on the forwarder’s disk until your system has accepted it.
Integrations and agents send their log lines, metrics and traces to the hub with OTLP over HTTPS. The backend stamps each batch with its sender's identity, writes it to the log, metric and trace stores and sends the forwarder copies of what the configuration selects. The forwarder is an OpenTelemetry collector that runs the telemetry-forwarding document and receives the copies on its receiver otlp/weikio. It gets the audit records from the trail itself, which keeps them for 400 days up to 8 GiB, and each audit exporter keeps the records in a queue on disk until the destination accepts them. The forwarder sends on to destinations such as Azure Monitor, Splunk HEC or your SIEM.
How forwarding works#
The installation has one telemetry-forwarding document or none. Its spec.collector is a complete OpenTelemetry collector configuration that Weik.io keeps as you wrote it. Its spec.audit.exporters lists the exporters that receive the audit records.
Before Weik.io stores a revision, it checks the document and tests your configuration with the collector it ships. If the collector can’t start your configuration, the apply fails with the collector’s error message. The forwarder then keeps running its current revision.
A pipeline receives the platform’s telemetry when it lists the receiver otlp/weikio. Weik.io sets the resource attributes that identify the sender of each record. An integration can’t put another environment’s name on its records.
| Attribute | Value |
|---|---|
service. |
weikio |
service.name |
project/ for an integration, such as finance/, or weikio-, weikio-apim, weikio- and weikio- for the platform’s processes |
deployment. |
The environment of an integration |
weikio. |
The project of an integration |
weikio. |
The API of a call through the gateway |
weikio. |
The installation, on every record |
weikio., weikio., weikio. |
The IDs, which a rename doesn’t change |
To send each environment’s data to a different place, route on deployment.environment.name. Records from the hub’s processes have no environment. If you filter on the environment and want those records too, let records with weikio.ingest.sender == "hub" through the filter.
Group an integration’s telemetry by weikio.integration.uid and weikio.environment.uid.
Store the secrets#
Store every credential the configuration needs as a platform secret, and refer to it in the file as ${secret:<name>}. Run weikio platform secret set and type the value at its prompt. You don’t see the value as you type:
$ weikio platform secret set siem-tokenOnly platform administrators set the values. Weik.io keeps them encrypted and passes them to the forwarder with each revision that uses them. You can’t delete a secret the document uses until you remove it from the configuration and apply the document again.
Start from an example#
The examples below ship with Weik.io and pass its checks as written. Each starts with a comment on where every signal and the audit records go, and on which secrets to store.
A SIEM that accepts OTLP#
# Telemetry forwarding to a SIEM that receives OTLP over HTTP.
#
# logs to the SIEM over OTLP/HTTP
# metrics to the SIEM over OTLP/HTTP
# traces to the SIEM over OTLP/HTTP
# audit to the SIEM over OTLP/HTTP, queued on the forwarder's disk until the SIEM accepts them
#
# Every signal waits on the forwarder's disk and is retried until the SIEM takes it. Write the SIEM's address below,
# then store the bearer token this file reads:
#
# weikio platform secret set siem-token
#
# and apply it with 'weikio apply -f otlp-siem.yaml'.
apiVersion: weik.io/v1
kind: telemetry-forwarding
spec:
audit:
exporters: [otlp_http/siem]
collector:
extensions:
file_storage/siem:
directory: /var/lib/otelcol/file_storage
create_directory: true
fsync: true
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 55
spike_limit_percentage: 15
exporters:
otlp_http/siem:
endpoint: https://siem.example.com:4318
headers:
Authorization: "Bearer ${secret:siem-token}"
sending_queue:
storage: file_storage/siem
sizer: bytes
queue_size: 268435456
batch:
flush_timeout: 1s
min_size: 65536
max_size: 4194304
retry_on_failure:
max_elapsed_time: 0
service:
extensions: [file_storage/siem]
pipelines:
logs:
receivers: [otlp/weikio]
processors: [memory_limiter]
exporters: [otlp_http/siem]
metrics:
receivers: [otlp/weikio]
processors: [memory_limiter]
exporters: [otlp_http/siem]
traces:
receivers: [otlp/weikio]
processors: [memory_limiter]
exporters: [otlp_http/siem]Splunk HEC#
In this example, the forwarder sends the audit records to Splunk’s HTTP Event Collector. It also sends log lines there, but only those of production deployments and of the hub’s services:
# Telemetry forwarding to Splunk through its HTTP Event Collector.
#
# logs to Splunk HEC: the logs of production deployments and of the hub's own services
# metrics not forwarded
# traces not forwarded
# audit to Splunk HEC, queued on the forwarder's disk until Splunk accepts them
#
# Log lines and audit records share one queue on the forwarder's disk, retried until Splunk takes it. Write the HEC
# address below, then store the HEC token this file reads:
#
# weikio platform secret set splunk-hec-token
#
# and apply it with 'weikio apply -f splunk-hec.yaml'.
apiVersion: weik.io/v1
kind: telemetry-forwarding
spec:
audit:
exporters: [splunk_hec]
collector:
extensions:
file_storage/splunk:
directory: /var/lib/otelcol/file_storage
create_directory: true
fsync: true
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 55
spike_limit_percentage: 15
# Keeps the log lines of production deployments and of the hub; the platform writes both attributes from what the
# sender's credential proves, so a deployment elsewhere cannot pass by claiming production. The environment's name
# is a label: group a deployment's lines by weikio.environment.uid and weikio.integration.uid, which a rename leaves.
filter/production:
error_mode: ignore
log_conditions:
- resource.attributes["deployment.environment.name"] != "production" and resource.attributes["weikio.ingest.sender"] != "hub"
exporters:
splunk_hec:
endpoint: https://splunk.example.com:8088/services/collector
token: ${secret:splunk-hec-token}
sending_queue:
storage: file_storage/splunk
sizer: bytes
queue_size: 268435456
batch:
flush_timeout: 1s
min_size: 65536
max_size: 4194304
retry_on_failure:
max_elapsed_time: 0
service:
extensions: [file_storage/splunk]
pipelines:
logs:
receivers: [otlp/weikio]
processors: [memory_limiter, filter/production]
exporters: [splunk_hec]Azure Monitor#
In this example, the forwarder sends metrics and traces to Azure Monitor through its OTLP ingestion. It sends the audit records to a SIEM. Azure Monitor accepts the data from a service principal that has the Monitoring Metrics Publisher role on the data collection rule:
# Telemetry forwarding to Azure Monitor, with the audit trail to a SIEM.
#
# logs not forwarded
# metrics to Azure Monitor through its OTLP ingestion, as deltas
# traces to Azure Monitor through its OTLP ingestion
# audit to a SIEM over OTLP/HTTP, queued on the forwarder's disk until the SIEM accepts them
#
# Azure Monitor accepts the data from a service principal that holds the Monitoring Metrics Publisher role on the data
# collection rule. Write its tenant and client ids, the two ingestion endpoints of the rule and the SIEM's address below,
# then store the two secrets this file reads:
#
# weikio platform secret set azure-monitor-secret
# weikio platform secret set siem-token
#
# and apply it with 'weikio apply -f azure-monitor.yaml'.
apiVersion: weik.io/v1
kind: telemetry-forwarding
spec:
audit:
exporters: [otlp_http/siem]
processors: [attributes/audit]
collector:
extensions:
azure_auth:
service_principal:
tenant_id: 00000000-0000-0000-0000-000000000000
client_id: 00000000-0000-0000-0000-000000000000
client_secret: ${secret:azure-monitor-secret}
scopes:
- https://monitor.azure.com/.default
file_storage/siem:
directory: /var/lib/otelcol/file_storage
create_directory: true
fsync: true
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 55
spike_limit_percentage: 15
# Azure Monitor takes metrics as deltas, and the platform sends cumulative ones.
cumulative_to_delta: {}
# Audit records reach the SIEM without the browser's user agent.
attributes/audit:
actions:
- key: user_agent.original
action: delete
exporters:
otlp_http/azure:
traces_endpoint: https://example-dce.westeurope-1.ingest.monitor.azure.com/datacollectionRules/dcr-00000000000000000000000000000000/streams/Microsoft-OTLP-Traces/otlp/v1/traces
metrics_endpoint: https://example-dce.westeurope-1.metrics.ingest.monitor.azure.com/datacollectionRules/dcr-00000000000000000000000000000000/streams/Custom-Metrics-Otel/otlp/v1/metrics
auth:
authenticator: azure_auth
sending_queue:
sizer: bytes
queue_size: 67108864
batch:
flush_timeout: 200ms
min_size: 1048576
max_size: 8388608
otlp_http/siem:
endpoint: https://siem.example.com:4318
headers:
Authorization: "Bearer ${secret:siem-token}"
sending_queue:
storage: file_storage/siem
sizer: bytes
queue_size: 268435456
batch:
flush_timeout: 1s
min_size: 65536
max_size: 4194304
retry_on_failure:
max_elapsed_time: 0
service:
extensions: [azure_auth, file_storage/siem]
pipelines:
metrics:
receivers: [otlp/weikio]
processors: [memory_limiter, cumulative_to_delta]
exporters: [otlp_http/azure]
traces:
receivers: [otlp/weikio]
processors: [memory_limiter]
exporters: [otlp_http/azure]If you send to Application Insights with the collector’s azure_monitor exporter instead, it cuts each attribute value to the 8,192 bytes that Application Insights keeps. To see the exporter’s warning about the cut, run weikio forwarding status --lines.
Apply the forwarding document#
A platform administrator applies the file with weikio apply:
$ weikio apply -f otlp-siem.yamlThe apply prints the revision and the collector that checked it, such as revision 9, checked by otelcol-contrib 0.160.0, starting now. Then see what the forwarder sends:
$ weikio forwarding statusFor each forwarder, you see the revision it runs and where each signal and the audit records go. You also see what each exporter sent and failed to send in the last hour.
Add --lines for the collector’s last warnings and errors, or --watch to follow the status. You see the same in the admin UI under Settings › Operations › Telemetry forwarding.
If your configuration selects a signal that Weik.io doesn’t keep, the deployments start sending it at their next start. weikio apply restarts none of them and prints a line about it.
To stop forwarding, run weikio forwarding delete.
Audit records#
The forwarder gets every audit record from the audit trail and, after a restart, goes on from where it stopped. An audit exporter keeps the records on the forwarder’s disk until the destination accepts them. Set each audit exporter’s sending_queue.storage to a file_storage extension listed in service.extensions, and its retry_on_failure.max_elapsed_time to 0. Without both settings, the apply fails.
While a destination is unreachable, its records fill its queue on disk and then stay in the trail. The trail keeps records for 400 days and up to 8 GiB. When the destination is back, the forwarder sends it the records that the queue and the trail still have.
Each audit record arrives as one log record with the event name weikio.audit.record.
| Attribute | What it has |
|---|---|
weikio. |
The action |
weikio. |
The outcome |
weikio. |
The actor |
weikio. |
The target’s name |
weikio. |
The reason given |
weikio. |
The commit a pipeline applied the change from |
weikio. |
The diff of the record before and after |
client. |
The caller’s address |
A record can arrive more than once and out of order. In your system, discard a second record with the same log.record.uid, and sort by the record’s time. Because the trail goes on from the backup’s last record after a restore, don’t remove repeats by weikio.audit.seq.
After the first apply, the forwarder sends the records written from then on. To send older ones, run weikio forwarding audit-from <time>. It first prints how many records it sends. To skip the records not yet sent, run weikio forwarding audit-from now.
The collector’s azure_monitor exporter keeps records only in memory and discards a record after a few failed tries. In the Azure Monitor example, the forwarder sends the audit records to a SIEM through an exporter that queues them on disk instead.
What can lose a record#
- When a destination answers that a record is invalid, the collector discards the record.
- A host that stops without warning can lose the last records written to the disk queue. To prevent it, write each record through to the disk with
fsync: trueon the storage, as in the shipped examples.
A discarded record counts as failed and raises the problem forwarding-exporter-failing.
Renaming an audit exporter or its storage starts an empty queue. The records in the old queue aren’t sent, but they stay in the trail.
When the forwarder stops, the collector can log Exporting failed. Dropping data. with interrupted due to shutdown. The forwarder sends the audit records in the disk queue when it starts again.
A receiver of your own#
A receiver you add to spec.collector listens inside the hub, where anything running at the hub can reach it but nothing outside can. Give such a receiver authentication.