Lightweight OTel stack

Install an OpenTelemetry collector with Jaeger and Prometheus, then send traces and metrics from kagent and Agent Substrate to it.

Deploy a small observability stack based on OpenTelemetry (OTel) that collects traces and metrics from both kagent and Agent Substrate. The stack includes the following components:

Traces: Distributed tracing with Jaeger.
Metrics: Time-series metrics collection with Prometheus.
Collection: Telemetry collection and routing with the OpenTelemetry Collector.
Logs: No logging backend. The collector prints the logs that it receives to its own output.

This stack follows the setup that Agent Substrate uses for its own local testing, and it needs less memory than the OTel stack with Grafana.

About the stack

The stack has no logging backend. The collector prints the logs that it receives to its own output, which is enough to confirm that logs arrive. To store and query audit events, use the OTel stack, which adds Loki.

    flowchart LR
    subgraph sources["Your cluster"]
        kagent["kagent controller<br/>and agent runtimes"]
        substrate["Agent Substrate"]
    end
    collector["OTel collector"]
    subgraph backends["Backends"]
        jaeger["Jaeger"]
        prom["Prometheus"]
    end
    kagent -->|traces, logs| collector
    substrate -->|traces, logs, metrics| collector
    kagent -->|metrics scrape| prom
    collector --> jaeger
    collector -->|metrics scrape| prom
    classDef boundary fill:#a78bfa26,stroke:#a78bfa,stroke-width:2px
    classDef inner fill:#80808033,stroke:#9ca3af,stroke-width:1px
    class sources,backends boundary
    class kagent,substrate,collector,jaeger,prom inner

For what kagent and Agent Substrate each send, see About the stack in the OTel stack guide.

Before you begin

Install kagent.
Create your first agent, so that you have a Harness and an AgentTemplate to send requests to.

Install Jaeger

Install Jaeger in all-in-one mode, which stores traces in memory and needs no other components.

helm upgrade --install jaeger jaeger \
  --repo https://jaegertracing.github.io/helm-charts \
  --version 4.4.7 \
  --namespace telemetry --create-namespace \
  --values - <<EOF
provisionDataStore:
  cassandra: false
allInOne:
  enabled: true
storage:
  type: memory
agent:
  enabled: false
collector:
  enabled: false
query:
  enabled: false
EOF

Install Prometheus

Install the Prometheus server on its own, without the exporters and Alertmanager that the chart includes by default. The values configure two scrape jobs, one for the metrics that the collector serves and one for the kagent controller.

Install the chart.

helm upgrade --install prometheus prometheus \
  --repo https://prometheus-community.github.io/helm-charts \
  --version 29.33.0 \
  --namespace telemetry \
  --values - <<EOF
alertmanager:
  enabled: false
kube-state-metrics:
  enabled: false
prometheus-node-exporter:
  enabled: false
prometheus-pushgateway:
  enabled: false
server:
  persistentVolume:
    enabled: false
extraScrapeConfigs: |
  - job_name: otel-collector
    scrape_interval: 15s
    static_configs:
    - targets: ["otel-collector.telemetry.svc.cluster.local:8889"]
  - job_name: kagent-controller
    scrape_interval: 15s
    scheme: https
    authorization:
      credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_config:
      insecure_skip_verify: true
    static_configs:
    - targets: ["kagent-controller-metrics.kagent.svc.cluster.local:8443"]
EOF

The kagent controller serves its metrics over HTTPS with a self-signed certificate, and it accepts a scrape only from a ServiceAccount that holds its metrics reader role. The kagent-controller job therefore skips certificate verification and sends the Prometheus ServiceAccount token.

Verify that Prometheus is running.

kubectl get pods -n telemetry -l app.kubernetes.io/name=prometheus

Install the OTel collector

Install a collector that receives OTLP on ports 4317 for gRPC and 4318 for HTTP. The collector sends traces to Jaeger, serves metrics on port 8889, and prints logs to its own output. The configuration uses the contrib collector image, because the Prometheus exporter is not part of the core image.

helm upgrade --install otel-collector opentelemetry-collector \
  --repo https://open-telemetry.github.io/opentelemetry-helm-charts \
  --version 0.172.0 \
  --namespace telemetry \
  --values - <<EOF
fullnameOverride: otel-collector
mode: deployment
image:
  repository: otel/opentelemetry-collector-contrib
ports:
  metrics-export:
    enabled: true
    containerPort: 8889
    servicePort: 8889
    protocol: TCP
config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
  processors:
    batch: {}
  exporters:
    debug:
      verbosity: basic
    otlp_grpc/jaeger:
      endpoint: jaeger.telemetry.svc.cluster.local:4317
      tls:
        insecure: true
    prometheus:
      endpoint: 0.0.0.0:8889
      resource_to_telemetry_conversion:
        enabled: true
  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [batch]
        exporters: [otlp_grpc/jaeger]
      logs:
        receivers: [otlp]
        processors: [batch]
        exporters: [debug]
      metrics:
        receivers: [otlp]
        processors: [batch]
        exporters: [prometheus]
EOF

Send kagent telemetry to the collector

Save the current revision of your Harness and AgentTemplate pair. A later step uses it to tell when kagent recompiles the pair with the new settings. The command first waits for any recompile that is still in progress, such as one from an earlier Helm upgrade, so that it saves a finished revision.

for i in $(seq 1 60); do
  REVISIONS=$(kubectl get agenttemplate my-first-agent -n kagent \
    -o jsonpath='{.status.harnesses[0].desiredRevision} {.status.harnesses[0].latestSuccessfulRevision}')
  [ "${REVISIONS% *}" = "${REVISIONS#* }" ] && break
  sleep 5
done
export OLD_REVISION=${REVISIONS#* }
echo "Current revision: $OLD_REVISION"

Upgrade the kagent Helm release to export traces and logs to the collector, and to serve controller metrics.

helm upgrade kagent \
  oci://ghcr.io/kagent-dev/kagent/helm/kagent \
  --version 1.0.0-alpha2 \
  --namespace kagent \
  --reuse-values \
  --values - <<EOF
otel:
  tracing:
    enabled: true
    exporter:
      otlp:
        endpoint: http://otel-collector.telemetry.svc.cluster.local:4317
        protocol: grpc
        insecure: true
  logging:
    enabled: true
    exporter:
      otlp:
        endpoint: http://otel-collector.telemetry.svc.cluster.local:4317
        protocol: grpc
        insecure: true
controller:
  metrics:
    enabled: true
EOF

Allow Prometheus to scrape the controller. Bind the metrics reader role that the kagent chart creates to the Prometheus ServiceAccount.

kubectl create clusterrolebinding kagent-metrics-reader-prometheus \
  --clusterrole=kagent-metrics-reader \
  --serviceaccount=telemetry:prometheus-server

Wait for the controller to roll out.

kubectl rollout status deployment/kagent-controller -n kagent --timeout=300s

Wait for kagent to recompile the pair. The controller rebuilds each pair after the controller restarts, and an AgentInstance that you create before the rebuild finishes starts from the previous revision, without the new settings. The following command prints Recompiled when the new revision is ready.
```
for i in $(seq 1 60); do
  [ "$(kubectl get agenttemplate my-first-agent -n kagent \
    -o jsonpath='{.status.harnesses[0].latestSuccessfulRevision}')" != "$OLD_REVISION" ] \
    && echo "Recompiled" && break
  sleep 5
done
```
If the command finishes without printing Recompiled, the upgrade did not change the pair, for example because the settings were already in place.

Send Agent Substrate telemetry to the collector

Upgrade the Agent Substrate Helm release. A single otel.endpoint setting turns on traces, logs, and metrics for every Agent Substrate component. Agent Substrate keeps 1% of its traces by default, so the command keeps every trace instead. Lower samplingRatio again for production.

helm upgrade substrate \
  oci://ghcr.io/kagent-dev/substrate/helm/substrate \
  --version 0.2.0-beta5 \
  --namespace ate-system \
  --reuse-values \
  --set otel.endpoint=http://otel-collector.telemetry.svc.cluster.local:4317 \
  --set otel.traces.samplingRatio=1.0 \
  --wait --timeout 10m

Send a request

Create a new AgentInstance. An AgentInstance keeps the runtime configuration that it was created with, so only a new AgentInstance exports traces.
```
kagent create agent-instance --harness my-first-harness --agent-template my-first-agent
```

Confirm that the AgentInstance runs the current revision of the pair. The command waits until kagent finishes compiling the pair, then compares that revision with the one that the AgentInstance started from. If the command prints Outdated, the AgentInstance was created from an earlier revision, and exports without the new settings. Create another AgentInstance, and run the command again.

for i in $(seq 1 60); do
  REVISIONS=$(kubectl get agenttemplate my-first-agent -n kagent \
    -o jsonpath='{.status.harnesses[0].desiredRevision} {.status.harnesses[0].latestSuccessfulRevision}')
  [ "${REVISIONS% *}" = "${REVISIONS#* }" ] && break
  sleep 5
done
INSTANCE_REVISION=$(kagent get agent-instance -o json \
  | jq -r '[.agentInstances[] | select(.agentTemplate.name == "my-first-agent")] | sort_by(.createdAt) | last | .preparedRevision')
[ "$INSTANCE_REVISION" = "${REVISIONS#* }" ] && echo "Current" || echo "Outdated"

Send a request to produce telemetry.

export INSTANCE_ID=$(kagent get agent-instance -o json \
  | jq -r '[.agentInstances[] | select(.agentTemplate.name == "my-first-agent")] | sort_by(.createdAt) | last | .id')
kagent invoke --agent-instance $INSTANCE_ID --task "What is 2+2?"

Review the telemetry

Review the traces in Jaeger.
1. Forward the Jaeger query port, and leave the command running.
  kubectl port-forward -n telemetry svc/jaeger 16686:16686
2. In your browser, open Jaeger at http://localhost:16686.
3. From the Service list, select my-first-agent-my-first-harness, and click Find Traces. To see Agent Substrate’s own work, select ateapi, atenet-router, or atelet instead.
Review the metrics in Prometheus.
1. Forward the Prometheus port, and leave the command running.
  kubectl port-forward -n telemetry svc/prometheus-server 9090:80
2. In your browser, open Prometheus at http://localhost:9090/targets, and confirm that the otel-collector and kagent-controller targets are UP.
3. Open the Query page, and run a query such as ate_workerpool_ready_workers. For more metrics, see Metrics.
Check that the collector receives logs. Each line reports a batch of log records from one source.
```
kubectl logs -n telemetry deploy/otel-collector | grep '"otelcol.signal": "logs"'
```

Clean up

Turn off telemetry export in kagent and Agent Substrate.

helm upgrade kagent \
  oci://ghcr.io/kagent-dev/kagent/helm/kagent \
  --version 1.0.0-alpha2 \
  --namespace kagent --reuse-values \
  --set otel.tracing.enabled=false \
  --set otel.logging.enabled=false \
  --set controller.metrics.enabled=false
helm upgrade substrate \
  oci://ghcr.io/kagent-dev/substrate/helm/substrate \
  --version 0.2.0-beta5 \
  --namespace ate-system --reuse-values \
  --set otel.endpoint="" \
  --set otel.traces.samplingRatio=0.01

Remove the role binding, the collector, the backends, and the telemetry namespace.

kubectl delete clusterrolebinding kagent-metrics-reader-prometheus
helm uninstall otel-collector prometheus jaeger -n telemetry
kubectl delete namespace telemetry

Next steps

TracingRead the spans and attributes of an agent request. MetricsReview the metrics that kagent and Agent Substrate report. OTel stackAdd Loki and Grafana to store logs and view every signal in one place.

OTel stack Tracing