Appearance

Metrics, logs, and traces

Choose telemetry by question, then connect collection, correlation, retention and sensitive-data boundaries.

What you will understand

  • Distinguish resource metrics from application and request telemetry
  • Explain Prometheus scraping and Grafana visualization
  • Trace OpenTelemetry context across services without exposing sensitive data

Builds on: Debug network and storage failures

Metrics summarize behavior as measurements over time: request counts, latency distributions, queue depth and memory use. Logs record discrete events with contextual fields. Traces connect spans representing operations across a distributed request. They overlap but do not replace one another. A latency metric can reveal that a problem affects many requests, a trace can show which dependency consumed time in a sampled request, and a log can explain the application decision at that point.

Kubernetes resource metrics, commonly exposed through Metrics Server for kubectl top and resource autoscaling, are not a complete observability system. They do not automatically provide application request rates, durable log retention or request traces. Production collection usually combines additional agents, backends and instrumentation. Name the signal, collection interval, retention period and expected coverage before claiming that the system is observable.

A panel depends on collection, storage and a correct query

Trace application measurements through Prometheus into Grafana while explaining cardinality.

Chart viewStep 1 of 4
Instrument a meaningful signalThe application exports a request counter or latency histogram. Installing a dashboard does not create this instrumentation.. Application, Exports bounded metric labels, active. Metrics endpoint, Counters, gauges, histograms, active. Prometheus, Discovers, scrapes, stores time series, idle. Grafana, Queries configured data source, idle. Alert rule, Evaluates a useful symptom, idleApplication: Exports bounded metric labels
ApplicationExports bounded metric labelsIn focus
Metrics endpoint: Counters, gauges, histograms
Metrics endpointCounters, gauges, histogramsIn focus
Prometheus: Discovers, scrapes, stores time series
PrometheusDiscovers, scrapes, stores time seriesContext
Grafana: Queries configured data source
GrafanaQueries configured data sourceContext
Alert rule: Evaluates a useful symptom
Alert ruleEvaluates a useful symptomContext
ApplicationExports bounded metric labelsIn focus
Metrics endpointCounters, gauges, histogramsIn focus
PrometheusDiscovers, scrapes, stores time seriesContext
GrafanaQueries configured data sourceContext
Alert ruleEvaluates a useful symptomContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Dashboard is stale or empty
Instrument a meaningful signal

The application exports a request counter or latency histogram. Installing a dashboard does not create this instrumentation.

Read the complete visual sequence
  1. Instrument a meaningful signal

    The application exports a request counter or latency histogram. Installing a dashboard does not create this instrumentation.

  2. Scrape and retain

    Prometheus reads the endpoint at configured intervals and stores series. Each distinct label set is a separate series with a cost.

  3. Query the correct aggregate

    Grafana queries its data source using the intended labels and time range. Raw counters normally need rate-style treatment for throughput.

  4. Alert on actionable change

    An alert evaluates a sustained symptom and links to investigation. Keep request IDs and unbounded user labels out of metric dimensions.

Dashboard is stale or empty

The application serves traffic but the collection path fails.

  1. Requests still succeed

    The application exports or logs recent work while its serving path remains available.

  2. Scrape fails

    Prometheus cannot reach or read the target, so new samples are missing.

  3. Inspect freshness before trusting the panel

    A chart can show old data or no data. Check target status, timestamps, query labels and time range before scaling based on it.

About this model

Illustrative scrape-based metrics stack; installation is optional and metric names require instrumentation.

Walkthrough: scrape useful measurements and label them carefully

Prometheus commonly discovers targets and scrapes their metrics endpoints. A counter increases over time except for resets, while a gauge represents a current value that may rise or fall. A histogram records a distribution through buckets and related values; a single average can hide a slow tail. Query the appropriate rate or distribution rather than comparing raw cumulative counts across processes with different start times.

This training Prometheus configuration scrapes one endpoint. It is a Prometheus server configuration fragment, not a Kubernetes resource; a production deployment needs installation, service discovery, security and retention configuration.

scrape_configs:
  - job_name: application
    scrape_interval: 15s
    static_configs:
      - targets: ["api.learning.svc.cluster.local:9090"]
        labels:
          environment: training

PromQL expressions:

# Example application metric: only works if instrumentation exports it.
sum by (service) (rate(http_requests_total[5m]))
# A resource signal, when a suitable collector exports this metric.
sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))

Every unique metric name and label set creates a time series. Labels such as service, method and a normalized route template often support useful aggregation. User IDs, request IDs, full URLs and arbitrary error messages can create unbounded cardinality, memory cost and sensitive-data exposure. Put a request identifier in logs or trace correlation fields rather than making it a metric label. Even apparently bounded labels need review when teams can supply arbitrary values.

Grafana queries configured data sources and visualizes their results; it is not automatically the collector or storage engine. A blank panel may reflect a wrong time range, missing label filter, unavailable source or absent instrumentation. Compare a panel with the raw backend query and its target health before concluding the application is idle. An alert should link to a user-visible symptom and a useful investigation, rather than firing on every harmless restart.

Carry context across calls, then correlate signals

Trace parent/child spans, structured log correlation and Collector processing with redaction.

Flow viewStep 1 of 4
Start a request traceInstrumentation creates a span with a service identity and operation. Useful attributes describe the operation without copying sensitive payloads.. Client request, Starts trace context, idle. API span, Service name + operation, active. Worker span, Child context across call, idle. Structured log, Trace ID + useful event, idle. OTel Collector, Receive, process, export, idle. Telemetry backend, Query traces and logs, idleClient request: Starts trace context
Client requestStarts trace contextContext
API span: Service name + operation
API spanService name + operationIn focus
Worker span: Child context across call
Worker spanChild context across callContext
Structured log: Trace ID + useful event
Structured logTrace ID + useful eventContext
OTel Collector: Receive, process, export
OTel CollectorReceive, process, exportContext
Telemetry backend: Query traces and logs
Telemetry backendQuery traces and logsContext
Client requestStarts trace contextContext
API spanService name + operationIn focus
Worker spanChild context across callContext
Structured logTrace ID + useful eventContext
OTel CollectorReceive, process, exportContext
Telemetry backendQuery traces and logsContext
Animation controls
Speed
0.5×
1×
2×
Explore a scenario
Normal behavior
Context propagation is missing
Start a request trace

Instrumentation creates a span with a service identity and operation. Useful attributes describe the operation without copying sensitive payloads.

Read the complete visual sequence
  1. Start a request trace

    Instrumentation creates a span with a service identity and operation. Useful attributes describe the operation without copying sensitive payloads.

  2. Propagate across the dependency

    The API passes trace context so the worker span can be connected to the same request rather than appearing as an unrelated operation.

  3. Correlate a log event

    A structured application event records trace/span IDs and relevant safe fields, allowing a trace investigation to find the corresponding log.

  4. Process and export signals

    The Collector can batch, filter and export telemetry. Sampling and queue limits affect coverage; a missing trace is not proof of no request.

Context propagation is missing

The downstream service creates a new unrelated trace.

  1. API trace exists

    The entry operation is instrumented correctly.

  2. Dependency starts a separate trace

    Without propagated context, the worker operation is recorded but cannot automatically be linked as the child of the API span.

  3. Repair the call boundary

    Configure client/server instrumentation and async propagation, then send a known request and verify parent-child linkage.

About this model

Instrumented services propagate compatible context; traces can be sampled.

Practice: correlate one request without collecting its payload

kubectl top pods -n learning
kubectl logs deployment/api -n learning --since=10m --tail=100
kubectl get events -n learning --sort-by=.metadata.creationTimestamp
# Inspect only the telemetry components actually installed in the lab.
kubectl get pods -A

The observability lab distinguishes baseline kubectl evidence from optional Prometheus, Grafana and OpenTelemetry installations. Do not expect the sample metric names to exist without their exporters or application instrumentation. First send one known request and confirm that its counter changes, its structured log appears and its trace is exported where those systems are installed.

OpenTelemetry defines vendor-neutral instrumentation and signal handling, including SDKs and a Collector that can receive, process and export telemetry. Instrumentation creates spans and propagates context across calls; a Collector does not magically infer missing application spans. Correlate structured logs with trace and span IDs and stable resource attributes such as service name. Preserve context across asynchronous boundaries deliberately. Sampling reduces trace volume but means the absence of a trace is not proof the request never happened.

Pitfalls: telemetry has its own failure and trust boundaries

A saturated collector or unavailable backend can drop or delay evidence while the application continues serving. Set bounded queues and retention appropriate to the operational need. Redact credentials and avoid recording raw authorization headers, personal data or full prompts by default. Limit access to telemetry because it can expose tenant names, resource identifiers and infrastructure details even without payloads. During an incident, compare signal freshness with wall-clock time before trusting a reassuring stale chart.

Check your understanding

The dashboard shows no requests, but an application log contains a fresh successful request. Is the log necessarily wrong?

No. The metrics path may be missing instrumentation, failing its scrape, using a mismatched query label or showing the wrong time range. Compare the raw metrics endpoint, target health and backend query. Different signals have different collection paths, and disagreement is a reason to test those paths rather than discard inconvenient evidence.

Next, package deployment configuration reproducibly and use these signals to decide whether scaling or a release change actually improves the application.

Primary documentation

Go to the source

Reviewed 2026-10-27 · Kubernetes 1.37 · Related local exercise

Search the book

Search loads when you open it.