go-observability

Go Observability

Observability is not optional for production services. Every service must produce structured logs, expose metrics, and propagate trace context. Use the stdlib log/slog for logging and OpenTelemetry for tracing and metrics.

Community default. A project skill or agent-config rule that explicitly supersedes go-observability takes precedence over this one.

Detailed reference material, loaded on demand:

• references/slog.md — handler setup, logger injection, child loggers, LogValuer redaction, trace-ID handler, hot paths, runtime levels.
• references/tracing.md — tracer provider, spans, propagation, sampling, profile labels, shutdown.
• references/metrics.md — metric definitions, HTTP instrumentation.
• references/alerting.md — RED/USE, histograms vs summaries, burn-rate alerts, alert hygiene, exemplars.

Read a reference file only when the section below is not enough.

1. Structured Logging with slog

Use log/slog (Go 1.21+) with a JSON handler in production. Every line is key-value pairs, never a formatted sentence:

// ✅ Good — structured, leveled
logger.Info("user created", slog.String("user_id", user.ID), slog.Duration("latency", elapsed))

// ❌ Bad — unparseable in production
log.Printf("user %s created in %v", user.ID, elapsed)

Inject the logger as a dependency; never reach for a package-level global. Derive child loggers with logger.With(...) so component, method and request ID are attached once instead of at every call site. See references/slog.md.

Log levels — use them consistently

Level	Use for
Debug	Verbose diagnostic info, disabled in production
Info	Normal operations: request received, job completed
Warn	Recoverable issues: retry succeeded, deprecated usage
Error	Failures requiring attention: DB down, external call failed

NEVER log at Error level for expected conditions (user not found → Info or Warn).

Sensitive data — NEVER log

• Passwords, tokens, API keys
• Full credit card numbers, SSNs
• Personal identifiers: email addresses, phone numbers, IP addresses where they identify a person
• Raw request bodies containing PII

// ✅ Good — an opaque internal ID, no personal data
logger.Info("auth attempt", slog.String("user_id", userID), slog.Bool("success", ok))

// ❌ Bad — leaks a credential and a personal identifier
logger.Info("auth attempt", slog.String("password", password))
logger.Info("auth attempt", slog.String("user", email))

Log the stable internal ID and join to personal data in the system that is allowed to hold it. Log retention, search access and third-party log sinks rarely meet the bar that applies to the user table.

Make redaction a property of the type: implement slog.LogValuer on secrets and on structs that hold personal data, so slog.Any(user) logs only the safe fields everywhere.

Cost and control

• Hot paths use logger.LogAttrs and guard expensive attributes with logger.Enabled.
• A slog.LevelVar in HandlerOptions lets the level change at run time without a deploy.
• slog.NewMultiHandler (Go 1.26+) fans out to several handlers.

2. Distributed Tracing with OpenTelemetry

Start a span for every operation worth seeing on a waterfall — inbound requests, DB calls, outbound HTTP, meaningful business steps — and always defer span.End().

Rules the reference examples follow:

• Name the span after the operation (GetUser, db.query), never after the fully-qualified function, and never doStuff.
• On failure call span.RecordError(err) and span.SetStatus(codes.Error, ...), or the trace shows a green span for a failed request.
• Pass ctx down the whole chain. A context.Background() mid-chain silently starts a new trace and breaks the parent link.
• Sample with ParentBased(TraceIDRatioBased(r)) so services agree on a trace; keep errors and slow requests with tail sampling in the collector.
• Profiles are the fourth signal: continuous profiling in production, with bounded pprof.Labels (route, tenant tier) to split a hot function.

Setup and examples in references/tracing.md.

3. Metrics with OpenTelemetry / Prometheus

Type	Use for	Example
Counter	Monotonically increasing values	Requests total, errors total
Gauge	Values that go up and down	Active connections, queue depth
Histogram	Distribution of values	Request latency, response size

Naming is <namespace>_<subsystem>_<name>_<unit>:

http_request_duration_seconds     ✅ (unit in name)
http_requests_total               ✅ (counter with _total suffix)
db_connections_active             ✅ (gauge, no suffix needed)
user_signups                      ❌ (missing _total for counter)
requestLatency                    ❌ (camelCase, no unit)

Keep cardinality bounded. Label with the route pattern, method and status — never a user ID, request ID or raw r.URL.Path, each of which mints a new time series per value and eventually takes the metrics backend down.

Definitions and middleware in references/metrics.md.

4. Connecting Logs, Traces, and Metrics

A log line without a trace ID cannot be joined to the request that produced it. Wrap the handler once so every *Context log call adds trace_id and span_id from ctx — the wrapper must also override WithAttrs and WithGroup, or child loggers lose it. Attach exemplars (trace IDs) to latency histograms to jump from a p99 spike to an offending trace.

5. Alerting and SLOs

• Measure RED (rate, errors, duration) per route for services and USE (utilization, saturation, errors) for pools and queues.
• Latency is a Histogram, never a Summary — summary quantiles cannot be aggregated across instances. Put buckets around the SLO threshold.
• Page on SLO burn rate over a long and a short window together; send slower burns to a ticket.
• Alert on symptoms, not causes; rate() not irate(); absent() for series that must exist; a runbook link on every page.

Queries and exemplar wiring: references/alerting.md.

6. Graceful Shutdown of Telemetry

The batch span processor holds spans in memory. Call tp.Shutdown(ctx) with its own timeout on the way out, or the last seconds before a crash — the interesting ones — never reach the collector.

Both examples in references/tracing.md.

Verification Checklist

1. All logging uses log/slog with structured key-value pairs, not fmt.Printf or log.Printf
2. Logger is injected as a dependency, not used as a global
3. No sensitive data (passwords, tokens, PII) in log output
4. Trace context is propagated through all function calls via context.Context
5. Spans are created for significant operations (DB calls, HTTP requests, business logic)
6. Spans record errors with span.RecordError(err) and set error status
7. Metrics follow naming conventions: _seconds, _total, _bytes
8. No high-cardinality labels (user IDs, request IDs) in metrics
9. Telemetry providers are shut down gracefully on service exit
10. Trace IDs are included in log entries for correlation
11. Latency recorded as histograms; paging alerts are SLO burn-rate alerts with runbooks