Production Observability: Logs, Metrics, and Traces That Actually Debug
From the three observability pillars to correlation IDs, log levels, RED/USE, and alerts that do not numb the team.
Ali Mortazavi
Founder, Paradise Code
Without observability, production is hope
When a user sees an error, the first question is “where,” not “maybe what.” Structured logs, golden metrics, and distributed traces are the three legs of the answer. One alone is not enough: logs without metrics are a flood, metrics without traces are blind, traces without business context are decoration.
From day one, carry `requestId`/`correlationId` from the edge through queues and databases.
Structured logs with field discipline
JSON with stable fields: level, service, environment, user/tenant when needed, error code, and a short message. Prefer those fields over long free text. Redact PII and secrets—logs are an attack surface too.
Take levels seriously: debug off by default in production, info for meaningful events, warn for recoverable issues, error for needs-action. If everything is error, nothing is.
Golden metrics in RED and USE styles
For request services: Rate, Errors, Duration (RED). For resources: Utilization, Saturation, Errors (USE). Watch p95/p99, not only averages. Put business metrics—successful signup, successful payment—beside system metrics so you know where the real outage is.
Control label cardinality; userId on metrics explodes time series.
Distributed tracing for critical paths
Focus trace sampling on money paths and error paths. Spans should name domain operations, not only middleware. When latency rises, traces show whether it is DB, an external dependency, or a queue.
Unlimited sampling is expensive; targeted sampling beats blind 100%.
Actionable alerts, not noise
An alert needs an owner, meaning, and first action. Alert on user-facing symptoms (payment errors, public API latency) not every brief CPU spike. Alert fatigue is more dangerous than carefully limited alerting.
Give each service a golden dashboard and a one-page runbook: how to diagnose, how to mitigate, how to escalate.
Post-incident culture
After an incident, connect “what signal was missing?” to the observability backlog. A new tool without questions is shelfware. Observability matures when every sensitive feature ships with its own metrics/logs.
That is the difference between a maintainable system and a terrifying one at 2 a.m.
Frequently asked questions
Is OpenTelemetry required?
An open standard for traces/metrics helps a lot, especially across services and vendors. Field contracts matter more than tool brands.
How long should we retain logs?
Depends on compliance; operationally often 7–30 hot days plus colder archive. Control cost with rate and sampling.
Do we need frontend logs too?
Yes for client errors and vitals, with sampling and without PII. Tie them to server correlation when possible.
Insights
Need these ideas implemented in your product?
Paradise Code supports you from consult to full delivery.