Production debugging should begin with evidence rather than assumptions.

I usually start by identifying the affected users, the time window, the expected behavior, and the actual behavior.

That incident frame turns a vague report into a bounded question. Logs, metrics, traces, deployment history, and recent configuration changes can then be compared without changing several variables at once.

Article map
ReportBound impactCorrelate signalsTest one hypothesisVerify

Define impact before reading logs

Capture when the behaviour started, which route or workflow is affected, whether every user sees it, and what a successful result looks like. Record identifiers that can be searched safely, such as a request ID, trace ID, job ID, or redacted resource ID.

Check whether the problem is active, intermittent, or already recovered. That determines whether the first job is mitigation, evidence preservation, or a controlled reproduction.

Correlate signals instead of collecting screenshots

Metrics show when behaviour changed and how broad the change is. Logs explain discrete events. Traces show the path and timing of one request across boundaries. A shared trace or request identifier turns those views into one investigation.

Compare the first bad window with a known-good window. Then overlay deployments, feature flags, configuration changes, dependency incidents, and queue depth. A correlation is a lead, not proof, but it gives the next test a direction.

Change one thing and verify the whole path

Prefer the smallest reversible mitigation that protects users: pause a worker, disable a flag, route around a dependency, or roll back a known change. Preserve the evidence needed for the root-cause investigation before it disappears.

Verification is more than watching an error counter fall. Repeat the affected journey, check latency and success metrics, inspect downstream queues, confirm no new error class appeared, and document the remaining uncertainty.

Step by step

An evidence-first investigation

  1. Write the expected and actual behaviour in one sentence each.
  2. Bound the users, operations, services, and time window affected.
  3. Find one reliable identifier and follow it through logs and traces.
  4. Compare metrics and telemetry with a known-good period and the change timeline.
  5. State one falsifiable hypothesis and run the smallest safe test.
  6. Mitigate reversibly, verify the complete workflow, and record follow-up work.

Code & commands

Working notes

Narrow a LogQL search

{app="api"} |= "error" | json
  | trace_id="4bf92f3577b34da6a3ce929d0e0e4736"

Check a bounded window

logcli query '{app="api"} |= "error"'   --since=30m   --limit=100   --stats

Structured error context

const context = span?.spanContext();

logger.error("checkout failed", {
  release: process.env.RELEASE_SHA,
  trace_id: context?.traceId,
  span_id: context?.spanId,
  error: error.message,
});

Inspect before rollback

kubectl rollout status deployment/api
kubectl rollout history deployment/api
kubectl rollout undo deployment/api
kubectl rollout status deployment/api

AI-assisted workflow

Use AI to narrow the work, not outsource judgment.

AI is most useful after the incident has been bounded. Give it a small, redacted evidence pack and ask for hypotheses or query drafts—not a declaration of root cause.

Generate testable hypotheses

Example prompt

Analyze this sanitized incident evidence.

Return:
1. Observed facts only
2. Up to three hypotheses
3. Evidence that would support or reject each hypothesis
4. The safest next read-only check

Do not:
- claim a root cause
- invent missing telemetry
- recommend a write or rollback yet

Evidence:
{{metrics, trace summary, deployment timeline}}

Human checks

  • Redact secrets, personal data, tokens, and customer payloads before sharing.
  • Reject hypotheses that do not predict a specific observable result.
  • Run the next check yourself and feed back the actual result.

Draft a bounded log query

Example prompt

Write one LogQL query for this investigation.

Known labels:
- app="api"
- environment="production"

Known fields after JSON parsing:
- trace_id
- level
- route

Need: errors for TRACE_ID in a 15-minute window.
Require a positive stream selector. Do not use broad regex labels.
Return the query and explain each filter.

Human checks

  • Set the time range in the log tool before running the query.
  • Inspect query statistics and narrow further if it scans too much data.
  • Treat the result as evidence, not an AI-generated conclusion.

Failure modes & troubleshooting

When the first path fails

The log search returns too much data to inspect.

Start with a positive stream selector, a narrow time range, and one identifier. Add parsing and filters only after the base query works.

Logs from two services cannot be connected.

Propagate trace context across the request boundary and include trace and span IDs in structured log records.

A deployment lines up with the failure, but rollback does not recover the workflow.

Treat timing as correlation. Check configuration, schema changes, dependency state, cached data, and queued work before declaring a cause.

The primary error rate falls but customers still see delays.

Verify downstream queues, retries, saturation, and end-to-end latency. Recovery of one signal is not recovery of the system.

Keep

Practical takeaways

  • Scope first; an unbounded incident produces unbounded searching.
  • Correlated telemetry is more useful than a larger pile of telemetry.
  • A good hypothesis predicts what evidence should change when tested.
  • Mitigation ends user impact; verification proves the mitigation worked.