Back to Blog
Logs, Metrics, Traces—Oh My! Observability That Matters!!

Logs, Metrics, Traces—Oh My! Observability That Matters!!

Parth Shah

Why “It Works on Prod” Isn’t Funny Anymore

When your entire company fits on a Zoom grid, the pager often routes straight to the developer who wrote the code. Good observability means you fix bugs at 2 AM without a crystal ball—or better yet, you spot them before users do.

Three Pillars, One Budget

PillarTooling (lean stack)When to invest first
MetricsPrometheus → GrafanaImmediate: uptime & latency
LogsLoki (Grafana Cloud)Sprint 2: debug traces
TracesOpenTelemetry → TempoAfter you hit micro-service count ≥ 3

If money is tight, start with Prometheus + Grafana Cloud free tier—you get 10 k series and alerts out of the box.

1 | Instrument the Golden Signals

“Everything” is unmaintainable; track RED + USE:

  • Rate – requests per second
  • Errors – 4xx / 5xx per second
  • Duration – latency percentiles
  • Utilization – CPU, memory
  • Saturation – queue length
  • Error – hardware or disk faults

Below is a minimal Spring Boot actuator scrape:

management:
  endpoints:
    web:
      exposure:
        include: health,metrics,prometheus
server:
  servlet:
    context-path: /api

Prometheus job:

- job_name: "todo-api"
  metrics_path: "/api/actuator/prometheus"
  static_configs:
    - targets: ["todo-blue:8080","todo-green:8080"]

2 | Alert Only on What Hurts Users

Bad:

High CPU usage on node-12

Good:

SLO violation: p95 latency > 250 ms for route /tasks

GrafanaAlert YAML:

expr: histogram_quantile(0.95, sum(rate(http_server_requests_seconds_bucket[5m])) by (le, route)) > 0.25
for: 3m
labels:
  severity: page
annotations:
  runbook: https://runbooks.todo.com/p95-latency

3 | Add Traces When Logs Aren’t Enough

OpenTelemetry Java agent—no code change:

java -javaagent:opentelemetry-javaagent.jar       -Dotel.resource.attributes=service.name=todo-api       -Dotel.exporter.otlp.endpoint=http://tempo:4317       -jar todo-api.jar

Now you can click a slow request in Grafana and jump straight into its trace waterfall. The first time you catch an N+1 SQL query in 15 s, you’ll wonder how you lived without it.

4 | Dashboards That Fit on One Screen

PanelWhy it earns real estate
p95 Latency (per route)Performance health
Error Rate (4xx, 5xx)Immediate alarms
DB Connection Pool UsageEarly deadlock signal
Queue Lag (Kafka/SQS)Back-pressure insights
Release Marker AnnotationsCorrelate deploys with spikes

Rule of thumb: < 6 graphs; if you need to squint, you need another dashboard.

5 | Cost Hacks

  • Sample traces at 10 % except when X-Debug: true.
  • Loki log labels sparingly: {level="error", app="todo-api"}. Unbounded label cardinality → $$$.
  • Ship metrics to Cloud but store logs locally on a 7-day retention EBS; archive to S3 Glacier after.

6 | Post-Incident Review Template

  1. Timeline – UTC timestamps, who did what.
  2. User impact – errors served, revenue lost.
  3. Root cause – single sentence, no blame.
  4. Detection gap – why alerts fired late / never.
  5. Next steps – patch + monitoring upgrade PR numbers.

Keep it in your repo under /postmortems/YYYY-MM-DD-incident.md so newcomers learn fast.

Key Takeaways

  • Start with metrics + alerts today; add logs and traces as your service map grows.
  • Alert only on user-facing SLIs; silence the noisy CPU charts.
  • Keep dashboards minimal—engineers remember pictures, not 42 graphs.
  • Observability is insurance for lean teams: small investment now, big payout when prod burns.

Your future self at 2 AM will thank you for every histogram bucket you add today.

ObservabilityMonitoringStartups