Three signals, one context
diagram
| Signal | Answers | Tip |
|---|---|---|
| Metrics | Is something wrong? How much? | RED for services (Rate, Errors, Duration); USE for resources |
| Traces | Where is it slow or failing? | Propagate traceparent across HTTP and queues |
| Logs | Why exactly? | Structured JSON with traceId, tenantId, requestId |
SLOs
- SLI: a measured ratio, e.g. "% of publish API requests under 500 ms and non-5xx".
- SLO: the target, e.g. 99.9% over 30 days. The error budget is 0.1%.
- Alert on burn rate (budget consumed too fast), not on CPU at 80%.
Multi-tenant extras
Tag every signal with tenantId, since "is it one customer or everyone?" is the first incident question. Watch the cardinality cost in metrics: use exemplars or logs for per-tenant detail.
Sources & further learning
Videos, courses, docs and books I recommend for this topic.
OpenTelemetry — What is OpenTelemetry?Docs · OpenTelemetry · opentelemetry.ioSite Reliability Engineering (free online) — SLOs chapterBook · Google · sre.googleObservability EngineeringBook · Majors, Fong-Jones, Miranda · oreilly.comDistributed tracing explained — video searchVideo · YouTube · youtube.com
Related topics
Architecture PatternsUnderstood
Circuit Breaker, Retry, Timeout & Bulkhead
The resilience toolkit — fail fast, retry safely, bound waiting, and isolate resources so one slow dependency can't take down the whole system.
Intermediate6 sources
Architecture PatternsApplied in production
Microservices Architecture
Structure a system as independently deployable services, each owning a business capability and its data, communicating over the network.
Intermediate5 sources
Cloud & DeliveryLearning
Kubernetes Essentials for Architects
The objects that matter — Deployment, Service, Ingress, ConfigMap/Secret, HPA — and the settings that decide reliability (probes, requests/limits, PDBs).
Intermediate4 sources