What is observability? Logs, metrics, and traces explained

Observability is a measure of how well you can infer the internal state of a system from its external outputs. The term comes from control theory, where it was introduced by the engineer Rudolf E. Kálmán in 1960 for linear dynamical systems. In software, observability means you can answer open-ended questions about why a system behaves the way it does — not just "is it up?" but "why is it slow for these users?", "which service is the bottleneck?", and "what happened right before the error rate spiked?"

A system is observable if you can determine its behavior from the data it produces, without adding more instrumentation. If an application only tells you whether it is online or offline, that is monitoring, not observability. If it emits logs, metrics, and traces that let you reconstruct the full chain of events behind any incident, it is observable. The practical goal is the ability to debug unknown problems in a distributed system whose parts you cannot fully reason about ahead of time.

The three pillars: logs, metrics, and traces

Observability in modern practice rests on three complementary types of telemetry data, sometimes called the three pillars:

• Metrics are numeric measurements collected over time, such as request rate, error rate, CPU usage, memory use, and latency percentiles. They are aggregated and cheap to store and query, which makes them ideal for dashboards and alerts. Metrics answer "What is happening?" in aggregate — they tell you a value went up or down, but not why or for which specific request.
• Logs are timestamped records of discrete events, written by applications, libraries, and the operating system. Each log line describes a single event: a request arrived, a user logged in, an error was thrown. Logs are rich in context but large in volume, so they are filtered and searched rather than continuously visualized. They answer "Why did this happen?" with the detail a human can inspect.
• Traces follow a single request's path through every service and component it touches, recording the duration of each step (each step is called a span). Because a single user request often fans out across many services — an API gateway, an authentication service, a database, a cache — traces are the only one of the three that reconstructs the end-to-end journey of one specific request. They answer "Where exactly did this request slow down or fail?"

The three work together. Metrics alert you that something is wrong ("error rate is up"), traces show you the path and where the failure happened, and logs supply the surrounding context to identify the root cause. No single pillar is sufficient on its own.

Monitoring vs. observability

Monitoring and observability are related but not the same. Monitoring is the act of collecting, processing, and displaying real-time data about a system in order to detect known problems — it is built around known questions and predefined dashboards, alerts, and thresholds. Observability is the broader property that lets you investigate unknown problems: with rich telemetry in place, you can ask new questions and trace a problem to its cause even when you did not anticipate that specific failure mode.

One useful way to see the difference: monitoring tells you when something is wrong; observability helps you understand why. A monitoring system with thresholds can page you when latency spikes. It is observability — the ability to dig into logs and traces and correlate them — that lets you discover the spike was caused by a database connection pool exhaustion triggered by a slow third-party API. Good observability makes monitoring far more effective, because every alert can be followed by a fast, evidence-based investigation.

The four golden signals

In Google's Site Reliability Engineering book, Rob Ewaschuk defines four golden signals — the minimum set of metrics that give a good picture of a user-facing service's health. They were introduced by Google in 2014 and are widely reused by SRE and DevOps teams:

• Latency — the time it takes to serve a request, ideally reported separately for successful and failed requests (a fast error is not a good response).
• Traffic — how much demand is being placed on the system, measured in requests per second, queries per second, or similar throughput units.
• Errors — the rate of requests that fail, whether explicitly (HTTP 500s) or implicitly (a 200 response with wrong content).
• Saturation — how close the system is to its capacity limits, for the resource that is most constrained (CPU, memory, database connections, queue depth, disk I/O).

Tracking these four signals covers the essential dimensions of health; teams add more metrics as specific services require, but the golden signals are the recommended baseline.

White-box vs. black-box monitoring

Another useful distinction from the Google SRE book is between two flavors of monitoring:

• White-box monitoring relies on internal signals and metrics that the system exposes about itself, through logs, interfaces that emit internal statistics, or instrumentation you add to your own code. It tells you what is happening inside the system and is essential for debugging.
• Black-box monitoring tests externally visible behavior the way an actual user would experience it, for example by periodically requesting a public endpoint and checking that it responds correctly within a time limit. It proves that the user-visible service works, regardless of how its internals are implemented.

Production environments need both: black-box checks catch genuine user-facing outages, while white-box metrics and traces explain why an outage happened and let you respond before it worsens.

Putting observability into practice

Bridging the theory to implementation is where observability tools come in. Open-source stacks like Prometheus (metrics) and Grafana (visualization, dashboards, and alerting) handle the metric pillar; your application emits numeric counters, gauges, and histograms that a metrics server scrapes and stores as time series. Log aggregation and search platforms collect log lines from many hosts into one searchable store. Tracing engines record spans and reconstruct the request path across services. Managed platforms such as Datadog, New Relic, and Grafana Cloud bundle logs, metrics, and traces into a single tool so you can correlate all three without running several systems.

Whatever stack you choose, the principles are the same: instrument your code so it emits useful telemetry, keep metrics cheap and log volume manageable, and use the three pillars together rather than expecting any one of them to answer every question.

FAQ

What is the difference between monitoring and observability?

Monitoring is the practice of collecting and displaying data about a system to detect known problems, using predefined dashboards, alerts, and thresholds. Observability is the property of a system that lets you ask new questions and investigate unknown problems from the telemetry it emits. Monitoring tells you something is wrong; observability helps you understand why.

Which is better for my app: metrics, logs, or traces?

You need all three working together. Metrics are cheap and good for alerting and dashboards, logs provide the detailed context of individual events, and traces show the end-to-end path of a single request across services. Trying to debug a distributed system with only one pillar leaves important questions unanswered.

What are the four golden signals?

The four golden signals, defined in Google's Site Reliability Engineering book, are latency (how long requests take), traffic (how much demand the system is handling), errors (the rate of failed requests), and saturation (how close the system is to capacity). Together they give a baseline picture of a user-facing service's health.

Do I need a full observability platform, or can I start small?

Start small and grow. Many teams begin with an open-source metrics stack (Prometheus plus Grafana) for the metric pillar, then add structured logging to a searchable store, and add traces only when they need to debug cross-service latency. The three-pillar principles apply at any scale, and you can adopt a managed platform later to bundle them together.