Watchtower

Watchtower closes the gap between “this service is unhealthy” and the specific request that shows why. You give it a window; it gives you every trace, log and metric inside that window, grouped.

​

Picking a window

Three ways in:

• A time range — 15m, 30m, 1h, 6h or 24h.
• A point in time — a timestamp plus a margin (±1m, ±5m, ±15m, ±1h), for when a customer tells you roughly when it broke.
• An alert — the Recent Alerts panel lists recent firings; clicking one jumps to the window it fired on.

Clicking a bar in the throughput histogram narrows to that 60-second bucket and re-derives every panel below it.

​

What you get

Throughput and failures — spans per 60s bucket, errored spans in red. This is the shape of the incident. RED metrics per service — rate, errors and duration, so you can see which service actually changed rather than which one is loudest. Log patterns — Drain3 template mining collapses the window’s log lines into a handful of patterns with counts. Expanding a pattern lists the trace ids that produced it, so you can go straight from “this error is new” to a request that shows it. Trace patterns — the same grouping over service and operation. Most failures / slowest operations — ranked by errored span count and by p95/p99. Raw output — the actual log and span rows behind all of the above, with a substring filter. Aggregates tell you where to look; this is where you read what happened.

​

How the culprit is chosen

When a trace contains several errored spans, Watchtower reports the error roots — an errored span with no errored descendant. Depth is the obvious alternative and it is wrong: it ranks correctly along a single chain but is meaningless across branches, and it silently discards one of two independent failures. When there are several roots, Watchtower says so (“3 independent failures”) rather than tie-breaking into one confident-looking answer.

​

Silent callees

An error root with no child at all means the callee never answered — no span, no log, nothing. That absence is the finding. It is the one case where the evidence is something that is missing rather than something that is present, and no metric or log line can show it to you.

​

Pattern grouping

Log and trace grouping runs through a separate Drain3 service. It masks UUIDs, hex ids, IP addresses, emails, timestamps and numbers before the template tree sees them — Drain groups by token position, so unmasked ids would put every line in its own cluster. The service is a soft dependency. If it is unreachable, the two pattern panels are dropped and a banner is shown; the rest of the window still works.