# Watchtower

> From a time range to the request that proves it — and the log line that explains it

Watchtower closes the gap between "this service is unhealthy" and the specific request
that shows why. You give it a window; it gives you every trace, log and metric inside
that window, grouped.

> **Note:** Watchtower is deterministic. It groups, counts and joins — nothing infers a cause. The
  alert already said something was wrong, and the log line already says why.

## Picking a window

Three ways in:

- **A time range** — 15m, 30m, 1h, 6h or 24h.
- **A point in time** — a timestamp plus a margin (±1m, ±5m, ±15m, ±1h), for when a
  customer tells you roughly when it broke.
- **An alert** — the Recent Alerts panel lists recent firings; clicking one jumps to the
  window it fired on.

Clicking a bar in the throughput histogram narrows to that 60-second bucket and
re-derives every panel below it.

## What you get

**Throughput and failures** — spans per 60s bucket, errored spans in red. This is the
shape of the incident.

**RED metrics per service** — rate, errors and duration, so you can see which service
actually changed rather than which one is loudest.

**Log patterns** — Drain3 template mining collapses the window's log lines into a handful
of patterns with counts. Expanding a pattern lists the trace ids that produced it, so you
can go straight from "this error is new" to a request that shows it.

**Trace patterns** — the same grouping over service and operation.

**Most failures / slowest operations** — ranked by errored span count and by p95/p99.

**Raw output** — the actual log and span rows behind all of the above, with a substring
filter. Aggregates tell you where to look; this is where you read what happened.

## How the culprit is chosen

When a trace contains several errored spans, Watchtower reports the **error roots** — an
errored span with no errored descendant.

Depth is the obvious alternative and it is wrong: it ranks correctly along a single chain
but is meaningless across branches, and it silently discards one of two independent
failures. When there are several roots, Watchtower says so ("3 independent failures")
rather than tie-breaking into one confident-looking answer.

### Silent callees

An error root with **no child at all** means the callee never answered — no span, no log,
nothing. That absence is the finding. It is the one case where the evidence is something
that is missing rather than something that is present, and no metric or log line can show
it to you.

## Pattern grouping

Log and trace grouping runs through a separate Drain3 service. It masks UUIDs, hex ids,
IP addresses, emails, timestamps and numbers **before** the template tree sees them —
Drain groups by token position, so unmasked ids would put every line in its own cluster.

The service is a soft dependency. If it is unreachable, the two pattern panels are
dropped and a banner is shown; the rest of the window still works.