The dashboard we stopped looking at.

The board was built by Mihail in the autumn of 2022, when he was still excited about us. It had forty-two tiles arranged in seven rows of six, and every tile had a thin amber border that pulsed when the underlying metric crossed a threshold. It looked, from the doorway of the war room in our second-floor office on Rua das Janelas Verdes, like the cockpit of something serious. We were proud of it for about eleven weeks.

By February it had become wallpaper. By June it had become a kind of indoor weather — pulsing slightly all the time, ignored by people walking past, occasionally turning the room a faint amber when the p99 latency on the order-ingest service drifted past 480 ms for no reason anyone could explain and then drifted back. We stopped pointing at it during standups. The intern stopped asking what the tiles meant. The CTO, on his rare visits from the Tallinn office, would glance at it and say "good, good" and we would all nod.

The truth was that nobody was watching it. Not really. We were watching PagerDuty, which was watching the same metrics through a less ambitious lens, and which had the rude virtue of waking us up. The dashboard was decorative; the pager was operational. We were building the wrong instrument for the wrong audience and we did not notice because the building of it had felt productive — a thing engineers do instead of asking what alerting is actually for.

A dashboard nobody trusts is worse than no dashboard. It absorbs attention without paying it back. — Pia Halvorsen, retro notes, 17 January

I want to be specific about the failure modes, because "alert fatigue" is one of those phrases that has been worn so smooth by conference talks it barely catches anymore. Our fatigue had a shape. It was: a Slack channel called #alerts-prod with 2,300 messages a week, of which roughly 87% were the same six warnings firing on the same six hosts. It was Marko, our newest hire, asking on his second Monday whether p99_orders_ms_high was something he should care about, and three of us saying "no, that one's always like that" without irony.

It was the dashboard tile labeled queue_depth_orders showing 14,000 at 09:32 on a Tuesday, which would have been alarming in 2022 and was now, in our institutional memory, "just how Tuesdays look after the Polish locale runs its overnight reconciliations." We had stopped asking why. We had stopped looking. The dashboard was a museum of metrics we no longer interrogated, and the alerts were a doorbell rung by ghosts.

The thing about a dashboard is that it is a statement of what the team believes matters. Ours, in retrospect, was a statement that we were not sure, that we wanted everything in front of us in case any of it turned out to be the important thing. It was, to use a word we have since adopted from Pia, cowardly. We had built a screen that absolved us of the harder work of editing.

What follows is the story of the page that made us edit it. It is not a triumphant story. It involves an engineer named Sami crying quietly in a hotel room in Porto, a Slack thread of forty-one replies arguing about whether to roll back a config change, and a single line in a Helm values file with a transposed digit that should not have been able to do what it did.

It began at 04:11 UTC on Sunday, 23 April, with a PagerDuty notification titled OrderIngestLatencyHigh — multi-region. Sami, who was in Porto for a wedding, acknowledged it from his phone in 47 seconds, which is good, and then spent the next eleven minutes trying to find his laptop charger in a hotel room he did not entirely remember checking into, which is human.

By 04:23 the page had escalated. Three more engineers were on, plus me, plus eventually our manager Helga, who had her phone on her bedside table in Tallinn and answered with the particular flat voice of someone who has been here before. We had eleven Datadog dashboards open between us. None of them showed what was happening, because what was happening was not a latency problem. What was happening was that the staging instance of our order-ingest service had been deployed with thirty replicas instead of three, due to a transposed digit in a Helm values.yaml file Marko had touched the previous Friday afternoon and forgotten to revert.

The thirty replicas had started up cleanly. They had then, all of them, opened persistent connections to the production Kafka cluster — because staging and production shared brokers, a decision made in 2021 we kept meaning to reverse — and begun consuming from the live orders.v3 topic, which they had no business reading. Production consumers slowed. Latency on the customer-facing API rose. The dashboard tile for p99 latency went amber, then red, then nobody looked at it because we were arguing in a Zoom call about whether to roll back the last production deploy, which had nothing to do with anything.

We rolled back the wrong service. We rolled it back twice. At 04:42 Helga, with a clarity that I think about often, said "nobody touch anything for sixty seconds and someone read the goddamn changes." Marko, who had not slept either, opened the Helm chart, read the transposed digit, and said the word "oh" in a way I will not forget. By 04:58 the staging replicas were scaled down. By 05:04 latency was back to baseline. By 05:11 Sami was crying, quietly, in his hotel room, and I was on a call with him pretending I could not hear it.

The dashboard, throughout, had done nothing useful. It had pulsed. It had shown forty-two tiles, of which precisely zero pointed at the actual problem, which was a config-management failure that no panel had ever been designed to detect. We had instrumented for performance and forgotten to instrument for identity — for which workload, deployed by whom, against which broker, was doing the consuming. The board was a map of where we had looked before. The incident was somewhere we had not.