on-call

The recurring incident your ticket system cannot see.

A ticket records a resolution. Your log records the third time.

A ticket system is built to close things. That is its job, and it does it well: something breaks, a reference is allocated, work happens, a resolution is recorded, the record shuts. Every incident is complete and self-contained by design.

Recurrence is the one thing that structure cannot represent. The fourth time a problem comes back it gets a fourth reference, with its own summary and its own resolution, and nothing in the record says it is the same problem. The knowledge that it is lives in the head of whoever responded, and only if the same person responded each time.

What recurrence looks like from inside the ticket system.

Six months of tickets that are one underlying problem
Reference Raised Summary as filed Closed as
INC-771402/04/2026Intermittent timeouts on checkoutCould not reproduce
INC-810619/05/2026Payment page slow for some customersResolved, restarted the service
TICK-184219/07/2026Checkout p95 above targetResolved, config rollback
INC-902104/08/2026Elevated error rate, checkoutResolved, restarted the service

Four references, four summaries written by four people in a hurry, three different resolutions, and two of them are "restarted the service". Nothing here says these are one problem. Read down the table and it looks like four unrelated bad afternoons, which is exactly how it gets treated at the quarterly review.

What the same six months look like in a log.

Now the same period from the notes of the people who responded, with the identifiers left in. The tickets are still there, but so is the thing the tickets do not carry.

02/04/2026
INC-7714. #checkout timing out. vpn.internal.example looked unhealthy but recovered before I could capture anything. Closing as not reproducible, noting it here in case
19/05/2026
INC-8106. Same shape as April. Restarted the gateway, vpn.internal.example settled. p95 back to 210ms
19/07/2026
TICK-1842. #checkout p95 480ms. Tunnel to vpn.internal.example dropping every few hours, rolled the config back
04/08/2026
INC-9021. Third time on vpn.internal.example since April. Restarted again. Someone needs to own this
The same host name appears in all four entries and in none of the four ticket summaries. The interval between recurrences is 47 days, then 61, then 16, so it is not only recurring, it is accelerating. Both facts are readable only because the responders wrote the hostname down each time.

The shortening interval is the finding. Any one of these afternoons is an annoyance you work through, and four of them with a closing gap is a component degrading toward a real outage. Nobody saw it because nobody was looking at the four together, and there was nowhere the four appeared together.

Why the ticket system cannot fix this itself.

This is not a criticism of ticket tooling, and adding a "related incidents" field does not solve it. Somebody still has to know the relationship exists in order to fill that field in, and the person filing the fourth ticket at eleven at night has not read the first three. The link that would have to be made is the one nobody has yet made.

The reason the notes work is that they are not organised by incident. They are organised by time and they contain names, so anything that keeps happening leaves the same name in the log repeatedly, without anyone deciding in advance that it was a category worth tracking. That is a property of the format rather than an effort anyone made.

A note format that survives an on-call shift.

The constraint is that this has to be writable while something is on fire, at two in the morning, by somebody who is not going to follow a template. Anything longer than a few lines gets skipped exactly when it matters. What follows is short enough to survive that.

~/log/oncall.md
04/08/2026
INC-9021 #checkout errors up from 22:40.
vpn.internal.example unhealthy again, third time since April.
Restarted the gateway at 23:05, p95 480ms back to 215ms.
Tried draining one node first, no effect.
Same shape as INC-8106 in May.

Five lines, one per thing worth retrieving later. The reference, the tag, the host, the numbers, and one line naming what did not work. The last line is a gift to whoever gets paged next, and it is the line most people leave out because it feels like admitting to a wasted twenty minutes.

The one habit that makes the difference is writing vpn.internal.example rather than "the tunnel". At 23:05 there is only one tunnel and the shorthand is obvious. In November there are four, and the entry that says "the tunnel" drops out of every search that would have found this pattern. There is more on that habit in the first article below.

What this does not catch.

It only sees recurrence that somebody wrote down. If four different people responded to those four incidents and one of them keeps a log, you find one quarter of the pattern, and one quarter of a pattern looks like noise. This approach gets much stronger with a shared log and it is genuinely weak with a solo one, which is worth being honest about before anyone builds a process on it.

It also cannot tell you the cause. Everything on this page establishes that four events share a name and that the gaps between them are closing. Why the host degrades is a question for the host, and the notes have done their job when they get somebody to ask it seven weeks earlier than they otherwise would have.

And it depends on a habit holding during the worst shifts. The notes from a calm week are always good. The notes from the night everything broke are the ones that decide whether any of this works, and those are the hardest to write.

The smallest version of this.

At the end of the next incident, before closing the ticket, write five lines into a plain text file with a date above them. Put the reference, the host, and the number in. It costs about ninety seconds and it does nothing at all the first time. The fourth time, it is the only place the pattern exists.

Keep reading.

blk/txt reads log files you already keep and makes them searchable by date, reference, and tag. The FAQ covers accounts and privacy, and the trial runs on text you have already written, with no account to make first.