TokayOps
03:14:07 Webhook

Wake the right person.

TokayOps sits between Alertmanager and the engineer who has to fix it. It turns each Alertmanager group into one live escalation, works out who is on call at this instant, escalates until somebody answers, and closes the loop in Slack or Telegram.

Install it Follow the alert

One container your Postgres source-available

One alert group, one night UTC
03:14:07 Created

Wake somebody once, not on every repeat.

Grouping is Alertmanager's job, and it stays there. The first delivery opens one Alert Group and starts one escalation. Every repeat afterwards merges into it: alerts added, cleared, or re-fired land on the same timeline, the chat message is edited in place, and nobody is woken a second time for the same thing.

The TokayOps Alert Groups view. Four cards, each one Alertmanager group, each holding two firing alerts. KubeAPIServerDown on the platform team, PaymentGatewayTimeout on payments and OriginUnreachable on edge are all critical and still triggered, with nobody named against Ack by. IngressLatencyHigh on edge is a warning and already acknowledged by Dana Whitfield. Every card names its team, who is on call for it, and how long it has been firing. The TokayOps Alert Groups view. Four cards, each one Alertmanager group, each holding two firing alerts. KubeAPIServerDown on the platform team, PaymentGatewayTimeout on payments and OriginUnreachable on edge are all critical and still triggered, with nobody named against Ack by. IngressLatencyHigh on edge is a warning and already acknowledged by Dana Whitfield. Every card names its team, who is on call for it, and how long it has been firing.
Alert Groups, filtered to what is still open
03:14:08 Assigned

Find who is on call now.

The alert's team label picks the team, its severity picks the escalation policy, and anything unlabelled lands in triage rather than nowhere. The schedule answers who is on call at this instant - after the rotation was edited, after somebody swapped a weekend, across the night the clocks change - and that answer sticks: editing the rotation cannot redirect a page already in flight.

Why that is harder than it looks
The TokayOps schedule calendar for the payments team, shown in UTC. Today's row is highlighted with Dana Whitfield on L1 and Tomas Lindqvist as L2 backup. Tomorrow the rotation hands over at 09:00 to a single slot holding two people, Marcus Ellery and Priya Raghavan, who then stay on for the following days. The TokayOps schedule calendar for the payments team, shown in UTC. Today's row is highlighted with Dana Whitfield on L1 and Tomas Lindqvist as L2 backup. Tomorrow the rotation hands over at 09:00 to a single slot holding two people, Marcus Ellery and Priya Raghavan, who then stay on for the following days.
The payments rotation, in the schedule's own timezone
03:14:08 Notified

Page until somebody answers.

Every member of the on-call group is messaged at the same moment, with the team channel alongside, and one failed message does not hold up the rest. Steps carry their own delays and timeouts, and nothing is dropped if a server dies mid-escalation - the step is picked up again rather than lost.

Delivers on
  • slack live
  • telegram live
  • sms planned
  • phone call planned
The TokayOps escalation policy list. Each policy is drawn as a chain of steps: the edge policy goes Start, DM, then after five minutes a channel; the payments policy goes Start, DM to the on-call group, after two minutes a channel, after five more a DM to the named backup. The TokayOps escalation policy list. Each policy is drawn as a chain of steps: the edge policy goes Start, DM, then after five minutes a channel; the payments policy goes Start, DM to the on-call group, after two minutes a channel, after five more a DM to the named backup.
Escalation policies, one per team
03:16:41 Acknowledged

Take ownership without leaving the chat.

Buttons in Slack, buttons in Telegram. Ack claims the group and stops the escalation at the same moment, so the next step cannot fire after somebody has answered. Resolve closes it. The message is edited in place, so the channel never fills up with copies of itself.

Lifecycle
  1. new
  2. processing
  3. triggered
  4. acknowledged
  5. resolved
  6. closed
The timeline of the HighErrorRate alert group in TokayOps. The group is created, and in the same second three alerts are added to it: HighErrorRate, HighLatency and PodCrashLoop. Two and a half minutes later a final entry reads Alert group acknowledged, by Dana Whitfield. The timeline of the HighErrorRate alert group in TokayOps. The group is created, and in the same second three alerts are added to it: HighErrorRate, HighLatency and PodCrashLoop. Two and a half minutes later a final entry reads Alert group acknowledged, by Dana Whitfield.
The same group's timeline, after the Ack
03:41:02 Resolved

Runs on your machines.

The last alert cleared and the group closed itself. Point TokayOps at a Postgres and the whole control plane runs on machines you own - every alert group, timeline and schedule lives in your database. What leaves is the notification itself, to the chat provider you chose.

Start it 
curl -fLO https://raw.githubusercontent.com/tokayops/tokayops/main/docker-compose.prod.yml
curl -fL -o .env https://raw.githubusercontent.com/tokayops/tokayops/main/.env.example

# In .env: generate ENCRYPTION_KEY, JWT_SECRET and DB_PASSWORD, set
# APP_ENV=production and TOKAY_SELF_URL. Then keep the file - the keys
# have to be the same on every start.
docker compose -f docker-compose.prod.yml up -d

That pulls :latest, the newest stable release. What a version number promises.

  1. start Postgres and one Go binary come up together. Health lands on :9090, the UI on :8080.
  2. create the first admin One CLI call inside the container. The first account on an empty database becomes admin.
  3. connect a channel Add a Slack or Telegram integration, then point the Alertmanager webhook at TokayOps.
one container
A single Go binary. No agent, no sidecar, no message broker to operate.
your postgres
The schema lives in a database you already back up. Your system of record stays in your Postgres; notification payloads are sent only to the providers you configure.
no per-seat bill
Add the whole team. Nothing about the price changes.
FSL-1.1-Apache-2.0
Source available today; every release converts to Apache-2.0 two years after it ships.
tagged releases
Semantic versions on GHCR, 0.1.0 being the current one. :latest follows them; TOKAY_TAG or a digest pins one.
Installation guide Read the source
Blog All posts