How canaries work

A canary is a live A/B test of one patch. It runs on your real traffic, splits sessions into two arms, measures the fault's own metric plus a set of always-on guardrails, and reaches a verdict. The design goals are that it concludes as early as the data allows, that it never degrades performance without rolling back immediately, and that every session sees a stable manifest for its whole lifetime.

Session bucketing and arms

A canary has two arms:

• Baseline: your original manifest, served untouched.
• Candidate: the patch applied to the tools/list response.

Each session is assigned to an arm deterministically, with no coordination between the collector and the backend. The assignment is a hash of the session id and the canary id:

bucket = bigEndianUint64(first 8 bytes of sha256(sessionId + ":" + canaryId)) mod 10000
arm    = candidate  when  bucket < round(candidateShare * 10000)

A candidateShare of 0.5 sends buckets 0 through 4999 to the candidate arm. The same function runs in the TypeScript SDK, the Python SDK, the Go CLI, and the backend analysis, so every side agrees on which arm a session was in without exchanging anything.

The arm is computed once, at a session's first tools/list, and pinned for the session's lifetime along with the delivery-state snapshot it was computed from. A session never sees the manifest change mid-conversation.

Primary metric per fault type

Each fault type carries the one metric a canary judges the patch on. The metric's direction rides with the measurement, so the test never guesses whether higher or lower is better.

The primary metric decides whether the candidate wins. The gates decide whether it is allowed to keep running at all.

Gates

Gates are the always-on performance promise. They are separate from the primary-metric verdict: the sequential test decides win or lose on the fault's own metric, while gates are instant guardrails against collateral damage. A breach on any gate rolls the canary back immediately, whatever the primary metric is doing.

There are seven gates. Each compares the candidate arm against the baseline arm and fires only when the candidate is credibly worse, so small samples never breach on noise.

Gates need a minimum sample before they judge: at least 30 subject-tool calls per arm for the duration gate, and at least 30 sessions per arm for the session-level gates. Below that a gate reports insufficient data rather than a false pass. Every credible-difference test uses a one-sided 95 percent level.

Thresholds are the defaults. An environment policy can tighten them; a malformed override is dropped rather than applied, so a bad policy row can never widen a gate open by accident.

Sequential testing, in plain language

The primary metric is judged by a sequential test (an mSPRT, a mixture sequential probability ratio test). In plain terms:

• The canary is looked at once per sweep, on the cumulative data so far.
• It concludes as early as the data allows. A clear win or a clear loss is called quickly; a marginal effect runs longer.
• It is anytime-valid. The false-positive guarantee holds no matter how many times the data is peeked at, which is what lets Mend check hourly without inflating error.
• It is two-sided. A credible degradation triggers a revert with the same guarantee that a credible improvement triggers a promote.
• Any gate breach rolls back immediately, independent of the sequential verdict.

The defaults are a 5 percent false-positive rate and a 20 percent minimum relative effect (the smallest change worth acting on, which the test tunes itself to detect fastest). A look needs at least 30 sessions in each arm to count.

Before a canary launches, a feasibility check walks the expected trajectory and estimates how long it would take to conclude. If that lands within about four weeks the canary runs as a standard A/B. If it would take longer, or if there is too little traffic, Mend falls back to before/after mode.

Before/after mode, and why it is weaker

When traffic is too thin for a randomized split to conclude in a reasonable time, Mend runs the patch at 100 percent and compares a window before the patch applied against a window after. It uses Bayesian posteriors on each window and fires when the credible interval excludes zero.

Before/after mode carries less weight on purpose. There is no randomization, so anything that changed with time (a traffic mix shift, a model upgrade on the client side, a seasonal pattern) confounds the comparison. Every verdict from this path is labeled weak evidence, and downstream policy (autopilot, digest, badges in the UI) treats it accordingly. A randomized canary is labeled A/B evidence.

Local runbook

For running Mend against a local environment, the canary sweep is the thing that advances every running canary one step: take a metrics snapshot, evaluate the gates, run the sequential verdict, and act on it.

Inspect a canary

Read a canary's current state, gate board, and latest verdict through the Canaries view in the dashboard, or through the region tRPC procedures mend.canaryDetail and mend.canariesByEnvironment.

Force-stop a stuck canary

Stop a canary immediately with the mend.stopCanary mutation. It takes the canaryId, a short human reason, and an optional target status of rolled_back, concluded_revert, or inconclusive. This is the manual override when a canary is stuck or you want it gone now; it writes to the audit log like any other Mend state change.

Re-run a sweep

The sweep runs hourly on a cron in each region. To trigger it by hand against a local or test region:

# Advance every running canary once and wait for the summary.
curl -X POST 'http://localhost:<api-region-port>/internal/canary-sweep?wait=true' \
  -H 'Authorization: Bearer <internal-auth>'

Without ?wait=true the endpoint schedules the sweep and returns 202 immediately. In code, runCanarySweep({ environmentId }) scopes a sweep to one environment; called with no options it sweeps every running canary in the region.