🐤 Canary Deployment

Step 1: Deploy canary (new version)
  Stable version → most traffic
  Canary version → small slice of traffic

Step 2: Monitor and evaluate
  Error rates? Response times? User feedback?

Step 3a: If healthy, increase traffic
  Stable → most → some → little → none
  Canary → small → larger → most → full rollout

Step 3b: If problems, rollback
  Route traffic back to stable (canary removed)
  A small slice of users was affected.

Time    v1 (old)    v2 (canary)    Action
─────   ─────────   ────────────   ─────────────
Start   Most        None           Deploy canary
Soon    Most        Small slice    Initial traffic
Later   Most        Larger slice   Metrics look good
Later   Some        Some           Split traffic
Later   Little      Most           Almost complete
End     None        Full rollout   Full rollout done

Regular deployment:
  Bug can affect lots of users at once

Canary deployment:
  Bug affects a small slice of users
  Detected and rolled back
  Many users may not see the bug

Staging ≠ Production

Canary lets you test with:
  - Real traffic patterns
  - Real data
  - Real scale
  - Real user behavior

Not "does it work in staging?"
But "does it work for real users?"

Metrics guide the rollout decision.

Small slice to canary, most traffic to stable

Simple and common.
Any user might hit either version.

Specific users get canary:
  - Internal employees first
  - Beta testers
  - Users who opted in

Same user usually gets the same version (sticky routing).

Canary in one region first:
  Australia → canary
  Most other users → stable

If Australia is fine, expand.

Small slice of requests from internal users in one region
→ Very narrow canary for initial testing

Error Metrics:
  - 5xx error rate
  - Exception count
  - Failed requests

Performance Metrics:
  - Response time (p50, p95, p99)
  - Throughput
  - CPU/Memory usage

Business Metrics:
  - Conversion rate
  - Revenue per user
  - Engagement metrics

Compare canary vs stable:

Metric          Stable    Canary    Status
─────────────   ─────     ─────     ─────
Error rate      Low       Higher    ⚠️ Warning
p99 latency     Fast      Similar   ✓ Acceptable
CPU usage       Normal    Normal    ✓ Acceptable

Canary error rate noticeably higher → Investigate!

Automated system:
  1. Deploy canary
  2. Collect metrics for a short window
  3. Compare to baseline
  4. If healthy → increase traffic
  5. If unhealthy → automatic rollback
  6. Repeat until full rollout

No human intervention for normal cases.

Flagger: Kubernetes canary automation
Argo Rollouts: GitOps progressive delivery
Spinnaker: Multi-cloud canary support
LaunchDarkly: Feature flags + metrics

✓ New features with risk
✓ Performance-critical changes
✓ Major version upgrades
✓ When you need data before committing

✗ Bug fixes (probably just roll out)
✗ Emergency patches (need speed)
✗ Low-risk changes
✗ Very small user base

Tiny slice → small slice → medium slice → most users → full rollout

Try not to jump straight from “tiny” to “all users.”
Gradual increase helps catch issues.

If your canary group is too small, you might not see problems reliably.
If your canary group is large enough, the signals are easier to trust.

Aim for a sample size that gives you meaningful data.

Sticky sessions:
  Same user → typically the same version

Avoid: User refreshes, gets different version

Define BEFORE deploying:
  - Error rate stays close to the stable baseline
  - Tail latency stays close to the stable baseline
  - No increase in support tickets

Clear criteria enable automation.

Just a little data → looks fine → full rollout
→ Problem appears later under real load

Give enough time to surface issues.

CPU usage fine, but users can't checkout!

Monitor business metrics, not just infra.

Canary deploy happens overnight, most people are asleep.
Errors spike, no response.

Automate rollback on metric thresholds.

User switches between v1 and v2.
State inconsistencies.

Ensure sticky routing!