Skip to content
Shakeride

Draft, not published · Updated

Running a cloud gameday for your team without breaking production

A gameday puts a team in front of a failure on purpose, so the first time they meet it is not at three in the morning. Done in production, it needs months of preparation; done in isolated accounts, it can start next week.

Decide what you want to learn

Pick one question before you pick a failure: can the on-call engineer find the right dashboard, does the runbook still match the system, do two teams hand over cleanly. A gameday with no question ends with a list of everything.

Keep it away from production

Give each participant or pair their own account with the failure already in place. Nobody waits on a shared environment, and nobody's mistake spills into a colleague's exercise or a customer's traffic.

Run it against a clock

Announce the time limit and keep to it. Pressure is part of what you are rehearsing, and a fixed end makes the debrief happen while the details are fresh.

The debrief is the deliverable

Close with a short, blameless review.

  • What did people check first, and was it the right first check?
  • Where did the documentation help, and where did it send someone the wrong way?
  • Which one change to tooling, alerting or runbooks comes out of today, and who owns it?

All guides

Practise on a real failure

A live AWS account, already broken, and a clock. Fix it with the console and command line, then read a report of how you worked.