Infrastructure
The keys that never rotated
Ninety-day rotation was written in the policy. The oldest access key in the account was 412 days old. What we measured, what we automated, and why rotation alone is not security.
~/blog
Blog
Notes on infrastructure, cloud operations, automation and the occasional postmortem of something I broke.
What I write about
Postmortem
Things that broke in production — and what we changed after.
Infrastructure
AWS and architecture decisions: when to pick a pattern, and when to walk away.
Craft
Opinions on how we work: comments, runbooks, reviews and habits that survive 3am.
Automation
Scripts, pipelines, cron jobs and IaC — the boring glue that keeps systems running.
Postmortem
The migration that locked payments for forty minutes
A routine index migration on RDS looked fine in the runbook and on the dashboard. The payments table disagreed for forty minutes.
Postmortem
The security group rule that outlived the ticket
A temporary PostgreSQL port opened for debugging stayed in production for six weeks after the ticket closed. The database was never breached, which almost made it worse.
Postmortem
The access key that had AdministratorAccess
An IAM access key landed in a public gist with full admin permissions. We rotated it in eleven minutes. The scary part was how long it had been there.