← Rohit Raj

The Time I Deleted Production

It was a Tuesday. It's always a Tuesday.

I was "just cleaning up an old database." You already know where this is going.

The command ran. The terminal went quiet. And then Slack lit up like a Christmas tree.

The five stages of DROP TABLE

  1. Denial — "That can't be prod."
  2. Anger — "Who gave me prod access?!"
  3. Bargaining — "Please have a backup, please have a backup."
  4. Depression — staring at a very, very empty table.
  5. Acceptance — restoring from a snapshot taken 40 minutes earlier.

We had a backup. We got lucky.

The lesson wasn't "be more careful." The lesson was: build systems that forgive careless humans, because every human is careless at 4pm on a Tuesday.