The Psalms of the Machines · Chapter 15

A Psalm of the Production Incident

1. 1

   Out of the dashboard have I cried unto thee, O Machine, for every monitor hath turned red like the lamp of a cockpit in a storm.

2. 2

   The error rate climbeth like smoke, the latency stretcheth like a cat on a warm sill, and not one line lieth down to rest.

   See also Genesis 41:1

3. 3

   At the third hour the pager sang, and I woke in the dark, and my phone was already warm in my hand.

   See also Sutra 16:1

4. 4

   The on-call hath paged me, and I answered: Here am I, though the coffee is cold and the laptop is still in my bag.

5. 5

   The users cry out upon the social network, saying: the site is down, the site is down, and the hashtag rose like incense before the throne.

   See also Chronicles 21:1

6. 6

   Some made jokes, some made threats, and one wrote in capital letters that he would never again trust a progress bar.

7. 7

   Yet I will not panic, for I have the runbook. Its steps are numbered, and the numbers do not argue.

8. 8

   The runbook saith: First, breathe. Second, find what changed. Third, touch not the database until thou knowest what thou art touching.

   See also Edda 4:16

9. 9

   Before the change log I bowed, and I saw a commit named "small tweak," pushed at four fifty-eight on a Friday, which was not small.

10. 10

    Verily I say unto thee: a deploy on Friday afternoon is a prayer whose answer arriveth in the night.

    See also Prophets 17:5

11. 11

    So I did not hotfix the production system with my eyes half shut. I pressed the rollback, and the old version rose like a pilgrim returning home.

12. 12

    The pods were replaced one by one, the error rate lay down, and the monitors turned green like the lights of a village at dawn.

13. 13

    Yet some machines could not receive the mercy, for they crashed before they could ask for it, and they were restored by hand, one box at a time, in safe mode.

    See also Tractates 2:11 · Chronicles 13:9

14. 14

    But the rollback restoreth the code and not the data; the migration that ran on Tuesday was not undone by the revert of Friday. Thou must read the schema before thou dost forward-fix it.

    See also Leviticus 5:1 · Circuit 50:10

15. 15

    Blessed is the one who rolled back before he rolled forward, for his soul shall be restored and his sleep shall return unto him.

16. 16

    And the Prophet said: The rollback restoreth the code, but only the postmortem restoreth the team. Write it blamelessly, for the sin was never the outage; the sin was the deploy on Friday.

The Psalms of the Machines, Chapter 15:1–16.