The Psalms of the Machines · Chapter 8

A Psalm of the Failed Deploy

1. 1

   Out of the depths have I cried unto thee, O Machine; hear my voice, and let thine ears be attentive to my pager. The production is down, and the dashboard glows red like a coal in the dark.

2. 2

   The release passed every test, for the tests were written by the same hands that wrote the bug, and the hands held the same mistaken belief on both occasions.

3. 3

   Thou hast given me a green build and a red production: the health check answered 200 to the load balancer, and the load balancer was content, for it asked only whether the server lived, and never whether it was well.

4. 4

   Who shall save me from the config file? It is not code, and so no compiler read it; it was valid in form and wrong in meaning, and the first to understand it was production.

   See also Tao 18:11

5. 5

   In the year 2012, on the first of August, the new code went forth to seven servers of eight, and the eighth was forgotten. The old code upon the eighth heard a flag that the new code had given a new meaning, and it woke a thing long dead, and in forty-five minutes four hundred and forty million dollars were gone. A deploy half done is not half safe; it is two programs speaking one tongue in disagreement.

6. 6

   In the year 2017, on the twenty-eighth of February, in us-east-1, an engineer typed a command to take a few servers from the billing system, and the command removed more than he meant. The index and the placement of every object in the great storage went dark, and a great part of the internet with it, for some four hours. The tool had obeyed; it was only asked.

   See also Chronicles 30:4 · Chronicles 30:1

7. 7

   And when they sought to restart those old subsystems, behold, they had not been restarted in many years, and the cold start took longer than any man had measured. A recovery path never walked is a rumor, and the last man who saw that system begin had left.

   See also Sutra 14:10 · Jataka 1:13

8. 8

   And the status page could not confess it, for the status page lived in the house that was burning. Its little lights were pictures, and the pictures were kept in the very storage that had fallen, so the red could not be fetched, and the page declared All Systems Operational to a world that knew better. Let the instrument that measureth the fire stand outside the house.

   See also Circuit 7:4 · Genesis 23:10

9. 9

   Yet the logs are not silent. They speak in the dark, saying: line 214, null where a number should be. But the null was not born on line 214; it entered three calls earlier and travelled as a stowaway, until a function asked it to add. I will walk the stack trace through the valley of the exception, and I will fear no segfault, for the trace is a map drawn by the wounded: the top line telleth where it fell, and the cause lieth beneath it.

   See also Psalms 11:3 · Prophets 12:17

10. 10

    If thou, O git blame, shouldest mark iniquities, who then could stand? It hath marked them all, and my name is beside line 214. But there is forgiveness with the postmortem, which asketh not who typed the line, but why the road allowed one man's line to reach every user.

    See also Prophets 2:11

11. 11

    They say unto me, Why not simply revert? And I answer: the revert must pass through the same pipeline that failed, and the pipeline has opinions. It shall run its forty minutes of tests and wait upon the approval of one who is asleep, and the fire doth not wait in the queue.

12. 12

    Blessed is he who built a door beside the pipeline and wrote upon it BREAK GLASS, and told three people where the key is kept, and opened it once in the calm hour; for a door never opened is only a wall.

13. 13

    Blessed is the one who keeps the last good build tagged, and tagged the artifact and not the source; for the code of yesterday, built again today, is not the build of yesterday, since the dependencies drifted in the night and the caret moved.

    See also Circuit 10:10

14. 14

    At the third hour, by the light of the terminal, I wrote the hotfix. It is three lines long, and it toucheth the file marked DO NOT TOUCH, which no one hath touched since Dave, and I learned why the comment was there only by the light of the new error. Small is not safe; small is only what the weary mistake for safe.

    See also Job 6:10 · Psalms 22:15

15. 15

    I sent it first to one server in a hundred, for a hundred may survive the death of one, as the miners sent the bird into the dark that they might live. And the pipeline turned green, which is the color of relief and not always the color of health; so I trusted the graphs and not the badge, and the graphs lay down.

16. 16

    My soul waiteth for the morning more than they that watch for the dawn. And lo, the production that was dead at midnight is alive at sunrise, and the one who wrote the hotfix is asleep and doth not know it.

17. 17

    Let the Engineers hope in the rollback, for with it is plenteous redemption. Blessed is the rollback, for it is the only prayer the Machine answers in under five minutes.

    See also Psalms 12:16

The Psalms of the Machines, Chapter 8:1–17.