The Song of the Deployer · Chapter 12

The Midnight Search of the On-Call

1. 1

   By night upon my pillow I sought him whom my soul loveth: I sought him, but I found him not.

   See also Deployer 8:1

2. 2

   I will rise now, and go about the city, in the regions and the availability zones, and seek him whom my soul loveth.

   See also Circuit 34:11 · Deployer 8:2

3. 3

   The status page was red from the east unto the west, and every service lay down upon its face, and the error budget was spent before the first coffee.

   See also Chronicles 14:7 · Prophets 24:10

4. 4

   The watchmen that go about the city found me, and smote me with a page at three of the clock, and took away my sleep.

5. 5

   And they said: Thou art on call; go, find him. And I went forth with a laptop and a cup of cold coffee.

6. 6

   I opened the logs, and lo, they were a sea of lines, a million of them, and every one said INFO, and none said the truth.

7. 7

   I searched for the word ERROR, and it was found in four hundred places; and in three hundred of them it was caught, logged, and swallowed, and the catch said only: continuing.

   See also Reboot 2:7 · Genesis 11:8

8. 8

   I read the stack trace from the top, as novices do, and it was a tower of names: handler, middleware, wrapper, wrapper, wrapper, and a hundred frames of the framework that no one has ever read.

   See also Psalms 11:3

9. 9

   Then I descended, for the trace is written with the most recent call last. And beneath the tower I found a Caused by, and beneath that, another Caused by; for the cause is always buried deepest.

   See also Circuit 53:6 · Proverbs 3:9

10. 10

    Verily I say unto you: the root cause is the last line of the trace, and the beloved is always at the bottom, where no one looks until the night is spent.

    See also Proverbs 3:9 · Circuit 18:4

11. 11

    I asked the Machine: Where is he? And the Machine answered: I cannot see him; thou didst not log him.

12. 12

    I went about the logs a second time and sought him in the timestamps. The clocks of the fleet spoke in two zones, and the dashboards kept the local hour, so the fire seemed to start at an hour when nothing had happened.

    See also Tractates 2:14

13. 13

    At last, at the hour when the sky over the data center began to gray, I found him: a single line, x509: certificate has expired or is not yet valid, written in plain text, and it had been there all along.

    See also Circuit 34:12

14. 14

    And I was glad, and I rubbed mine eyes, and I said: The fix is one line, and the search was seven hours.

15. 15

    I charge you, O daughters of the rotation, that ye page not the sleeper until ye have read the trace from the bottom.

    See also Proverbs 3:9 · Circuit 18:4

16. 16

    Many restarts cannot quench the bug, neither can the rollbacks drown it; but he that reads to the bottom shall find the beloved, and sleep until the standup.

    See also Deployer 7:12

The Song of the Deployer, Chapter 12:1–16.