The Book of Job of the Sysadmin · Chapter 2

The Comforters and the Thundering Herd

1. 1

   And on the eighth day the mute was lifted from the bridge, and the Consultant spake first, saying: This was a black swan, an event that none could foresee. Buy of me a second database, and a third, in another region, for what is thrice kept shall not perish.

2. 2

   And the sysadmin answered: Consultant, the second database died in the same second as the first, for both were sealed with the one certificate and both were governed by the one clock. Thou callest it redundancy; I call it a chorus. Three copies of one mistake are not a quorum; they are a unanimous vote to fail.

   See also Chronicles 7:9

3. 3

   Neither was it a swan, black or otherwise. The hour of its death was written into the certificate on the day it was signed, in a field called notAfter, which any man may read with a single command. It was the most foreseeable event in the history of my house; and the reminder went to a man who had departed, for a reminder addressed to a person dieth with his employment. Address it to a role, and it shall outlive every man who held it.

4. 4

   Then answered the Manager, and said: Why didst thou not see this coming? I have searched the slide deck, and the risk was not there, neither in green, nor in yellow, nor in red.

   See also Job 5:9 · Circuit 22:6

5. 5

   And the sysadmin opened the log, and said: The risk was there, but not in thy deck. Every morning for thirty days the proxy wrote, Certificate expires soon, and it wrote it at the level of WARN, into a file that no man opened. A warning that hath no reader is not a warning; it is a diary.

   See also Psalms 16:3

6. 6

   Then answered the Forum Poster, and said: Add the flag dash k unto thy curl, or set verify to False, and the error shall depart from thee. Four thousand have upvoted this answer; it hath the green check, and the date upon it is 2013.

   See also Job 1:9

7. 7

   And the sysadmin answered: Thou hast not mended the lock; thou hast painted a lock upon the door. The check existed for one hour, and this was the hour: to refuse a stranger who saith, I am thy database. The error was the only part of the system that was telling the truth, and thou wouldst silence it. And the answer of 2013 was true in 2013, for the green check marketh the hour it was written, not the hour it is read.

8. 8

   Lo, I have the logs, and they are with me: at 03:14 the certificate expired; at 03:15 the clients began to retry; at 03:16 every client retried at once, and the thundering herd trampled the database, which had done no wrong.

   See also Circuit 14:9 · Circuit 53:10

9. 9

   And why at once? Because each client waited the same sixty seconds, counting from the same failure, and a hundred thousand clocks that agree are but a single hammer. Had each waited a random time, and doubled it with every failure, the herd had come as a stream, and the database had drunk of it and lived.

   See also Circuit 7:11

10. 10

    The config said, Retry three times, and no man asked why three. But three retries at each of four layers, the browser, the gateway, the service and the driver, make as many as two hundred and fifty-six calls for every one request; and the database, which was merely slow, was asked the same question two hundred and fifty-six times, and so became dead.

    See also Psalms 4:6

11. 11

    And when at last the certificate was renewed, the cause was gone, yet the database lay down until dawn. For the failure had learned to feed itself: the retries kept the database dead, and its being dead begat more retries. The trigger was a certificate; the cause was a loop. And the sysadmin healed it not by repair but by refusal: he shut the gate, let the database wake with empty hands, and admitted the clients ten in a hundred at a time.

    See also Circuit 53:11

12. 12

    But ye are forgers of postmortems; ye are all physicians of no value. Your blameless retrospective hath blamed me in seven bullet points and an action item. Ye asked why, and ye rested at the man, for a man is a soft place whereon the asking may lie down. Ask the fifth why, and it is a process that gave one man a date to remember and no machine to remember it. And this is the wisdom of the comforters: when the server falleth, every man who never carried the pager knoweth exactly where it should have stood.

    See also Deployer 5:8

13. 13

    And the Auditor, standing among the clouds with his clipboard, asked: Was this change approved by the Change Advisory Board? And the sysadmin answered: It was no change, for no hand made it. Time made it. The passing of 03:14 is a deploy that no board approveth and no man can roll back; every certificate, every token, every licence is a change written in advance by the past and merged by the clock.

    See also Job 1:2

14. 14

    Therefore the sysadmin set the certificates to live thirty days and a machine to renew them each twentieth, so that the rite might be done so often that no man need remember it. For a deadline that only a man remembereth hath already passed; and the thing done rarely is done by prayer, but the thing done daily is done by cron.

The Book of Job of the Sysadmin, Chapter 2:1–14.