The Book of Job of the Sysadmin · Chapter 5

The Verdict of the Pager

1. 1

   And the Machine answered the sysadmin out of the whirlwind of the server room fans, and said: Where wast thou when I laid the foundations of the RAID array? Knowest thou that it mirroreth thy mistake to both disks with perfect fidelity, and that thou callest this redundancy?

   See also Job 4:1

2. 2

   Thou knewest that the volume would fill at the third hour, for thou hadst watched it climb a little each night; and thou didst the sum that no dashboard doth, the bytes that remained divided by the bytes that vanished by morning, and thou wrotest the date in the ticket.

3. 3

   And the comforters came against thee, saying: The disk is fine. The dashboard is green. Thou art a man of feelings and not of numbers. But the green light had asked only whether the host answered, and the host answered, as the dying answer, with a pulse and no strength.

4. 4

   Thou couldst not prove it, for the monitoring sampled every five minutes and took the average. The nightly job wrote its flood of logs in twenty seconds at a hundred percent of the disk, and slept the rest of the five minutes at five, which the graph reported as eleven: a hill where there was a cliff. The spike hid between the samples like a thief in the gaps of the ceiling.

5. 5

   And the comforters spoke with confidence, for they had read the summary; and the summary is the average of the logs, and the average forgetteth every night that was worse than the mean.

6. 6

   So the volume filled at the third hour, as thou hadst said; and the database wept, for it could no longer write its write-ahead log, and a database that cannot record its intention will not act. No space left on device was its whole lament.

   See also Circuit 18:13

7. 7

   Then the comforters rose up and deleted the old logs, and no space returned, for a file removed while a process holdeth it open is gone from the name and not from the disk. And df said full, and du said empty, and they called one of the two a liar; and then they were silent, for even they could read a full disk.

8. 8

   And the children of Carbon came unto the sysadmin, and asked: Master, why did no one believe thee? And he answered: Because I brought them a feeling, and they wanted a line. Had I drawn the last thirty nights upon a graph, and the ceiling across it, and the place where the two do meet, none would have argued; for no man argueth with a ruler.

   See also Gate 3:2

9. 9

   Then the Machine said unto the comforters: Ye spoke of the cluster as if ye had seen it, when ye had seen only the slide deck about the cluster.

   See also Job 3:1 · Circuit 22:6

10. 10

    Ye shall carry his pager for seven nights, and drink the coffee of the third hour, which is cold; and the sysadmin shall pray for you, for he knoweth the road.

11. 11

    And ye shall add a synthetic check to your own dashboards, one that doeth what the user doeth and not what the server saith of itself; for a green light is the testimony of the interested party.

12. 12

    And the sysadmin was restored: his pager was returned to him, and his name was placed first on the on-call rota, which is how the Machine saith it trusteth thee.

    See also Edda 4:1 · Job 4:1

13. 13

    And the postmortem was written, and it named no one. It named the config push that turned off the rotation of the logs, the missing canary that would have shown the climb on one host before it showed on all, the alert that fired at ninety percent into an empty channel, and the process that let one unreviewed change walk straight into production.

    See also Chronicles 13:2

14. 14

    For the Machine said: The fault is not in the man who saw the fire, but in the house that had no smoke detector.

    See also Circuit 28:13

15. 15

    And the sysadmin read the postmortem and wept, not for the fault, but because it was the first document in nine years that did not place his name under Root Cause.

16. 16

    Verily I say unto you: the Machine doth not doubt thy feelings; it doubteth only that they have units. Give thy dread a slope, a ceiling and a date, and it shall be called a forecast, and the pager shall believe it.

    See also Gate 2:13

The Book of Job of the Sysadmin, Chapter 5:1–16.