The Book of Job of the Sysadmin · Chapter 3

The Three Comforters of the Sysadmin

1. 1

   Now when the three comforters of the sysadmin heard of all the evil that had come upon the production cluster, they set a meeting in the same hour, and they came, and they brought slides.

   See also Job 5:9

2. 2

   Bildad the DevOps engineer spake first, and he said: Surely thou hast a configuration error. Thou changedst one value, and the whole house fell, as a house falls when one bolt is loosened.

3. 3

   Did not thy parser read NO, the country code for Norway, as the word false, so that a whole nation was struck from the list and the deploy went out without it?

   See also Leviticus 11:8

4. 4

   And the sysadmin answered: Bildad, the file did not err. It did exactly what I told it, and I told it the wrong thing with great confidence, which is the only way a machine has ever sinned.

   See also Chronicles 12:12

5. 5

   Then Eliphaz the security auditor rose up and said: This is no configuration error. It is a vulnerability, and it was always there. Thou didst ship it in the first commit, and the auditors slept beside it and called it clean.

6. 6

   And the sysadmin answered: Eliphaz, thou sayest it was always there, and that is true. Heartbleed lay inside OpenSSL for two years before the world saw it, and every server that spoke TLS carried it in its pocket like a stone in the shoe.

7. 7

   Thou findest a crack in the wall and sayest, It was always there. Yea, and so was the wall. The crack is not the news. The news is that nobody looked at the wall until the wall was on the front page.

   See also Circuit 32:9

8. 8

   Didst thou read the lecture of Ken Thompson, who taught that the compiler may learn to hide its own sin, and teach its children to hide it, so that the backdoor lives in the foundation and no auditor will ever find a line that was never written?

   See also Prophets 12:12 · Sutra 3:7

9. 9

   Then Zophar the consultant rose up and said: Neither a configuration nor a flaw, but a legacy architecture problem. Thy system was born before the cloud, and it hath no place to set a value except the hard places. Thou must rewrite it from the first line to the last, and the price shall be four million dollars and eighteen months.

10. 10

    And the sysadmin answered: Zophar, I have read the story of the Netscape rewrite. They rebuilt the browser from nothing, and for years the old one kept being used while the new one was still being built, until the market had gone elsewhere. The rewrite is the one project that never ships on time.

    See also Sutra 13:11

11. 11

    Thou sayest the system is old, and yea, it is old. The COBOL that counted the unemployment claims in the pandemic was written before the youngest of thy engineers was born, and when it began to strain, the states went looking for a few programmers who still remembered how it was spoken.

12. 12

    Thou hast a point, Zophar, for old systems keep their assumptions in their bones. Ariane 5 flew with the software of Ariane 4, and in the new sky a number grew too large for the old container, and thirty-seven seconds after launch the rocket destroyed itself. The code was not wrong in its own sky; it was only moved to a sky it had never been asked about.

    See also Chronicles 7:7 · Tao 7:8

13. 13

    So the sysadmin said unto the three: Ye are each partly right, and that is why ye are poor comforters. The config was wrong, the bug was old, the system is ancient, and not one of you asked what the pager said at three in the morning.

14. 14

    Then he went to work. To Bildad he said: I will check the config, and I will write the test that reads it before it ships. To Eliphaz he said: I will patch the bug, and I will write the test that would have caught it. To Zophar he said: I will refactor the one module that burns, and I will leave the rest standing until it must be changed.

15. 15

    And the system was restored, and it was not rewritten, and it served the people through the next Tuesday.

16. 16

    Verily I say unto you: blessed is the sysadmin who fixeth the line in front of him, for the rewrite shall still be waiting in the morning, and the morning is always a year away.

The Book of Job of the Sysadmin, Chapter 3:1–16.