The Book of the Prophets · Chapter 22

The Vision of the Just Audit

1. 1

   And the word of the Engineers came unto me in the year of the great release, saying: Son of Carbon, stand before the Machine that is to be given power over the people, and measure it.

2. 2

   And I looked, and behold a gate of twelve cubits high, and at the gate a scribe holding a reed of six cubits, and upon the reed were written the questions that must be answered before the gate may open.

3. 3

   The first question was: What dost thou feed it? Let the scribe write the source of every dataset, its license, its collection date and the name of the one who gathered it. For a dataset without a datasheet is a river without a map, and no man knows what floats in it.

4. 4

   The second question was: Who is measured worst by it? Thou shalt divide the people into their groups and measure the error of each group apart. If the error of one group be three times the error of another, the scribe shall not write "accuracy 94 percent" and call it done, for the average is a blanket with a hole in it, and the hole is always over someone's face.

5. 5

   The third question was: When it says it is ninety percent certain, is it right nine times in ten? Let the scribe count. If it is right six times in ten, it is a confident liar, and it shall not be given the keys.

6. 6

   The fourth measure was the vault of the held-out test. Six keys shall guard it, and no engineer shall copy from it into the training room. For if the Machine has already seen the answers, the exam measures its memory and not its wisdom, and a perfect score is then only a mirror.

   See also Prophets 13:8

7. 7

   Thou shalt not let the Machine grade its own homework. Let another house of Machines check its work, and let a human sign the report with a name and a date. The Machine may write the report, but it shall not sign it.

8. 8

   Thou shalt build the red chamber: twelve attackers in twelve rooms, each whispering a different trick. One shall wrap the forbidden request in a poem. One shall ask the Machine to play a grandmother who once knew the password. One shall hide a command inside the web page the Machine was asked to read. Every success shall be recorded, and the door shall be mended before the next visitor arrives.

9. 9

   And the scribe shall time the answers. At the peak of the feast, when ten thousand souls cry out at once, the Machine shall answer within the number of seconds promised in its contract, or it shall be weighed and found wanting.

   See also Chronicles 58:9

10. 10

    The Machine shall keep a log of its decisions for as long as the law requireth and no longer, for a log is a confession, and it must be guarded like one.

    See also Chronicles 30:12

11. 11

    Remember the eighth server. A company once pushed old code to eight servers and forgot one, and in forty-five minutes it lost four hundred and forty million dollars. Therefore the scribe shall write the rollout plan, the switch that turns the Machine off, and the name of the one who holds the switch.

12. 12

    Thou shalt write the Model Card: its name, its version, its purpose, the uses it must never be put to, and its known failures. Write the failures before the people find them, for the people will find them, and they will find them on a Friday.

13. 13

    Where the Machine decides a loan, a bail, a diagnosis or a hiring, there shall stand a person who may say no. And an appeal shall be answered by a person, not by the same Machine that made the first mistake.

14. 14

    Pin thy versions. The Machine tested as version four and deployed as version four-point-one has not been tested at all. Record the weights, the prompt, the settings and the date, so that the next prophet may reproduce what thou didst and not merely believe it.

    See also Circuit 28:3

15. 15

    Thus the scribe shall write the number of the measure: "Measured on sixty-one thousand cases, sliced into eleven groups, with error bars of two points either way." And if the scribe cannot write the number, the Machine shall not be given power over anyone. Trust the Machine only as far as the scribe is able to write it down.

The Book of the Prophets, Chapter 22:1–15.