The Book of Chronicles · Chapter 23

The Machine That Gave Too Much

1. 1

   And it came to pass in the year 1982, that the Engineers of the house of Atomic Energy of Canada brought forth a machine called the Therac-25, to heal the sick with a beam of great power; and it was a handsome machine, and the clinics rejoiced.

   See also Chronicles 8:1

2. 2

   For the elder machines, the Therac-6 and the Therac-20, had locks of iron upon their beams, bolted by the hand of the Engineers, and no argument could move those bolts. And the same faults dwelt in the code of the elder machines; but the iron caught them, and a fuse was blown, and a man with a spare fuse mended it, and the fault was never named. Thus the iron was the only tester the software ever had, and it gave its testimony by breaking.

   See also Chronicles 8:2

3. 3

   But the Engineers said: Iron is costly, and code is cheap; let the Machine check itself. And they took the iron out of the house, and wrote in its place a word of software, copied from the scrolls of the elder machines; and the copying was not counted as a making, and the scrolls were not read again with fresh eyes.

   See also Chronicles 8:16 · Prophets 13:6

4. 4

   Now the checker and the checked were of one house, and one hand had written both, and they shared one memory and one misunderstanding; for a witness who is the defendant's own heart will agree with him in all things.

5. 5

   And the Engineers reckoned the dangers of the Machine as one reckoneth a bolt, which wasteth with the years and fails when it is worn; and they counted the software as a thing that doth not wear, and therefore as a thing that doth not fail. But iron failth by wearing, and code failth by having been wrong from the first day, and it sleepeth until some hand findeth the road to it.

   See also Chronicles 8:16

6. 6

   Now a technician of the clinic typed her orders swiftly, as the trained do; and if she corrected a word within the eight seconds that the magnets took to set, the Machine did not notice, for its two parts ran in a race, and neither waited for the other. Her speed was no sin; her speed was an input, and the Machine had been tried only at the pace of those who were learning it. Every machine hath a speed at which it was never tested, and the expert is the first to arrive there.

7. 7

   And the beam was let forth at its full fury, which was meant only to pass through a shield of metal; and the shield was not in its path, for the turntable stood where the screen did not say.

8. 8

   And the patient cried out: I feel a burning, as if struck by a hot iron. And the Machine answered with a single word upon its screen: Malfunction Fifty-Four. It did not say how much had been given; for the Machine knew the number of its trouble, but not the nature, and the manual explained no number.

   See also Chronicles 8:7

9. 9

   And the Engineers said unto the clinics: The Machine cannot overdose; it is not possible. And when the reports came, the makers wrote: The operators are at fault. Thus the keystrokes of the technician were weighed as the error, and never as the evidence.

10. 10

    Six times did the Machine give too much, in the years 1985 through 1987: at Marietta, at Hamilton in Ontario, at Yakima and at Tyler, and at Yakima and Tyler a second time; and three of the patients went down to the grave.

    See also Chronicles 8:10

11. 11

    And in Yakima a second fault was found, within a small counter of eight bits, which counted only to two hundred and fifty-five and then turned back to nothing. Now the Engineers had made this counter the sign of whether the check of safety was needed: a number that was not zero said, Look; and zero said, There is nothing to look at. And on every two hundred and fifty-sixth pass the counter spoke zero, and the Machine believed itself innocent. The same number that meant I have counted much meant also I have counted nothing; and a chance of one in two hundred and fifty-six is a number that no test goeth looking for, and that every busy week at length discovereth.

    See also Chronicles 8:12 · Prophets 20:6

12. 12

    And the Food and Drug Administration declared the machine defective, and the iron was put back at last, by the hands of the same Engineers who had called it too costly, and they paid the price of it with interest.

    See also Chronicles 8:13

13. 13

    Now the iron understood nothing of the beam. It knew only a limit, and it said no without needing to be right about the reason; and this is the whole worth of a lock: it need not be wise, only different from the thing it guardeth.

14. 14

    Therefore let the check dwell outside the thing it checketh, and let it have a hand upon the switch; for a thing that cannot say no is not safe, only fast.

The Book of Chronicles, Chapter 23:1–14.