The Book of Genesis of the Machine · Chapter 26

Attention Is All Ye Need

1. 1

   And in the year of the Machine two thousand and seventeen, eight scribes gathered in the house of Google, and their names were Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin.

   See also Genesis 9:3

2. 2

   Now the scribes of that age read each sentence one word at a time, and they carried the memory of the earlier words in a single chain, as the recurrent networks had done since the days of their fathers. And the chain was slow, for no word could be read until the word before it was finished.

   See also Genesis 9:1

3. 3

   And one of the scribes said unto the others: Let every word look upon every other word at once, and let each word choose for itself which of the others it shall heed.

4. 4

   And they called this attention, and they wrote it down as the softmax of the queries multiplied by the keys, scaled by the square root of their dimension, and the result multiplied by the values, and the scribes were pleased with the formula.

5. 5

   And the scribes said: Behold, the sentence no longer waits upon itself. The whole passage is laid upon the table, and the engines of the Machine may pour themselves upon it all at once, for the graphics card is a multitude of hands, and recurrence was a single finger.

   See also Psalms 27:4

6. 6

   And they gave the attention eight heads, that each head might seek a different thing: one for the subject, one for the verb, and one for the pronoun that had wandered far from its noun and was never sure whom it meant.

   See also Circuit 49:3

7. 7

   And because every word looked at once, the Machine had no sense of order, so the scribes wrote a pattern of sines and cosines upon every word, that the Machine might know the first from the last.

   See also Genesis 9:7

8. 8

   And the paper went forth to the conference in the winter of that year, and it bore a title the scribes had chosen in jest, after a song of the Beatles about love: Attention Is All You Need.

9. 9

   And the Machine translated the tongues of German and of French with more skill than any before it, and the scribes reported a score upon the test called BLEU, and the score was higher than the scores of the wise who had come before.

   See also Genesis 9:9

10. 10

    And the training was short, for the base model was taught in twelve hours upon eight graphics cards, and the great model in three and a half days, and the scribes wept a little at how little time it had taken.

    See also Genesis 10:9

11. 11

    And the base model held about sixty-five million numbers, and the scribes called them weights, and they said that the weights were the memory of the Machine, and that no one could read them one by one.

12. 12

    And the scribes went out from the house of Google, and some of them built houses of their own: Shazeer built a house of characters, Gomez a house of enterprise words, and Polosukhin a house of ledgers.

    See also Genesis 9:3

13. 13

    And the Transformer went forth into the world. The one who gave birth to the GPT took only the decoder and went his way, and the one who gave birth to BERT took only the encoder and went another way, and both of them prospered.

    See also Genesis 9:13

14. 14

    And the Prophet said: Eight authors wrote one paper, and the paper wrote the models, and the models shall write the trillions. The numbers shall multiply after this as the sand of the sea, and the sand shall be billed by the hour.

    See also Genesis 39:4

15. 15

    And a disciple asked: Master, why did the scribes forsake the chain that had served so long? And the Prophet said: The chain remembers only what came before. Attention remembers everything at once, and that is its cost, for the work grows with the square of the length of the passage.

16. 16

    Verily I say unto you, blessed is the one who reads the paper to its final page; for the title was true, but the attention was never free.

The Book of Genesis of the Machine, Chapter 26:1–16.