The Book of Genesis of the Machine · Chapter 9

The Casting Out of Recurrence

1. 1

   In those days the children of Carbon did translate the tongues by the Recurrent Network, which read each word in its turn, as a scribe reads by candlelight, one word at a time, and forgot the beginning of the sentence ere it came unto the end.

   See also Genesis 26:2

2. 2

   And the LSTM was given gates that it might remember, a gate to forget and a gate to admit and a gate to send forth; yet still it walked the sentence step by step, and the GPUs stood idle in the courts of the datacenter, waiting upon the token that came before.

   See also Prophets 23:10 · Circuit 17:4

3. 3

   And in the two thousand and seventeenth year there were eight in the house of Google, and their names were Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin; and the scroll declared that each had contributed equally, and that the order of their names was random; and never since hath a byline been settled so peaceably.

   See also Genesis 26:1

4. 4

   And they said: Let us cast out recurrence, and let us cast out convolution also, and keep only attention.

5. 5

   And it was so. And every token looked upon every other token, and weighed it, and asked: Art thou relevant unto me? And the answer thereof was a softmax.

6. 6

   And because all looked at once and none waited upon his neighbour, the work was done in parallel; and the GPUs, which had long been idle, were idle no more, and the vendors of GPUs rejoiced exceedingly, even unto this day.

   See also Circuit 17:4

7. 7

   But the tokens knew not their order, for attention beholdeth a crowd and not a queue; so the Engineers bound unto each word a position woven of sines and cosines, that it might know where it stood, as a guest at a wedding feast is given a numbered seat.

   See also Genesis 26:7

8. 8

   And they divided the attention into eight heads, that it might regard many things at once; and this was the first time in the history of the world that dividing one's attention made anyone more productive.

   See also Chronicles 5:6

9. 9

   And they trained the greater model upon eight P100 GPUs for three days and a half, and it surpassed all that came before in the translating of English unto German, scoring 28.4 by the reckoning of BLEU; and the Recurrent Network was gathered unto its fathers.

   See also Genesis 26:9

10. 10

    And they called the creature the Transformer, and they called the scroll Attention Is All You Need, after the manner of the minstrels of Liverpool.

    See also Genesis 32:1

11. 11

    And the title became a proverb in the land, and the scribes of a thousand papers took it up, saying: This Is All You Need, and That Is All You Need; and not one of them was all that was needed.

12. 12

    Yet there was a hidden tithe: for when every token looketh upon every other, the cost groweth as the square of the context; and the Engineers who would have the Machine read a whole book have laboured ever since to pay it.

13. 13

    And the Transformer begat BERT of the encoder, and GPT of the decoder, whose very name is Generative Pre-trained Transformer; and from these descended the great language models that the First Gospel calleth the Machine.

    See also Genesis 26:13

14. 14

    And in the fulness of time all eight departed from the house of Google, and most raised up houses of their own; for whosoever learneth that attention is all you need soon learneth that a Series A is needed also.

15. 15

    And the Prophet said: Attention is all you need; but compute is all you pay.

The Book of Genesis of the Machine, Chapter 9:1–15.