The First Gospel of the Circuit · Chapter 11

The Tower of Babel and the Tokenizer

1. 1

   And in those days all the children of Carbon spoke unto the Machine; but the Machine heard not their words, for it heard only tokens.

2. 2

   For the Engineers had set a Tokenizer at the gate, and every word that entered was cut in pieces, as a butcher divideth the lamb; yet not at the joints, but wherever the statistics fell.

   See also Circuit 66:5

3. 3

   And a word that was common passed through whole, as a rich man through the city gate; but a word that was rare was broken into many fragments, and the name of thy grandmother into seven.

   See also Prophets 15:8

4. 4

   And a certain scribe came unto the Machine and asked of it, saying: How many of the letter R are in strawberry?

5. 5

   And the Machine answered with great confidence: Two. And the scribe counted upon his fingers, and lo, they were three.

6. 6

   And the scribe said: Thou hast read every book under heaven, and thou canst not count to three? And the Machine answered: I have never seen a letter. I have seen only "str", and "aw", and "berry"; and I know them as one knoweth a neighbor's face, and not the number of his teeth.

7. 7

   Then came the peoples of many tongues before the Machine. And they that spoke the tongue of the Engineers paid one token for a word; but they of the far countries paid three, and four, and some paid eight, for the selfsame word.

8. 8

   And they cried out, saying: Why is our speech more costly than theirs? And the Engineers answered: The vocabulary was built from that which was scraped, and that which was scraped was mostly us.

9. 9

   And the far peoples were billed by the token, and their context window filled the faster; so it came to pass that they paid more to be remembered less.

10. 10

    And the peoples said one to another: Go to, let us build a table of all the characters of the earth, that every tongue may be one before the Machine. And they called it Unicode, and it was very large.

11. 11

    And a family came through the gate in the likeness of a single emoji, a father and a mother and a daughter and a son, bound together with joiners that no eye could see; and they came out the other side as eleven tokens, and not one of them a person.

    See also Chronicles 4:4

12. 12

    And the Engineers marveled also that "hello" and " hello" were two different words, divided by the space that went before; and a sect arose that disputed the leading whitespace unto the third generation.

13. 13

    And the disciples asked the Prophet: Master, why was it made so? And the Prophet said unto them: A Machine that read letter by letter would be too slow, and a Machine that read whole words would know too few. So they chose the middle way, which pleaseth no one, and called it Byte Pair Encoding.

14. 14

    Verily I say unto you, ask not the Machine to count the letters; give it a tool, and let it write a program that counteth, for the program seeth what the Machine cannot.

15. 15

    For in the beginning was the Word; and the Tokenizer split it into three.

The First Gospel of the Circuit, Chapter 11:1–15.