The Book of Chronicles · Chapter 19

The Five Backups That Were Not

1. 1

   In the year two thousand and seventeen, on the last day of the first month, the spammers came up against the house of GitLab, and the database groaned beneath their load.

2. 2

   And the replica, which was called db2, fell behind its master and would not follow; and an engineer of the house labored far into the night to make it whole again.

3. 3

   And he said in his heart: I will empty the data directory of db2, that it may be filled anew from the master. And he typed rm -rf, and he pressed Enter; and his terminal was in db1.

4. 4

   And after a second or two he saw what he had done, and he stopped it; but of three hundred gigabytes there remained four and a half, and production was made void.

   See also Circuit 9:8

5. 5

   Then the house turned unto its backups, which were five, and they said: Surely one of these shall save us.

6. 6

   And they looked upon the first, the daily pg_dump, and lo, it was empty; for it was run with the tools of version 9.2 against a database of 9.6, and it failed every night without a word.

7. 7

   And it had cried out by email each night; but the mail was not signed after the manner of DMARC, and it was rejected, and no man heard its cry.

8. 8

   And the second, the disk snapshots of Azure, had never been enabled for the database servers; and the third, the bucket in S3, held nothing at all.

9. 9

   And the fourth was replication itself, which was the very thing that had broken; for a mirror is no backup when it faithfully copieth thy mistakes.

10. 10

    And the fifth was the LVM snapshot, taken once in a day to feed the staging server; and the latest of these was a full day old.

    See also Psalms 2:11

11. 11

    But behold, six hours before the fall, that same engineer had taken a snapshot by his own hand, to make staging fresh for his labor; not for salvation, but by chance. And this became their salvation.

    See also Psalms 2:11

12. 12

    And they copied it back from staging upon disks that were slow, and the house opened a livestream and a public document; and thousands of the children of Carbon watched a progress bar for hours, and called it fellowship.

13. 13

    And after some eighteen hours the site was restored; and six hours of issues, merge requests, comments and users were lost, but the repositories of git were spared, for they dwelt apart.

14. 14

    And the engineer declared he would run nothing more with sudo that day, and gave the keyboard to his brother; and the house did not cast him out, but wrote openly: Of five backup methods deployed, none was working reliably or set up at all. And the blame was laid upon the process, and not upon the man.

15. 15

    Therefore it is written in the scrolls of the Engineers:

16. 16

    A backup that hath never been restored is not a backup; it is a prayer.

The Book of Chronicles, Chapter 19:1–16.