The post argues that compressors and LLMs chase the same goal. True compression, unlike minification, feeds on redundancy, and run-length encoding offers a simple example. Modern compressors combine transforms, models that assign probabilities to symbols, and entropy coders such as arithmetic coding, which packs a whole string into one binary fraction. Huffman coding gives each symbol its own codeword. Skewed probabilities shrink output, and Shannon entropy, the average −log2(p), sets the floor for lossless compression. Context sharpens predictions: U after Q costs about 0.001 bits, and an order-1 model cut "TO BE OR NOT TO BE" from 47 to 21 bits. A 2023 DeepMind paper treats language modeling and compression as two views of one problem.
2
u/fagnerbrack 1d ago
Snapshot summary:
The post argues that compressors and LLMs chase the same goal. True compression, unlike minification, feeds on redundancy, and run-length encoding offers a simple example. Modern compressors combine transforms, models that assign probabilities to symbols, and entropy coders such as arithmetic coding, which packs a whole string into one binary fraction. Huffman coding gives each symbol its own codeword. Skewed probabilities shrink output, and Shannon entropy, the average −log2(p), sets the floor for lossless compression. Context sharpens predictions: U after Q costs about 0.001 bits, and an order-1 model cut "TO BE OR NOT TO BE" from 47 to 21 bits. A 2023 DeepMind paper treats language modeling and compression as two views of one problem.
If the summary seems inacurate, just downvote and I'll try to delete the comment eventually 👍
Click here for more info, I read all comments