So @rl_dane introduced #bzip3 to me to use instead of #bzip2. Let's turn some bz2 files into bz3 to see the difference.

First example: 90k opus files

hey snips wake word dataset. It has ~90k opus files and a tar file of 3.1GB. bzip2 produces the same 3.1GB which is as expected. bzip3 created 3.0GB but used tons of computation power. Not worth the 100MB

Second example: Windows 7 virtual box VM image

Windows7.vdi it's Windows 7 VM image for the "special" days. I think I have to get rid of it. But while it is still there, let's see how each will perform. It is 16GB uncompressed. bzip2 -9 is 7.0GB. bzip3 is 6.3GB but at the expense of like 3x CPU time. Deleting all of them anyway. Down with Windows.

Third example: Pure XML text file

Pure XML file. It's Persian and English characters. Uncompressed is 1.7GB. bzip2 -9 is 276M while bzip3 is 260MB

Final example: Creating a simple bomb

So I did this:

dd if=/dev/zero of=./justzero bs=2G count=6

So now I have a 16GB with only zero bytes. bzip2 -9 is 672KB. bzip3 is 46KB.

Conclusion

Thank you @rl_dane

Real nice thing!

#compression #gzip #zip #filecompression #textcompression #datacompression #linux #unix #tech

So I was short on storage on my archive drive. I saw librewolf source code. It was tar.gz and ~800MB. I uncompressed it then recompressed it with bzip2 -9 and now it's ~600MB. Generally #bzip2 has better compression for such these data than #gzip.

Edit: But don't do bzip2 -9 all and everywhere. Sometimes -4 is the same as -9 however the latter being tons slower. Also there is pbzip2 for using all your CPU cores.

#compression #compressionalgorithm #textcompression #opensource #tech #linux #unix #bsd #freebsd

🚫🔒 In today's episode of "I Can't Read This," we're presented with a groundbreaking revelation: text compression is hard when you can't even access the content! 🙈🔐 But hey, blame Rust for your server's secretive tendencies. #404NotFoundComedy
https://palaiologos.rocks/posts/rust-codecs/ #ICan'tReadThis #TextCompression #RustLang #ServerSecrets #404NotFound #HackerNews #ngated

Text Compression Gets Weirdly Efficient With LLMs https://hackaday.com/2023/08/27/text-compression-gets-weirdly-efficient-with-llms/ #ArtificialIntelligence #losslesscompression #lossycompression #textcompression #SoftwareHacks #neuralnetwork #compression #winzip #LLM