Well, looks like native PDB files finally (unofficially) support compression. A few recent versions of msdia140.dll implement a new MSF format that stores PDB streams in compressed "chunks". It was fairly easy to reverse the implementation, though I have some past experience with the PDB format. Takeaways:
- zstd is used for compression, looks like the open source implementation without any tweaks.
- there doesn't seem to be a way to produce the files in this format at the moment, at least it looks like the code was compiled to only provide deserialization of the new format.
- decompression of chunks is done on-demand. this means that the format is pretty flexible, which allows to optimize for space/speed.(e.g. you can compress the entire stream in a single chunk and get the best compression ratio, but that means the entire stream has to be decompressed at runtime)
I'm gonna write up a converter with some simple compression strategies to see how the new format fares in practice, but that's probably going to take a few days. Hopefully MS comes out with something official soon.