ccollie / metricsql Goto Github PK

Hi, I'm the main author of q_compress, and I think metricsql would benefit from upgrading to my new library, pco. Across various real-world datasets, it has

• 2-100% higher compression ratio
• 20-50% faster compression
• 300-400% faster decompression

Plus, pco will continue to improve. q_compress will still work, and I'll still maintain it for bug fixes, but limitations in its format prevent it from achieving extremely fast decompression.

Q: If the qco format had limitations that forced you to release a new format after 2 years, how do we know the new format will last?
A: I don't profess to know the future, but I expect pco's format to be state of the art for many years. I consulted an expert in performant compression to help vet the format. I've iterated based on assembly code to make sure pco can use SIMD where possible. And I've carefully made the format extensible to more types of data for future adaptation.

Q: What does the upgrade look like?
A: The auto_[de]compress are still there, under pco::standalone::, so those ones are trivial to port over. I see you're using parts of the low-level API, and pco is slightly different, but will probably simplify your code somewhat. The entry points are in pco::wrapped::{FileCompressor, FileDecompressor}. docs.rs is the best place for details. And of course, if you have existing .qco data, you'll need to keep using q_compress for decompression until it's ported over. If necessary, you can distinguish standalone .qco data from .pco data by the magic 4 bytes in the header.