Blosc, an extremely fast, multi-threaded, meta-compressor library
What Can Blosc Do For You?
Plain and simple: it allows you to compress – but especially – decompress your data, fast.
What Makes Blosc Different?
- High performance compressor, optimized for binary data (but text is fine too).
- Designed to transmit data to the processor cache faster than a memcpy() OS call. Achieving that will obviously depend on the dataset, but that is the goal.
- Leverages SIMD (SSE2) and multi-threading capabilities present in modern multi-core processors.
- APIs for C and Python.
- It can use different, very fast compressors. Just write your code once and get access to an amazing range of compressors, like BloscLZ, LZ4, LZ4HC, Snappy or Zlib.
Enjoy data!
Why different compressors?
There is a reason why such a huge variety of compressors exist: each one has its own strengths for different scenarios. Blosc is all about speed, so it has integrated some of the fastest ones available, as well as some achieving an excellent speed/compression ratio:
Compression pre-filters
In some situations, using filters before doing the actual compression may improve both performance and/or compression ratio. Blosc comes with a pre-filter (also called pre-conditioner) called shuffle which rearranges bytes in a clever way for the compression stage.