Codec Details

FAQ

Well, okay, I'm writing this before I publish the URL, so not so much "frequently asked questions" as "things I thought you might be wondering", but if I called this section "TITYMBW" the first thing I would have to answer would be "WTF is 'TITYMBW'?"

Can I be informed of updates?

Sure! This benchmark is updated whenever a new version of Squash is released, so you can just subscribe to the squash-annonce mailing list. We send one message to announce the updated version of both the library and the benchmark.

Will you add «insert compression codec»?

In order to be included the benchmark the software must be supported by Squash. If there is a specific codec you're interested in, please file an issue against Squash.

If the codec is reliable (it has to pass Squash's unit tests), works on Linux, accessible from C or C++, and open source the odds are good I would be willing to write a plugin, or at least merge a pull request.

If the codec doesn't meet all the above critera I still may be willing to accept a pull request for a plugin, but keep in mind that all the machines I run this on are running Linux with various architectures, so even if there is a plugin for a Windows-only, or x86-only, etc., library it probably will not show up for every machine. For example, I would like to add a Windows API plugin, but it will not show up on any of the Linux machines' results. Ditto for OS X.

How are the values calculated?

The benchmark collects the compressed size, compression time, and decompression time. Those are then used to calculate the values used in the benchmark:

Ratio

uncompressed size ÷ compressed size

Compression Speed

uncompressed size ÷ compression time

Decompression Speed

uncompressed size ÷ decompression time

Round Trip Speed

(2 × uncompressed size) ÷ (compression time + decompression time)

Sizes are presented using binary prefixes—1 KiB is 1024 bytes, 1 MiB is 1024 KiB, and so on.

What about memory usage?

I would really like to include that data. If you have a good way to capture that data on the C side (in benchmark.c) I would be very happy to merge it, and integrate it into the results.

Please use quixdb/squash-benchmark#2 to discuss this.

Is time CPU time or wall-clock?

CPU time. Wall-clock data is actually captured but not presented. I'm willing to consider pull requests if you can find a good way to display it, as well as a good reason.

What about multi-threading?

That's a tricky question. For one thing it would explode the number of data points (for example, LZMA would have 4 codecs with 9 levels each, multiplied the number the number of cores—that's 288 different configurations on an 8-core machine per dataset). Obviously this would make generating the benchmark even slower, but the main problem is presenting the data in a way which doesn't destroy the usability for people who are interested in single-threaded compression/decompression.

Can you switch the graphs to use a logarithmic scale?

You can. Just click on the label for either axis and it will toggle between linear and logarithmic. I know this isn't obvious—I'd be happy to merge a PR if you can improve that.

I don't want to switch the default because I think that linear is probably better for most people. Logarithmic tends to be better if all you care about is compression ratio, not speed.

Can you make it easier to compare machines or datasets (instead of codecs)?

I would like to (see quixdb/squash-benchmark-web#2 and quixdb/squash-benchmark-web#3 for discussion). It is probably going to require downloading all the data from all the machines, which is around 3.7 MiB, though HTTP compression should help (oh, the irony of being stuck with zlib).

Can you add a feature (chart, table, etc.) to the bencmark?

No promises, but I am looking for ideas—I'm certainly willing to at least listen to the request. Please file an issue.

My library isn't performing as well as I think it should!

Sorry. I am trying to keep conditions as fair and as close to real-world as I can, but I'm certainly willing to discuss any concerns you have with methodology.

The Squash library typically adds very little overhead, but if you have ideas on how to improve the plugin for your library I'm happy to accept patches.

If the issue is performance on an architecture you don't have access to (e.g., ARM), I'm willing to provide SSH access to most of the machines included in this benchmark to people working on open source libraries. If you would like access to one to help you optimize your code just let me know.

Can you add «insert machine, CPU, architecture, OS, etc.»?

Only if I have, or at least have access to, a machine which fits that description.

In general I include what I have available. That tends to be some newish Intel CPUs, some older Intel CPUs that haven't yet found their way to the electronics recycler, and some ARM SBCs. If you would like to donate other hardware I'm willing to add it to the benchmark.

What compiler flags were used?

Flags vary a bit since different plugins require different flags, but as far as performance related flags are concerned, plugins are currently compiled with -O3 -flto -march=native -mtune=native.

If you are the author of one of the libraries and would like for your plugin to use different flags when compiled as part of Squash, please file an issue. As long as the flags are safe, we will respect your wishes.

This doesn't work in my browser.

I'm sorry, but if you're using an old browser (probably Internet Explorer) you'll have to upgrade. I'm willing to consider patches, but I will not be putting in any effort to make this page display in older browsers unless someone wants to pay me to do it—and it's not something I enjoy, so they would have to pay pretty well.

If your browser is up to date, make sure JavaScript is enabled. Cookies, Flash, etc. aren't necessary, but JS definitely is.

Are there any better benchmarks?

There are different benchmarks, which may or may not be better, depending on what you're interested in. Most benchmarks use command line programs instead of libraries, which is a bit easier so they tend to include many more codecs, but usually don't include nearly as many different machines. Those benchmarks include:

• Large Text Compression Benchmark
• Squeeze Chart
• Maximum Compression
• World Compression Challenge

The only other benchmark I'm aware of focusing on libraries is fsbench. It includes more compression codecs (though fewer total options), as well as hash functions (cryptographic and non-cryptographic) and some other cryptographic functions. It is run on a few different machines against a tarball of Silesa with all files truncated to 1 MiB.

If I'm missing any modern, up-to-date benchmarks please let me know.

Can I have the raw data?

Of course! The table in the "choose a machine" section includes a link to a CSV which you can import into your favorite spreadsheet application (or at least your least-hated spreadsheet application). If you don't have one yet, LibreOffice Calc is a good choice.

Additionally, you can grab a copy from the the data folder of the squash-benchmark-web git repository.

If you do something interesting with it please let us know! Or, even better, submit a pull request so everyone can benefit from your brilliance!

The data itself is CC0 licensed. That said, we would certainly appreciate attribution.

Can I link to a specific configuration?

Some things can be configured by passing parameters in the query string:

dataset

Dataset to show (currently xargs.1), the default is selected randomly

machine

Machine to show (currently phalanx), the default is selected randomly

speed

Transfer speed (in KiB/s) for the Transfer + Processing chart (currently 128000)

speed-scale

The default scale for the speed axis of charts (linear or logarithmic, currently linear)

visible-plugins

A comma-separated list of plugins to show in the scatter plots. All other plugins will be disabled, though they can be re-enabled by clicking on their entry in the legend.

hidden-plugins

A comma-separated list of plugins to hide in the scatter plots. Note that, if used, this parameter overrides visible-plugins

For example, your current configuration would be: https://quixdb.github.io/squash-benchmark/?dataset=xargs.1&machine=phalanx&speed=128000&speed-scale=linear.

Note that all fields are optional; you can provide as many or few of them as you like. Also, please be aware that this isn't necessarily stable—we may change the format when adding new features to the benchmark.

Discussion