Skip to content

Why do we need the --ultra flag? #435

@JoeUX

Description

@JoeUX

There are 22 compression levels. To use levels 20, 21, and 22 the program requires not only the standard numeric level flag but that we also add --ultra to the command. Why the extra hassle?

If it's related to the fact that these compression levels use more memory, and that it's potentially disruptive enough that people should have to include extra flags to unlock these levels, I suggest changing the word "ultra" to something that clearly conveys the issue so people won't forget why it's required. Maybe memboost or highmem...

Activity

self-assigned this
on Oct 30, 2016
Cyan4973

Cyan4973 commented on Oct 30, 2016

@Cyan4973
Contributor

As stated by the man/help page, these levels require more memory.
This is an understatement.
Most importantly, the increased memory requirement will impact not only the compression side, but also the (future) decompression side.

To require a lot of memory on the decompression side is not welcomed. It could be considered an attack, trying to take away memory from the receiving system.
For its own protection, a decoder is free to reject a frame which requires too much memory.
So, in short, these frames are more likely to be rejected by 3rd party decoders.

As long as you remain in the x64 PC world, using the official zstd cli, there will be no such problem. Here, memory is plentyful.
Problems will start to happen when trying to share such frames with lighter systems, such as, for example, a mobile 32-bits ARM platform. These platforms will typically reject frames built with levels 21 and 22. It can be even worse, as systems are allowed to set their own limit.

In order to reduce the number of misfits, a memory threshold is recommended, keeping most systems compatible with each other. This value is 8 MB. It's used by level 19.

So basically, when setting --ultra, it makes the compressor go beyond the threshold, and reduce the number of systems which will be able to decode the compressed frame. It's a non-issue as long as the frame will be consumed by some laptop/desktop/server class, typically x64 Intel chip platform.

If you like the setting, and are tired of having to type it manually every time, you can alias the command, so that it will be enabled all the time (and it isn't a problem for faster modes) :
alias zstd='zstd --ultra'

Cyan4973

Cyan4973 commented on Nov 3, 2016

@Cyan4973
Contributor

Hope it answered your question. ping me it not

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @Cyan4973@JoeUX

      Issue actions

        Why do we need the --ultra flag? ยท Issue #435 ยท facebook/zstd