Skip to content

Allocate memory dynamically #4541

@HT-7

Description

@HT-7

Is your feature request related to a problem? Please describe.
zst reserves the entire specified memory shortly after starting, even if the amount of processed data is nowhere close.

Describe the solution you'd like
If possible, I suggest only allocating as much memory as needed at a time, rather than immediately reserving the entire specified memory.

Describe alternatives you've considered
This is also the behaviour of LZMA-based compressors (lzma, xz, 7z).

Activity

self-assigned this
on Dec 8, 2025
Cyan4973

Cyan4973 commented on Dec 8, 2025

@Cyan4973
Contributor

Note that, even though zstd can allocate a buffer with a size that is proportional to the window size, the buffer is only allocated, and not initialized.
In most modern Operating System, it means no real memory is really reserved for the buffer at the moment it's allocated, right until it is actually needed.

This should cover many scenarios where the library is blind and must allocate in advance the maximum possible memory in case it needs it (streaming mostly).

But also note that, in many other scenarios, where exact needs are known upfront, zstd is smart enough to only reserve what it needs. For example, if one requires a 1 GB window size, and then provides a 1 KB file to compress, zstd will notice the discrepancy, and automatically size down the requirements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @Cyan4973@HT-7

      Issue actions