Respect `zstd.CompressionParameter.content_size_flag` for streaming `zstd` when possible

[socketpair]

Feature or enhancement

Proposal:

from compression import zstd

options = {
   zstd.CompressionParameter.content_size_flag: 1
}

# zstd.open() implementation should call set_pledged_input_size(10)
# should zstd set CompressionParameter.content_size_flag=1 automatically if uncompressed_size passed? I don't know. need to choose API.
with zstd.open("file.zst", "w", options=options, uncompressed_size=10) as f:
    f.write(b"x"*8)  # OK
    f.write(b"aa")  # OK
    f.write(b"z")  # should fail because too many data written. I guess in zstd internal code

# should fail on context exit (because to few data written).  I guess in zstd internal code
with zstd.open("file.zst", "w", options=options, uncompressed_size=10) as f:
    f.write(b"z")  

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

[picnixz]

Actually, I just observed that uncompressed_size doesn't exist. I think what you want is to have the "overall" size to be respected in advance. But I don't think it should be provided at zstd level. Because set_pledged_input_size is meant to control the content size written to the current frame. The notion of what is a frame may also differ. With uncompressed_size, this means that you assume you have a single frame. And in this case, I think it's better to directly use ZstdCompressor, call set_pledged_input_size and then multiple compress and then flush.

cc @emmatyping

[socketpair]

@picnixz

1. Yes, there is no such kwarg, it's just my proposition.
2. Neither Zstd.open nor ZstdFile allow to pass ZstdCompressor
3. Yes, ZstFile uses single frame for all operations. It's not said explicitly (which is also bug in documentation)

Why jot to add an useful option.

Or, possibly, make internal ZstdCompressor instance public.

[picnixz]

Neither Zstd.open nor ZstdFile allow to pass ZstdCompressor

I meant to use ZstdCompressor instead of zstd.open. They are public: https://docs.python.org/3/library/compression.zstd.html#compression.zstd.ZstdCompressor.

Yes, there is no such kwarg, it's just my proposition

For me, open() has semantics for when you do not know in advance the exact size you'll be reading/writing. So using the explicit compressor would be better. Likewise, adding this
option to the dict of CompressionOptions may not work because of zstd's internals.

Now, adding a parameter that is unused in some reading modes seems to be the norm for ZstdFile (for instance, level must not be given in reading mode).

Yes, ZstdFile uses single frame for all operations. It's not said explicitly (which is also bug in documentation

Actually, I should amend what I said. ZstFile implements flush(mode=...) as well but:

The mode argument can be FLUSH_BLOCK or FLUSH_FRAME. Abuse of this
method will reduce compression ratio, use it only when necessary.

So it's not just for single frame operations.

Why jot to add an useful option.

Because there is (IMO) a simpler (maintainable) alternative that exists. OTOH, we could expose ZstdFile._compressor.set_pledged_input_size directly for that instead of having it as an additional keyword argument. Is there a reason why not just using the ZstdCompressor object directly?

[socketpair]

@picnixz because file-like access api is simple. It exists in zstd module exactly bu this reason, I guess. For example, shutil.copyfileobj() can be used.

In my case, I need to compress data with size known in advance (few gigabytes). So, I would like to have flexible way of tuning compressor. Pass compressor as an option? Perfect. Access compressor after opening? Also good. Any of these options would be OK for me. Possibly other, that allows to pass expected size to the compressor.

[picnixz]

Pass compressor as an option? Perfect

That wouldn't be possible IMO because open is for read or write and Python doesn't like to mix different APIs in general (more generally, having mutually exclusive parameter names is a design we usually try to avoid).

Access compressor after opening? Also good

That would break the encapsulated design so not a good idea either.

Possibly other, that allows to pass expected size to the compressor

In this case, solely exposing set_pledged_input_size on the file object may be fine, but I also don't like having a method that exposes such meaning.

@emmatyping What do you think?

[picnixz]

It exists in zstd module exactly bu this reason

I think it was meant to provide minimal API, but set_pledged_input_size may have been too niche. I think the API was more designed to be a simple wrapper on file objects, not to be a compressor behaving like a file object.

[socketpair]

Exposing set_pledged_input_size() as a file object method sounds good for me.

[emmatyping]

I didn't respond immediately here because I've been mulling over whether adding this is a good idea. I think for set_pledged_input_size, the answer is no for a couple of reasons:

1. ZstdFile (and compression.zstd.open) are high level interfaces that are meant to handle high level streaming scenarios.
2. The semantics are not obvious if you write more than one frame. set_pledged_input_size needs to be called every time a new ZstdCompressor is initialized to ensure each frame writes the content size header. But that feels like a violation of the abstraction of ZstdFile to expose the fact that multiple ZstdCompressor instances are used.

I think a kwarg to ZstdFile could be useful, I'd probably call it pledged_size (to associate it with set_pledged_input_size).

We'd then be able to track how much is written internally to ZstdFile and handle flushes and multiple frames.

[emmatyping]

Actually upon thinking about an implementation further, I don't think we can have pledged_size work either. It could only work for a single frame since we need to know the size of frames ahead of time, and we won't know that until users call flush(FLUSH_FRAME).

So I think you'd be better served by using your own ZstdCompressor.

[Suor]

Passing compressor to zstd.open() will be in line with passing zstd_dict there, which is already implemented.