[Python] Streaming Deserialization Support For Python

[chaokunyang]

Feature Request

Add Python stream deserialization support by reusing C++ stream-backed Buffer behavior.

Is your feature request related to a problem? Please describe

Current Python workflows usually deserialize from bytes/Buffer. For transport streams, users need extra buffering/framing glue.

Design update (based on PR #3307)

Reference: #3307

1. API shape

• Keep existing Fory.deserialize(...) API unchanged.
• Support stream input through Buffer(stream_obj) where stream_obj provides read(size).
• No new deserialize_from_stream(...) API required in this phase.

2. C++ bridge

• Add Fory_PyCreateBufferFromStream in cpp/fory/python/pyfory.cc.
• Bridge adapts Python stream read(size) into C++ std::istream and constructs stream-backed fory::Buffer.

3. Buffer integration

• In python/pyfory/buffer.pyx, constructor detects stream-like objects and creates a stream-backed buffer.
• Bound checks/reads use C++ ensure_size to fetch more bytes on demand.

4. Behavior model

• Sequential objects are decoded by reusing the same Buffer(stream) instance.
• No seek/tell rewind requirement in this design.

5. Mode compatibility

• Shared Buffer behavior is used by existing deserialize paths and validated for xlang/native modes.
• Cython path is explicitly covered by new tests; semantics remain aligned for existing API usage.

6. Tests

• Add Python tests for:
• stream-backed primitive and string reads,
• tiny fragmented stream reads,
• sequential multi-object decode,
• truncated stream failure.

Describe alternatives you've considered

• Python-only stream loop that materializes bytes before decode.
• New dedicated Python stream deserialize API in this phase.

Additional context

Relevant files (PR #3307):

• python/pyfory/buffer.pyx
• python/pyfory/includes/libpyfory.pxd
• python/pyfory/tests/test_buffer.py
• python/pyfory/tests/test_stream.py
• cpp/fory/python/pyfory.cc

[aryash45]

hey @chaokunyang can i assist you o this issue. please reply if its cool with you

[asadjan4611]

@chaokunyang Please assign this issue to me,i want to work on it .

[ulekarsiddhant0-boop]

Hi @chaokunyang, I am Siddhant (ulekarsiddhant0-boop). I am currently writing a GSoC proposal for a Persistence Layer and NumPy modernization. I’d like to help implement the Python Streaming Deserialization support as it aligns perfectly with my project goals. I’ll start by looking at the C++ implementation to ensure parity.

[chaokunyang]

Hi @ulekarsiddhant0-boop , This feature has be implemented already.