diff --git a/peps/pep-0782.rst b/peps/pep-0782.rst index b1310c08d0f..cbf4c974b9f 100644 --- a/peps/pep-0782.rst +++ b/peps/pep-0782.rst @@ -80,8 +80,9 @@ Create, Finish, Discard Create a :c:type:`PyBytesWriter` to write *size* bytes. - If *size* is greater than zero, allocate *size* bytes for the - returned buffer. + If *size* is greater than zero, allocate *size* bytes, and set the + writer size to *size*. The caller is responsible to write *size* + bytes using :c:func:`PyBytesWriter_GetData`. On error, set an exception and return NULL. @@ -107,14 +108,14 @@ Create, Finish, Discard Similar to :c:func:`PyBytesWriter_Finish`, but resize the writer using *buf* pointer before creating the :class:`bytes` object. - Pseudo-code:: + Set an exception and return ``NULL`` if *buf* pointer is outside the + internal buffer bounds. + + Function pseudo-code:: Py_ssize_t size = (char*)buf - (char*)PyBytesWriter_GetData(writer); return PyBytesWriter_FinishWithSize(writer, size); - Set an exception and return ``NULL`` if *buf* pointer is outside the - internal buffer bounds. - .. c:function:: void PyBytesWriter_Discard(PyBytesWriter *writer) Discard a :c:type:`PyBytesWriter` created by :c:func:`PyBytesWriter_Create`. @@ -128,7 +129,9 @@ High-level API .. c:function:: int PyBytesWriter_WriteBytes(PyBytesWriter *writer, const void *bytes, Py_ssize_t size) - Write *size* bytes of *bytes* into the *writer*. + Grow the *writer* internal buffer by *size* bytes, + write *size* bytes of *bytes* at the *writer* end, + and add *size* to the *writer* size. If *size* is equal to ``-1``, call ``strlen(bytes)`` to get the string length. @@ -138,8 +141,9 @@ High-level API .. c:function:: int PyBytesWriter_Format(PyBytesWriter *writer, const char *format, ...) - Similar to ``PyBytes_FromFormat()``, but write the output directly - into the writer. + Similar to ``PyBytes_FromFormat()``, but write the output directly at + the writer end. Grow the writer internal buffer on demand. + Then add the written size to the writer size. On success, return ``0``. On error, set an exception and return ``-1``. @@ -153,7 +157,7 @@ Getters .. c:function:: void* PyBytesWriter_GetData(PyBytesWriter *writer) - Get the writer data. + Get the writer data: start of the internal buffer. The pointer is valid until :c:func:`PyBytesWriter_Finish` or :c:func:`PyBytesWriter_Discard` is called on *writer*. @@ -182,16 +186,22 @@ Low-level API On success, return ``0``. On error, set an exception and return ``-1``. - *size* must be positive or zero. + *size* can be negative to shrink the writer. .. c:function:: void* PyBytesWriter_GrowAndUpdatePointer(PyBytesWriter *writer, Py_ssize_t size, void *buf) Similar to :c:func:`PyBytesWriter_Grow`, but update also the *buf* pointer. + The *buf* pointer is moved if the internal buffer is moved in memory. + The *buf* relative position within the internal buffer is left + unchanged. + On error, set an exception and return ``NULL``. - Pseudo-code:: + *buf* must not be ``NULL``. + + Function pseudo-code:: Py_ssize_t pos = (char*)buf - (char*)PyBytesWriter_GetData(writer); if (PyBytesWriter_Grow(writer, size) < 0) { @@ -207,6 +217,10 @@ Overallocation overallocate the internal buffer to reduce the number of ``realloc()`` calls and so reduce memory copies. +:c:func:`PyBytesWriter_Finish` trims overallocations: it shrinks the +internal buffer to the exact size when creating the final :class:`bytes` +object. + Thread safety ------------- @@ -273,8 +287,8 @@ Example creating the bytes string ``b"abc"``, with a fixed size of 3 bytes:: return PyBytesWriter_Finish(writer); } -GrowAndUpdatePointer() example ------------------------------- +``GrowAndUpdatePointer()`` example +---------------------------------- Example using a pointer to write bytes and to track the written size. @@ -310,22 +324,84 @@ Create the bytes string ``b"Hello World"``:: } +Update ``PyBytes_FromStringAndSize()`` code +------------------------------------------- + +Example of code using the soft deprecated +``PyBytes_FromStringAndSize(NULL, size)`` API:: + + PyObject *result = PyBytes_FromStringAndSize(NULL, num_bytes); + if (result == NULL) { + return NULL; + } + if (copy_bytes(PyBytes_AS_STRING(result), start, num_bytes) < 0) { + Py_CLEAR(result); + } + return result; + +It can now be updated to:: + + PyBytesWriter *writer = PyBytesWriter_Create(num_bytes); + if (writer == NULL) { + return NULL; + } + if (copy_bytes(PyBytesWriter_GetData(writer), start, num_bytes) < 0) { + PyBytesWriter_Discard(writer); + return NULL; + } + return PyBytesWriter_Finish(writer); + + +Update ``_PyBytes_Resize()`` code +--------------------------------- + +Example of code using the soft deprecated ``_PyBytes_Resize()`` API:: + + PyObject *v = PyBytes_FromStringAndSize(NULL, size); + if (v == NULL) { + return NULL; + } + char *p = PyBytes_AS_STRING(v); + + // ... fill bytes into 'p' ... + + if (_PyBytes_Resize(&v, (p - PyBytes_AS_STRING(v)))) { + return NULL; + } + return v; + +It can now be updated to:: + + PyBytesWriter *writer = PyBytesWriter_Create(size); + if (writer == NULL) { + return NULL; + } + char *p = PyBytesWriter_GetData(writer); + + // ... fill bytes into 'p' ... + + return PyBytesWriter_FinishWithPointer(writer, p); + + Reference Implementation ======================== `Pull request gh-131681 `__. -The implementation allocates internally a :class:`bytes` object, so -:c:func:`PyBytesWriter_Finish` just returns the object without having -to copy memory. +Notes on the CPython reference implementation which are not part of the +Specification: -For strings up to 256 bytes, a small internal raw buffer of bytes is -used. It avoids having to resize a :class:`bytes` object which is -inefficient. At the end, :c:func:`PyBytesWriter_Finish` creates the -:class:`bytes` object from this small buffer. +* The implementation allocates internally a :class:`bytes` object, so + :c:func:`PyBytesWriter_Finish` just returns the object without having + to copy memory. -A free list is used to reduce the cost of allocating a -:c:type:`PyBytesWriter` on the heap memory. +* For strings up to 256 bytes, a small internal raw buffer of bytes is + used. It avoids having to resize a :class:`bytes` object which is + inefficient. At the end, :c:func:`PyBytesWriter_Finish` creates the + :class:`bytes` object from this small buffer. + +* A free list is used to reduce the cost of allocating a + :c:type:`PyBytesWriter` on the heap memory. Backwards Compatibility @@ -334,6 +410,10 @@ Backwards Compatibility There is no impact on the backward compatibility, only new APIs are added. +``PyBytes_FromStringAndSize(NULL, size)`` and ``_PyBytes_Resize()`` APIs +are soft deprecated. No new warnings is emitted when these functions are +used and they are not planned for removal. + Prior Discussions =================