From 622cf70061fc10cb7be545003dde8caac8ef2430 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Tue, 25 Mar 2025 16:23:02 -0700 Subject: [PATCH 1/8] Several updates based on Michal Gorny's feedback on DPO https://discuss.python.org/t/pep-694-pypi-upload-api-2-0/76316/2 Summary: * Add some error return code descriptions where they were underspecified * Fix some misspellings and poorly worded text * Combine the canceling and deleting of upload files sections * Simplify the rules for replacing a staged but not yet published file * Add open question about deferring stage previews --- peps/pep-0694.rst | 72 +++++++++++++++++++++++------------------------ 1 file changed, 36 insertions(+), 36 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index ffdfa2e94c5..d4d0da99275 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -6,7 +6,8 @@ Status: Draft Type: Standards Track Topic: Packaging Created: 11-Jun-2022 -Post-History: `27-Jun-2022 `__ +Post-History: `27-Jun-2022 `__, + `06-Jan-2025 `__ Abstract @@ -409,6 +410,7 @@ If the server determines that upload should proceed, it will return a ``201 Crea an empty body, and a ``Location`` header pointing to the URL that the file content should be uploaded to. The :ref:`status ` of the session will also include the filename in the ``files`` mapping, with the above ``Location`` URL included in under the ``link`` sub-key. +Otherwise the server **MUST** return a ``409 Conflict``. .. IMPORTANT:: @@ -529,7 +531,7 @@ these responses has no content. The final chunk of data **MUST** include the ``Upload-Complete: ?1`` header, since at that point the entire file has been uploaded. -With both chunked and non-chunked uploads, once completed successfully, the file **MUST** not be +With both chunked and non-chunked uploads, once completed successfully, the file **MUST NOT** be publicly visible in the repository, but merely staged until the upload session is :ref:`completed `. If the server supports :ref:`previews `, the file **MUST** be visible at the ``stage`` :ref:`URL `. Partially uploaded chunked files **SHOULD @@ -541,9 +543,9 @@ multiple chunks: - A client **MUST NOT** perform multiple ``POST`` requests in parallel for the same file to avoid race conditions and data loss or corruption. -- If the offset provided in ``Upload-Offset`` is not ``0`` or correctly specifies the byte offset of - the next chunk in an incomplete upload, then the server **MUST** respond with a ``409 Conflict``. - This means that a client **MAY NOT** upload chunks out of order. +- If the offset provided in ``Upload-Offset`` is not ``0`` and does not correctly specify the byte + offset of the next chunk in an incomplete upload, then the server **MUST** respond with a ``409 + Conflict``. This means that a client **MUST NOT** upload chunks out of order. - Once a file upload has completed successfully, you may initiate another upload for that file, which **once completed**, will replace that file. This is possible until the entire session is @@ -578,47 +580,30 @@ multiple chunks as per the above protocol. .. _cancel-an-upload: -Canceling an In-Progress Upload -+++++++++++++++++++++++++++++++ +Canceling and Deleting File Uploads ++++++++++++++++++++++++++++++++++++ -If a client wishes to cancel an upload of a specific file, for instance because they need to upload -a different file, they may do so by issuing a ``DELETE`` request to the upload resource URL of the -file they want to delete. - -A successful cancellation request **MUST** respond with a ``204 No Content``. - -Once deleting, a client **MUST NOT** assume that the previous upload resource URL can be reused. - - -Delete a Partial or Fully Uploaded File -+++++++++++++++++++++++++++++++++++++++ - -Similarly, for files which have already been completely uploaded, clients can delete the file by -issuing a ``DELETE`` request to the upload resource URL. +A client can cancel an in-progress upload for a file, or delete a file that has been completely +uploaded. In both cases, the client performs this by issuing a ``DELETE`` request to the upload +resource URL of the file they want to delete. A successful deletion request **MUST** response with a ``204 No Content``. -Once deleting, a client **MUST NOT** assume that the previous upload resource URL can be reused. +Once canceled or deleted, a client **MUST NOT** assume that the previous upload resource URL can be reused. Replacing a Partially or Fully Uploaded File ++++++++++++++++++++++++++++++++++++++++++++ -To replace a session file, the file upload **MUST** have been previously completed or deleted. It -is not possible to replace a file if the upload for that file is incomplete. Clients have two -options to replace an incomplete upload: - -- :ref:`Cancel the in-progress upload ` by issuing a ``DELETE`` to the upload - resource URL for the file they want to replace. After this, the new file upload can be initiated - by beginning the entire :ref:`file upload ` sequence over again. This means - providing the metadata request again to retrieve a new upload resource URL. Client **MUST NOT** - assume that the previous upload resource URL can be reused after deletion. +To replace a session file, the file upload **MUST** have been previously completed, canceled, or +deleted. It is not possible to replace a file if the upload for that file is in-progress. -- :ref:`Complete the in-progress upload ` by uploading a zero-length chunk - providing the ``Upload-Complete: ?1`` header. This effectively truncates and completes the - in-progress upload, after which point the new upload can commence. In this case, clients - **SHOULD** reuse the previous upload resource URL and do not need to begin the entire :ref:`file - upload ` sequence over again. +To replace a session file, clients should :ref:`cancel and delete the in-progress upload +` by issuing a ``DELETE`` to the upload resource URL for the file they want to +replace. After this, the new file upload can be initiated by beginning the entire :ref:`file upload +` sequence over again. This means providing the metadata request again to retrieve a +new upload resource URL. Client **MUST NOT** assume that the previous upload resource URL can be +reused after deletion. .. _session-status: @@ -720,6 +705,9 @@ If a session is published that has no staged files, the operation is effectively where a new project name is being reserved. In this case, the new project is created, reserved, and owned by the user that created the session. +If an error occurs, the appropriate ``4xx`` code should be returned, as described in the +:ref:`session-errors` section. + .. _session-token: @@ -932,6 +920,18 @@ The user that created the session will become the owner of the new project. Open Questions ============== +Defer Stage Previews +-------------------- + +:ref:`Stage previews ` are an important and useful feature for testing new version +wheel uploads before they are published. They'd allow us to effectively decommission +``test.pypi.org``, which has well-known deficiencies. + +However, the ability to preview stages before they're published does complicate the protocol and +this proposal. We could defer this feature for later, although if we do, we should still keep the +optional ``nonce`` for token generation, in order to be easily future proof. + + Multipart Uploads vs tus ------------------------ From a5aacbb0dc1cdd00ee26f7085650276edb089f9c Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Tue, 25 Mar 2025 16:31:20 -0700 Subject: [PATCH 2/8] Fix a typo --- peps/pep-0694.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index d4d0da99275..a37997cb747 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -923,7 +923,7 @@ Open Questions Defer Stage Previews -------------------- -:ref:`Stage previews ` are an important and useful feature for testing new version +:ref:`Stage previews ` are an important and useful feature for testing new version wheel uploads before they are published. They'd allow us to effectively decommission ``test.pypi.org``, which has well-known deficiencies. From b89504da0c453d201e0872f4337d1e434d2ff766 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Tue, 1 Apr 2025 10:50:15 -0700 Subject: [PATCH 3/8] The last chunk can be accepted asynchronously The server is allowed to respond with a 202 for the last chunk, to indicate it's processing that last chunk asynchronously. This fleshes out the process for checking for completed process of the file in that case, and improves the file status key definitions. --- peps/pep-0694.rst | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index a37997cb747..60c69e30816 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -298,6 +298,7 @@ the following keys: wishes to communicate to the end user. These notices are specific to the overall session, not to any particular file in the session. + .. _session-links: Session Links @@ -330,8 +331,15 @@ The ``files`` key contains a mapping from the names of the files uploaded in thi sub-mapping with the following keys: ``status`` - A string with the same values and semantics as the :ref:`session status key `, - except that it indicates the status of the specific referenced file. + A string with valid values ``partial``, ``pending``, ``complete``, and ``error``. If a file + upload has not seen an ``Upload-Complete: ?1`` header, then ``partial`` will be returned. If + ``Upload-Complete: ?1`` resulted in a ``202 Accepted``, then ``pending`` will be returned until + asynchronous processing of the last chunk and the full file has been completed. If a ``201 + Created`` was returned, or the last chunk processing is finished, ``complete`` will be returned. + If there was an error during upload, then clients should not assume the file is in any usable + state, ``error`` will be returned and it's best to :ref:`cancel or delete ` + the file and start over. This action would remove the file name from the ``files`` key of the + :ref:`session status response body `. ``link`` The *absolute* URL that the client should use to reference this specific file. This URL is used @@ -523,10 +531,13 @@ For the second chunk representing bytes 1000 through 1999, include the following These requests would continue sequentially until the last chunk is ready to be uploaded. For each successful chunk, the server **MUST** respond with a ``202 Accepted`` header, except for -the final chunk, which **MUST** be a ``201 Created``, and as with non-chunked uploads, the body of -these responses has no content. +the final chunk, which **MUST** be either: -.. _complete-the-upload: +* ``201 Created`` if the server accepts and processes the last chunk synchronously, completing the + file upload. +* ``202 Accepted`` if the server accepts the last chunk, but must process it asynchronously. In + this case, the client should query the :ref:`session status ` periodically until + the uploaded :ref:`file status ` transitions to ``complete``. The final chunk of data **MUST** include the ``Upload-Complete: ?1`` header, since at that point the entire file has been uploaded. From c48fa6a406cedc8af0e439f28afbcfd253036618 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Tue, 1 Apr 2025 10:54:25 -0700 Subject: [PATCH 4/8] Open question on whether to allow parallel file uploads --- peps/pep-0694.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 60c69e30816..69f2741940f 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -942,6 +942,15 @@ However, the ability to preview stages before they're published does complicate this proposal. We could defer this feature for later, although if we do, we should still keep the optional ``nonce`` for token generation, in order to be easily future proof. +Allow parallel file uploads +--------------------------- + +While this PEP explicitly disallows upload of file chunks in parallel, it current is silent on +whether multiple files can be uploaded to a stage in parallel. The question is whether we should +explicitly disallow parallel file uploads, or leave it as a server option. This PEP should not +require parallel file uploads, since that could potentially impose an unacceptable processing burden +for some server implementations. + Multipart Uploads vs tus ------------------------ From 1df3de5bdeea83c32c65beea9f81a58edfb05d9e Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Tue, 1 Apr 2025 13:54:58 -0700 Subject: [PATCH 5/8] Fix reference link typo --- peps/pep-0694.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 69f2741940f..9993750df8a 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -537,7 +537,7 @@ the final chunk, which **MUST** be either: file upload. * ``202 Accepted`` if the server accepts the last chunk, but must process it asynchronously. In this case, the client should query the :ref:`session status ` periodically until - the uploaded :ref:`file status ` transitions to ``complete``. + the uploaded :ref:`file status ` transitions to ``complete``. The final chunk of data **MUST** include the ``Upload-Complete: ?1`` header, since at that point the entire file has been uploaded. From 6e9eba8c3bf5639cac91748dd5403d7c01a8eb70 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Mon, 14 Apr 2025 15:04:51 -0700 Subject: [PATCH 6/8] Be explicit that the server MAY accepted parallel file uploads --- peps/pep-0694.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 9993750df8a..2079169bd2d 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -418,7 +418,9 @@ If the server determines that upload should proceed, it will return a ``201 Crea an empty body, and a ``Location`` header pointing to the URL that the file content should be uploaded to. The :ref:`status ` of the session will also include the filename in the ``files`` mapping, with the above ``Location`` URL included in under the ``link`` sub-key. -Otherwise the server **MUST** return a ``409 Conflict``. +If the server determines the upload cannot proceed, it **MUST** return a ``409 Conflict``. The +server **MAY** allow parallel uploads of files, but is not required to. + .. IMPORTANT:: From 65dd90ff0328709238c02c9c8dc66753bb52ecd3 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Mon, 14 Apr 2025 15:11:40 -0700 Subject: [PATCH 7/8] Post-History --- peps/pep-0694.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 2079169bd2d..7d59969757f 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -8,6 +8,7 @@ Topic: Packaging Created: 11-Jun-2022 Post-History: `27-Jun-2022 `__, `06-Jan-2025 `__ + `14-Apr-2025 `__ Abstract From ea511aef92dc1432913a11b05bb9ce83f22cafb2 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Mon, 14 Apr 2025 15:13:33 -0700 Subject: [PATCH 8/8] Parallel uploads will be allowed, but not required --- peps/pep-0694.rst | 9 --------- 1 file changed, 9 deletions(-) diff --git a/peps/pep-0694.rst b/peps/pep-0694.rst index 7d59969757f..db2ef5fdb24 100644 --- a/peps/pep-0694.rst +++ b/peps/pep-0694.rst @@ -945,15 +945,6 @@ However, the ability to preview stages before they're published does complicate this proposal. We could defer this feature for later, although if we do, we should still keep the optional ``nonce`` for token generation, in order to be easily future proof. -Allow parallel file uploads ---------------------------- - -While this PEP explicitly disallows upload of file chunks in parallel, it current is silent on -whether multiple files can be uploaded to a stage in parallel. The question is whether we should -explicitly disallow parallel file uploads, or leave it as a server option. This PEP should not -require parallel file uploads, since that could potentially impose an unacceptable processing burden -for some server implementations. - Multipart Uploads vs tus ------------------------