RESUMABLE: feature request - chunk granularity

[LPardue]

The authors received an off-list message from a server implementer. They have an existing resumable upload deployment based on a custom protocol and are working to integrate support for draft-ietf-httpbis-resumable-upload.

One of the features that the custom protocol provides is the notion of a chunk granularity. This particular protocol uses an HTTP response header field to advertise a value in bytes. Clients that upload chunks are expected to use an integer multiple of the chunk granularity value (modulo the last chunk, which can be a fragment of it). For example , if a value of 262144 bytes was advertised, a client could send chunks of 262,144, or 524,288, or 2,621,440 bytes.

Other custom reliable upload protocols depend on out-of-band advertisements of this concept. For example, see these requirements listed on a webpage [1]:

Chunk size must be divisible by 256 KiB (256x1024 bytes). Round your chunk size to the nearest multiple of 256 KiB. Note that the final chunk of an upload that fits within a single chunk is exempt from this requirement.

draft-ietf-httpbis-resumable-upload does not support this notion of chunk granularity. The protocol does support in-band advertisement of limits via the Upload-Limit field but it defines only the min-append-size and max-append-size properties. The chunk granularity falls somewhere between these two - the range of chunk sizes can be between a min and a max, but a set of sizes are allowed.

There's a few ways we might be able to support this draft-ietf-httpbis-resumable-upload. However, the key point is that this is a feature request that we need to first consider if we want to support.

[1] https://developers.cloudflare.com/stream/uploading-videos/resumable-uploads/#requirements

[danielresnick]

Thanks @LPardue - adding some more detail here based on some internal discussions:

1. Technically we can represent a hacky/limited form of chunk granularity via min-append-size=max-append-size=chunk_granularity*N. However this artifically constraints the client into sending chunks of a fixed size when really any value of N would have been OK.

2. Even if our server uses the workaround described above, some spec complaint implementations might run into trouble with this as min-append-size + max-append-size are SHOULD from a client point of view, not MUST.

3. Clients that include an unaligned chunk in Upload Creation (with Upload-Complete: ?0) are more troublesome. AFAICT the spec doesn't expect clients which include chunks in Upload Creation requests to respect min-append-size/max-append-size at all -- that makes sense given the name, but means these clients can hypothetically get stuck entirely, especially if they (or the server they're talking to) don't support interim HTTP responses. However it seems somewhat unlikely that a "generic" HTTP client would implement the resumable upload spec in this manner, since it would require the client to do feature detection first which is a bit awkward. Instead "Optimistic Upload Creation" seems to be the likely default path for generic HTTP clients (e.g. this appears to be the path NSURLSession has taken on iOS).

4. Despite these points we still think it would be valuable to include a granularity concept in the spec. Ideally it would apply to both Upload Creation (with Upload-Complete: ?0) and Upload Append requests. Based on my reading of the spec I expect it would land as a SHOULD rather than a MUST, and we'd be OK with that. It'd serve as a clear signal to client implementors that this is something that at least some servers care about, so if they are aiming to be widely interoperarable they will want to consider respecting it.

5. Regardless of whether this lands in the spec or not, our tentative plan (to be maximally interoperarable with as many clients as possible) is to accept as much data as possible for Upload Append requests (up to the nearest granularity boundary), and then fail the request with a 4xx error, including an Upload-Offset header. I think this should allow a spec complaint client (that is unaware of min-append-size / max-append-size and/or a hypothetical granularity constraint) to pick up where they left off, ideally via the provided Upload-Offset in the error response, but otherwise via the Offset Retrieval mechanism. 400 is probably the more "correct" code, but it's tempting to (ab)use 409 CONFLICT in this scenario since clients need to deal with this anyway. It might be helpful for the spec to say that servers MAY return Upload-Offset in other error responses so that clients are more likely to take advantage of this to avoid the extra RT for recovery.

Curious for others' thoughts on this!

[gstrauss]

With HTTP/1.1 pipelining, and HTTP/2 and later streams, requests can be pipelined. There is diminishing returns reducing the number of requests as large append sizes are increased to being even larger request sizes. Would specifying a list of acceptable and precise chunk sizes work for you? 256k, 512k, 1M, 2M How many more would you list?

[danielresnick]

Agreed there are diminishing returns. I think that would be an acceptable alternative (assuming you mean specifying via a server advertised list as opposed to a hard-coded list in the spec), though I think I personally prefer the ergonomics of leaving it up to the client to send in multiples of a server specified granularity. In practice we have different "dropzones" with granularity ranging from 256KiB to 8MiB (with several steps along the way). We'd likely produce a separate list per "dropzone" and could almost certainly make do with < 5 items in the list.

[LPardue]

@danielresnick Are you able to explain or provide more detail about why granular chunk sizes are desirable? I'm aware it can/is done already for custom protocols but it's not clear what the motivation is. (We can speculate but I'd rather here something concrete from current implementers)

[danielresnick]

I'm not an expert in this area but from what I gather it allows multiple simplifications/optimizations throughout our stack. It helps align writes to our underlying filesystem block size (which supports replication & different encoding schemes). Data is divided into atomic blocks/units for replication, deduplication, caching, garbage collection, etc so having clear guarantees around intermediate write granularity helps simplify metadata management and can allow more efficent data packing.

[LPardue]

Thanks!

Do you have any sense if the set of values tends to benefit from being an arithmetic progession (i.e a constant delta between values) or if you have a non-lonear range like 266K, 512K, 1M, 2M, 8M

I kinda like the idea of a list to be explicit and allow flexibility in the sequence. The downside might be that a server provides nonsense values in the set and there'd need to be some additional parsing/validation logic defined to be defensive to that. For example, an unsorted list, a list with duplicates, etc. However, any format for expressing a granularity would need some validation against other hints, so a list might not push that much further

[danielresnick]

I had a look but wasn't able to find any concrete data on this unfortunately. I tend to think a non-linear range would be more client friendly/flexible though

[Acconut]

Thanks for the additional input, @danielresnick. It seems that there are a few cases where upload systems have to be restricted to a certain granularity. Since this isn't the first report on this, I think we should consider these situations better in the draft.

I'm in favor of adding a granularity value to Upload-Limit. On the question of whether the value should be a single integer indicating a number that the append size must be fully divisible by or a list of allowed append sizes, I would side with the single integer, as its handling on the client- and server-side appear simpler to me. However, if other's experience shows that this isn't sufficient to accommodate limits of existing infrastructure, I'm happy to change my mind.

In my interpretation, the server is currently already allowed to drop trailing bytes, thereby making the appended data fit the required granularity. However, this isn't very clear from the document and we could improve the text.

5. Regardless of whether this lands in the spec or not, our tentative plan (to be maximally interoperarable with as many clients as possible) is to accept as much data as possible for Upload Append requests (up to the nearest granularity boundary), and then fail the request with a 4xx error, including an Upload-Offset header. I think this should allow a spec complaint client (that is unaware of min-append-size / max-append-size and/or a hypothetical granularity constraint) to pick up where they left off, ideally via the provided Upload-Offset in the error response, but otherwise via the Offset Retrieval mechanism. 400 is probably the more "correct" code, but it's tempting to (ab)use 409 CONFLICT in this scenario since clients need to deal with this anyway. It might be helpful for the spec to say that servers MAY return Upload-Offset in other error responses so that clients are more likely to take advantage of this to avoid the extra RT for recovery.

A 4xx status code fits the semantics best. However, the draft currently recommends to not retry on generic 4xx response codes: https://httpwg.org/http-extensions/draft-ietf-httpbis-resumable-upload.html#section-4.4.1-6.4.1. Not every client will retry on a 409 Conflict response because the draft had been written assuming that a compliant client would never trigger a 409 response and therefore not need a retry. However, with required granularities, even a compliant client might run into these statuses. With the current text, a 5xx would likely be needed to make it work with compliant clients. That doesn't appear as correctly, so maybe we have to revisit handling of 409 on the client-side.

[danielresnick]

Great point, I'd missed that the spec suggests not to retry unexpected 4xx responses. So yes I agree that within the current spec a 5xx response is most appropriate to maximize interoperability. I didn't quite understand your last sentence though - mind clarifying? Are you saying you think that this is not ideal and are therefore thinking maybe the draft spec should be refined so as to allow a different / more specific recovery mechanism for clients that are spec complaint (in a minimal sense) and opt not to support an optional granularity constraint?

[Acconut]

I didn't quite understand your last sentence though - mind clarifying? Are you saying you think that this is not ideal and are therefore thinking maybe the draft spec should be refined so as to allow a different / more specific recovery mechanism for clients that are spec complaint (in a minimal sense) and opt not to support an optional granularity constraint?

Not entirely. My thinking is that with the current spec, a 5xx status code is needed to trigger the client to fetch the correct offset (which isn't the offset the client expects since representation data was discarded). However, this doesn't seem like a server error which 5xx status codes indicate, in particular if we extend Upload-Limit to explicitly communicate the granularity. A 4xx would be more suitable, but require the spec to be updated so clients react properly. Does that help you?

[danielresnick]

Thanks for the clarification, @Acconut. That makes perfect sense.

I agree that relying on a 5xx response feels like a necessary but semantically incorrect workaround under the current spec. A 4xx response is definitely more appropriate, and updating the spec to guide clients on how to handle it would be the ideal long-term solution.

Your comment, "Not every client will retry on a 409 Conflict response because the draft had been written assuming that a compliant client would never trigger a 409 response", got me thinking about other scenarios though...

Even with a perfectly compliant client, a 409 Conflict might occur due to factors outside its control. For example, network-level retries (e.g., from a proxy or load balancer) or internal server-side race conditions could result in a chunk being processed more than once. While the HTTP spec (I think?) discourages uninformed retries of non-idempotent methods like PATCH, it's a condition that robust server infrastructure should probably handle, and consequently, a robust client should probably anticipate.

Given this, a client aiming for wide interoperability should likely be prepared to receive a 409 Conflict at any point and use the Offset Retrieval mechanism to recover. If that's a fair assumption, then perhaps recommending a 409 response for this granularity mismatch isn't a significant departure from what a robust client should already implement.

This leads to a practical question for a major client implementation. @guoye-zhang, out of curiosity, how does the current macOS/iOS client (using NSURLSession) handle this today? Will it attempt to recover from an "unexpected" 409 Conflict on a PATCH request by fetching the latest offset, or does it treat it as a fatal error? How does its behavior compare when receiving a generic 5xx response?