Multipart upload with CSE using RSA key pair

[sid22]

Problem:

My code is roughly

    val s3ClientObject =
      S3Client
        .builder()
        .credentialsProvider(
              StaticCredentialsProvider.create(
                AwsBasicCredentials.create(
                  spec.accessKey.get,
                  getSecretKey(metadataEncryptionUtils)
                )
              )
        )
       .region(REGION.US_EAST_1).build()

    val s3AsyncClientObject =
      S3AsyncClient
        .builder()
        .credentialsProvider(
              StaticCredentialsProvider.create(
                AwsBasicCredentials.create(
                  spec.accessKey.get,
                  getSecretKey(metadataEncryptionUtils)
                )
              )
        )
       .region(REGION.US_EAST_1).build()

Now I create S3EncryptionClient by wrapping above such as

    val encObject = 
      S3EncryptionClient
        .builder()
        .rsaKeyPair(userKeys)
        .enableLegacyUnauthenticatedModes(true)
        .enableLegacyWrappingAlgorithms(true)
        .wrappedClient(s3ClientObject)
        .wrappedAsyncClient(s3AsyncClientObject)
        .enableDelayedAuthenticationMode(true)
        .build()

I am able to use this encObject to do operations like creating bucket etc. I am also able to upload files to s3 bucket.

However, when i try to upload a large file ( say ~200MB ) with multi part upload it fails with following error

aused by: software.amazon.awssdk.crt.http.HttpException: Amount of data streamed out does not match the previously declared length.
	at software.amazon.awssdk.http.crt.internal.response.CrtResponseAdapter.onResponseComplete(CrtResponseAdapter.java:108) ~[thirdparty-intellij-deps.jar:?]
	at software.amazon.awssdk.crt.http.HttpStreamResponseHandlerNativeAdapter.onResponseComplete(HttpStreamResponseHandlerNativeAdapter.java:58) ~[thirdparty-intellij-deps.jar:?]
Exception in thread "AwsEventLoop 9" java.lang.IllegalStateException: Encountered fatal error in publisher
	at software.amazon.awssdk.utils.async.SimplePublisher.panicAndDie(SimplePublisher.java:339)
	at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:226)
	at software.amazon.awssdk.utils.async.SimplePublisher.send(SimplePublisher.java:128)
	at software.amazon.awssdk.utils.async.InputStreamConsumingPublisher.doBlockingWrite(InputStreamConsumingPublisher.java:58)
	at software.amazon.awssdk.core.async.BlockingInputStreamAsyncRequestBody.writeInputStream(BlockingInputStreamAsyncRequestBody.java:76)
	at software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.doBlockingWrite(InputStreamWithExecutorAsyncRequestBody.java:108)
	at software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.lambda$subscribe$0(InputStreamWithExecutorAsyncRequestBody.java:81)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: Must use either different key or iv for GCM encryption
	at java.base/com.sun.crypto.provider.CipherCore.checkReinit(CipherCore.java:1088)
	at java.base/com.sun.crypto.provider.CipherCore.update(CipherCore.java:662)
	at java.base/com.sun.crypto.provider.AESCipher.engineUpdate(AESCipher.java:380)
	at java.base/javax.crypto.Cipher.update(Cipher.java:1869)
	at software.amazon.encryption.s3.internal.CipherSubscriber.onNext(CipherSubscriber.java:52)
	at software.amazon.encryption.s3.internal.CipherSubscriber.onNext(CipherSubscriber.java:16)
	at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:267)
	at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
	... 10 more

If I directly use the s3ClientObject it works.

Solution:

Is there some limitation on CSE with multi part uploads ?

[kessplas]

Hello Sid22,

The Async S3 Encryption Client uses the Java implementation Reactive Streams to send data to S3. This is generally more efficient, but for multipart uploads, there are issues, because Reactive Streams do not have an equivalent to mark/reset or seek in "traditional" input/output streams. Because of this, when individual parts are retried, the cipher cannot keep track of its state and replay the data it has already encrypted to the server, and you see errors like the one you have posted. If you still need to use the Async client, you can set the enableMultipartPutObject(true) option which guards against retries; the request would still fail under adverse network conditions, but you at least get a proper modeled error. We are working on a better fix for this but there is currently no ECD.

Alternatively, for the most robust multipart upload solution, you can set the same option (enableMultipartPutObject(true)) in the (non-Async) S3EncryptionClient, which uses "traditional" input/output streams for multipart upload using the putObject API. You can also use the "low-level" multipart upload API in the same client (createMultipartUpload/uploadPart/completeMultipartUpload) but you will need to be careful to always upload the parts in order, to avoid issues with decryption later on, if the parts are shuffled, and avoid retrying individual parts for the same reason as above.

Let us know if you have any further questions, thanks!

[sid22]

@justplaz thanks for the detailed explanation. I modified my code to add enableMultipartPutObject(true) to the S3EncryptionClient and re-tried.

However, i still see the same error. On a side note we are using the "low level" multi part upload API ( createMultipartUpload/uploadPart/completeMultipartUpload )

We have a method which expects an object of S3Client interface and uses it with low level multipart methods to do the upload.

If i pass S3Client object directly the multi part upload succeeds so it is not an issue of improper login in the upload method.

[kessplas]

I see, that makes sense then. If you're using the the low-level multipart upload API, then setting enableMultipartPutObject(true) won't have any affect, it only modifies the behavior of putObject, in other words, it enables high-level multipart upload.

For low-level multipart upload, you need to ensure that each part is encrypted sequentially and in the correct order. Otherwise, the encryption will fail. Please refer to the low-level multipart upload example for an example of how to upload parts in sequence.

Let us know if you are still having issues, thanks!

[sid22]

@justplaz thanks for the example, one key thing we are not doing is

// Set sdkPartType to SdkPartType.LAST for last part of the multipart upload.
//
// Note: Set sdkPartType parameter to SdkPartType.LAST for last part is required for Multipart Upload in S3EncryptionClient to call cipher.doFinal()

We were uploading parts sequentially in order but were not setting the above anywhere. I will modify the code to do this and then revert back

[sid22]

I added check in our code to manually ensure we tag the last part properly. Still however getting the same error but from a different code part now within the SDK

Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Amount of data streamed out does not match the previously declared length.
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) ~[thirdparty-intellij-deps.jar:?]
	at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47) ~[thirdparty-intellij-deps.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223) ~[thirdparty-intellij-deps.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:218) ~[thirdparty-intellij-deps.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:182) ~[thirdparty-intellij-deps.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159) ~[thirdparty-intellij-deps.jar:?]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79) ~[thirdparty-intellij-deps.jar:?]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$null$0(MakeAsyncHttpRequestStage.java:103) ~[thirdparty-intellij-deps.jar:?]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:240) ~[thirdparty-intellij-deps.jar:?]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:163) ~[thirdparty-intellij-deps.jar:?]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907) ~[?:?]
	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: software.amazon.awssdk.crt.http.HttpException: Amount of data streamed out does not match the previously declared length.
	at software.amazon.awssdk.http.crt.internal.response.CrtResponseAdapter.onResponseComplete(CrtResponseAdapter.java:108) ~[thirdparty-intellij-deps.jar:?]
	at software.amazon.awssdk.crt.http.HttpStreamResponseHandlerNativeAdapter.onResponseComplete(HttpStreamResponseHandlerNativeAdapter.java:58) ~[thirdparty-intellij-deps.jar:?]
Exception in thread "AwsEventLoop 7" java.lang.IllegalStateException: Encountered fatal error in publisher
	at software.amazon.awssdk.utils.async.SimplePublisher.panicAndDie(SimplePublisher.java:339)
	at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:226)
	at software.amazon.awssdk.utils.async.SimplePublisher.send(SimplePublisher.java:128)
	at software.amazon.awssdk.utils.async.InputStreamConsumingPublisher.doBlockingWrite(InputStreamConsumingPublisher.java:58)
	at software.amazon.awssdk.core.async.BlockingInputStreamAsyncRequestBody.writeInputStream(BlockingInputStreamAsyncRequestBody.java:76)
	at software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.doBlockingWrite(InputStreamWithExecutorAsyncRequestBody.java:108)
	at software.amazon.awssdk.core.internal.async.InputStreamWithExecutorAsyncRequestBody.lambda$subscribe$0(InputStreamWithExecutorAsyncRequestBody.java:81)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: Must use either different key or iv for GCM encryption
	at java.base/com.sun.crypto.provider.CipherCore.checkReinit(CipherCore.java:1088)
	at java.base/com.sun.crypto.provider.CipherCore.update(CipherCore.java:662)
	at java.base/com.sun.crypto.provider.AESCipher.engineUpdate(AESCipher.java:380)
	at java.base/javax.crypto.Cipher.update(Cipher.java:1869)
	at software.amazon.encryption.s3.internal.CipherSubscriber.onNext(CipherSubscriber.java:52)
	at software.amazon.encryption.s3.internal.CipherSubscriber.onNext(CipherSubscriber.java:16)
	at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:267)
	at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
	... 10 more

@justplaz is there any check around the type of RSA keys that work ? I generated a fresh sample RSA key and got the above error. Adding the RSA key here for reference.

key:-

    "-----BEGIN RSA PRIVATE KEY-----\n" +
      "MIIEpAIBAAKCAQEAnc+Cy5uUz+/5G5gtuwjnxhq0VY2TffCQQVV0CzK/KM2mhfrI\n" +
      "CEH/stDPIT4Alf/75H/hW03IY3PDSxjRLqVKMKLpd9SkGQHcn38aa2Kb2eX4Ar77\n" +
      "KCk2G0wcAm/mM//gJmAe5AFQ90z32C9HEz11P6uUrkp772wx2/vJSjcG5kzSWpAj\n" +
      "zMHF0EplNL41aBFDtDyJ9hSoJmRPE9gAHM2nS7otdGu4l0l9skFw1mkZtjMxirjU\n" +
      "CqqLiugLVsJnyajrlS72wZuc0Hp6xp7NH0ZGtkFN0r58z4Qoycn/W6DBNZtG1XAg\n" +
      "aniP0i3lGo/hh1UrGCA/4DrITBtDfuBWccbkoQIDAQABAoIBAQCKu3iSok2ql555\n" +
      "QclCGcwX/jX22CWHm8pVhVgk2BHxPwlb02Gy0MKHYsYUxTsiow3AjSOCbtjxhT10\n" +
      "cXbD+Q9FvpJchBVW3qojlUuWh/PXFTJ4x4hogAJO8RPWmKTZpeJaGjpN21JgdcuU\n" +
      "w8tKAMdol+B3cIePraAPckQ8+C8amaEQ8NIlACD7lD8CcYKNk3PyBd1bNzlF7atP\n" +
      "5KybPBJBOrupE9AVT4jd+mbBzrbfCkiISa5JiuEfMcgbThNMYPHakofMhDZL62CG\n" +
      "luo/BohnMBFzsJDGg/LFrBSUUPlC0BX6/zfBicyiODtc3AFujRGCR6gYNrisTI1s\n" +
      "iTSflOW5AoGBAMryCrElACRw4QDS3yNX8WkoMBDlYSC3r9dcbdUL3EX2LiiLDoD/\n" +
      "lVF1WKxnG7ZR5cPs2kcawWrwquA7fuXZUYjsKTFh+QLl/D0G8L9aCt1qMGlHLKy7\n" +
      "ozFyVxlvh1Lec9IgtdC8YKPJfDN8JoJQQV76ReHA6FuFvHv6jrSIim/fAoGBAMcQ\n" +
      "2qz4htFsrtkiEjxlAGWVD4qls9aAXhnUgorUeX3a02Jx+25rMR6SgmyrI5UB69YG\n" +
      "OaniekV23aFZNlf5QzYjzwt8Nv9CeSNkD2m/k5+feYf5JJM3bm2ktxN0wu3yKMtU\n" +
      "OLHpHLhW+3twwSCI2c1hpQln3p41DIr1Noyozzt/AoGAJ14mdtCPo4IGE6vUPz3r\n" +
      "BZQXJt/oJHmdcbBrWd2QID4uHA1Fhf6OT5vs1Jy3wnlGkegbO5nUFVOUQiUoa5vp\n" +
      "dh8hqoOv00Eb2hbDksr7upHDzFhTMTrA4HGmtbdtz8R5QTS5MEGqmXsXTcFykurQ\n" +
      "k4UHE1DhggeCVaZ4Eks+V48CgYEApcskxcEr0AqbyZ413/UjEnfGfOwrTvCU7yBu\n" +
      "JSB3m1mAitJx3XILc/IEDGuw8+6otBV1O0e0HFy2lCZQO48P6myCiYdH6us7Jz20\n" +
      "FJgJZH2W46eeTbpyD4GLNPofS7xPO6GGoq6LTACt7Q5o2yb/d63mnWHUKKH4M1et\n" +
      "uhLynhMCgYBzD3lczr2yVV3+VkRbAbjjprYGFansmJVI3U03uSg1zYtaAxNhUMcT\n" +
      "EMx+KziFxcSBSFzEMNoJ7ttEVtUqgX6yrxivr9B4nw1znFLAno4fs+O/3Ax4+qvZ\n" +
      "5IPYZyCdCRiNTluwgI45Xedv/XwF56wHxZIv3WN6Edjd9la7QzdSFQ==\n" +
      "-----END RSA PRIVATE KEY-----"