Skip to content

docs: Add troubleshooting guide on new S3 checksums #684

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sbernauer merged 2 commits into from
Sep 5, 2025
Merged

docs: Add troubleshooting guide on new S3 checksums #684

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1fd81ff
docs: Add troubleshooting guide on new S3 checksums
sbernauer Sep 5, 2025
879856b
Apply suggestions from code review
sbernauer Sep 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions docs/modules/airflow/pages/troubleshooting/index.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
= Troubleshooting

== S3 Logging: An error occurred (411) when calling the PutObject operation: Length Required

If Airflow is trying to access S3 (e.g. for remote task logging) and throws the following error

[source,text]
----
{s3_task_handler.py:190} ERROR - Could not write logs to s3://***/***-task-logs/dag_id=test/run_id=manual__2025-05-22T08:14:17.917519+00:00/task_id=extract/attempt=1.log
Traceback (most recent call last):
[..]
botocore.exceptions.ClientError: An error occurred (411) when calling the PutObject operation: Length Required
----

then this is an indication that your S3 store doesn't support the new AWS S3 checksums.
Ideally you should upgrade your S3 store to a more recent version that supports the new checksums (if there is any).
If this is not possible, you can work around this by setting the following env vars

[source,yaml]
----
webservers: # Also add to other roles!
envOverrides:
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
----

See e.g. https://github.com/minio/minio/issues/20845[this MinIO issue] for details.
1 change: 1 addition & 0 deletions docs/modules/airflow/partials/nav.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
*** xref:airflow:usage-guide/operations/pod-placement.adoc[]
*** xref:airflow:usage-guide/operations/pod-disruptions.adoc[]
*** xref:airflow:usage-guide/operations/graceful-shutdown.adoc[]
* xref:nifi:troubleshooting/index.adoc[]
* xref:airflow:reference/index.adoc[]
** xref:airflow:reference/crds.adoc[]
*** {crd-docs}/airflow.stackable.tech/airflowcluster/v1alpha1/[AirflowCluster {external-link-icon}^]
Expand Down
Loading