Skip to content

fix(config): set MD5 checksums for DigitalOcean and GCS flavors since they don't support CRC32C, and explicitly set CRC32C for MinIO #6487

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fulmicoton-dd wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(config): set MD5 checksums for DigitalOcean and GCS flavors since they don't support CRC32C, and explicitly set CRC32C for MinIO #6487

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 29 additions & 8 deletions quickwit/quickwit-config/src/storage_config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -370,20 +370,26 @@ impl S3StorageConfig {
Some(StorageBackendFlavor::DigitalOcean) => {
self.force_path_style_access = true;
self.disable_multi_object_delete = true;
// doesn't support CRC32C
self.checksum_algorithm = ChecksumAlgorithm::Md5;

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve explicit checksum_algorithm with flavors

When an S3 config uses a flavor and also sets checksum_algorithm: disabled (or md5 for MinIO/Garage), apply_flavor() runs after deserialization and this unconditional assignment replaces the user-provided value; docs/configuration/storage-config.md documents checksum_algorithm as the supported control while disable_checksums is deprecated. For example, a DigitalOcean or GCS user disabling checksums to work around provider/proxy failures will still send Content-MD5, and MinIO/Garage users who choose disabled will get CRC32C. Please apply these flavor values only as defaults when the field was omitted, or otherwise keep honoring explicit config.

Useful? React with 👍 / 👎.

}
Some(StorageBackendFlavor::Garage) => {
self.region = Some("garage".to_string());
self.force_path_style_access = true;
self.checksum_algorithm = ChecksumAlgorithm::Crc32c;
}
Some(StorageBackendFlavor::Gcs) => {
self.disable_multi_object_delete = true;
self.disable_multipart_upload = true;
// doesnt support CRC32C via the S3 SDK
self.checksum_algorithm = ChecksumAlgorithm::Disabled;
// Quickwit comes with a "native" GCS client that supports CRC32C
// when compiled with the GCS feature flag.
self.checksum_algorithm = ChecksumAlgorithm::Md5;
}
Some(StorageBackendFlavor::MinIO) => {
self.region = Some("minio".to_string());
self.force_path_style_access = true;
self.checksum_algorithm = ChecksumAlgorithm::Crc32c;
}
_ => {}
}
Expand Down Expand Up @@ -704,40 +710,55 @@ mod tests {
let s3_storage_config_yaml = r#"
flavor: digital_ocean
"#;
let s3_storage_config: S3StorageConfig =
let mut s3_storage_config: S3StorageConfig =
serde_yaml::from_str(s3_storage_config_yaml).unwrap();

s3_storage_config.apply_flavor();
assert_eq!(
s3_storage_config.flavor,
Some(StorageBackendFlavor::DigitalOcean)
);
assert!(!s3_storage_config.disable_checksums);
assert_eq!(s3_storage_config.checksum_algorithm, ChecksumAlgorithm::Md5);
Comment thread
fulmicoton marked this conversation as resolved.
}
{
let s3_storage_config_yaml = r#"
flavor: garage
"#;
let s3_storage_config: S3StorageConfig =
let mut s3_storage_config: S3StorageConfig =
serde_yaml::from_str(s3_storage_config_yaml).unwrap();
s3_storage_config.apply_flavor();

assert_eq!(s3_storage_config.flavor, Some(StorageBackendFlavor::Garage));
assert!(!s3_storage_config.disable_checksums);
assert_eq!(
s3_storage_config.checksum_algorithm,
ChecksumAlgorithm::Crc32c
);
}
{
let s3_storage_config_yaml = r#"
flavor: gcs
"#;
let s3_storage_config: S3StorageConfig =
let mut s3_storage_config: S3StorageConfig =
serde_yaml::from_str(s3_storage_config_yaml).unwrap();

s3_storage_config.apply_flavor();
assert_eq!(s3_storage_config.flavor, Some(StorageBackendFlavor::Gcs));
assert!(!s3_storage_config.disable_checksums);
assert_eq!(s3_storage_config.checksum_algorithm, ChecksumAlgorithm::Md5);
}
{
let s3_storage_config_yaml = r#"
flavor: minio
"#;
let s3_storage_config: S3StorageConfig =
let mut s3_storage_config: S3StorageConfig =
serde_yaml::from_str(s3_storage_config_yaml).unwrap();

s3_storage_config.apply_flavor();
assert_eq!(s3_storage_config.flavor, Some(StorageBackendFlavor::MinIO));
assert!(!s3_storage_config.disable_checksums);
assert_eq!(
s3_storage_config.checksum_algorithm,
ChecksumAlgorithm::Crc32c
);
}
}
}
Loading