Skip to content

chore: Bump stackable-operator to get updated s3 region structs #696

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
NickLarsenNZ wants to merge 4 commits into from
Closed

chore: Bump stackable-operator to get updated s3 region structs #696

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
57b8797
chore: Bump stackable-operator to get updated s3 region structs
NickLarsenNZ Mar 11, 2025
77b8f1a
chore: Add S3 bucket region to config
NickLarsenNZ Mar 11, 2025
6d0358f
chore: Update changelog
NickLarsenNZ Mar 11, 2025
4dde62e
chore: Update changelog
NickLarsenNZ Mar 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ All notable changes to this project will be documented in this file.
- Support Apache Druid `31.0.1` and `30.0.1`, remove `26.0.0` ([#685]).
- BREAKING: Adjust default memory limits of coordinator from `512Mi` to `768Mi` and middlemanager from `1Gi` to `1500Mi` ([#685]).
- Support configuring JVM arguments ([#693]).
- BREAKING: Add s3 bucket region to config ([#696]).

### Changed

Expand All @@ -24,6 +25,7 @@ All notable changes to this project will be documented in this file.
[#677]: https://github.com/stackabletech/druid-operator/pull/677
[#693]: https://github.com/stackabletech/druid-operator/pull/693
[#685]: https://github.com/stackabletech/druid-operator/pull/685
[#696]: https://github.com/stackabletech/druid-operator/pull/696

## [24.11.1] - 2025-01-09

Expand Down
16 changes: 8 additions & 8 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

38 changes: 16 additions & 22 deletions Cargo.nix
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 3 additions & 5 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,8 @@ edition = "2021"
repository = "https://github.com/stackabletech/druid-operator"

[workspace.dependencies]
stackable-versioned = { git = "https://github.com/stackabletech/operator-rs.git", features = [
"k8s",
], tag = "stackable-versioned-0.5.0" }
stackable-operator = { git = "https://github.com/stackabletech/operator-rs.git", tag = "stackable-operator-0.85.0" }
stackable-versioned = { git = "https://github.com/stackabletech/operator-rs.git", features = ["k8s"], tag = "stackable-versioned-0.5.0" }
stackable-operator = { git = "https://github.com/stackabletech/operator-rs.git", tag = "stackable-operator-0.87.2" }
product-config = { git = "https://github.com/stackabletech/product-config.git", tag = "0.7.0" }

anyhow = "1.0"
Expand All @@ -31,7 +29,7 @@ serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
serde_yaml = "0.9"
snafu = "0.8"
strum = { version = "0.26", features = ["derive"] }
strum = { version = "0.27", features = ["derive"] }
tokio = { version = "1.40", features = ["full"] }
tracing = "0.1"

Expand Down
6 changes: 3 additions & 3 deletions crate-hashes.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

28 changes: 28 additions & 0 deletions deploy/helm/druid-operator/crds/crds.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -665,6 +665,20 @@ spec:
minimum: 0.0
nullable: true
type: integer
region:
default:
name: us-east-1
description: |-
Bucket region used for signing headers (sigv4).

This defaults to `us-east-1` which is compatible with other implementations such as Minio.

WARNING: Some products use the Hadoop S3 implementation which falls back to us-east-2.
properties:
name:
default: us-east-1
type: string
type: object
tls:
description: Use a TLS connection. If not specified no TLS will be used.
nullable: true
Expand Down Expand Up @@ -795,6 +809,20 @@ spec:
minimum: 0.0
nullable: true
type: integer
region:
default:
name: us-east-1
description: |-
Bucket region used for signing headers (sigv4).

This defaults to `us-east-1` which is compatible with other implementations such as Minio.

WARNING: Some products use the Hadoop S3 implementation which falls back to us-east-2.
properties:
name:
default: us-east-1
type: string
type: object
tls:
description: Use a TLS connection. If not specified no TLS will be used.
nullable: true
Expand Down
2 changes: 2 additions & 0 deletions rust/operator-binary/src/crd/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ pub const DS_DIRECTORY: &str = "druid.storage.storageDirectory";
pub const DS_BUCKET: &str = "druid.storage.bucket";
pub const DS_BASE_KEY: &str = "druid.storage.baseKey";
pub const S3_ENDPOINT_URL: &str = "druid.s3.endpoint.url";
pub const S3_REGION_NAME: &str = "aws.region";
pub const S3_ACCESS_KEY: &str = "druid.s3.accessKey";
pub const S3_SECRET_KEY: &str = "druid.s3.secretKey";
pub const S3_PATH_STYLE_ACCESS: &str = "druid.s3.enablePathStyleAccess";
Expand Down Expand Up @@ -389,6 +390,7 @@ impl v1alpha1::DruidCluster {
Ok(result)
}

#[allow(clippy::type_complexity)]
pub fn build_role_properties(
&self,
) -> HashMap<
Expand Down
5 changes: 4 additions & 1 deletion rust/operator-binary/src/druid_controller.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,8 @@ use crate::{
DRUID_CONFIG_DIRECTORY, DS_BUCKET, EXTENSIONS_LOADLIST, HDFS_CONFIG_DIRECTORY, JVM_CONFIG,
JVM_SECURITY_PROPERTIES_FILE, LOG_CONFIG_DIRECTORY, MAX_DRUID_LOG_FILES_SIZE,
OPERATOR_NAME, RUNTIME_PROPS, RW_CONFIG_DIRECTORY, S3_ACCESS_KEY, S3_ENDPOINT_URL,
S3_PATH_STYLE_ACCESS, S3_SECRET_KEY, STACKABLE_LOG_DIR, ZOOKEEPER_CONNECTION_STRING,
S3_PATH_STYLE_ACCESS, S3_REGION_NAME, S3_SECRET_KEY, STACKABLE_LOG_DIR,
ZOOKEEPER_CONNECTION_STRING,
},
discovery::{self, build_discovery_configmaps},
extensions::get_extension_list,
Expand Down Expand Up @@ -734,6 +735,8 @@ fn build_rolegroup_config_map(
};

if let Some(s3) = s3_conn {
conf.insert(S3_REGION_NAME.to_string(), Some(s3.region.name.to_string()));
Copy link
Member

@sbernauer sbernauer Mar 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think according to https://druid.apache.org/docs/latest/development/extensions-core/s3/#aws-region aws.region needs to go into the JVM args, not the Druid config.
Via env var AWS_REGION sounds even easier to me

Copy link
Member Author

@NickLarsenNZ NickLarsenNZ Mar 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does, but that also seems strange to be since you might want to connect to buckets in multiple regions.

The S3 extension needs testing with AWS S3 buckets:

  • Deep storage
  • S3 Ingestion

I'm just trying to get a working baseline.


conf.insert(
S3_ENDPOINT_URL.to_string(),
Some(s3.endpoint().context(ConfigureS3Snafu)?.to_string()),
Expand Down