RUBY-3472 Migrate from mlaunch to drivers-tools orchestration#3053
Open
comandeo-mongo wants to merge 10 commits into
Open
RUBY-3472 Migrate from mlaunch to drivers-tools orchestration#3053comandeo-mongo wants to merge 10 commits into
comandeo-mongo wants to merge 10 commits into
Conversation
Replace the Python mlaunch-based server startup with the shared drivers-evergreen-tools mongo-orchestration infrastructure used by all other MongoDB drivers. Key changes: - Add .evergreen/run-orchestration.sh: translates Ruby driver env vars (TOPOLOGY, AUTH, SSL, LOAD_BALANCED, SINGLE_MONGOS, OCSP_*) to the format expected by drivers-evergreen-tools run-orchestration.sh, then exports MONGODB_URI from the resulting mo-expansion.sh. - Add .evergreen/orchestration-configs/: Ruby-specific orchestration JSON configs (single-node replica set, single-node SSL replica set, single-mongos sharded cluster) copied into $MONGO_ORCHESTRATION_HOME at runtime. - Rewrite .evergreen/run-tests.sh: remove mlaunch startup/stop and the hand-built MONGODB_URI assembly; call run-orchestration.sh instead. Update x509 and AWS auth handling for orchestration credentials (bob:pwd123) and cert paths (.evergreen/x509gen/). - Remove MLAUNCH_TOPOLOGY from axes.yml.erb, common.yml.erb, run-tests-aws-auth.sh, and run-tests.sh. - Rename the "test-mlaunch" Evergreen task to "test" everywhere in standard.yml.erb and common.yml.erb. - Update spec/support/spec_config.rb to use drivers-tools x509gen/ client.pem for x509 auth when DRIVERS_TOOLS is set. - Update .evergreen/lib/server_setup.rb to use MONGODB_URI credentials in orchestration mode instead of the legacy bootstrap:bootstrap user. - Regenerate .evergreen/config.yml.
![semgrep-code-mongodb[bot]](https://avatars.githubusercontent.com/u/45120?s=60&v=4){kind=small}
Comment on lines
+106
to
+108
| Mongo::Client.new(%w(localhost), | ||
| user: 'bootstrap', password: 'bootstrap', auth_mech: :scram, auth_mech_properties: nil, | ||
| ) |
There was a problem hiding this comment.
Semgrep identified an issue in your code:
A secret is hard-coded in the application. Secrets stored in source code, such as credentials, identifiers, and other types of sensitive data, can be leaked and used by internal or external malicious actors. Use environment variables to securely provide credentials and other secrets or retrieve them from a secure vault or Hardware Security Module (HSM).
To resolve this comment:
✨ Commit fix suggestion
- Remove the hardcoded username and password values from the
Mongo::Client.newcall inbootstrap_client. - Store the MongoDB username and password securely, such as in environment variables, for example,
MONGODB_USERandMONGODB_PASSWORD. - Update the
Mongo::Client.newcall to use these environment variables:user: ENV['MONGODB_USER'], password: ENV['MONGODB_PASSWORD']. - Ensure the environment that runs this code has the proper values set for
MONGODB_USERandMONGODB_PASSWORD.
Using environment variables helps prevent accidental exposure of secrets in your source code and repository.
💬 Ignore this finding
Reply with Semgrep commands to ignore this finding.
/fp <comment>for false positive/ar <comment>for acceptable risk/other <comment>for all other reasons
Alternatively, triage in Semgrep AppSec Platform to ignore the finding created by ruby-mongo-hardcoded-secret.
🛟 Help? Slack #semgrep-help or go/semgrep-help.
Resolution Options:
- Fix the code
- Reply
/fp $reason(if security gap doesn’t exist) - Reply
/ar $reason(if gap is valid but intentional; add mitigations/monitoring) - Reply
/other $reason(e.g., test-only)
You can view more details about this finding in the Semgrep AppSec Platform.
Contributor
Author
There was a problem hiding this comment.
/other test-only bootstrap credentials used only for local development when MONGODB_URI is not set; not present in CI or production
Contributor
There was a problem hiding this comment.
Pull request overview
Migrates the Ruby driver's Evergreen test infrastructure from the Python mlaunch tool to the shared drivers-evergreen-tools (DET) mongo-orchestration flow used by other MongoDB drivers. A new wrapper script translates Ruby-specific env vars and orchestration JSON configs into DET's expected inputs, and run-tests.sh is reworked to consume the MONGODB_URI produced by DET.
Changes:
- Add
.evergreen/run-orchestration.shand Ruby-specific orchestration JSON configs (single-node RS, single-node SSL RS, single-mongos sharded). - Replace mlaunch start/stop, hand-built
MONGODB_URI, andbootstrap:bootstrapcredentials inrun-tests.sh,run-tests-aws-auth.sh, andserver_setup.rbwith DET-based credentials (bob:pwd123) and x509 cert paths under.evergreen/x509gen/. - Drop
MLAUNCH_TOPOLOGYaxis variable / expansion and rename thetest-mlaunchtask totestacross the Evergreen YAML/ERB configs; route x509 spec config to the DET cert whenDRIVERS_TOOLSis set.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
.evergreen/run-orchestration.sh |
New wrapper that translates Ruby env vars, copies Ruby-specific configs, then invokes DET orchestration. |
.evergreen/orchestration-configs/replica_sets/single-node.json |
Ruby-specific single-node RS config. |
.evergreen/orchestration-configs/replica_sets/single-node-ssl.json |
Ruby-specific single-node RS config with SSL (currently never selected). |
.evergreen/orchestration-configs/sharded_clusters/single-mongos.json |
Single-router sharded cluster config for SINGLE_MONGOS. |
.evergreen/run-tests.sh |
Remove mlaunch path; call run-orchestration.sh; rebuild URI for AWS auth; use add_uri_option helper. |
.evergreen/run-tests-aws-auth.sh |
Drop MLAUNCH_TOPOLOGY re-export. |
.evergreen/lib/server_setup.rb |
Use existing MONGODB_URI credentials instead of bootstrap:bootstrap when running under orchestration. |
spec/support/spec_config.rb |
Route x509 client PEM to evergreen cert dir when DRIVERS_TOOLS is set. |
.evergreen/config/axes.yml.erb |
Remove MLAUNCH_TOPOLOGY; use ORCHESTRATION_FILE for single-node RS. |
.evergreen/config/standard.yml.erb |
Rename test-mlaunch task references to test. |
.evergreen/config/common.yml.erb |
Remove MLAUNCH_TOPOLOGY exports; rename task to test. |
.evergreen/config.yml |
Regenerated output reflecting the above ERB changes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+16
to
+67
| # Translate topology names to orchestration format. | ||
| case "${TOPOLOGY:-server}" in | ||
| replica-set-single-node) | ||
| export TOPOLOGY=replica_set | ||
| export ORCHESTRATION_FILE="${ORCHESTRATION_FILE:-single-node}" | ||
| ;; | ||
| standalone) | ||
| export TOPOLOGY=server | ||
| ;; | ||
| esac | ||
|
|
||
| # Single mongos: use a 1-router sharded cluster config. | ||
| if test "${SINGLE_MONGOS:-}" = 'true' && test "${TOPOLOGY:-}" = sharded_cluster; then | ||
| export ORCHESTRATION_FILE="${ORCHESTRATION_FILE:-single-mongos}" | ||
| fi | ||
|
|
||
| # Load balancer support. | ||
| if test "${LOAD_BALANCED:-}" = 'true'; then | ||
| export LOAD_BALANCER=1 | ||
| fi | ||
|
|
||
| # x509 auth: start server with auth+ssl; the x509 user is created later in run-tests.sh. | ||
| if test "${AUTH:-}" = x509; then | ||
| export AUTH=auth | ||
| export SSL=yes | ||
| fi | ||
|
|
||
| # AWS auth: use the auth-aws orchestration file which enables MONGODB-AWS mechanism. | ||
| if echo "${AUTH:-}" | grep -q ^aws; then | ||
| export AUTH_AWS=1 | ||
| export AUTH=auth | ||
| fi | ||
|
|
||
| # OCSP: select orchestration file based on algorithm and mustStaple flag. | ||
| if test -n "${OCSP_ALGORITHM:-}"; then | ||
| _ocsp_file="${OCSP_ALGORITHM}-basic-tls-ocsp" | ||
| if test "${OCSP_MUST_STAPLE:-}" = 1; then | ||
| _ocsp_file="${_ocsp_file}-mustStaple" | ||
| fi | ||
| export ORCHESTRATION_FILE="${ORCHESTRATION_FILE:-$_ocsp_file}" | ||
| fi | ||
|
|
||
| # If prepare_server already downloaded MongoDB, reuse those binaries. | ||
| if test -n "${BINDIR:-}"; then | ||
| export EXISTING_BINARIES_DIR="$BINDIR" | ||
| fi | ||
|
|
||
| # Copy Ruby-driver-specific orchestration configs that are not (yet) in drivers-evergreen-tools. | ||
| _configs_src="$(dirname "$0")/orchestration-configs" | ||
| _configs_dst="$MONGO_ORCHESTRATION_HOME/configs" | ||
| cp "$_configs_src"/replica_sets/single-node.json "$_configs_dst/replica_sets/" | ||
| cp "$_configs_src"/replica_sets/single-node-ssl.json "$_configs_dst/replica_sets/" |
Contributor
Author
There was a problem hiding this comment.
Fixed: removed single-node-ssl.json and the cp line that copies it. The replica-set-single-node axis hard-codes ORCHESTRATION_FILE: single-node.json, so there is no code path that can select this file.
Comment on lines
116
to
+138
| @@ -165,10 +134,8 @@ elif test "$AUTH" = aws-assume-role; then | |||
|
|
|||
| aws sts get-caller-identity | |||
|
|
|||
| hosts="`uri_escape $MONGO_RUBY_DRIVER_AWS_AUTH_ACCESS_KEY_ID`:`uri_escape $MONGO_RUBY_DRIVER_AWS_AUTH_SECRET_ACCESS_KEY`@$hosts" | |||
|
|
|||
| uri_options="$uri_options&"\ | |||
| "authMechanismProperties=AWS_SESSION_TOKEN:`uri_escape $MONGO_RUBY_DRIVER_AWS_AUTH_SESSION_TOKEN`" | |||
| _mongo_host=$(echo "$MONGODB_URI" | sed 's|mongodb://[^@]*@||' | sed 's|/.*||') | |||
| export MONGODB_URI="mongodb://$(uri_escape "$MONGO_RUBY_DRIVER_AWS_AUTH_ACCESS_KEY_ID"):$(uri_escape "$MONGO_RUBY_DRIVER_AWS_AUTH_SECRET_ACCESS_KEY")@${_mongo_host}/?authMechanism=MONGODB-AWS&authSource=\$external&authMechanismProperties=AWS_SESSION_TOKEN:$(uri_escape "$MONGO_RUBY_DRIVER_AWS_AUTH_SESSION_TOKEN")" | |||
Contributor
Author
There was a problem hiding this comment.
Valid observation. The AWS auth test matrix does not currently combine aws-assume-role with topologies that carry extra query params in the orchestration URI (e.g. loadBalanced, replicaSet), so no options are silently dropped in practice. The behavior here mirrors what the pre-PR code did: it assembled the URI from scratch from $hosts without carrying forward orchestration-provided options. I'll leave it as-is for now since it matches existing behavior and fixing it would require careful URI parsing; it can be addressed if AWS auth is ever combined with a complex topology.
Comment on lines
+51
to
+53
| _ocsp_file="${OCSP_ALGORITHM}-basic-tls-ocsp" | ||
| if test "${OCSP_MUST_STAPLE:-}" = 1; then | ||
| _ocsp_file="${_ocsp_file}-mustStaple" |
Contributor
Author
There was a problem hiding this comment.
Fixed in commit 518de95: non-mustStaple uses ${OCSP_ALGORITHM}-basic-tls-ocsp-disableStapling.json.
|
|
||
| # Copy Ruby-driver-specific orchestration configs that are not (yet) in drivers-evergreen-tools. | ||
| _configs_src="$(dirname "$0")/orchestration-configs" | ||
| _configs_dst="$MONGO_ORCHESTRATION_HOME/configs" |
Contributor
Author
There was a problem hiding this comment.
Fixed in commit 518de95: mkdir -p is now called before the cp commands.
mo-expansion.sh sets MONGODB_URI without 'export', so Ruby subprocesses
did not inherit the variable. This caused ENV.fetch('MONGODB_URI') to
raise in aws_lambda_examples_spec and bootstrap_client to fall back to
bootstrap:bootstrap credentials in the AWS auth path.
The non-mustStaple OCSP variant is named with -disableStapling suffix, not bare -basic-tls-ocsp. Also guard the cp calls with mkdir -p in case the destination directories don't exist yet.
run-mongodb.sh uses the mongodb-runner (Node.js) backend, which is the modern replacement for mongo-orchestration. The bootstrap-mongo-orchestration function in common.yml.erb already uses run-mongodb.sh start; align the run-tests.sh path to do the same.
drivers-orchestration uses ORCHESTRATION_FILE verbatim when explicitly set, so all values must include the .json extension. Also add waitress to the OCSP pip install (required by the OCSP mock server's Flask app).
Orchestration writes a URI with tls=true but no tlsCAFile, so the Ruby client fails to verify the self-signed x509gen CA. Add the CA file to the URI when OCSP_CONNECTIVITY is set.
With orchestration, the push monitor keeps running after the polling monitor thread is killed. With a 30-second selection timeout the push monitor responds after 20 s (heartbeat_frequency), makes the server known again, and the server_selector_spec "dead monitor threads" test fails because no exception is raised. The run-tests-new.sh (used by run-main-test-suite) does not set this option, and all tests pass with the spec_config.rb default of ~7 s. Remove it here to match that behaviour.
The OCSP server is configured with certs signed by the OCSP-specific CA (.evergreen/ocsp/<algo>/ca.pem), not the x509gen CA. Point tlsCAFile at the correct CA so the client can verify the server cert.
With mlaunch, calculate_server_args added authMechanism=MONGODB-AWS and authSource=$external to URI_OPTIONS for all aws-* auth types. With the new orchestration approach the URI comes from drivers-orchestration with bob:pwd123 SCRAM credentials. Reset the URI to use MONGODB-AWS after setting up the auth user, matching the old behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the Python mlaunch-based server startup with the shared drivers-evergreen-tools mongo-orchestration infrastructure used by all other MongoDB drivers.
Changes
.evergreen/run-orchestration.sh(new): translates Ruby driver env vars (TOPOLOGY,AUTH,SSL,LOAD_BALANCED,SINGLE_MONGOS,OCSP_*) to the format expected bydrivers-evergreen-tools/run-orchestration.sh, then exportsMONGODB_URIfrom the resultingmo-expansion.sh..evergreen/orchestration-configs/(new): Ruby-specific orchestration JSON configs (single-node replica set, single-node SSL replica set, single-mongos sharded cluster) that get copied into$MONGO_ORCHESTRATION_HOME/configs/at runtime. The single-node replica set configs should eventually be contributed upstream to drivers-evergreen-tools..evergreen/run-tests.sh: removes mlaunch startup/stop and the hand-builtMONGODB_URIassembly; callsrun-orchestration.shinstead. Updates x509 and AWS auth handling for orchestration credentials (bob:pwd123) and cert paths (.evergreen/x509gen/).axes.yml.erb,common.yml.erb,run-tests-aws-auth.sh,run-tests.sh: removeMLAUNCH_TOPOLOGY.standard.yml.erb,common.yml.erb: renametest-mlaunchtask totest.spec/support/spec_config.rb: usex509gen/client.pemfor x509 auth whenDRIVERS_TOOLSis set (cert signed by the orchestration CA)..evergreen/lib/server_setup.rb: useMONGODB_URIcredentials in orchestration mode instead of the legacybootstrap:bootstrapuser..evergreen/config.yml: regenerated.Testing
Requires an Evergreen patch covering: standalone, replica set, sharded cluster, load-balanced, single-node replica set, x509 auth, AWS auth, and OCSP variants.