Skip to content

skip full wipe for --binary-path with source repos#6561

Draft
p-datadog wants to merge 1 commit intofix/binary-path-source-reposfrom
fix/binary-path-incremental-sync
Draft

skip full wipe for --binary-path with source repos#6561
p-datadog wants to merge 1 commit intofix/binary-path-source-reposfrom
fix/binary-path-incremental-sync

Conversation

@p-datadog
Copy link
Copy Markdown
Member

Summary

Builds on #6560. This is a small follow-up that makes repeated build.sh invocations faster when using --binary-path with a local source repo.

Today, every build.sh call with --binary-path runs clean-binaries first — wiping everything in binaries/ — then copies the source repo back in. For a repo like dd-trace-rb (~85MB), that's a full delete and full re-copy on every invocation, even when nothing in the repo has changed.

This moves clean-binaries into the flat-copy path only (artifact-based tracers like Java, .NET, PHP). For the source-repo path, rsync --delete handles keeping the destination in sync incrementally — it only transfers files that actually changed and removes files that no longer exist in the source. When nothing changed, rsync does nothing.

What changed

Depends on #6560 — please review that first.

-clean-binaries                    # always wipe everything
 if source-repo detected; then
-    rsync ... "$BINARY_PATH/" "$expected_repo_dir/"
+    rsync ... --delete "$BINARY_PATH/" "$expected_repo_dir/"   # incremental sync
 else
+    clean-binaries                 # wipe only for flat artifacts
     cp -r "$BINARY_PATH"/* ./
 fi

This is the same pattern that utils/scripts/watch.sh uses — rsync with delete for incremental sync into binaries/.

What it feels like in practice

First run — same as before, full copy:

$ ./build.sh ruby --binary-path ~/code/dd-trace-rb
Copying source repo into binaries/dd-trace-rb/
# ~85MB transferred

Second run, nothing changed — rsync skips everything:

$ ./build.sh ruby --binary-path ~/code/dd-trace-rb
Copying source repo into binaries/dd-trace-rb/
# nothing transferred, finishes instantly

Second run, one file changed — rsync transfers just that file:

$ ./build.sh ruby --binary-path ~/code/dd-trace-rb
Copying source repo into binaries/dd-trace-rb/
# only the changed file transferred

Artifact-based workflows (Java jars, .NET tarballs, etc.) are unchanged — they still get clean-binaries before the flat copy.

Test plan

  • Verify first run with --binary-path ~/code/dd-trace-rb does a full copy
  • Verify second run with no changes transfers nothing
  • Verify second run after modifying a file only transfers that file
  • Verify artifact-based tracers still get clean-binaries before flat copy

Generated with Claude Code

For the source-repo path, rsync --delete keeps the destination in
sync incrementally — only transferring changed files and removing
stale ones. This makes repeated build.sh invocations with the same
--binary-path skip the copy when nothing changed, instead of doing
a full wipe and re-copy of ~85MB+ repos every time.

The flat-copy path (artifact-based tracers) still runs clean-binaries
as before.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

utils/build/build.sh                                                    @DataDog/system-tests-core

p-datadog pushed a commit to p-datadog/system-tests that referenced this pull request Mar 20, 2026
Reverts:
- 1f95bf3 selected test detection fixed
- 69e93c4 terminate self more efficiently
- 100a53d exit faster when nothing is collected
- 848db3c use rsync instead of cp
- 42f9453 use dtr

These are replaced by:
- DataDog#6558 (fail fast when no tests are selected)
- DataDog#6560 (fix --binary-path for source repos)
- DataDog#6561 (skip full wipe for --binary-path)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
p-datadog pushed a commit to p-datadog/system-tests that referenced this pull request Mar 20, 2026
* 'fix/binary-path-incremental-sync' of https://github.com/DataDog/system-tests:
  skip clean-binaries for source repos, use rsync --delete instead
@datadog-prod-us1-3
Copy link
Copy Markdown

datadog-prod-us1-3 bot commented Mar 20, 2026

⚠️ Tests

Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 166 Tests failed

tests.appsec.test_asm_standalone.Test_SCAStandalone_Telemetry_V2.test_telemetry_sca_enabled_propagated[akka-http] from system_tests_suite (Datadog) (Fix with Cursor)
AssertionError: Missing telemetry config item for 'appsec_sca_enabled'
assert None is not None

self = <tests.appsec.test_asm_standalone.Test_SCAStandalone_Telemetry_V2 object at 0x7fe43dfc6660>

    def test_telemetry_sca_enabled_propagated(self):
        self.assert_standalone_is_enabled(self.r0, self.r1)
    
        configuration_by_name: dict[str, list[dict]] = {}
        for data in interfaces.library.get_telemetry_data():
...
tests.appsec.test_asm_standalone.Test_SCAStandalone_Telemetry_V2.test_telemetry_sca_enabled_propagated[akka-http] from system_tests_suite (Datadog) (Fix with Cursor)
AssertionError: Missing telemetry config item for 'appsec_sca_enabled'
assert None is not None

self = <tests.appsec.test_asm_standalone.Test_SCAStandalone_Telemetry_V2 object at 0x7f45e08066f0>

    def test_telemetry_sca_enabled_propagated(self):
        self.assert_standalone_is_enabled(self.r0, self.r1)
    
        configuration_by_name: dict[str, list[dict]] = {}
        for data in interfaces.library.get_telemetry_data():
...
tests.appsec.test_asm_standalone.Test_SCAStandalone_Telemetry_V2.test_telemetry_sca_enabled_propagated[jersey-grizzly2] from system_tests_suite (Datadog) (Fix with Cursor)
AssertionError: Missing telemetry config item for 'appsec_sca_enabled'
assert None is not None

self = <tests.appsec.test_asm_standalone.Test_SCAStandalone_Telemetry_V2 object at 0x7f79008213a0>

    def test_telemetry_sca_enabled_propagated(self):
        self.assert_standalone_is_enabled(self.r0, self.r1)
    
        configuration_by_name: dict[str, list[dict]] = {}
        for data in interfaces.library.get_telemetry_data():
...
View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ea5c9b5 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant