ci: fix sccache C++ caching on Windows for Python builds#174
Merged
cpsievert merged 7 commits intoposit-dev:mainfrom Mar 6, 2026
Merged
ci: fix sccache C++ caching on Windows for Python builds#174cpsievert merged 7 commits intoposit-dev:mainfrom
cpsievert merged 7 commits intoposit-dev:mainfrom
Conversation
Split the Python CI workflow into build and test phases. The build job compiles the abi3 wheel once per OS, then test jobs download and install it across Python 3.10-3.13. This avoids redundant Rust compilation — particularly on Windows where builds take ~30 min each. Also removes path filters so Python tests run on all repo changes, since the build overhead is now minimal. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This reverts commit d8b29c5.
sccache sees zero C/C++ compilations on Windows because the cc crate doesn't auto-detect sccache as a compiler wrapper with MSVC. Setting CC_WRAPPER=sccache explicitly routes cl.exe calls through sccache, enabling caching of DuckDB and other C/C++ dependency builds. The first run will still be slow (all cache misses), but subsequent runs should benefit from ~300 cached C/C++ compilation results. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CC_WRAPPER didn't work because the cc crate doesn't check that env var. The real issue: when the cc crate finds MSVC cl.exe via find_msvc_tools, it bypasses the RUSTC_WRAPPER-based sccache fallback that works on Linux. Setting CC/CXX='sccache cl' on Windows makes the cc crate's env_tool parser recognize "sccache" as a known wrapper and "cl" as the compiler. This routes the ~21-minute libduckdb-sys C++ build through sccache. First run will still be slow (cache seeding), but subsequent runs should see C/C++ cache hits and significantly faster Windows builds. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The previous attempt failed because sccache couldn't find bare 'cl' - MSVC tools aren't on PATH by default. ilammy/msvc-dev-cmd runs vcvarsall to set up the MSVC environment, putting cl.exe on PATH so that 'sccache cl' works as a CC/CXX wrapper. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…indows Remove the build/test split since sccache should make Windows builds fast enough (~5 min) that compiling per Python version is acceptable. The diff from main is now minimal: remove path filters, add MSVC setup + CC/CXX env vars for sccache C++ caching on Windows. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ilammy/msvc-dev-cmdand settingCC/CXX='sccache cl'Problem
Windows Python builds take ~30 minutes vs ~7 min on other platforms. Investigation of sccache stats revealed that Windows gets zero C/C++ cache hits while Ubuntu gets ~350:
The root cause: the
cccrate's MSVCcl.exedetection (viafind_msvc_tools) bypasses theRUSTC_WRAPPER-based sccache fallback that works automatically on Linux/macOS. Thelibduckdb-sysC++ build (~350 source files) was compiling from scratch every run.Fix
Two additions to the workflow:
ilammy/msvc-dev-cmd— Runsvcvarsall.batto putcl.exeon PATHCC/CXX='sccache cl'(Windows only) — Makes thecccrate'senv_toolparser recognizesccacheas a known wrapper andclas the compilerResults
After the sccache cache was seeded (first run: 350 C/C++ misses, 0 write errors), the next run showed:
The Rust sccache is still warming up (55% hit rate), so further improvement is expected on subsequent runs as more Rust compilation results are cached.
Path filter removal
Previously the Python workflow only ran on changes to
ggsql-python/and the workflow file itself. Since the Python bindings depend on the core Rust library, changes elsewhere (parser, writer, reader) could break them silently. With Windows builds cut in half (~15 min and improving), running Python tests on all changes is now practical.Test plan
🤖 Generated with Claude Code