Skip to content

CI: stop rebuilding everything in every step!#8678

Open
rustyrussell wants to merge 14 commits intoElementsProject:masterfrom
rustyrussell:guilt/ci-no-rebuild
Open

CI: stop rebuilding everything in every step!#8678
rustyrussell wants to merge 14 commits intoElementsProject:masterfrom
rustyrussell:guilt/ci-no-rebuild

Conversation

@rustyrussell
Copy link
Contributor

@rustyrussell rustyrussell commented Nov 11, 2025

Our build steps are not working, in the sense we are rebuilding every time.

Changelog-None: CI only.

@rustyrussell rustyrussell added this to the v25.12 milestone Nov 11, 2025
@madelinevibes madelinevibes added the Status::Ready for Review The work has been completed and is now awaiting evaluation or approval. label Nov 12, 2025
@endothermicdev
Copy link
Collaborator

Will the make check-gen-updated job halt the rest of the matrix if it fails? My only nit is it could be a bit wasteful on unnecessary runs if it's running in parallel. Needs a Changelog entry to test.

@rustyrussell
Copy link
Contributor Author

Will the make check-gen-updated job halt the rest of the matrix if it fails? My only nit is it could be a bit wasteful on unnecessary runs if it's running in parallel. Needs a Changelog entry to test.

Indeed. I'll make the tests depend on this, so it will abort.

@rustyrussell rustyrussell force-pushed the guilt/ci-no-rebuild branch 3 times, most recently from 190cc1a to eceb7cc Compare January 28, 2026 00:22
More future proof, better for self-hosted runners.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
@rustyrussell rustyrussell force-pushed the guilt/ci-no-rebuild branch 2 times, most recently from e427d7c to db91469 Compare February 16, 2026 03:39
check-source-bolt needs devtools, and check-python needs a full build.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
It's done by setup.sh, so simply set the env var for that.  Don't try
to reinstall valgrind or call `sudo apt-get update -qq` either, since
setup.sh does that too.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Our CI didn't run the Tor tests, and when we do, they fail
(particularly tests/test_gossip.py::test_tor_port_onions and
test_static_tor_onions):

```
lightningd-2 2026-02-16T01:06:28.607Z INFO    connectd: dev_report_fds: 3 -> hsm fd
lightningd-2 2026-02-16T01:06:28.607Z INFO    connectd: dev_report_fds: 4 -> gossipd fd
lightningd-2 2026-02-16T01:06:28.607Z INFO    connectd: dev_report_fds: 5 -> listener (connection_in)
lightningd-2 2026-02-16T01:06:28.607Z INFO    connectd: dev_report_fds: 5 name IPv4 socket 127.0.0.1:46045
lightningd-2 2026-02-16T01:06:28.607Z **BROKEN** connectd: dev_report_fds: 6 open but unowned? fd mode: socket
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
No point spending 1m24 seconds for *each* different compiler variant.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Using RUST_TARGET=release is slower than RUST_TARGET=debug, *but*
the images are much bigger:

RUST_TARGET=release: 8m53s  130MB
RUST_TARGET=debug:   ???    339MB

So create our own profile which gives the best of both worlds.

RUST_TARGET=small:   ???    132MB

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Make cdump-enumstr use libccan.a, rather than explicit ccan object: it's the
last one.

And mark submodcheck phony, as it is.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This means we don't have to include them in our bundle.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This keeps it simpler for future expansion, and also means we won't
rebuild them as much.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Here are the speeds and sizes of different compressors:

gzip -1:  0m20.914s	789012066
gzip -2:  0m23.846s	775576167
gzip -3:  0m26.088s	765520181
gzip -4:  0m28.264s	748485795
gzip -5:  0m35.916s	736567221
gzip -6:  0m49.906s	730337022
gzip -7:  1m2.391s	728726884
bzip2:    1m32.191s	711695852
gzip -8:  1m51.655s	727303811
gzip -9:  2m51.123s	726946796

Uncompressing time:

bzip2:			0m43.353s
gzip -5:		0m8.318s

The downside is that this is only useful on a *freshly built* tree: it will include
everything config.vars and newer.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…builds.

For CI we want to make sure we don't rebuild.  CC=false doesn't work because
we reply on the output of "$CC -dumpmachine" for the external build directory.

So we would use:

	make CC=devtools/cc-nobuild CARGO=false

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
@rustyrussell rustyrussell force-pushed the guilt/ci-no-rebuild branch 2 times, most recently from b67eba7 to ee8481e Compare February 16, 2026 04:45
We use the same fragment everywhere for consistency, even though some
stages don't actually call make.

The magic is:
 CARGO=false CC=devtools/cc-nobuild SUPPRESS_GENERATION=1

Which causes us to fail if we want to rebuild.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…rce step.

As pointed out by Alex Myers: we don't want to waste time running the
complex steps if the generated files are not up-to-date.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Status::Ready for Review The work has been completed and is now awaiting evaluation or approval.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants