Improve build system and CI/CD pipeline#140
Open
Mes0903 wants to merge 4 commits into
Open
Conversation
Replace 'scripts/build-image.sh' with a source-build path organized around artifact classes. 'scripts/build-artifacts.sh' becomes the local user interface, while the actual source recipes live under 'scripts/prebuilt' and can be shared by local builds and CI. The local CLI owns artifact selection and local cleanup policy. It parses requests such as 'image', 'rootfs', 'test-tools', and 'all', handles flags such as '--clean-build' and '--full-rebuild', and decides whether to derive 'ext4.img'. 'scripts/prebuilt/artifact-recipes.sh' owns the Buildroot and Linux source operations, publishes raw artifacts, and keeps recipe-specific work out of the 'Makefile'. Default rootfs and test-tools rootfs now use separate Buildroot output directories. The default output produces 'rootfs.cpio' and the local 'ext4.img'. The test-tools output produces 'test-tools.img' with the selected payloads. This keeps default rootfs artifacts stable when the test-tools recipe adds a larger filesystem payload. The test-tools recipe is explicit. 'scripts/prebuilt/test-tools-recipe.sh' normalizes selections such as 'x11' and 'directfb2', and 'scripts/build-artifacts.sh' accepts flags like '--x11' and '--directfb2-test' directly. DirectFB2 and X11 are independent recipe entries. Local Make keeps the file-target model for guest artifacts. 'Image', 'rootfs.cpio', and 'test-tools.img' are ordinary targets from 'mk/external.mk', and missing files are fetched through the shared release download helper. Source rebuilds are requested through 'make build-artifacts', so changing a guest config means the developer explicitly asks the source-build path to refresh local artifacts. Cleanup semantics are made explicit as well. Generated stamp files move under '.stamps', 'make clean' removes Buildroot output directories, 'distclean' removes external source trees, '--clean-build' refreshes selected outputs while keeping source checkouts and download cache, and '--full-rebuild' starts from freshly cloned pinned revisions. This keeps the fast local Buildroot-cache workflow available while still giving developers a complete rebuild path when they need one. Move recipe pins into 'artifact-recipe.env' and rename the Meson RISC-V cross file to make its role clear. CI recipe keys own release freshness; local Make is responsible for existing files, missing-file downloads, and explicit source rebuilds.
Replace the old PR drift job and standalone prebuilt workflow with a resolver/materializer model inside the main workflow. The new flow treats guest artifacts as a planned input to the test jobs: first decide where every class should come from, then materialize exactly that plan before tests or publishing consume it. The resolver computes the current recipe key for each registered artifact class from '.ci/prebuilt/artifact-inputs.sh'. It compares those keys with the rolling release manifest and with any '.stamps/prebuilt-local' entries restored from the workflow-local raw cache. The output is a 'KEY=VALUE' plan containing one action per class: 'use-action-cache', 'download-release', or 'build'. The materializer executes that plan without making new policy decisions. Restored cache entries are accepted when their raw artifact and recipe-key stamp match the current recipe key. Release-backed entries are downloaded through the shared transport helper. Build entries are passed to '.ci/prebuilt/build-plan-artifacts.sh', which calls the same 'scripts/prebuilt' source recipes used by local 'make build-artifacts'. The workflow-local cache stores raw artifacts plus '.stamps/prebuilt-local'. The rolling release stores compressed '.bz2' artifacts plus 'prebuilt.sha1'. CI uses raw artifacts for fast same-workflow or nearby-workflow reuse, and publishes a compact rolling release for local Make and test jobs that consume prebuilt artifacts. Move package and verification work into '.ci/prebuilt/package.sh' and '.ci/prebuilt/verify-package.sh'. 'guest-artifact-build' uploads raw artifacts for test jobs and packaged artifacts for 'publish-prebuilt'. The publish job runs on master after the required tests have exercised the same materialized artifacts and the resolver has reported that the rolling release needs an update. The workflow layout is cleaned up at the same time. Device smoke scripts move under '.ci/device-smoke', GitHub output and formatting helpers move under '.ci/github', and 'append-github-output.sh' validates the requested outputs before appending them. Publishing follows the main workflow's resolved artifact plan, so the old '.ci/publish-prebuilt.sh' and '.github/workflows/prebuilt.yml' are removed. Add '.github/actions/provide-guest-artifacts' so Linux and macOS test jobs consume artifacts through the same contract. Test jobs either download a current rolling release or download the raw artifact bundle from the current workflow run. 'PREBUILT_FORBID_BUILD' keeps test jobs from compiling guest artifacts after the build job has already made the release/cache/build decision.
Extend the prebuilt release manifest so it records transport integrity as well as source recipe state. 'prebuilt.sha1' now contains sha1sum entries for each compressed artifact, followed by the virtual '*.recipe-key' entries used by the resolver. The two kinds of entries answer different questions and are intentionally kept in the same manifest. Recipe keys decide freshness. The resolver compares the current class recipe key with the release recipe key to decide whether the rolling release matches the current checkout. Archive checksums verify that the downloaded '.bz2' bytes match the manifest the resolver and materializer are using. Update '.ci/prebuilt/package.sh' to write archive checksum entries for every class output before writing the recipe-key entries. Update '.ci/prebuilt/verify-package.sh' to require both entry types and to recompute each compressed archive checksum before publish. This catches packaging mistakes before the rolling release is updated. Make 'scripts/prebuilt/download-release-artifact.sh' fetch 'prebuilt.sha1' before downloading an artifact. The helper verifies the archive checksum before decompression and can also compare the manifest recipe key with an expected key supplied by CI. Local Make gets byte verification for downloaded release artifacts, and CI gets an extra guard against release-manifest or asset drift between resolve and materialize. Pass the expected class recipe key from '.ci/prebuilt/materialize-artifacts.sh' when materializing a 'download-release' action. After the helper verifies the archive and recipe key, the materializer decompresses the raw artifact and stamps '.stamps/prebuilt-local' with the current recipe key. A mismatch fails before stale bytes are marked as current. Update '.ci/prebuilt/resolve-artifacts.sh' so release recipe keys are accepted from manifests that contain archive checksum entries for the class outputs. A manifest that lacks those entries is outside the current release contract and resolves as stale, which drives CI toward cache or source build. Remove the unused 'release_manifest_available' output from the resolver and the materializer's ignored-key list. Also document the bootstrap boundary in code: forks and mirrors may bootstrap from source when 'prebuilt.sha1' returns HTTP 404, while network failures and other HTTP errors remain fatal so external release-state problems stay visible.
Add 'docs/build-system-and-ci.md' as the build-system and CI overview. The document follows the execution flow through local Make file targets, explicit source builds, the shared artifact class contract, CI recipe keys, resolver decisions, materialization, packaging, testing, publishing, and fork/mirror bootstrap behavior. Update 'README.md' to match the same model. The first-run instructions describe missing guest artifacts as release downloads, local downloads verify the archive checksum in 'prebuilt.sha1', and source rebuilds stay explicit through 'make build-artifacts'. The artifact-build section describes local Make behavior and CI recipe keys as separate responsibilities. Update 'docs/networking.md' after moving device smoke scripts under '.ci/device-smoke'.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review completed
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Re-trigger cubic
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improve build system and CI/CD pipeline
This pull request restructures the guest artifact build flow used by semu.
The main reason for this change is the ordering problem in the old CI/CD flow. When a branch was merged into
master, the device tests could request the rolling release prebuilt artifacts before the new prebuilt release had been built and published. That meant the tests could boot with stale release artifacts even though the current checkout required new guest artifacts, so the merge-time CI/CD path could fail for reasons unrelated to the emulator changes being tested.The old PR flow also compared guest artifact inputs against the
masterbranch. When a PR changed a guest config, every later push to that PR could trigger another source rebuild, even if the new commit did not touch any guest config or recipe input. This made PR iteration expensive because the decision was tied to branch drift instead of the exact artifact recipe required by the current checkout.The old flow also made the build and test pipeline depend heavily on GitHub-specific behavior. Cache restore, workflow artifacts, release downloads, publish decisions, and test setup were coupled together in the workflow. This PR redraws that architecture around clearer layers: source-build recipes, artifact resolution, artifact materialization, packaging/publishing, and test consumption. GitHub Actions remains the current execution environment, but the GitHub-specific pieces are now adapters around those layers rather than the place where the artifact policy lives.
Local development stays close to Make’s file-target model. Missing guest artifacts can still be downloaded from the rolling prebuilt release, while source rebuilds are explicit through
make build-artifacts. This keeps the normal local path predictable and preserves the Buildroot cache workflow developers rely on when iterating on guest configuration.CI owns the artifact decision model. The workflow computes recipe keys for the current checkout, compares them with restored workflow-local raw artifacts and the rolling release manifest, then materializes a plan as cache reuse, release download, or source build. The resolver decides where artifacts should come from, and the materializer performs that plan.
The rolling release contract is also tightened. Recipe keys continue to decide whether a release artifact matches the current source recipe, while archive checksums verify that downloaded
.bz2bytes match the published manifest before decompression and stamping. These checks cover different failure modes, so the release manifest now records both archive checksum entries and virtual recipe-key entries.Local build flow
make build-artifactsreplaces the old image-oriented source build entrypoint with an artifact-class based flow. The local CLI selects which classes to build, handles cleanup policy, and delegates the actual source recipes to shared helpers underscripts/prebuilt/.The default rootfs and test-tools rootfs now use separate Buildroot output directories. This keeps optional test payloads such as X11 and DirectFB2 from changing the default
rootfs.cpioand derivedext4.imgpath.ext4.imgremains a local derived artifact built fromrootfs.cpio.Test-tools payload selection is explicit. Options such as
--x11and--directfb2-testare passed directly to the build command, and DirectFB2 no longer implicitly selects X11.CI artifact flow
CI is split into resolution and materialization. The resolver reads the current recipe inputs, restored cache stamps, and release manifest, then emits a plan for each artifact class. The materializer executes that plan without making new policy decisions.
Workflow-local raw artifacts are tracked with
.stamps/prebuilt-local/*.recipe-key. Those stamps allow restored CI cache entries to be checked against the current checkout before reuse. Test jobs consume either the raw artifact bundle produced by the current workflow run or the rolling prebuilt release through the same composite action.Publishing is part of the main workflow. After the required tests run against the same materialized artifact set, the publish job verifies the packaged artifacts and updates the rolling
prebuiltrelease when the resolver reports release drift.Release manifest
prebuilt.sha1now records both compressed archive checksums and recipe-key entries. The archive checksum entries verify transport integrity forImage.bz2,rootfs.cpio.bz2, andtest-tools.img.bz2. The recipe-key entries describe the source recipe that produced each artifact class.Local release downloads verify archive bytes before decompression. CI release materialization additionally verifies that the manifest recipe key still matches the resolver plan before stamping the raw artifact as current.
Fork and mirror repositories can bootstrap from source when the rolling release manifest returns HTTP 404. Other release download failures remain visible as release or network state problems.
Documentation
The README now describes missing guest artifacts as release downloads and source guest rebuilds as an explicit
make build-artifactsoperation.docs/build-system-and-ci.mddocuments the local build path, CI artifact contract, resolver/materializer model, package and publish flow, and fork/mirror bootstrap behavior.The networking document is updated to point at the moved device smoke script path.
Summary by cubic
Overhauled the guest artifact build and CI/CD flow to use a recipe-keyed resolver/materializer, preventing stale prebuilts in tests and making PRs faster and more reliable.
Refactors
.stamps/prebuilt-local.*.recipe-keyinprebuilt.sha1; downloads verify bytes before decompressing.make build-artifactsusingscripts/prebuilt/*; default Make targets continue to download prebuilts. Rootfs and test-tools now use separate Buildroot outputs. Test-tools payloads are explicit (--x11,--directfb2-test)../.github/actions/provide-guest-artifactsso tests consume either workflow raw artifacts or the rolling release through a single path. Removed the standaloneprebuiltworkflow.Migration
make checkto download verified prebuilts; rebuild from source withmake build-artifacts [image|rootfs|test-tools]and optional--x11/--directfb2-test, plus--clean-buildor--full-rebuildas needed.prebuiltrelease publishes from the main workflow onmasterafter tests; no manual publish job. CI concurrency no longer cancels publishing runs onmaster.PREBUILT_URLdefaults to the current repository; CI can bootstrap from source when the release manifest returns 404.Written for commit 80f204d. Summary will update on new commits.