Skip to content

feat: move from two-track to three-track container build #115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wdconinc merged 5 commits into from
Jan 5, 2026
Merged

feat: move from two-track to three-track container build #115

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c20e26e
feat: move from two-track to three-track container build
wdconinc Dec 30, 2025
741e5de
fix: improve comments
wdconinc Dec 30, 2025
6c36e30
docs: update architecture diagram for three-track build structure (#116)
Copilot Dec 30, 2025
6411d0e
fix: grammar
wdconinc Dec 30, 2025
b04090b
fix: eic_cuda: build target builder_concretization_custom
wdconinc Dec 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build-push.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -406,7 +406,7 @@ jobs:
arch: amd64
runner: ubuntu-latest
PLATFORM: linux/amd64
target: builder_concretization_default
target: builder_concretization_custom
fail-fast: false
steps:
- name: Free Disk Space (Ubuntu)
Expand Down
71 changes: 27 additions & 44 deletions containers/eic/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ ARG RUNTIME_IMAGE="debian_stable_base"
ARG INTERNAL_TAG="master"

##
## This docker build follows two tracks, in order to ensure that we build all packages
## This docker build follows three tracks, in order to ensure that we build all packages
## in a builder image, but install them in a runtime image, while at the same time
## avoiding a expensive filesystem copy operation at the end that breaks layering.
## avoiding an expensive filesystem copy operation at the end that breaks layering.
##
## The build is split in an infrequently-changing default environment, upon which
## an environment with custom versions (e.g. individual commits) is layered. The
Expand All @@ -18,15 +18,20 @@ ARG INTERNAL_TAG="master"
## The separation in a builder and runtime image is particularly relevant to end up with
## lightweight images for expensive build dependencies, such as for example CUDA.
##
## builder track runtime track
## ----------------------------------------------------------------------
## builder track: runtime track:
## concretization: installation: concretization/installation:
## ---------------------------------------------------------------------------------------
## builder_image runtime_image
## builder_concretization_default
## builder_installation_default -> runtime_concretization_default (copy spack.lock)
## \-> runtime_installation_default (from buildcache)
## builder_concretization_custom
## builder_installation_custom -> runtime_concretization_custom (copy spack.lock)
## \-> runtime_installation_custom (from buildcache)
## \-> builder_installation_default
## runtime_default
## (copy spack.lock from builder_installation_default)
## (install via buildcache)
## \-> builder_concretization_custom
## \-> builder_installation_custom
## \-> runtime_custom
## (copy spack.lock from builder_installation_custom)
## (install via buildcache)
##


Expand Down Expand Up @@ -105,14 +110,14 @@ EOF


## ========================================================================================
## runtime_concretization_default
## - runtime base with concretization of default versions (taken from equivalent builder)
## runtime_default
## - runtime base with installation of default versions (buildcache populated by builder)
## ========================================================================================
FROM ${DOCKER_REGISTRY}${RUNTIME_IMAGE}:${INTERNAL_TAG} AS runtime_concretization_default
FROM ${DOCKER_REGISTRY}${RUNTIME_IMAGE}:${INTERNAL_TAG} AS runtime_default
ARG TARGETPLATFORM

# Open Container Initiative labels
LABEL org.opencontainers.image.title="Electron-Ion Collider runtime concretization image (default configuration, $TARGETPLATFORM)"
LABEL org.opencontainers.image.title="Electron-Ion Collider runtime image (default configuration, $TARGETPLATFORM)"

## Copy our default environment
COPY --from=spack-environment . /opt/spack-environment/
Expand All @@ -126,17 +131,6 @@ COPY --from=builder_installation_default \
/opt/spack-environment/${ENV}/spack.* \
/opt/spack-environment/${ENV}/


## ========================================================================================
## runtime_installation_default
## - runtime base with installation of default versions (buildcache populated by builder)
## ========================================================================================
FROM runtime_concretization_default AS runtime_installation_default
ARG TARGETPLATFORM

# Open Container Initiative labels
LABEL org.opencontainers.image.title="Electron-Ion Collider runtime installation image (default configuration, $TARGETPLATFORM)"

# Installation (default environment, from buildcache)
RUN --mount=type=cache,target=/var/cache/spack \
--mount=type=secret,id=mirrors,target=/opt/spack/etc/spack/mirrors.yaml \
Expand All @@ -155,7 +149,7 @@ EOF
## builder_concretization_custom
## - builder base with concretization of custom versions
## ========================================================================================
FROM builder_installation_default AS builder_concretization_custom
FROM builder_concretization_default AS builder_concretization_custom
ARG TARGETPLATFORM

# Open Container Initiative labels
Expand Down Expand Up @@ -241,14 +235,14 @@ EOF


## ========================================================================================
## runtime_concretization_custom
## - runtime base with concretization of custom versions (taken from equivalent builder)
## runtime_custom
## - runtime base with installation of custom versions (buildcache populated by builder)
## ========================================================================================
FROM runtime_installation_default AS runtime_concretization_custom
FROM runtime_default AS runtime_custom
ARG TARGETPLATFORM

# Open Container Initiative labels
LABEL org.opencontainers.image.title="Electron-Ion Collider runtime concretization image (custom configuration, $TARGETPLATFORM)"
LABEL org.opencontainers.image.title="Electron-Ion Collider runtime image (custom configuration, $TARGETPLATFORM)"

# Set spack environment directory
ENV SPACK_ENV=/opt/spack-environment/${ENV}/epic
Expand All @@ -263,18 +257,7 @@ COPY --from=builder_installation_custom \
/opt/spack-environment/packages.yaml \
/opt/spack-environment/


## ========================================================================================
## runtime_installation_custom
## - runtime base with installation of custom versions (buildcache populated by builder)
## ========================================================================================
FROM runtime_concretization_custom AS runtime_installation_custom
ARG TARGETPLATFORM

# Open Container Initiative labels
LABEL org.opencontainers.image.title="Electron-Ion Collider runtime installation image (custom configuration, $TARGETPLATFORM)"

# Installation (default environment, from buildcache)
# Installation (custom environment, from buildcache)
RUN --mount=type=cache,target=/var/cache/spack \
--mount=type=secret,id=mirrors,target=/opt/spack/etc/spack/mirrors.yaml \
<<EOF
Expand All @@ -286,13 +269,13 @@ EOF


## ========================================================================================
## final image, based on runtime_installation_custom
## final image, based on runtime_custom
## ========================================================================================
FROM runtime_installation_custom AS final
FROM runtime_custom AS final
ARG TARGETPLATFORM

# Open Container Initiative labels
LABEL org.opencontainers.image.title="Electron-Ion Collider runtime installation image (custom configuration, $TARGETPLATFORM)"
LABEL org.opencontainers.image.title="Electron-Ion Collider runtime image (custom configuration, $TARGETPLATFORM)"

## Ensure views directories, not symlinks
RUN <<EOF
Expand Down
29 changes: 15 additions & 14 deletions docs/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,29 +4,30 @@ The EIC container infrastructure uses a multi-stage build approach with separate

## Build Strategy

The container build follows a two-track approach:
The container build follows a three-track approach:

```mermaid
flowchart TB
subgraph "Builder Track"
subgraph "Builder Concretization Track"
A[builder_image<br/>debian_stable_base] --> B[builder_concretization_default<br/>Concretize spack environment]
B --> C[builder_installation_default<br/>Build packages]
C --> D[builder_concretization_custom<br/>Concretize custom versions]
D --> E[builder_installation_custom<br/>Build custom packages]
B --> C[builder_concretization_custom<br/>Concretize custom versions]
end

subgraph "Builder Installation Track"
B --> D[builder_installation_default<br/>Build packages]
C --> E[builder_installation_custom<br/>Build custom packages]
end

subgraph "Runtime Track"
F[runtime_image<br/>debian_stable_base] --> G[runtime_concretization_default<br/>Copy spack.lock from builder]
G --> H[runtime_installation_default<br/>Install from buildcache]
H --> I[runtime_concretization_custom<br/>Copy custom spack.lock]
I --> J[runtime_installation_custom<br/>Install custom from buildcache]
J --> K[Final Image<br/>eic_ci / eic_xl]
F[runtime_image<br/>debian_stable_base] --> G[runtime_default<br/>Copy spack.lock, install from buildcache]
G --> H[runtime_custom<br/>Copy custom spack.lock, install from buildcache]
H --> K[Final Image<br/>eic_ci / eic_xl]
end

C -.->|spack.lock| G
C -.->|buildcache| H
E -.->|spack.lock| I
E -.->|buildcache| J
D -.->|spack.lock| G
D -.->|buildcache| G
E -.->|spack.lock| H
E -.->|buildcache| H
```

## Multi-Architecture Support
Expand Down
Loading