docker_*.sh: add kvm support, X11 authentication, update README.md instructions. Bump docker image to v.0.2.7 with pinned hash under CircleCI#2036
Conversation
There was a problem hiding this comment.
Pull request overview
This PR enhances the Docker wrapper scripts to enable KVM acceleration and X11 GUI support for QEMU board testing. Previously tested only under QubesOS, these changes allow the scripts to work with standard Linux environments running KVM-accelerated QEMU with GTK windows.
Key changes:
- Added
/dev/kvmdevice passthrough to enable KVM hardware acceleration in containers - Added X11 socket and authentication mounting to support GUI applications (GTK windows)
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 18 comments.
| File | Description |
|---|---|
| docker_repro.sh | Added KVM device passthrough and X11 authentication mounts to enable hardware-accelerated QEMU with GUI support |
| docker_local_dev.sh | Added KVM device passthrough and X11 authentication mounts to enable hardware-accelerated QEMU with GUI support |
| docker_latest.sh | Added KVM device passthrough and X11 authentication mounts to enable hardware-accelerated QEMU with GUI support |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tlaurion
force-pushed
the
bugfix-qemu_kvm
branch
from
c127e5a to
510b43e
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 18 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tlaurion
force-pushed
the
bugfix-qemu_kvm
branch
4 times, most recently
from
61b1ccf to
79e115f
Compare
tlaurion
force-pushed
the
bugfix-qemu_kvm
branch
2 times, most recently
from
b26b8c5 to
f4c3a85
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 7 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tlaurion
force-pushed
the
bugfix-qemu_kvm
branch
from
f4c3a85 to
8d60dec
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tlaurion
force-pushed
the
bugfix-qemu_kvm
branch
from
8d60dec to
f959057
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tlaurion
force-pushed
the
bugfix-qemu_kvm
branch
2 times, most recently
from
8c27511 to
c18ec25
Compare
tlaurion
force-pushed
the
bugfix-qemu_kvm
branch
from
e482742 to
9d184eb
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 14 changed files in this pull request and generated 17 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tlaurion
force-pushed
the
bugfix-qemu_kvm
branch
from
b1889a4 to
f399ac1
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 14 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tlaurion
force-pushed
the
bugfix-qemu_kvm
branch
from
f399ac1 to
a5657e5
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 13 out of 14 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tlaurion
force-pushed
the
bugfix-qemu_kvm
branch
from
a5657e5 to
d8349e6
Compare
- Add shared Docker helpers and wrapper-specific help while keeping full env reference in common.sh - Introduce digest tooling (get_digest, pin-and-run, fetch_nix_installer) and pin DOCKER_REPRO_DIGEST - Rework reproducibility checks (config digest comparison, remote resolution) and improve docker load/Xauthority handling - Update README and QEMU docs; tighten initrd kexec-seal-key parsing Signed-off-by: Thierry Laurion <insurgo@riseup.net>
Upgrade the Docker development environment image from v0.2.5 to v0.2.7. This version adds coreboot-utils as a dependency, which is required by diffoscope for ROM analysis and component extraction (ifdtool, cbfsutils). The image has been verified to be reproducible (unchanged content after push to docker.io). Changes: .circleci/config.yml: - Pin all Docker image references from :v0.2.5 to @sha256:5f890f3d1b6b57f9e567191695df003a2ee880f084f5dfe7a5633e3e8f937479 (v0.2.7) - Add version comment for human readability (# Docker image: tlaurion/heads-dev-env:v0.2.7) - All four job definitions (prep_env, build_and_persist, build, save_cache) updated - Ensures reproducible CI builds matching published image exactly docker/DOCKER_REPRO_DIGEST: - Update pinned digest from v0.2.5 to v0.2.7 - sha256:5f890f3d1b6b57f9e567191695df003a2ee880f084f5dfe7a5633e3e8f937479 - Add version comment for human readability flake.nix: - Add coreboot-utils dependency - Comment: "consumed by diffoscope for ifdtool cbfsutils etc" - Required for ROM analysis and investigation workflows docker/get_digest.sh, docker/pin-and-run.sh: - Update example references from v0.2.6 to v0.2.7 - Improve documentation consistency Testing: - Reproducibility checks against tlaurion/heads-dev-env:v0.2.7 - CircleCI workflow validated with pinned digest - All existing functionality confirmed working Summary of changes: - 1 configuration update: .circleci/config.yml (docker image pinning) - 1 digest file update: docker/DOCKER_REPRO_DIGEST - 1 build dependency: flake.nix (coreboot-utils) - 2 documentation updates: docker/get_digest.sh, docker/pin-and-run.sh example references Signed-off-by: Thierry Laurion <insurgo@riseup.net>
…us failing because race condition without sleep Signed-off-by: Thierry Laurion <insurgo@riseup.net>
tlaurion
force-pushed
the
bugfix-qemu_kvm
branch
from
d8349e6 to
b94a0a9
Compare
DO_WITH_DEBUG redirects stdout/stderr through tee for logging, which breaks interactive password prompts by interfering with TTY access. Both increment_tpm_counter calls already redirect output to /dev/null, so DO_WITH_DEBUG provided no logging benefit while breaking prompts. This allows TPM owner password prompts to display correctly on console when TPM counters need to be created or incremented. Signed-off-by: Thierry Laurion <insurgo@riseup.net>
tlaurion
force-pushed
the
bugfix-qemu_kvm
branch
from
b94a0a9 to
699a1a7
Compare
|
Ok enough. Next steps will be in other pull requests. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 17 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| docker run "${DOCKER_OPTS_ARRAY[@]}" -v "${host_workdir}:${container_workdir}" -w "${container_workdir}" "${image}" -- "$@" | ||
| local status=$? | ||
| if [ "${DOCKER_XAUTH_TEMP:-0}" = "1" ] && [ -n "${DOCKER_XAUTH_FILE}" ]; then | ||
| rm -f "${DOCKER_XAUTH_FILE}" || true | ||
| fi |
There was a problem hiding this comment.
Temporary Xauthority cleanup only happens after docker run returns. If the caller traps SIGINT (as the wrappers do) and exits early, the mktemp-created Xauthority file can be left behind. Consider registering an EXIT/signal trap in common.sh (or exposing a cleanup helper) that removes DOCKER_XAUTH_FILE when DOCKER_XAUTH_TEMP=1, so cleanup also happens on Ctrl-C and other early exits.
| if [ -n "${manifest_digest}" ]; then | ||
| print_digest_info "${image%@*}@${manifest_digest}" "${manifest_digest}" "registry API" "" | ||
| echo "${image%@*}@${manifest_digest}" | ||
| echo "${manifest_digest}" | ||
|
|
||
| # Offer to pull the exact image so the local Docker daemon has a repo@digest entry. | ||
| if [ "${auto_yes}" = 1 ]; then | ||
| echo "Auto-pull enabled: pulling ${image} (progress will be shown)..." >&2 | ||
| if ! docker pull "${image}" 2>&1 | sed -u 's/^/ /'; then | ||
| exit 1 | ||
| fi | ||
| local_repo_digest=$(docker inspect --format='{{index .RepoDigests 0}}' "${image}" 2>/dev/null || true) | ||
| if [ -n "${local_repo_digest}" ]; then | ||
| echo "${local_repo_digest}" | ||
| echo "${local_repo_digest#*@}" | ||
| exit 0 | ||
| fi | ||
| # else fall through and print the manifest digest as best-effort | ||
| exit 0 |
There was a problem hiding this comment.
When a manifest digest is found via the registry API, the script immediately prints repo@digest + digest to stdout, and then (if the user opts to pull / --yes) prints a second repo@digest + digest pair after the pull. This breaks the stated contract of emitting a single 2-line result and can confuse callers that parse the first/second line. Consider deferring stdout output until after the pull decision (or only printing the pulled result when a pull occurs, and writing intermediate info to stderr only).
| wait_for_usb_devices() { | ||
| TRACE_FUNC | ||
| if [ ! -d /sys/bus/usb/devices ] || [ ! -r /proc/uptime ]; then | ||
| DEBUG "USB sysfs or uptime not available, skipping wait" | ||
| return | ||
| fi | ||
|
|
||
| local start now elapsed | ||
| start=$(awk '{print $1}' /proc/uptime) | ||
| DEBUG "Waiting for USB peripheral devices (not just hubs) - max 2s timeout" | ||
|
|
||
| local iteration=0 | ||
| while :; do | ||
| iteration=$((iteration + 1)) | ||
|
|
||
| # Check for actual USB peripheral devices (format: bus-port like 1-1, 5-3) | ||
| # Root hubs are named usb1, usb2, etc. - we want devices downstream from them | ||
| # Pattern: /sys/bus/usb/devices/[0-9]*-[0-9]*/idVendor (e.g., 1-1, 5-3.2) | ||
| local peripheral_count=0 | ||
| if [ -d /sys/bus/usb/devices ]; then | ||
| # Count devices matching bus-port pattern (not usb* root hubs) | ||
| for dev in /sys/bus/usb/devices/*-*/idVendor; do | ||
| if [ -r "$dev" ]; then | ||
| peripheral_count=$((peripheral_count + 1)) | ||
| fi | ||
| done | ||
| fi | ||
|
|
||
| now=$(awk '{print $1}' /proc/uptime) | ||
| elapsed=$(awk -v s="$start" -v n="$now" 'BEGIN{printf "%.3f", n - s}') | ||
|
|
||
| if [ $peripheral_count -gt 0 ]; then | ||
| DEBUG "USB peripheral devices ready after ${elapsed}s (iteration $iteration): found $peripheral_count device(s)" | ||
| return | ||
| fi | ||
|
|
||
| # Timeout after 2 seconds | ||
| if awk -v s="$start" -v n="$now" 'BEGIN{exit (n - s > 2.0) ? 0 : 1}'; then | ||
| DEBUG "USB wait timeout at ${elapsed}s (iter $iteration): only found $peripheral_count peripheral device(s)" | ||
| return | ||
| fi | ||
| done |
There was a problem hiding this comment.
wait_for_usb_devices() uses a tight polling loop with no sleep/backoff and executes multiple awk processes per iteration. In initrd this can burn CPU for up to the full 2s timeout and slow boot on low-power devices. Consider adding a small bounded sleep/backoff (e.g., 10–50ms) or using an event-driven wait (udev settle if available) while keeping the overall timeout.
1 participant
TODO after merge:
Docker Workflows and Reproducibility Tooling Refactor
Overview
This changeset introduces comprehensive improvements to Heads' Docker-based build system, focusing on reproducibility, maintainability, and user experience.
Files Changed
Modified Files
.circleci/config.yml
v0.2.5to digest-pinnedv0.2.7tlaurion/heads-dev-env:v0.2.5tlaurion/heads-dev-env@sha256:5f890f3d1b6b57f9e567191695df003a2ee880f084f5dfe7a5633e3e8f937479README.md
Major documentation expansion including:
docker_repro.sh,docker_local_dev.sh,docker_latest.sh)HEADS_*variables)qemu-imgsnapshot workflowdocker_latest.sh
docker/common.shfor shared logicusage()functionresolve_docker_imagefor digest resolutionrequire_dockercheckDOCKER_LATEST_DIGESToverridedocker_local_dev.sh
docker/common.shusage()functionmaybe_rebuild_local_imagefor conditional rebuildsHEADS_CHECK_REPRODUCIBILITY=1)ensure_nix_and_flakesinfrastructureHEADS_SKIP_DOCKER_REBUILD=1docker_repro.sh
docker/common.shusage()functionDOCKER_REPRO_DIGESTfile/env varresolve_docker_imagewith digest supportflake.nix
initrd/bin/kexec-seal-key
[ \t]to[[:space:]]for better portabilitytargets/qemu.md
New Files
docker/DOCKER_REPRO_DIGEST
./docker_repro.shsha256:5f890f3d1b6b57f9e567191695df003a2ee880f084f5dfe7a5633e3e8f937479v0.2.7docker/check_reproducibility.sh
./docker/check_reproducibility.sh [local_image] [remote_image]compare_image_reproducibilityin common.shHEADS_MAINTAINER_DOCKER_IMAGEoverridedocker/common.sh
Shared helper library providing:
ensure_nix_and_flakes()— Nix installation and flakes setup with interactive promptsresolve_docker_image()— Image reference resolution with digest pinning supportmaybe_rebuild_local_image()— Conditional Docker image rebuild when flake files are dirtykill_usb_processes()— Automatic USB device cleanup (scdaemon/pcscd)build_docker_opts()— Docker runtime options construction (volumes, X11, USB, etc.)run_docker()— Container execution wrapper with cleanupget_remote_config_digest()— Fetch remote image config digest via skopeo/APIget_local_image_id()— Extract local image ID (config digest)compare_image_reproducibility()— Compare local vs remote image digestsresolve_repro_remote_image()— Default reproducibility remote image resolutionrequire_docker()— Docker availability checkdocker/fetch_nix_installer.sh
--version nix-X.Y.Zor--url <custom-url>.sha256from releases.nixos.org when available./docker/fetch_nix_installer.sh --version nix-2.33.2docker/get_digest.sh
-y/--yesfor auto-pullrepo@digestand rawsha256:...for easy scripting./docker/get_digest.sh [-y] <image:tag>./docker/get_digest.sh tlaurion/heads-dev-env:v0.2.7docker/pin-and-run.sh
get_digest.shand exportsDOCKER_LATEST_DIGEST./docker/pin-and-run.sh [-y] [-w wrapper] <image:tag> -- <command>./docker/pin-and-run.sh tlaurion/heads-dev-env:v0.2.7 -- make BOARD=x230./docker_latest.shif no wrapper specified-yfor non-interactive auto-pullKey Features
1. Reproducibility Infrastructure
docker_repro.shnow use immutable@sha256:...referencesHEADS_CHECK_REPRODUCIBILITY=1compares local builds vs published images2. Code Consolidation
3. Enhanced Documentation
HEADS_*andDOCKER_*variables4. Developer Tools
HEADS_MAINTAINER_DOCKER_IMAGE5. Improved Help System
--help)docker/common.shdirectly for complete environment documentationOlder notes
My bad: I always only tested qemu boards under QubesOS (which as of now still doesn't permit nested xen->kvm virtualization)
This change permits to run things under qemu+kvm
Tested
Note: TPM DUK is mitigation for any OSes not providing efifb in their initramfs, prior of specialized drm+gpu drivers being loaded (at LUKS passphrase prompt early in init/systemd).
ISO tested:
Also:
TODO:
404f56fdf6a63226fddc97e0059703f696ad47bfad2881838845647e8b04799a /root/heads/build/x86/EOL_t480-hotp-maximized/heads-EOL_t480-hotp-maximized-v0.2.1-2921-ge931ff1.rom