Add bootc image reference

[bshephar]

To support image mode RHEL deployments, this change adds the reference to the bootc container image.

Jira: https://issues.redhat.com/browse/OSPRH-14326

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bshephar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [bshephar]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/b2a7f196aab34a5da9b3d0e5840c74b9

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 54m 58s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 24m 20s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 34m 24s
❌ adoption-standalone-to-crc-ceph-provider FAILURE in 1h 33m 55s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 38m 31s

[bshephar]

/test openstack-operator-build-deploy-kuttl

[bshephar]

/test openstack-operator-build-deploy-kuttl

[bshephar]

recheck FAILURE

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/83b1f4bb3c79438e81bcbac3ef97c6dd

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 23m 34s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 20m 52s
❌ cifmw-crc-podified-edpm-baremetal FAILURE in 1h 25m 58s
✔️ adoption-standalone-to-crc-ceph-provider SUCCESS in 3h 07m 45s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 43m 11s

[bshephar]

Unrelated failures:

❯ curl -s "https://logserver.rdoproject.org/21/1321/e320a18f35f52482ca40de94b9c0b121ec5619e2/github-check/cifmw-crc-podified-edpm-baremetal/54b00d4/ci-framework-data/logs/openstack-k8s-operators-openstack-must-gather/namespaces/openstack/pods/repo-setup-edpm-deployment-ipam-openstack-edpm-ipam-prjfm/logs/repo-setup-edpm-deployment-ipam-openstack-edpm-ipam.log" | tail -n10
fatal: [edpm-compute-0]: FAILED! => {"changed": true, "cmd": "set -euxo pipefail\npushd /var/tmp\ncurl -sL https://github.com/openstack-k8s-operators/repo-setup/archive/refs/heads/main.tar.gz | tar -xz\npushd repo-setup-main\npython3 -m venv ./venv\nPBR_VERSION=0.0.0 ./venv/bin/pip install ./\n# This is required for FIPS enabled until trunk.rdoproject.org\n# is not being served from a centos7 host, tracked by\n# https://issues.redhat.com/browse/RHOSZUUL-1517\nupdate-crypto-policies --set FIPS:NO-ENFORCE-EMS\n./venv/bin/repo-setup current-podified -b antelope\npopd\nsed -i '/AppStream/a exclude=\"libvirt*10.10.0-4*,qemu*9.1.0-9*\"' /etc/yum.repos.d/repo-setup-centos-appstream.repo\nsed -i '/AppStream/a exclude=\"libvirt*10.10.0-4*,qemu*9.1.0-9*\"' /etc/yum.repos.d/centos.repo || true\nrm -rf repo-setup-main\n", "delta": "0:00:20.046624", "end": "2025-02-27 17:57:59.442203", "msg": "non-zero return code", "rc": 2, "start": "2025-02-27 17:57:39.395579", "stderr": "+ pushd /var/tmp\n+ tar -xz\n+ curl -sL https://github.com/openstack-k8s-operators/repo-setup/archive/refs/heads/main.tar.gz\n\ngzip: stdin: unexpected end of file\ntar: Child returned status 1\ntar: Error is not recoverable: exiting now", "stderr_lines": ["+ pushd /var/tmp", "+ tar -xz", "+ curl -sL https://github.com/openstack-k8s-operators/repo-setup/archive/refs/heads/main.tar.gz", "", "gzip: stdin: unexpected end of file", "tar: Child returned status 1", "tar: Error is not recoverable: exiting now"], "stdout": "/var/tmp /home/cloud-admin", "stdout_lines": ["/var/tmp /home/cloud-admin"]}

PLAY RECAP *********************************************************************
edpm-compute-0             : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

[fmount]

+1 from me but adding someone from the core team to approve

[abays]

Does this by any chance need a change to pkg/dataplane/util/version.go as well? Just checking because I was reviewing something similar this weekend and, in that case, a change was needed in the file [1].

[1] https://github.com/openstack-k8s-operators/openstack-operator/pull/1319/files#diff-80f6225dc117813e2cfef98d6356561b01b979f012f2bdc7ff98113f232a2da6

[stuggi]

looks good to me, but I'd say lets wait for it to land after the FR2 branching

[slagle]

I think we might want to hold on this until we have a targeted release (FR3 perhaps), or at least a plan for how this would be set downstream.

[bshephar]

/test openstack-operator-build-deploy-kuttl

[bshephar]

Does this by any chance need a change to pkg/dataplane/util/version.go as well? Just checking because I was reviewing something similar this weekend and, in that case, a change was needed in the file [1].

[1] https://github.com/openstack-k8s-operators/openstack-operator/pull/1319/files#diff-80f6225dc117813e2cfef98d6356561b01b979f012f2bdc7ff98113f232a2da6

Indeed, thanks. I've updated that reference now.

[bshephar]

/hold

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/8a7e81f5cf344053aa42e0e8ec46bc3b

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 51m 11s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 19m 01s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 24m 21s
❌ adoption-standalone-to-crc-ceph-provider RETRY_LIMIT in 15m 03s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 35m 41s

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/f5d13c3eea9f4f5189e97bd00d1af6cb

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 57m 09s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 15m 17s
❌ cifmw-crc-podified-edpm-baremetal FAILURE in 59m 03s
❌ adoption-standalone-to-crc-ceph-provider FAILURE in 1h 41m 32s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 35m 42s
❌ openstack-operator-kuttl FAILURE in 29m 48s (non-voting)

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/024472b06ee04063a8fb671099f882a9

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 37m 49s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 09m 11s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 25m 26s
✔️ adoption-standalone-to-crc-ceph-provider SUCCESS in 3h 22m 10s
❌ openstack-operator-tempest-multinode FAILURE in 1h 28m 24s
❌ openstack-operator-kuttl FAILURE in 26m 53s (non-voting)

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/e57172d5fc37470e9c97b47ea2dfb6ea

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 33m 32s
❌ podified-multinode-edpm-deployment-crc FAILURE in 1h 34m 47s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 27m 34s
✔️ adoption-standalone-to-crc-ceph-provider SUCCESS in 3h 16m 02s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 25m 24s
❌ openstack-operator-kuttl FAILURE in 24m 45s (non-voting)

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/b2f3fda118804bd79cfdd122a8eef28b

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 53m 35s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 11m 44s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 39m 50s
❌ adoption-standalone-to-crc-ceph-provider FAILURE in 1h 35m 21s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 28m 13s
❌ openstack-operator-kuttl FAILURE in 26m 27s (non-voting)

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/db6b5fd20f934ddbafdf79b278765451

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 51m 46s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 09m 40s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 27m 59s
❌ adoption-standalone-to-crc-ceph-provider FAILURE in 1h 37m 34s
❌ openstack-operator-tempest-multinode RETRY_LIMIT in 12m 41s
❌ openstack-operator-kuttl FAILURE in 25m 59s (non-voting)

[rabi]

Not sure why we would need this? Can we not leverage the same OsContainerImage for this? Having both bootc and non-bootc container images in OpenStackVersion does not look like a good design. If we expect customers to use nodesets with both bootc and non-bootc images in a deployment, they've to patch nodesets I think.

[bshephar]

Hey Rabi,

It has been 9 months since I've looked at this. But I believe the issue is that OsContainerImage refers to the image that contains the qcow2 file that's used to provision nodes with Metal3. Whereas the variable I was adding here refers specifically to the bootc container image for the OS. So for example, here:
https://github.com/openstack-k8s-operators/openstack-operator/pull/1321/files#diff-a8669e419dcd80822e0cf7a87d6d1c9444012a2a58b7df32d1d17059aee51d83R330

That Ansible variable would be used to lifecycle the OS via bootc update or bootc switch for example. You wouldn't use this variable to Provision nodes in the NodeSet via Metal3 since the image doesn't contain the qcow2 file.

I can rebase this if you want to actively discuss and pursue this. I'm not too sure how your requirements have evolved since I left, but I'm still happy to work with you on it if it's something you need.

[bshephar]

So, the way I had envisioned this working was:

We would ship two artifacts:

1. The Bootc QCOW2 image. Which is packaged in a way that it can be consumed by Metal3. (Steve was working on this already: https://github.com/openstack-k8s-operators/edpm-image-builder/tree/main/bootc).

2. The Bootc container image itself.

From a user perspective. They would consume the QCOW2 image to build their nodes during the provisioning process. During the Ansible deployment of the NodeSet, it would update or switch to whatever the user has defined here with this new variable.

The value of doing it that way is that the user never needs to build their own QCOW2 image, since that is quite an arduous experience. Since they would need to build the QCOW2, package it into the container, publish the container somewhere, etc. There's some complexity in making this compatible with Metal3 - or at least there was.

So, by having the separation of these variables, you're allowing the user to easily build their own bootc image using just a Containerfile that derives FROM the EDPM base image and makes whatever modifications they need. They can then provide it via the new variable in this PR and Ansible will perform the bootc switch during the deployment.

The benefit from a Red Hat support perspective is that users are always using a known good and tested base image during the deployment. So it narrows the scope of troubleshooting and should ensure that you have bootable nodes 90% of the time, with the scope of user modified changes being limited to the bootc switch side. If the node fails to reboot, you still have a known good image to rollback to for troubleshooting.

[openshift-merge-robot]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[rabi]

From a user perspective. They would consume the QCOW2 image to build their nodes during the provisioning process. During the Ansible deployment of the NodeSet, it would update or switch to whatever the user has defined here with this new variable.

OK, I'm not sure if we should keep two container images for bootc in openstackversion. That can also cause confusion and inconsistency. From long term perspective when we move to BMHs/metal3 directly consuming container images, just a new OsContainerImage during update would suffice. Till that time IMO we should just provide an ansible var for a new image if the user wants os update during minor update (using bootc switch and update).

[bshephar]

From a user perspective. They would consume the QCOW2 image to build their nodes during the provisioning process. During the Ansible deployment of the NodeSet, it would update or switch to whatever the user has defined here with this new variable.

OK, I'm not sure if we should keep two container images for bootc in openstackversion. That can also cause confusion and inconsistency. From long term perspective when we move to BMHs/metal3 directly consuming container images, just a new OsContainerImage during update would suffice. Till that time IMO we should just provide an ansible var for a new image if the user wants os update during minor update (using bootc switch and update).

How do you plan to ship this? I guess that's the question. You will have two bespoke images regardless. One for the qcow2, one for the bootc image. Even for the major version upgrades they still need to change the Ansible var in that case. Or, we need a way of doing it for them. Once the nodes are physically provisioned, everything else will be via bootc upgrade. And the issue becomes that the image that bootc references != OsContainerImage if OsContainerImage is going to point to the container for the qcow2 image.

But that's doable without the ref to this image here. Just needs some thought about how the images are shipped, and what that minor and major upgrade path is going to look like.

[rabi]

But that's doable without the ref to this image here. Just needs some thought about how the images are shipped, and what that minor and major upgrade path is going to look like.

Yeah, interim shipping could be adhoc if we support both. Once we move to RHOS19 with RHEL10(bootc) and use bootc deploy interface from ironic as the only supported way, we don't have to maintain both I guess.