[multiple] Co-locate provisionserver with metal3 to prevent DHCP failures by mnietoji · Pull Request #3691 · openstack-k8s-operators/ci-framework

mnietoji · 2026-02-17T16:34:49Z

[multiple] Co-locate provisionserver with metal3 to prevent DHCP failures

When metal3-dnsmasq pod restarts during a node's DHCP lease renewal on the
provisioning network (172.23.0.0/24), NetworkManager fails to renew and sets
ipv4.method=disabled. NMState operator then preserves this disabled state,
causing permanent loss of provisioning network connectivity on that node.

The issue occurs when OpenStackProvisionServer and metal3 pods run on
different nodes. If metal3 restarts while a node is attempting DHCP renewal,
the temporary unavailability of metal3-dnsmasq causes the renewal to fail.

Solution:
Automatically detect the node running metal3 pod (via k8s-app=metal3 label)
and configure provisionServerNodeSelector in baremetalSetTemplate to schedule
OpenStackProvisionServer on the same node. This ensures provisioning network
connectivity is maintained because metal3-static-ip-manager maintains a static
IP (172.23.0.3) on the metal3 node regardless of dnsmasq restarts.

Signed-off-by: Miguel Angel Nieto Jimenez <mnietoji@redhat.com>
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

softwarefactory-project-zuul · 2026-02-17T18:29:40Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/c72a795e19ee4ef1994a72979d3b02ba

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 53m 36s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 19m 11s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 40m 42s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 20s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 52s
❌ cifmw-pod-pre-commit FAILURE in 8m 40s
✔️ cifmw-molecule-devscripts SUCCESS in 10m 12s

openshift-ci · 2026-02-17T23:01:26Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign michburk for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

softwarefactory-project-zuul · 2026-02-18T00:58:11Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/419cbfb4c38c4c90a492120cc24c67ca

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 55m 45s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 22m 44s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 42m 54s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 43s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 00s
❌ cifmw-pod-pre-commit FAILURE in 8m 25s
✔️ cifmw-molecule-devscripts SUCCESS in 10m 26s

softwarefactory-project-zuul · 2026-02-18T08:46:57Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/bb09852a1750455a9f54d8a9a5c3f190

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 48m 51s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 17m 15s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 35m 25s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 54s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 00s
❌ cifmw-pod-pre-commit FAILURE in 7m 52s
✔️ cifmw-molecule-devscripts SUCCESS in 10m 17s

danpawlik

Thank you for PR LGTM.
Could you share testproject result via DM ?

softwarefactory-project-zuul · 2026-02-23T00:08:51Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/48d00299c0da4681a65b751e604fc62b

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 22m 33s
❌ podified-multinode-edpm-deployment-crc FAILURE in 23m 41s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 30m 42s
❌ cifmw-crc-podified-edpm-baremetal-minor-update FAILURE in 2h 09m 04s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 59s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 30s
✔️ cifmw-pod-k8s-snippets-source SUCCESS in 4m 34s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 27s
✔️ cifmw-architecture-validate-hci SUCCESS in 3m 54s
✔️ cifmw-molecule-ci_gen_kustomize_values SUCCESS in 5m 07s
✔️ cifmw-molecule-kustomize_deploy SUCCESS in 4m 04s

softwarefactory-project-zuul · 2026-02-23T11:05:01Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/cb65dea83fcb4bd4924a69fc8156e64d

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 27m 43s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 19m 51s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 23m 57s
❌ cifmw-crc-podified-edpm-baremetal-minor-update FAILURE in 2h 14m 58s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 52s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 36s
✔️ cifmw-pod-k8s-snippets-source SUCCESS in 4m 35s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 28s
✔️ cifmw-architecture-validate-hci SUCCESS in 4m 02s
✔️ cifmw-molecule-ci_gen_kustomize_values SUCCESS in 5m 12s
✔️ cifmw-molecule-kustomize_deploy SUCCESS in 4m 18s

mnietoji · 2026-02-23T22:15:59Z

recheck

…ures When metal3-dnsmasq pod restarts during a node's DHCP lease renewal on the provisioning network (172.23.0.0/24), NetworkManager fails to renew and sets ipv4.method=disabled. NMState operator then preserves this disabled state, causing permanent loss of provisioning network connectivity on that node. The issue occurs when OpenStackProvisionServer and metal3 pods run on different nodes. If metal3 restarts while a node is attempting DHCP renewal, the temporary unavailability of metal3-dnsmasq causes the renewal to fail. Solution: Automatically detect the node running metal3 pod (via k8s-app=metal3 label) and configure provisionServerNodeSelector in baremetalSetTemplate to schedule OpenStackProvisionServer on the same node. This ensures provisioning network connectivity is maintained because metal3-static-ip-manager maintains a static IP (172.23.0.3) on the metal3 node regardless of dnsmasq restarts. Signed-off-by: Miguel Angel Nieto Jimenez <mnietoji@redhat.com> Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

softwarefactory-project-zuul · 2026-02-24T00:47:08Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/e0bffd51141e4e21b058b228aef2fff8

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 29m 41s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 19m 08s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 34m 54s
❌ cifmw-crc-podified-edpm-baremetal-minor-update FAILURE in 2h 16m 36s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 50s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 40s
✔️ cifmw-pod-k8s-snippets-source SUCCESS in 4m 54s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 44s
✔️ cifmw-architecture-validate-hci SUCCESS in 4m 30s
✔️ cifmw-molecule-ci_gen_kustomize_values SUCCESS in 5m 38s
✔️ cifmw-molecule-kustomize_deploy SUCCESS in 4m 39s

amartyasinha · 2026-02-24T04:48:15Z

recheck

softwarefactory-project-zuul · 2026-02-24T07:26:03Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/c69c9689e2e04aa0a1ceaaf172e60e01

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 36m 21s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 19m 25s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 31m 37s
❌ cifmw-crc-podified-edpm-baremetal-minor-update FAILURE in 2h 22m 35s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 25s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 35s
✔️ cifmw-pod-k8s-snippets-source SUCCESS in 4m 46s
✔️ cifmw-pod-pre-commit SUCCESS in 7m 51s
✔️ cifmw-architecture-validate-hci SUCCESS in 4m 14s
❌ cifmw-molecule-ci_gen_kustomize_values TIMED_OUT in 30m 45s
✔️ cifmw-molecule-kustomize_deploy SUCCESS in 4m 22s

danpawlik · 2026-02-24T08:02:34Z

recheck

softwarefactory-project-zuul · 2026-02-24T08:32:12Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/480004750ed042aba015d0bf8ab41162

✔️ openstack-k8s-operators-content-provider SUCCESS in 28m 15s
❌ podified-multinode-edpm-deployment-crc RETRY_LIMIT in 11m 20s
❌ cifmw-crc-podified-edpm-baremetal RETRY_LIMIT in 14m 34s
❌ cifmw-crc-podified-edpm-baremetal-minor-update RETRY_LIMIT in 14m 24s
✔️ cifmw-pod-zuul-files SUCCESS in 6m 11s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 10m 20s
✔️ cifmw-pod-k8s-snippets-source SUCCESS in 6m 37s
✔️ cifmw-pod-pre-commit SUCCESS in 9m 36s
✔️ cifmw-architecture-validate-hci SUCCESS in 5m 23s
✔️ cifmw-molecule-ci_gen_kustomize_values SUCCESS in 6m 15s
✔️ cifmw-molecule-kustomize_deploy SUCCESS in 4m 24s

amartyasinha · 2026-02-24T09:58:48Z

recheck

amartyasinha

/lgtm

danpawlik · 2026-02-24T10:57:53Z

recheck

softwarefactory-project-zuul · 2026-02-24T12:35:18Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/268e95b083da44ca8380b7b24f71af4a

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 36m 11s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 23m 02s
❌ cifmw-crc-podified-edpm-baremetal RETRY_LIMIT in 14m 29s
❌ cifmw-crc-podified-edpm-baremetal-minor-update RETRY_LIMIT in 14m 47s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 58s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 30s
✔️ cifmw-pod-k8s-snippets-source SUCCESS in 5m 06s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 24s
✔️ cifmw-architecture-validate-hci SUCCESS in 4m 16s
✔️ cifmw-molecule-ci_gen_kustomize_values SUCCESS in 5m 49s
✔️ cifmw-molecule-kustomize_deploy SUCCESS in 4m 24s

mnietoji force-pushed the dhcp_provisioning branch from 7038291 to 1c7cf00 Compare February 17, 2026 23:01

mnietoji force-pushed the dhcp_provisioning branch from 1c7cf00 to d8f15b7 Compare February 18, 2026 06:57

mnietoji force-pushed the dhcp_provisioning branch 4 times, most recently from c87b028 to 102e9ec Compare February 18, 2026 15:57

github-actions bot added the Ready For Review label Feb 18, 2026

mnietoji removed the Ready For Review label Feb 18, 2026

mnietoji force-pushed the dhcp_provisioning branch from 102e9ec to 829d9f4 Compare February 19, 2026 08:26

github-actions bot added the Ready For Review label Feb 19, 2026

danpawlik reviewed Feb 19, 2026

View reviewed changes

mnietoji force-pushed the dhcp_provisioning branch from 829d9f4 to 73f8928 Compare February 20, 2026 07:48

github-actions bot added Ready For Review and removed Ready For Review labels Feb 20, 2026

mnietoji force-pushed the dhcp_provisioning branch from 73f8928 to 8ab220a Compare February 20, 2026 11:43

github-actions bot added Ready For Review and removed Ready For Review labels Feb 20, 2026

mnietoji force-pushed the dhcp_provisioning branch from 8ab220a to 8a22795 Compare February 20, 2026 20:12

github-actions bot removed the Ready For Review label Feb 20, 2026

mnietoji force-pushed the dhcp_provisioning branch from 8a22795 to 4cd95ee Compare February 20, 2026 20:15

github-actions bot added the Ready For Review label Feb 20, 2026

mnietoji force-pushed the dhcp_provisioning branch from 4cd95ee to faca5a7 Compare February 22, 2026 17:25

github-actions bot removed the Ready For Review label Feb 22, 2026

mnietoji force-pushed the dhcp_provisioning branch 2 times, most recently from c739939 to 027282c Compare February 22, 2026 17:40

mnietoji force-pushed the dhcp_provisioning branch 3 times, most recently from b378912 to 9a374c6 Compare February 22, 2026 21:45

mnietoji changed the title ~~Fix provisioning network DHCP timeout race condition~~ [multiple] Co-locate provisionserver with metal3 to prevent DHCP failures Feb 22, 2026

mnietoji force-pushed the dhcp_provisioning branch from 9a374c6 to 1533740 Compare February 23, 2026 08:24

mnietoji added the Ready For Review label Feb 23, 2026

github-actions bot removed the Ready For Review label Feb 23, 2026

mnietoji force-pushed the dhcp_provisioning branch from 1533740 to d47fe74 Compare February 23, 2026 08:36

mnietoji enabled auto-merge (rebase) February 23, 2026 22:15

mnietoji force-pushed the dhcp_provisioning branch from d47fe74 to 4b70c3d Compare February 23, 2026 22:16

danpawlik approved these changes Feb 24, 2026

View reviewed changes

amartyasinha approved these changes Feb 24, 2026

View reviewed changes

openshift-ci bot assigned amartyasinha Feb 24, 2026

openshift-ci bot added the lgtm label Feb 24, 2026

Comments

Conversation

mnietoji commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

softwarefactory-project-zuul bot commented Feb 17, 2026

Uh oh!

openshift-ci bot commented Feb 17, 2026

Uh oh!

softwarefactory-project-zuul bot commented Feb 18, 2026

Uh oh!

softwarefactory-project-zuul bot commented Feb 18, 2026

Uh oh!

danpawlik left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

softwarefactory-project-zuul bot commented Feb 23, 2026

Uh oh!

softwarefactory-project-zuul bot commented Feb 23, 2026

Uh oh!

mnietoji commented Feb 23, 2026

Uh oh!

softwarefactory-project-zuul bot commented Feb 24, 2026

Uh oh!

amartyasinha commented Feb 24, 2026

Uh oh!

softwarefactory-project-zuul bot commented Feb 24, 2026

Uh oh!

danpawlik commented Feb 24, 2026

Uh oh!

softwarefactory-project-zuul bot commented Feb 24, 2026

Uh oh!

amartyasinha commented Feb 24, 2026

Uh oh!

amartyasinha left a comment

Choose a reason for hiding this comment

Uh oh!

danpawlik commented Feb 24, 2026

Uh oh!

softwarefactory-project-zuul bot commented Feb 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mnietoji commented Feb 17, 2026 •

edited

Loading

danpawlik left a comment •

edited

Loading