OCPBUGS-59925: Fix keepalived SIGTERM handling #5403
Conversation
Previously the SIGTERM that is sent when kubelet shuts down the pod was being ignored, which caused keepalived to be SIGKILLed and not able to send the priority 0 message so another node takes the VIP immediately. This caused ~5 seconds of delay when keepalived and haproxy were restarted during upgrades, and if this happened on a node where the local apiserver is also unavailable it causes a temporary API outage. There appears to have been two reasons for this: 1) The socat call blocks SIGTERM handling in the bash script 2) Sending SIGTERM to keepalived without waiting for it to complete can cause the container to exit before priority 0 is sent Using the "wait" command seems to make this work as expected. Now socat is started in the background and wait is used to keep the script from exiting. A delay is also added to allow keepalived time to shut down cleanly. Timeouts are not needed for any of these calls because kubelet will already send SIGKILL after the grace period expires.
openshift-ci-robot
added
jira/valid-reference
|
@cybertron: This pull request references Jira Issue OCPBUGS-59925, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/jira refresh |
openshift-ci-robot
added
jira/valid-bug
|
@cybertron: This pull request references Jira Issue OCPBUGS-59925, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/test e2e-metal-ipi-ovn-dualstack |
|
/test e2e-metal-ipi-ovn-dualstack Failed on ovnk, probably not related but would still like to see it green. |
| if pid=$(pgrep -o keepalived); then | ||
| kill -s SIGTERM "$pid" | ||
| # Give keepalived time to shut down | ||
| while pgrep -o keepalived; do sleep 1; done |
There was a problem hiding this comment.
killall -o -w -s SIGTERM keepalived maybe?
-o,--older-than kill processes older than TIME
-w,--wait wait for processes to die
or do we need the extra sleeps?
There was a problem hiding this comment.
nevermind. killall doesn't seem to match pgrep -o. Could pkill though.
There was a problem hiding this comment.
or could we do wait $pid it's a child process?
There was a problem hiding this comment.
Ah, we don't want to be fast, we want to sleep.
There was a problem hiding this comment.
Talked about this with Ross offline, but documenting here for future reference:
The reason I'm looping on pgrep instead of using wait is that sometimes the pid we get back from keepalived isn't a child of the main script and wait will fail. I suspect that may be a bug in itself - we use -o to get the oldest pid, which would presumably be the parent keepalived process that was started by the main script, but it seems that isn't always true. It may be that the oldest pid somehow isn't always the main keepalived process.
In any case, another advantage of using pgrep is we will wait until all of the keepalived processes have exited so we should know for certain that priority 0 was sent by the time that completes. It's a bit inelegant, but it seems to be the safest way to handle this.
|
/lgtm |
|
/verified by @rbbratta |
openshift-ci-robot
added
the
verified
|
@rbbratta: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/assign @umohnani8 |
|
/approved |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cybertron, rbbratta, umohnani8 The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
/retest-required |
|
@cybertron: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/label acknowledge-critical-fixes-only This is a fix to improve test stability, so I think it's valid to go in right now. |
openshift-ci
bot
added
the
acknowledge-critical-fixes-only
openshift-merge-bot
bot
merged commit e206063
into
openshift:main
|
@cybertron: Jira Issue Verification Checks: Jira Issue OCPBUGS-59925 Jira Issue OCPBUGS-59925 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Fix included in accepted release 4.21.0-0.nightly-2025-12-08-112148 |
|
/cherry-pick release-4.20 Spot checking some ci results, it looks like this has had the intended effect. Now instead of 6 seconds of disruption, we get 1 second (likely just a blip when the VIP fails over, which we can't reasonably fix with a cluster-hosted LB). |
|
@cybertron: new pull request created: #5507 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
6 participants
Previously the SIGTERM that is sent when kubelet shuts down the pod was being ignored, which caused keepalived to be SIGKILLed and not able to send the priority 0 message so another node takes the VIP immediately. This caused ~5 seconds of delay when keepalived and haproxy were restarted during upgrades, and if this happened on a node where the local apiserver is also unavailable it causes a temporary API outage.
There appears to have been two reasons for this:
can cause the container to exit before priority 0 is sent
Using the "wait" command seems to make this work as expected. Now socat is started in the background and wait is used to keep the script from exiting. A delay is also added to allow keepalived time to shut down cleanly.
Timeouts are not needed for any of these calls because kubelet will already send SIGKILL after the grace period expires.
- What I did
- How to verify it
- Description for the changelog