Disable Webhooks #466

bshephar · 2024-10-31T12:55:22Z

This change builds the Operator bundle with ENABLE_WEBHOOKS=false. This is due to the duplication of webhook logic since openstack-operator imports and runs this webhook at the OpenStackControlPlane level. Therefor, having the webhook run by the service operator is a needless duplication of this webhook logic.

Jira: https://issues.redhat.com/browse/OSPRH-11198
Depends-On: openstack-k8s-operators/openstack-k8s-operators-ci#114

openshift-ci · 2024-10-31T12:55:32Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bshephar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [bshephar]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

softwarefactory-project-zuul · 2024-10-31T15:02:40Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/e6ee4d6e038742faa0358b058025f546

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 01m 08s
❌ heat-operator-tempest-multinode FAILURE in 1h 11m 45s

bshephar · 2024-11-01T04:30:20Z

Yeah, so as we discussed on the call. This will break Kuttl tests. So we need to be able to dynamically set this for CI jobs - which is ideal since we do want to test web hooks in CI. But, ultimately, the bundle we build and ship should have the service level operator web hooks disabled so that they can be executed by the openstack-operator without duplication.

❯ curl -s "https://storage.googleapis.com/test-platform-results/pr-logs/pull/openstack-k8s-operators_heat-operator/466/pull-ci-openstack-k8s-operators-heat-operator-main-heat-operator-build-deploy-kuttl/1852209004751622144/build-log.txt" | grep "failed to call webhook"
    case.go:380: Internal error occurred: failed calling webhook "mheat.kb.io": failed to call webhook: Post "https://heat-operator-controller-manager-service.openstack-operators.svc:443/mutate-heat-openstack-org-v1beta1-heat?timeout=10s": dial tcp 10.128.0.75:9443: connect: connection refused
    case.go:380: Internal error occurred: failed calling webhook "mheat.kb.io": failed to call webhook: Post "https://heat-operator-controller-manager-service.openstack-operators.svc:443/mutate-heat-openstack-org-v1beta1-heat?timeout=10s": dial tcp 10.128.0.75:9443: connect: connection refused
    case.go:380: Internal error occurred: failed calling webhook "mheat.kb.io": failed to call webhook: Post "https://heat-operator-controller-manager-service.openstack-operators.svc:443/mutate-heat-openstack-org-v1beta1-heat?timeout=10s": dial tcp 10.128.0.75:9443: connect: connection refused

bshephar · 2024-11-01T04:40:11Z

/test heat-operator-build-deploy-kuttl

softwarefactory-project-zuul · 2024-11-02T12:19:28Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/cc47398324ea43fc8a7863f7cafc5c82

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 26m 45s
❌ heat-operator-tempest-multinode FAILURE in 1h 09m 56s

softwarefactory-project-zuul · 2024-11-05T05:34:00Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/9ac36f3b6ba341e6abf8a2fc21672ca7

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 25m 40s
❌ heat-operator-tempest-multinode FAILURE in 1h 10m 17s

bshephar · 2024-11-11T06:55:26Z

We need to merge this one to proceed any further here:
openshift/release#58391

bshephar · 2024-11-12T01:14:48Z

/test heat-operator-build-deploy-kuttl

softwarefactory-project-zuul · 2024-11-12T08:00:04Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/589eadecd7fc49e2b4cd73a423f7c1f9

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 25m 16s
❌ heat-operator-tempest-multinode FAILURE in 1h 09m 45s

softwarefactory-project-zuul · 2024-11-13T07:00:00Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/efff7273a75442b0941b043ecbe08afb

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 49m 58s
❌ heat-operator-tempest-multinode FAILURE in 1h 33m 40s

softwarefactory-project-zuul · 2024-11-20T15:34:26Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/1dfbed4e2db6453e9a382f5526c704d4

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 54m 15s
❌ heat-operator-tempest-multinode FAILURE in 1h 34m 58s

bshephar · 2024-11-20T21:34:05Z

So, the build is enabling webhooks:

❯ curl -s "https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openstack-k8s-operators_heat-operator/466/pull-ci-openstack-k8s-operators-heat-operator-main-heat-operator-build-deploy-kuttl/1859192010674540544/artifacts/heat-operator-build-deploy-kuttl/openstack-k8s-operators-build/build-log.txt" | grep ENABLE_WEB
[2/2] STEP 10/21: ARG ENABLE_WEBHOOKS=false
[2/2] STEP 18/21: ENV ENABLE_WEBHOOKS="${ENABLE_WEBHOOKS}"
+ oc patch bc/heat-operator-bundle --type=json '-p=[{"op": "add", "path": "/spec/strategy/dockerStrategy/env", "value": [{"name": "ENABLE_WEBHOOKS", "value": "true"}]}]'
STEP 2/17: ENV "ENABLE_WEBHOOKS"="true"
+ oc patch bc/openstack-operator-bundle --type=json '-p=[{"op": "add", "path": "/spec/strategy/dockerStrategy/env", "value": [{"name": "ENABLE_WEBHOOKS", "value": "true"}]}]'
STEP 2/17: ENV "ENABLE_WEBHOOKS"="true"

But Kuttl is failing because the webhook isn't responding:

❯ curl -s "https://storage.googleapis.com/test-platform-results/pr-logs/pull/openstack-k8s-operators_heat-operator/466/pull-ci-openstack-k8s-operators-heat-operator-main-heat-operator-build-deploy-kuttl/1859192010674540544/build-log.txt" | grep -E 'vheat|mheat'
    case.go:380: Internal error occurred: failed calling webhook "mheat.kb.io": failed to call webhook: Post "https://heat-operator-controller-manager-service.openstack-operators.svc:443/mutate-heat-openstack-org-v1beta1-heat?timeout=10s": dial tcp 10.128.0.98:9443: connect: connection refused
    case.go:380: Internal error occurred: failed calling webhook "mheat.kb.io": failed to call webhook: Post "https://heat-operator-controller-manager-service.openstack-operators.svc:443/mutate-heat-openstack-org-v1beta1-heat?timeout=10s": dial tcp 10.128.0.98:9443: connect: connection refused
    case.go:380: Internal error occurred: failed calling webhook "mheat.kb.io": failed to call webhook: Post "https://heat-operator-controller-manager-service.openstack-operators.svc:443/mutate-heat-openstack-org-v1beta1-heat?timeout=10s": dial tcp 10.128.0.98:9443: connect: connection refused

I must be missing something else.

bshephar · 2024-11-20T22:09:26Z

... Yeah, ARG doesn't work like that.

Makefile

This change builds the Operator bundle with ENABLE_WEBHOOKS=false. This is due to the duplication of webhook logic since openstack-operator imports and runs this webhook at the OpenStackControlPlane level. Therefor, having the webhook run by the service operator is a needless duplication of this webhook logic. Jira: https://issues.redhat.com/browse/OSPRH-11198 Signed-off-by: Brendan Shephard <bshephar@redhat.com>

rabi · 2024-11-21T15:02:42Z

You can still see the webhook definitions in the csv and webhooks registered with the controller manager
[1] https://logserver.rdoproject.org/66/466/2e89b036df8fb622dd793ee602363754d6849630/github-check/heat-operator-tempest-multinode/32f4769/controller/ci-framework-data/logs/openstack-k8s-operators-openstack-must-gather/csv/heat-operator.v0.0.1
[2] https://logserver.rdoproject.org/66/466/2e89b036df8fb622dd793ee602363754d6849630/github-check/heat-operator-tempest-multinode/32f4769/controller/ci-framework-data/logs/openstack-k8s-operators-openstack-must-gather/namespaces/openstack-operators/pods/heat-operator-controller-manager-7574fb7f67-cpz4h/logs/manager.log

This change adds the environment variable required to disable operator webhooks in the build. This will only disable webhooks in operators that have the option to do so in their specific Dockerfile. For example: openstack-k8s-operators/heat-operator#466 Signed-off-by: Brendan Shephard <bshephar@redhat.com>

bshephar · 2024-11-22T02:13:32Z

Yeah, so I guess we need to disable it like this:
https://github.com/openstack-k8s-operators/openstack-k8s-operators-ci/pull/114/files

dprince · 2024-11-22T02:16:01Z

@bshephar So my PR here would disable the validation/mutating webhooks for operators here: openstack-k8s-operators/openstack-operator#1185

bshephar · 2024-11-22T02:22:49Z

@bshephar So my PR here would disable the validation/mutating webhooks for operators here: openstack-k8s-operators/openstack-operator#1185

We wanted to run them from the openstack-operator though didn't we? Then disable them in each individual service operator.

bshephar · 2024-11-22T02:57:12Z

/retest

bshephar · 2024-12-09T05:09:39Z

This doesn't need to be done on an individual operator level. We will change the architecture for deploying openstack-operator and in doing so, we will stop running service webhooks.

openshift-ci bot requested review from frenzyfriday and slagle October 31, 2024 12:55

openshift-ci bot added the approved label Oct 31, 2024

bshephar force-pushed the disable-webhooks branch 3 times, most recently from d0d9edc to b0c5b47 Compare November 2, 2024 10:51

bshephar force-pushed the disable-webhooks branch from b0c5b47 to b4a76f5 Compare November 5, 2024 04:07

bshephar force-pushed the disable-webhooks branch 2 times, most recently from e9c55e2 to cff9550 Compare November 12, 2024 06:33

bshephar force-pushed the disable-webhooks branch from cff9550 to cf3f2de Compare November 13, 2024 05:09

bshephar force-pushed the disable-webhooks branch from cf3f2de to a0bd52f Compare November 20, 2024 11:07

bshephar force-pushed the disable-webhooks branch from a0bd52f to ebd2f8c Compare November 20, 2024 21:40

bshephar force-pushed the disable-webhooks branch from ebd2f8c to 9052c71 Compare November 21, 2024 02:29

rabi reviewed Nov 21, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

bshephar force-pushed the disable-webhooks branch from 9052c71 to 2e89b03 Compare November 21, 2024 06:19

bshephar mentioned this pull request Nov 22, 2024

Disable webhooks by default when possible openstack-k8s-operators/openstack-k8s-operators-ci#114

Closed

bshephar closed this Dec 9, 2024

bshephar deleted the disable-webhooks branch December 9, 2024 05:11

Disable Webhooks #466

Disable Webhooks #466

Uh oh!

Conversation

bshephar commented Oct 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Oct 31, 2024

Uh oh!

softwarefactory-project-zuul bot commented Oct 31, 2024

Uh oh!

bshephar commented Nov 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bshephar commented Nov 1, 2024

Uh oh!

softwarefactory-project-zuul bot commented Nov 2, 2024

Uh oh!

softwarefactory-project-zuul bot commented Nov 5, 2024

Uh oh!

bshephar commented Nov 11, 2024

Uh oh!

bshephar commented Nov 12, 2024

Uh oh!

softwarefactory-project-zuul bot commented Nov 12, 2024

Uh oh!

softwarefactory-project-zuul bot commented Nov 13, 2024

Uh oh!

softwarefactory-project-zuul bot commented Nov 20, 2024

Uh oh!

bshephar commented Nov 20, 2024

Uh oh!

bshephar commented Nov 20, 2024

Uh oh!

Uh oh!

rabi commented Nov 21, 2024

Uh oh!

bshephar commented Nov 22, 2024

Uh oh!

dprince commented Nov 22, 2024

Uh oh!

bshephar commented Nov 22, 2024

Uh oh!

bshephar commented Nov 22, 2024

Uh oh!

bshephar commented Dec 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bshephar commented Oct 31, 2024 •

edited

Loading

bshephar commented Nov 1, 2024 •

edited

Loading