[OSPRH-21020] Scrape all rabbitmq pods #780

vyzigold · 2025-10-31T11:16:31Z

RabbitMQ can be deployed with multiple replicas, while each replica
will export different metrics. Having a ScrapeConfig with a service
as a target means we're not scraping all metrics. This PR
uses the Endpoints object to retrieve IP addresses of all rabbitmq
related pods and creates a ScrapeConfig to scrape each pod through
its IP address. This keeps the pattern of using ScrapeConfigs for
configuring scraping.

Depends-On: infrawatch/feature-verification-tests#317

openshift-ci · 2025-10-31T11:16:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vyzigold

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [vyzigold]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

vyzigold · 2025-10-31T11:17:34Z

FVT will fail now, they need to be adjusted.

softwarefactory-project-zuul · 2025-10-31T13:17:45Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/26cc499d02c84360a3c384988c0b7989

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 46m 14s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 17m 18s
✔️ telemetry-openstack-meta-content-provider-master SUCCESS in 1h 55m 24s
❌ telemetry-operator-multinode-default-telemetry FAILURE in 1h 25m 33s
❌ functional-tests-osp18 FAILURE in 1h 35m 36s

A bugfix proposed in openstack-k8s-operators/telemetry-operator#780 changes how rabbitmq metrics are being scraped. The FVT needs to be adjusted for the tests to pass. For a limited time, both versions of telemetry-operator (pre-bugfix and post-bugfix) are allowed to pass the tests. A PR to remove the pre-bugfix tasks will be created in the future.

softwarefactory-project-zuul · 2025-10-31T15:59:23Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/977b5ae2e2344525b99154fa0f53868e

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 49m 29s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 16m 20s
✔️ telemetry-openstack-meta-content-provider-master SUCCESS in 1h 54m 22s
✔️ telemetry-operator-multinode-default-telemetry SUCCESS in 1h 28m 32s
❌ functional-tests-osp18 FAILURE in 1h 34m 07s

softwarefactory-project-zuul · 2025-10-31T18:31:36Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/43efd572382b4069a721e061d995281a

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 41m 45s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 16m 21s
✔️ telemetry-openstack-meta-content-provider-master SUCCESS in 2h 23m 19s
✔️ telemetry-operator-multinode-default-telemetry SUCCESS in 1h 22m 44s
❌ functional-tests-osp18 FAILURE in 2h 04m 25s

vyzigold · 2025-11-03T08:52:35Z

recheck

vyzigold · 2025-11-04T17:34:45Z

This needs a few adjustments to make it work with TLS disabled.

vyzigold · 2025-12-02T09:47:45Z

recheck

vyzigold · 2025-12-02T09:49:37Z

/retest

softwarefactory-project-zuul · 2025-12-02T12:52:14Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/acbb6851fe2f48f188852c5a972a4f6d

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 42m 04s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 21m 05s
✔️ telemetry-openstack-meta-content-provider-master SUCCESS in 3h 02m 48s
❌ telemetry-operator-multinode-default-telemetry RETRY_LIMIT in 9m 13s
❌ functional-tests-osp18 FAILURE in 2h 43m 27s

vyzigold · 2025-12-02T14:32:29Z

recheck tempest tests didn't seem to get executed

vyzigold · 2025-12-02T17:39:57Z

/retest

vyzigold · 2025-12-02T20:56:57Z

/retest

vyzigold · 2025-12-02T22:04:40Z

/retest

softwarefactory-project-zuul · 2025-12-08T22:56:05Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/c8414ce5d81c410889a2546337be8fa7

❌ openstack-k8s-operators-content-provider FAILURE in 19m 18s
❌ telemetry-operator-multinode-cloudkitty FAILURE in 1h 08m 22s
✔️ telemetry-openstack-meta-content-provider-master SUCCESS in 2h 00m 46s
⚠️ telemetry-operator-multinode-default-telemetry SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
❌ functional-tests-osp18 FAILURE in 1h 40m 47s

RabbitMQ can be deployed with multiple replicas, while each replica will export different metrics. Having a ScrapeConfig with a service as a target means we're not scraping all metrics. This PR uses the Endpoints object to retrieve IP addresses of all rabbitmq related pods and creates a ScrapeConfig to scrape each pod through its IP address. This keeps the pattern of using ScrapeConfigs for configuring scraping.

softwarefactory-project-zuul · 2025-12-09T11:33:26Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/757814351a5d4ccf887b757498f869c0

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 47m 41s
✔️ telemetry-operator-multinode-cloudkitty SUCCESS in 1h 38m 15s
✔️ telemetry-openstack-meta-content-provider-master SUCCESS in 2h 08m 53s
✔️ telemetry-operator-multinode-default-telemetry SUCCESS in 1h 28m 10s
❌ functional-tests-osp18 FAILURE in 1h 49m 22s

vyzigold · 2025-12-10T17:01:51Z

recheck - known issue with current-tested images missing

vyzigold · 2025-12-10T17:01:57Z

/retest

vyzigold · 2025-12-10T20:38:28Z

/retest

A bugfix proposed in openstack-k8s-operators/telemetry-operator#780 changes how rabbitmq metrics are being scraped. The FVT needs to be adjusted for the tests to pass. For a limited time, both versions of telemetry-operator (pre-bugfix and post-bugfix) are allowed to pass the tests. A PR to remove the pre-bugfix tasks will be created in the future.

jlarriba · 2025-12-12T10:38:38Z

/lgtm

vyzigold · 2025-12-12T14:18:28Z

/cherry-pick 18.0-fr4

openshift-cherrypick-robot · 2025-12-12T14:19:06Z

@vyzigold: #780 failed to apply on top of branch "18.0-fr4":

Applying: Scrape all rabbitmq pods
Using index info to reconstruct a base tree...
A	internal/controller/metricstorage_controller.go
A	internal/dashboards/openstack-rabbitmq.go
A	internal/metricstorage/const.go
A	internal/metricstorage/scrape_config.go
A	internal/metricstorage/scrape_config_mysqld_exporter.go
A	test/kuttl/tests/metricstorage/01-assert.yaml
A	test/kuttl/tests/metricstorage/04-assert.yaml
A	test/kuttl/tests/tls/02-assert.yaml
Falling back to patching base and 3-way merge...
Auto-merging tests/kuttl/suites/tls/tests/02-assert.yaml
Auto-merging tests/kuttl/suites/metricstorage/tests/04-assert.yaml
Auto-merging tests/kuttl/suites/metricstorage/tests/01-assert.yaml
CONFLICT (rename/delete): internal/metricstorage/scrape_config_mysqld_exporter.go deleted in Scrape all rabbitmq pods and renamed to pkg/metricstorage/scrape_config_mysqld_exporter.go in HEAD. Version HEAD of pkg/metricstorage/scrape_config_mysqld_exporter.go left in tree.
Auto-merging controllers/metricstorage_controller.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 Scrape all rabbitmq pods

Details

In response to this:

/cherry-pick 18.0-fr4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

RabbitMQ can be deployed with multiple replicas, while each replica will export different metrics. Having a ScrapeConfig with a service as a target means we're not scraping all metrics. This PR uses the Endpoints object to retrieve IP addresses of all rabbitmq related pods and creates a ScrapeConfig to scrape each pod through its IP address. This keeps the pattern of using ScrapeConfigs for configuring scraping. Cherry-pick from: openstack-k8s-operators#780

vyzigold marked this pull request as draft October 31, 2025 11:16

openshift-ci bot requested review from abays and dprince October 31, 2025 11:16

openshift-ci bot added do-not-merge/work-in-progress approved labels Oct 31, 2025

vyzigold force-pushed the fix_rabbit branch from a2b09f3 to 4ca548f Compare October 31, 2025 14:03

vyzigold mentioned this pull request Oct 31, 2025

[OSPRH-21020] Adjust rabbitmq tests infrawatch/feature-verification-tests#317

Merged

vyzigold force-pushed the fix_rabbit branch from 4ca548f to bf095e5 Compare October 31, 2025 16:07

vyzigold marked this pull request as ready for review November 3, 2025 08:04

openshift-ci bot removed the do-not-merge/work-in-progress label Nov 3, 2025

openshift-ci bot requested review from elfiesmelfie and viroel November 3, 2025 08:04

vyzigold force-pushed the fix_rabbit branch 5 times, most recently from 4435a50 to b4c2d73 Compare November 4, 2025 14:16

vyzigold marked this pull request as draft November 4, 2025 17:34

openshift-ci bot added the do-not-merge/work-in-progress label Nov 4, 2025

vyzigold force-pushed the fix_rabbit branch from b4c2d73 to 0d27f4e Compare November 4, 2025 22:46

vyzigold marked this pull request as ready for review November 5, 2025 07:21

openshift-ci bot removed the do-not-merge/work-in-progress label Nov 5, 2025

vyzigold force-pushed the fix_rabbit branch from f562b9d to 1fc827c Compare December 2, 2025 17:38

vyzigold force-pushed the fix_rabbit branch from 1fc827c to 8d099bc Compare December 2, 2025 18:00

vyzigold force-pushed the fix_rabbit branch from 8d099bc to 00aa5f5 Compare December 8, 2025 20:54

vyzigold force-pushed the fix_rabbit branch from 00aa5f5 to adc44e0 Compare December 9, 2025 09:23

vyzigold marked this pull request as ready for review December 12, 2025 07:29

openshift-ci bot removed the do-not-merge/work-in-progress label Dec 12, 2025

openshift-ci bot requested a review from elfiesmelfie December 12, 2025 07:29

openshift-ci bot assigned jlarriba Dec 12, 2025

openshift-ci bot added the lgtm label Dec 12, 2025

openshift-merge-bot bot merged commit 5f3d940 into openstack-k8s-operators:main Dec 12, 2025
6 checks passed

vyzigold mentioned this pull request Dec 16, 2025

[18.0-fr4] [OSPRH-21020] Scrape all rabbitmq pods #816

Merged

[OSPRH-21020] Scrape all rabbitmq pods #780

[OSPRH-21020] Scrape all rabbitmq pods #780

Uh oh!

Conversation

vyzigold commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Oct 31, 2025

Uh oh!

vyzigold commented Oct 31, 2025

Uh oh!

softwarefactory-project-zuul bot commented Oct 31, 2025

Uh oh!

softwarefactory-project-zuul bot commented Oct 31, 2025

Uh oh!

softwarefactory-project-zuul bot commented Oct 31, 2025

Uh oh!

vyzigold commented Nov 3, 2025

Uh oh!

vyzigold commented Nov 4, 2025

Uh oh!

vyzigold commented Dec 2, 2025

Uh oh!

vyzigold commented Dec 2, 2025

Uh oh!

softwarefactory-project-zuul bot commented Dec 2, 2025

Uh oh!

vyzigold commented Dec 2, 2025

Uh oh!

vyzigold commented Dec 2, 2025

Uh oh!

vyzigold commented Dec 2, 2025

Uh oh!

vyzigold commented Dec 2, 2025

Uh oh!

softwarefactory-project-zuul bot commented Dec 8, 2025

Uh oh!

softwarefactory-project-zuul bot commented Dec 9, 2025

Uh oh!

vyzigold commented Dec 10, 2025

Uh oh!

vyzigold commented Dec 10, 2025

Uh oh!

vyzigold commented Dec 10, 2025

Uh oh!

jlarriba commented Dec 12, 2025

Uh oh!

Uh oh!

vyzigold commented Dec 12, 2025

Uh oh!

openshift-cherrypick-robot commented Dec 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vyzigold commented Oct 31, 2025 •

edited

Loading