Skip to content

USHIFT-6951: Add metrics-server as optional MicroShift component#6808

Open
copejon wants to merge 5 commits into
openshift:mainfrom
copejon:ushift-6951/metrics-server
Open

USHIFT-6951: Add metrics-server as optional MicroShift component#6808
copejon wants to merge 5 commits into
openshift:mainfrom
copejon:ushift-6951/metrics-server

Conversation

@copejon
Copy link
Copy Markdown
Contributor

@copejon copejon commented Jun 5, 2026

Add metrics-server as an optional MicroShift component deployed via kustomize manifests and packaged as microshift-metrics-server and microshift-metrics-server-release-info RPM sub-packages.

This is PR 1/3 splitting #6763 into independently-mergeable patches. The three PRs can merge in any order.
Siblings: #6809 (kube-state-metrics), #6810 (node-exporter)

What's included

  • Kustomize manifests (assets/optional/metrics-server/) — Deployment, APIService, RBAC, NetworkPolicy, PDB, service-ca TLS, and openshift-monitoring namespace
  • Kubelet client cert provisioning — dedicated cert under kube-apiserver-to-kubelet-signer, applied as Secret + ConfigMap after the namespace exists (pkg/cmd/metrics.go)
  • Healthcheck registrationmergeWorkloads() function and metrics-server map entry in optional workload paths
  • RPM sub-packages%package metrics-server and %package metrics-server-release-info with per-file installs
  • Otel-collector drop-inotelcol.d/ directory structure and microshift-metrics-server.yaml scrape config; service file updated to load drop-in configs via bash wrapper
  • Auto-rebase integrationupdate_metrics_images() in rebase.sh + assets_metrics.yaml for all three exporters (shared infrastructure lives in this PR)
  • Test infra — RPM names added to test/bin/common.sh

Summary by CodeRabbit

  • New Features

    • Optional Kubernetes Metrics Server: namespace, service, RBAC, API registration, hardened deployment, and audit profiles.
    • Metrics Server integrated into observability: collector discovers and scrapes the Metrics Server endpoint.
    • Automated provisioning of TLS credentials for Metrics Server kubelet connectivity; service startup now loads extra observability configs.
  • Chores / Packaging

    • Optional RPMs, per-architecture image pinning, and extended release/manifest image-management workflow.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 5, 2026
@openshift-ci-robot
Copy link
Copy Markdown

openshift-ci-robot commented Jun 5, 2026

@copejon: This pull request references USHIFT-6951 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Add metrics-server as an optional MicroShift component deployed via kustomize manifests and packaged as microshift-metrics-server and microshift-metrics-server-release-info RPM sub-packages.

This is PR 1/3 splitting #6763 into independently-mergeable patches. The three PRs can merge in any order.

What's included

  • Kustomize manifests (assets/optional/metrics-server/) — Deployment, APIService, RBAC, NetworkPolicy, PDB, service-ca TLS, and openshift-monitoring namespace
  • Kubelet client cert provisioning — dedicated cert under kube-apiserver-to-kubelet-signer, applied as Secret + ConfigMap after the namespace exists (pkg/cmd/metrics.go)
  • Healthcheck registrationmergeWorkloads() function and metrics-server map entry in optional workload paths
  • RPM sub-packages%package metrics-server and %package metrics-server-release-info with per-file installs
  • Otel-collector drop-inotelcol.d/ directory structure and microshift-metrics-server.yaml scrape config; service file updated to load drop-in configs via bash wrapper
  • Auto-rebase integrationupdate_metrics_images() in rebase.sh + assets_metrics.yaml for all three exporters (shared infrastructure lives in this PR)
  • Test infra — RPM names added to test/bin/common.sh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Jun 5, 2026

Caution

Review failed

An error occurred during the review process. Please try again later.

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Adds metrics-server as an optional MicroShift component: Kubernetes manifests and kustomizations, RBAC and APIService registration, deployment and audit profiles, certificate CSR and runtime provisioning, OpenTelemetry drop-in and systemd wrapper, packaging/subpackages, and auto-rebase tooling to generate image overrides.

Changes

Metrics Server Optional Component

Layer / File(s) Summary
Namespace, ServiceAccount, ClusterRole and RBAC + APIService
assets/optional/metrics-server/00-namespace.yaml, assets/optional/metrics-server/01-service-account.yaml, assets/optional/metrics-server/01-cluster-role.yaml, assets/optional/metrics-server/01-cluster-role-binding.yaml, assets/optional/metrics-server/01-cluster-role-binding-auth-delegator.yaml, assets/optional/metrics-server/01-role-binding-auth-reader.yaml, assets/optional/metrics-server/04-service.yaml, assets/optional/metrics-server/04-api-service.yaml, assets/optional/metrics-server/kustomization.yaml
Namespace openshift-monitoring, ServiceAccount, ClusterRole and bindings, RoleBinding in kube-system, Service (HTTPS), and APIService registration for metrics.k8s.io/v1beta1.
Deployment, audit profiles, kustomizations, release JSONs
assets/optional/metrics-server/03-deployment.yaml, assets/optional/metrics-server/02-configmap-audit-profiles.yaml, assets/optional/metrics-server/kustomization.aarch64.yaml, assets/optional/metrics-server/kustomization.x86_64.yaml, assets/optional/metrics-server/release-metrics-server-aarch64.json, assets/optional/metrics-server/release-metrics-server-x86_64.json
Single-replica Deployment with TLS/kubelet client cert mounts, probes, hardened security context, audit-profile ConfigMap, arch-specific image overrides and pinned per-arch release JSONs.
Certificate CSR and runtime provisioning
pkg/cmd/init.go, pkg/util/cryptomaterial/certinfo.go, pkg/cmd/metrics.go, pkg/cmd/run.go
Adds metrics-server-kubelet-client CSR entry and minor CSR UserInfo adjustment, helper to compute cert dir, provisionMetricsServerCerts() reads cert/key/CA from disk and creates/updates TLS Secret and kubelet CA ConfigMap after namespace exists; background goroutine invokes provisioning.
Optional workloads merge
pkg/healthcheck/microshift_optional_workloads.go
Adds mergeWorkloads() and uses it to merge multiple optional workload lists per namespace instead of overwriting.
Observability / OpenTelemetry and packaging
packaging/observability/microshift-observability.service, packaging/observability/otelcol.d/microshift-metrics-server.yaml, packaging/rpm/microshift.spec
systemd ExecStart composes OTel Collector args including otelcol.d/ drop-ins; new Prometheus receiver and pipeline scraping metrics-server over HTTPS with Kubernetes endpoint discovery and TLS CA verification; RPM subpackages and install/%files entries for manifests, otel drop-in, and release JSONs.
Auto-rebase, assets registration, and tests
scripts/auto-rebase/rebase.sh, scripts/auto-rebase/assets.yaml, scripts/auto-rebase/assets_metrics.yaml, test/bin/common.sh
Adds OPTIONAL_COMPONENTS and update_metrics_images() to generate per-arch release JSON and kustomize image overrides for optional components; registers optional metrics assets for auto-rebase; updates test RPM lists to include new subpackages.

Sequence Diagram(s)

sequenceDiagram
  participant MicroShift
  participant Kustomize
  participant provisionMetricsServerCerts
  participant KubernetesAPI
  MicroShift->>Kustomize: apply metrics-server kustomization
  Kustomize->>KubernetesAPI: create Namespace/ServiceAccount/RBAC/Service/APIService/Deployment
  MicroShift->>provisionMetricsServerCerts: start background provisioning
  provisionMetricsServerCerts->>KubernetesAPI: wait for Namespace then Create/Update Secret+ConfigMap
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Suggested labels

ready-for-human-review


Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

  • Ignore

❌ Failed checks (1 error, 1 warning)

Check name Status Explanation Resolution
Container-Privileges ❌ Error Namespace manifest sets pod-security.kubernetes.io/* labels to 'privileged' level without documented justification, allowing privileged workloads despite container being properly hardened. Use restricted-v2 or baseline pod-security level for openshift-monitoring namespace, or document why privileged enforcement is required for metrics-server.
Docstring Coverage ⚠️ Warning Docstring coverage is 42.86% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (13 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and specifically describes the main change: adding metrics-server as an optional MicroShift component. It directly reflects the primary objective of the changeset.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PR adds no new Ginkgo tests or modifies no existing test files, so check is not applicable.
Test Structure And Quality ✅ Passed No Ginkgo tests added or modified in this PR. Repository uses testify framework, not Ginkgo. Check not applicable.
Microshift Test Compatibility ✅ Passed This PR adds no Ginkgo e2e tests—only manifests, build configuration, and infrastructure code. The custom check for MicroShift test compatibility is not applicable.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PR adds metrics-server component via manifests and Go code only. No new Ginkgo e2e tests are present, so the SNO compatibility check does not apply.
Topology-Aware Scheduling Compatibility ✅ Passed Deployment uses 1 replica, generic nodeSelector, and only tolerates master taint; no pod affinity, topology spread, or wildcard tolerations that break SNO/TNF/TNA/HyperShift.
Ote Binary Stdout Contract ✅ Passed No non-JSON stdout writes found in process-level code. All logging uses klog (writes to files/stderr), and all fmt usage is fmt.Errorf() in returns. OTE binary stdout contract is preserved.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed This PR adds metrics-server as an optional MicroShift component but does not add any Ginkgo e2e tests (no *_test.go files with Describe/Context/It patterns). The check is not applicable to this PR.
No-Weak-Crypto ✅ Passed No weak crypto, custom implementations, or insecure comparisons detected. Strong TLS cipher suites (AES-GCM, ChaCha20-Poly1305 with ECDHE) used.
No-Sensitive-Data-In-Logs ✅ Passed Logging statements only expose manifest paths and namespace names. Certificate keys, digests, and secrets are not logged to console/systemd logs.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

Tools execution failed with the following error:

Failed to run tools: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error)


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot requested review from eslutsky and pacevedom June 5, 2026 04:30
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Jun 5, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: copejon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 5, 2026
@copejon copejon force-pushed the ushift-6951/metrics-server branch from a185390 to 2f00a69 Compare June 5, 2026 06:29
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 9

🧹 Nitpick comments (1)
packaging/observability/microshift-observability.service (1)

11-11: ⚡ Quick win

Use an argv array instead of a concatenated shell string

Building ARGS as a string is brittle (word-splitting/path edge cases). Use a bash array and quoted expansion.

Proposed change
-ExecStart=/bin/bash -c 'ARGS="--config=file:/etc/microshift/observability/opentelemetry-collector.yaml"; for f in /etc/microshift/observability/otelcol.d/*.yaml; do [ -f "$$f" ] && ARGS="$$ARGS --config=file:$$f"; done; exec /usr/bin/opentelemetry-collector $$ARGS'
+ExecStart=/bin/bash -c 'args=(--config=file:/etc/microshift/observability/opentelemetry-collector.yaml); for f in /etc/microshift/observability/otelcol.d/*.yaml; do [ -f "$$f" ] && args+=(--config=file:$$f); done; exec /usr/bin/opentelemetry-collector "$${args[@]}"'
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packaging/observability/microshift-observability.service` at line 11, The
ExecStart line builds a brittle space-joined ARGS string; change it to build a
bash array and use quoted array expansion when executing the collector:
initialize ARGS=() (or ARGS=()), append elements as
ARGS+=(--config="file:/etc/microshift/observability/opentelemetry-collector.yaml")
and inside the loop for f in /etc/microshift/observability/otelcol.d/*.yaml; do
[ -f "$f" ] && ARGS+=(--config="file:$f"); done; finally exec
/usr/bin/opentelemetry-collector "${ARGS[@]}" so that each --config is a
separate argv element and you avoid word-splitting/path edge cases for the
ExecStart/ARGS usage.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@assets/optional/metrics-server/00-namespace.yaml`:
- Around line 7-9: The namespace-level PodSecurity labels
pod-security.kubernetes.io/enforce, pod-security.kubernetes.io/audit, and
pod-security.kubernetes.io/warn are set to "privileged" which is too permissive;
change their values to a safer profile (e.g., "baseline" or "restricted") unless
you have a documented exception, and ensure the three keys (enforce, audit,
warn) are updated consistently to the chosen profile so namespace-wide policies
default to the tightened posture.

In `@assets/optional/metrics-server/03-deployment.yaml`:
- Around line 67-70: The metrics-server container currently only specifies
resource requests; add a matching limits block under the same resources section
for the metrics-server container to enforce CPU and memory caps (e.g.,
limits.cpu: 100m and limits.memory: 100Mi). Update the Deployment's container
resources block (the metrics-server container in the manifest) to include both
requests and limits so it conforms to the "Resource limits (cpu, memory) on
every container" guideline.
- Around line 71-75: The container securityContext currently sets
allowPrivilegeEscalation, readOnlyRootFilesystem and runAsNonRoot but does not
drop Linux capabilities; update the Pod/Container spec by adding a capabilities
block under securityContext (e.g., in the same container spec that contains
allowPrivilegeEscalation/readOnlyRootFilesystem/runAsNonRoot) with
capabilities.drop set to ["ALL"] so all capabilities are removed by default and
then explicitly add back any minimal capabilities only if absolutely required
elsewhere; ensure the change is applied alongside the existing
terminationMessagePolicy and other securityContext fields.

In `@assets/optional/metrics-server/kustomization.yaml`:
- Around line 3-13: The kustomization.yaml resources list is missing the two
RPM-installed manifests; update the resources array (the same block that
currently lists 00-namespace.yaml, 03-deployment.yaml, 04-service.yaml, etc.) to
include network-policy-downstream.yaml and pod-disruption-budget.yaml so both
NetworkPolicy and PodDisruptionBudget are applied; add those two filenames to
the resources list in assets/optional/metrics-server/kustomization.yaml.

In `@pkg/cmd/metrics.go`:
- Around line 44-50: The polling closure passed to wait.PollUntilContextTimeout
currently treats any Get() error as "not ready" and keeps retrying; change the
logic in the closure used with wait.PollUntilContextTimeout so that after
calling clientset.CoreV1().Namespaces().Get(ctx, ns, metav1.GetOptions{}), if
err == nil return true,nil; if apierrors.IsNotFound(err) log and return
false,nil to keep retrying; for any other error return false, err (so the poll
stops and surfaces auth/TLS/transport failures). Ensure you import/use
apierrors.IsNotFound and update the closure around
clientset.CoreV1().Namespaces().Get accordingly.
- Around line 24-31: The current check uses
util.PathExists(metricsServerManifestPath) so provisioning proceeds whenever the
file exists even if the metrics component is not enabled; change the gating to
verify the component is enabled and that any configured kustomization paths are
present (e.g., read the configured kustomizationPaths or component flag used by
your installer and ensure they are enabled) before attempting cert provisioning.
Update the logic around util.PathExists(metricsServerManifestPath) to first
consult the metrics enablement/config (the same config that controls
installation) and validate each configured kustomization path exists, returning
early if the component is disabled or no configured paths are present.

In `@scripts/auto-rebase/assets_metrics.yaml`:
- Around line 34-37: Update the two YAML asset entries so they match the rest of
the PR naming convention: change the "file" values
"release-metrics-aarch64.json" and "release-metrics-x86_64.json" to
"release-metrics-server-aarch64.json" and "release-metrics-server-x86_64.json"
respectively in the assets_metrics.yaml file (the two "- file:" entries shown),
leaving the corresponding "ignore" values unchanged.

In `@scripts/auto-rebase/assets.yaml`:
- Around line 304-326: The asset list under the optional/metrics-server/ entry
is missing the two manifests that the packaging script expects; update the files
list for the optional/metrics-server/ dir to include entries for
network-policy-downstream.yaml and pod-disruption-budget.yaml so the asset set
matches packaging/rpm/microshift.spec; ensure the new entries use the same
indentation/format as the existing file: ... entries (e.g., add "- file:
network-policy-downstream.yaml" and "- file: pod-disruption-budget.yaml" under
the files: block for the optional/metrics-server/ dir).

In `@scripts/auto-rebase/rebase.sh`:
- Around line 1184-1187: The script currently skips unknown images when
release_tag is empty (checking release_tag, orig_image, component_dir) which
allows silent incomplete kustomization.*.yaml outputs; change the behavior to
fail-fast by replacing the continue path with a non-zero exit (e.g., >&2 echo
descriptive error including orig_image and component_dir, then exit 1) so the
rebase.sh run stops immediately on unmapped images and surfaces the problem to
CI.

---

Nitpick comments:
In `@packaging/observability/microshift-observability.service`:
- Line 11: The ExecStart line builds a brittle space-joined ARGS string; change
it to build a bash array and use quoted array expansion when executing the
collector: initialize ARGS=() (or ARGS=()), append elements as
ARGS+=(--config="file:/etc/microshift/observability/opentelemetry-collector.yaml")
and inside the loop for f in /etc/microshift/observability/otelcol.d/*.yaml; do
[ -f "$f" ] && ARGS+=(--config="file:$f"); done; finally exec
/usr/bin/opentelemetry-collector "${ARGS[@]}" so that each --config is a
separate argv element and you avoid word-splitting/path edge cases for the
ExecStart/ARGS usage.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 25baa466-07b1-4aa1-90d3-a9e96b03c015

📥 Commits

Reviewing files that changed from the base of the PR and between 93e835e and 2f00a69.

📒 Files selected for processing (27)
  • assets/optional/metrics-server/00-namespace.yaml
  • assets/optional/metrics-server/01-cluster-role-binding-auth-delegator.yaml
  • assets/optional/metrics-server/01-cluster-role-binding.yaml
  • assets/optional/metrics-server/01-cluster-role.yaml
  • assets/optional/metrics-server/01-role-binding-auth-reader.yaml
  • assets/optional/metrics-server/01-service-account.yaml
  • assets/optional/metrics-server/02-configmap-audit-profiles.yaml
  • assets/optional/metrics-server/03-deployment.yaml
  • assets/optional/metrics-server/04-api-service.yaml
  • assets/optional/metrics-server/04-service.yaml
  • assets/optional/metrics-server/kustomization.aarch64.yaml
  • assets/optional/metrics-server/kustomization.x86_64.yaml
  • assets/optional/metrics-server/kustomization.yaml
  • assets/optional/metrics-server/release-metrics-server-aarch64.json
  • assets/optional/metrics-server/release-metrics-server-x86_64.json
  • packaging/observability/microshift-observability.service
  • packaging/observability/otelcol.d/microshift-metrics-server.yaml
  • packaging/rpm/microshift.spec
  • pkg/cmd/init.go
  • pkg/cmd/metrics.go
  • pkg/cmd/run.go
  • pkg/healthcheck/microshift_optional_workloads.go
  • pkg/util/cryptomaterial/certinfo.go
  • scripts/auto-rebase/assets.yaml
  • scripts/auto-rebase/assets_metrics.yaml
  • scripts/auto-rebase/rebase.sh
  • test/bin/common.sh

Comment on lines +7 to +9
pod-security.kubernetes.io/enforce: privileged
pod-security.kubernetes.io/audit: privileged
pod-security.kubernetes.io/warn: privileged
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Namespace-level privileged Pod Security is overly permissive.

Setting enforce/audit/warn to privileged at Line 7-Line 9 weakens namespace isolation for all workloads. Prefer baseline/restricted unless a strict exception is required.

Suggested tightening
-    pod-security.kubernetes.io/enforce: privileged
-    pod-security.kubernetes.io/audit: privileged
-    pod-security.kubernetes.io/warn: privileged
+    pod-security.kubernetes.io/enforce: baseline
+    pod-security.kubernetes.io/audit: restricted
+    pod-security.kubernetes.io/warn: restricted

As per coding guidelines, "**/*.{yaml,yml,json}: Flag privileged: true ... in container/Kubernetes manifests".

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
pod-security.kubernetes.io/enforce: privileged
pod-security.kubernetes.io/audit: privileged
pod-security.kubernetes.io/warn: privileged
pod-security.kubernetes.io/enforce: baseline
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/warn: restricted
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@assets/optional/metrics-server/00-namespace.yaml` around lines 7 - 9, The
namespace-level PodSecurity labels pod-security.kubernetes.io/enforce,
pod-security.kubernetes.io/audit, and pod-security.kubernetes.io/warn are set to
"privileged" which is too permissive; change their values to a safer profile
(e.g., "baseline" or "restricted") unless you have a documented exception, and
ensure the three keys (enforce, audit, warn) are updated consistently to the
chosen profile so namespace-wide policies default to the tightened posture.

Comment on lines +67 to +70
resources:
requests:
cpu: 1m
memory: 40Mi
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Add container resource limits.

Only requests are set; limits are missing for metrics-server. Add CPU/memory limits to enforce predictable runtime behavior.

As per coding guidelines, "Resource limits (cpu, memory) on every container".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@assets/optional/metrics-server/03-deployment.yaml` around lines 67 - 70, The
metrics-server container currently only specifies resource requests; add a
matching limits block under the same resources section for the metrics-server
container to enforce CPU and memory caps (e.g., limits.cpu: 100m and
limits.memory: 100Mi). Update the Deployment's container resources block (the
metrics-server container in the manifest) to include both requests and limits so
it conforms to the "Resource limits (cpu, memory) on every container" guideline.

Comment thread assets/optional/metrics-server/03-deployment.yaml
Comment thread assets/optional/metrics-server/kustomization.yaml
Comment thread pkg/cmd/metrics.go
Comment on lines +24 to +31
exists, err := util.PathExists(metricsServerManifestPath)
if err != nil {
return err
}
if !exists {
klog.V(2).Infof("Metrics-server manifests not found at %s, skipping cert provisioning", metricsServerManifestPath)
return nil
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Gate provisioning on configured kustomization paths, not only file presence.

This runs even when the component is installed but not enabled, causing avoidable polling/timeouts on startup.

Suggested fix
 func provisionMetricsServerCerts(ctx context.Context, cfg *config.Config) error {
 	exists, err := util.PathExists(metricsServerManifestPath)
 	if err != nil {
 		return err
 	}
 	if !exists {
 		klog.V(2).Infof("Metrics-server manifests not found at %s, skipping cert provisioning", metricsServerManifestPath)
 		return nil
 	}
+
+	configuredPaths, err := cfg.Manifests.GetKustomizationPaths()
+	if err != nil {
+		return fmt.Errorf("reading configured kustomization paths: %w", err)
+	}
+	enabled := false
+	for _, p := range configuredPaths {
+		if p == metricsServerManifestPath {
+			enabled = true
+			break
+		}
+	}
+	if !enabled {
+		klog.V(2).Infof("Metrics-server manifests found but not configured in kustomization paths, skipping cert provisioning")
+		return nil
+	}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/cmd/metrics.go` around lines 24 - 31, The current check uses
util.PathExists(metricsServerManifestPath) so provisioning proceeds whenever the
file exists even if the metrics component is not enabled; change the gating to
verify the component is enabled and that any configured kustomization paths are
present (e.g., read the configured kustomizationPaths or component flag used by
your installer and ensure they are enabled) before attempting cert provisioning.
Update the logic around util.PathExists(metricsServerManifestPath) to first
consult the metrics enablement/config (the same config that controls
installation) and validate each configured kustomization path exists, returning
early if the component is disabled or no configured paths are present.

Comment thread pkg/cmd/metrics.go
Comment thread scripts/auto-rebase/assets_metrics.yaml Outdated
Comment thread scripts/auto-rebase/assets.yaml
Comment thread scripts/auto-rebase/rebase.sh
@copejon copejon force-pushed the ushift-6951/metrics-server branch from 2f00a69 to 2552538 Compare June 5, 2026 06:46
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@assets/optional/metrics-server/02-configmap-audit-profiles.yaml`:
- Around line 2-38: The embedded audit policy YAMLs are malformed because keys
and values are written with JSON-style quoted strings; update
metadata-profile.yaml, none-profile.yaml, request-profile.yaml and
requestresponse-profile.yaml inside the ConfigMap data to valid YAML by removing
the unnecessary quotes around keys (apiVersion, kind, metadata, name,
omitStages, rules, level) and unquoting list items (e.g., RequestReceived), and
ensure each rule is a proper mapping (e.g., - level: Metadata) with correct
indentation so each policy is valid Kubernetes audit policy YAML.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 070ff4b2-5dbd-4430-8057-ef1fec980558

📥 Commits

Reviewing files that changed from the base of the PR and between 2f00a69 and 2552538.

⛔ Files ignored due to path filters (3)
  • etcd/vendor/github.com/openshift/microshift/pkg/config/config.go is excluded by !**/vendor/**
  • etcd/vendor/github.com/openshift/microshift/pkg/config/dns.go is excluded by !**/vendor/**
  • etcd/vendor/github.com/openshift/microshift/pkg/util/cryptomaterial/certinfo.go is excluded by !**/vendor/**
📒 Files selected for processing (27)
  • assets/optional/metrics-server/00-namespace.yaml
  • assets/optional/metrics-server/01-cluster-role-binding-auth-delegator.yaml
  • assets/optional/metrics-server/01-cluster-role-binding.yaml
  • assets/optional/metrics-server/01-cluster-role.yaml
  • assets/optional/metrics-server/01-role-binding-auth-reader.yaml
  • assets/optional/metrics-server/01-service-account.yaml
  • assets/optional/metrics-server/02-configmap-audit-profiles.yaml
  • assets/optional/metrics-server/03-deployment.yaml
  • assets/optional/metrics-server/04-api-service.yaml
  • assets/optional/metrics-server/04-service.yaml
  • assets/optional/metrics-server/kustomization.aarch64.yaml
  • assets/optional/metrics-server/kustomization.x86_64.yaml
  • assets/optional/metrics-server/kustomization.yaml
  • assets/optional/metrics-server/release-metrics-server-aarch64.json
  • assets/optional/metrics-server/release-metrics-server-x86_64.json
  • packaging/observability/microshift-observability.service
  • packaging/observability/otelcol.d/microshift-metrics-server.yaml
  • packaging/rpm/microshift.spec
  • pkg/cmd/init.go
  • pkg/cmd/metrics.go
  • pkg/cmd/run.go
  • pkg/healthcheck/microshift_optional_workloads.go
  • pkg/util/cryptomaterial/certinfo.go
  • scripts/auto-rebase/assets.yaml
  • scripts/auto-rebase/assets_metrics.yaml
  • scripts/auto-rebase/rebase.sh
  • test/bin/common.sh
✅ Files skipped from review due to trivial changes (5)
  • assets/optional/metrics-server/kustomization.aarch64.yaml
  • test/bin/common.sh
  • scripts/auto-rebase/assets_metrics.yaml
  • assets/optional/metrics-server/release-metrics-server-x86_64.json
  • assets/optional/metrics-server/release-metrics-server-aarch64.json
🚧 Files skipped from review as they are similar to previous changes (19)
  • pkg/cmd/run.go
  • assets/optional/metrics-server/kustomization.x86_64.yaml
  • assets/optional/metrics-server/01-cluster-role-binding-auth-delegator.yaml
  • assets/optional/metrics-server/00-namespace.yaml
  • assets/optional/metrics-server/kustomization.yaml
  • assets/optional/metrics-server/01-cluster-role.yaml
  • scripts/auto-rebase/assets.yaml
  • assets/optional/metrics-server/01-role-binding-auth-reader.yaml
  • packaging/observability/microshift-observability.service
  • packaging/observability/otelcol.d/microshift-metrics-server.yaml
  • assets/optional/metrics-server/04-api-service.yaml
  • assets/optional/metrics-server/01-cluster-role-binding.yaml
  • pkg/cmd/init.go
  • pkg/healthcheck/microshift_optional_workloads.go
  • assets/optional/metrics-server/01-service-account.yaml
  • pkg/cmd/metrics.go
  • assets/optional/metrics-server/03-deployment.yaml
  • packaging/rpm/microshift.spec
  • scripts/auto-rebase/rebase.sh

Comment on lines +2 to +38
data:
metadata-profile.yaml: |-
"apiVersion": "audit.k8s.io/v1"
"kind": "Policy"
"metadata":
"name": "Metadata"
"omitStages":
- "RequestReceived"
"rules":
- "level": "Metadata"
none-profile.yaml: |-
"apiVersion": "audit.k8s.io/v1"
"kind": "Policy"
"metadata":
"name": "None"
"omitStages":
- "RequestReceived"
"rules":
- "level": "None"
request-profile.yaml: |-
"apiVersion": "audit.k8s.io/v1"
"kind": "Policy"
"metadata":
"name": "Request"
"omitStages":
- "RequestReceived"
"rules":
- "level": "Request"
requestresponse-profile.yaml: |-
"apiVersion": "audit.k8s.io/v1"
"kind": "Policy"
"metadata":
"name": "RequestResponse"
"omitStages":
- "RequestReceived"
"rules":
- "level": "RequestResponse"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical | ⚡ Quick win

Malformed YAML syntax in embedded audit profiles.

All four audit policy documents use JSON-style quoted keys ("apiVersion", "kind", etc.) instead of proper YAML syntax. Kubernetes audit policies should be written in standard YAML format without quoted keys.

🔧 Proposed fix
 data:
   metadata-profile.yaml: |-
-    "apiVersion": "audit.k8s.io/v1"
-    "kind": "Policy"
-    "metadata":
-      "name": "Metadata"
-    "omitStages":
+    apiVersion: audit.k8s.io/v1
+    kind: Policy
+    metadata:
+      name: Metadata
+    omitStages:
     - "RequestReceived"
-    "rules":
-    - "level": "Metadata"
+    rules:
+    - level: Metadata
   none-profile.yaml: |-
-    "apiVersion": "audit.k8s.io/v1"
-    "kind": "Policy"
-    "metadata":
-      "name": "None"
-    "omitStages":
+    apiVersion: audit.k8s.io/v1
+    kind: Policy
+    metadata:
+      name: None
+    omitStages:
     - "RequestReceived"
-    "rules":
-    - "level": "None"
+    rules:
+    - level: None
   request-profile.yaml: |-
-    "apiVersion": "audit.k8s.io/v1"
-    "kind": "Policy"
-    "metadata":
-      "name": "Request"
-    "omitStages":
+    apiVersion: audit.k8s.io/v1
+    kind: Policy
+    metadata:
+      name: Request
+    omitStages:
     - "RequestReceived"
-    "rules":
-    - "level": "Request"
+    rules:
+    - level: Request
   requestresponse-profile.yaml: |-
-    "apiVersion": "audit.k8s.io/v1"
-    "kind": "Policy"
-    "metadata":
-      "name": "RequestResponse"
-    "omitStages":
+    apiVersion: audit.k8s.io/v1
+    kind: Policy
+    metadata:
+      name: RequestResponse
+    omitStages:
     - "RequestReceived"
-    "rules":
-    - "level": "RequestResponse"
+    rules:
+    - level: RequestResponse
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@assets/optional/metrics-server/02-configmap-audit-profiles.yaml` around lines
2 - 38, The embedded audit policy YAMLs are malformed because keys and values
are written with JSON-style quoted strings; update metadata-profile.yaml,
none-profile.yaml, request-profile.yaml and requestresponse-profile.yaml inside
the ConfigMap data to valid YAML by removing the unnecessary quotes around keys
(apiVersion, kind, metadata, name, omitStages, rules, level) and unquoting list
items (e.g., RequestReceived), and ensure each rule is a proper mapping (e.g., -
level: Metadata) with correct indentation so each policy is valid Kubernetes
audit policy YAML.

@copejon copejon force-pushed the ushift-6951/metrics-server branch from 2552538 to 6f8a50b Compare June 5, 2026 07:24
@coderabbitai coderabbitai Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Jun 5, 2026
@copejon copejon force-pushed the ushift-6951/metrics-server branch from 6f8a50b to 82550e8 Compare June 5, 2026 07:37
@coderabbitai coderabbitai Bot removed the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Jun 5, 2026
@copejon copejon force-pushed the ushift-6951/metrics-server branch from 82550e8 to 9cc9d0e Compare June 5, 2026 07:54
Add metrics-server with Go cert provisioning, kustomize manifests,
healthcheck registration, auto-rebase support, RPM packaging, and
OpenTelemetry collector drop-in configuration.

Establishes shared infrastructure used by all metrics exporters:
- OPTIONAL_COMPONENTS in rebase.sh for CMO repo cloning
- update_metrics_images() for image digest resolution
- otelcol.d drop-in config directory and service wrapper
- mergeWorkloads() for shared-namespace healthcheck

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@copejon copejon force-pushed the ushift-6951/metrics-server branch from 9cc9d0e to 261c1a0 Compare June 5, 2026 08:12
…omponent

- Change metrics-server cert provisioning failure from Warning to Error level
  so failures are visible without being Fatal for an optional component.
- Split shared release JSON into per-component files. Each metrics component
  now ships its own release-{component}-{arch}.json with only its image ref.
- Update rebase.sh to generate per-component release JSONs during rebase.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
scripts/auto-rebase/rebase.sh (1)

1191-1193: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Do not skip unmapped manifest images.

Still valid from the earlier review: warning-and-continue here can emit a partial kustomization.${arch}.yaml, so the rebase succeeds with broken output.

Patch
                 if [[ -z "${release_tag}" ]]; then
-                    >&2 echo "WARNING: Unknown metrics image '${orig_image}' in ${component_dir}, skipping"
-                    continue
+                    >&2 echo "ERROR: Unknown metrics image '${orig_image}' in ${component_dir}"
+                    return 1
                 fi
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/auto-rebase/rebase.sh` around lines 1191 - 1193, The current block in
rebase.sh that checks if [[ -z "${release_tag}" ]] and then echoes a warning and
continues causes partial kustomization.${arch}.yaml output when an image
(orig_image) isn't in the mapping; instead, stop skipping: when release_tag is
empty, still emit the image into the kustomization entry (use orig_image as the
image reference or a sensible fallback) and log the warning but do not use
continue; update the code paths that write into kustomization.${arch}.yaml so
they handle unmapped images (referencing variables release_tag, orig_image and
component_dir) and ensure the resulting kustomization.${arch}.yaml always
includes an image entry for every manifest image.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/auto-rebase/rebase.sh`:
- Around line 1169-1176: The script assumes jq successfully finds a pullspec;
validate that the jq result is non-empty before writing files: after computing
release_tag and new_image (from the jq command that selects .from.name), check
if new_image is empty (or if new_digest parsed from new_image is empty), and if
so emit a clear error mentioning the release_tag and json_key and exit with a
non-zero status instead of writing component_release_json; apply the same guard
to the other identical block (the one that mirrors lines ~1196-1204) so
malformed release-*.json / kustomization.*.yaml are never created when the
payload tag is missing.

---

Duplicate comments:
In `@scripts/auto-rebase/rebase.sh`:
- Around line 1191-1193: The current block in rebase.sh that checks if [[ -z
"${release_tag}" ]] and then echoes a warning and continues causes partial
kustomization.${arch}.yaml output when an image (orig_image) isn't in the
mapping; instead, stop skipping: when release_tag is empty, still emit the image
into the kustomization entry (use orig_image as the image reference or a
sensible fallback) and log the warning but do not use continue; update the code
paths that write into kustomization.${arch}.yaml so they handle unmapped images
(referencing variables release_tag, orig_image and component_dir) and ensure the
resulting kustomization.${arch}.yaml always includes an image entry for every
manifest image.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 33a078aa-483a-4842-a2dd-2d3d185e3f96

📥 Commits

Reviewing files that changed from the base of the PR and between 82550e8 and 4aa1099.

⛔ Files ignored due to path filters (3)
  • etcd/vendor/github.com/openshift/microshift/pkg/config/config.go is excluded by !**/vendor/**
  • etcd/vendor/github.com/openshift/microshift/pkg/config/dns.go is excluded by !**/vendor/**
  • etcd/vendor/github.com/openshift/microshift/pkg/util/cryptomaterial/certinfo.go is excluded by !**/vendor/**
📒 Files selected for processing (27)
  • assets/optional/metrics-server/00-namespace.yaml
  • assets/optional/metrics-server/01-cluster-role-binding-auth-delegator.yaml
  • assets/optional/metrics-server/01-cluster-role-binding.yaml
  • assets/optional/metrics-server/01-cluster-role.yaml
  • assets/optional/metrics-server/01-role-binding-auth-reader.yaml
  • assets/optional/metrics-server/01-service-account.yaml
  • assets/optional/metrics-server/02-configmap-audit-profiles.yaml
  • assets/optional/metrics-server/03-deployment.yaml
  • assets/optional/metrics-server/04-api-service.yaml
  • assets/optional/metrics-server/04-service.yaml
  • assets/optional/metrics-server/kustomization.aarch64.yaml
  • assets/optional/metrics-server/kustomization.x86_64.yaml
  • assets/optional/metrics-server/kustomization.yaml
  • assets/optional/metrics-server/release-metrics-server-aarch64.json
  • assets/optional/metrics-server/release-metrics-server-x86_64.json
  • packaging/observability/microshift-observability.service
  • packaging/observability/otelcol.d/microshift-metrics-server.yaml
  • packaging/rpm/microshift.spec
  • pkg/cmd/init.go
  • pkg/cmd/metrics.go
  • pkg/cmd/run.go
  • pkg/healthcheck/microshift_optional_workloads.go
  • pkg/util/cryptomaterial/certinfo.go
  • scripts/auto-rebase/assets.yaml
  • scripts/auto-rebase/assets_metrics.yaml
  • scripts/auto-rebase/rebase.sh
  • test/bin/common.sh
✅ Files skipped from review due to trivial changes (4)
  • packaging/observability/microshift-observability.service
  • assets/optional/metrics-server/kustomization.yaml
  • assets/optional/metrics-server/kustomization.aarch64.yaml
  • test/bin/common.sh
🚧 Files skipped from review as they are similar to previous changes (20)
  • assets/optional/metrics-server/release-metrics-server-aarch64.json
  • assets/optional/metrics-server/01-service-account.yaml
  • pkg/healthcheck/microshift_optional_workloads.go
  • assets/optional/metrics-server/01-cluster-role.yaml
  • assets/optional/metrics-server/04-api-service.yaml
  • assets/optional/metrics-server/00-namespace.yaml
  • assets/optional/metrics-server/01-cluster-role-binding.yaml
  • assets/optional/metrics-server/02-configmap-audit-profiles.yaml
  • assets/optional/metrics-server/04-service.yaml
  • pkg/cmd/run.go
  • assets/optional/metrics-server/03-deployment.yaml
  • packaging/observability/otelcol.d/microshift-metrics-server.yaml
  • scripts/auto-rebase/assets_metrics.yaml
  • assets/optional/metrics-server/kustomization.x86_64.yaml
  • pkg/cmd/init.go
  • scripts/auto-rebase/assets.yaml
  • packaging/rpm/microshift.spec
  • assets/optional/metrics-server/01-role-binding-auth-reader.yaml
  • pkg/cmd/metrics.go
  • assets/optional/metrics-server/01-cluster-role-binding-auth-delegator.yaml

Comment thread scripts/auto-rebase/rebase.sh
copejon and others added 3 commits June 5, 2026 04:02
- Add capabilities.drop ALL to metrics-server container securityContext
- Fix release JSON filenames in assets_metrics.yaml to match actual files
- Fail-fast on unknown metrics image in rebase.sh instead of silent skip

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fail-fast when jq returns empty or null for release tag or image
lookups, preventing garbage data in release JSONs and kustomization
files.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Only retry when the namespace is not yet created (IsNotFound). Return
auth, TLS, and other permanent errors immediately instead of retrying
for 5 minutes with a misleading timeout.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@copejon
Copy link
Copy Markdown
Contributor Author

copejon commented Jun 5, 2026

@CodeRabbit review

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Jun 5, 2026

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented Jun 5, 2026

@copejon: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-tests-bootc-arm-el10 81264bb link true /test e2e-aws-tests-bootc-arm-el10
ci/prow/e2e-aws-tests-periodic 81264bb link true /test e2e-aws-tests-periodic
ci/prow/e2e-aws-tests-arm 81264bb link true /test e2e-aws-tests-arm
ci/prow/e2e-aws-tests-bootc-periodic-arm-el10 81264bb link true /test e2e-aws-tests-bootc-periodic-arm-el10
ci/prow/e2e-aws-tests-bootc-periodic-el10 81264bb link true /test e2e-aws-tests-bootc-periodic-el10
ci/prow/e2e-aws-tests-bootc-periodic-el9 81264bb link true /test e2e-aws-tests-bootc-periodic-el9
ci/prow/e2e-aws-tests-bootc-periodic-arm-el9 81264bb link true /test e2e-aws-tests-bootc-periodic-arm-el9
ci/prow/e2e-aws-tests-bootc-arm-el9 81264bb link true /test e2e-aws-tests-bootc-arm-el9
ci/prow/e2e-aws-tests-periodic-arm 81264bb link true /test e2e-aws-tests-periodic-arm
ci/prow/e2e-aws-tests-bootc-el9 81264bb link true /test e2e-aws-tests-bootc-el9
ci/prow/e2e-aws-tests 81264bb link true /test e2e-aws-tests
ci/prow/e2e-aws-tests-bootc-el10 81264bb link true /test e2e-aws-tests-bootc-el10

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants