ci-test-mapping: better memory limits#79729
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
WalkthroughResources '*' now specify ChangesCI Test Mapping Memory & Test Env
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related PRs
Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 15✅ Passed checks (15 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
openshift-ci
Bot
added
the
approved
|
/pj-rehearse pull-ci-openshift-eng-ci-test-mapping-main-lint |
|
@deepsm007: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
sosiouxme
force-pushed
the
20260526-ctm-memory-limits
branch
from
5f026ee to
935f7c3
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/config/openshift-eng/ci-test-mapping/openshift-eng-ci-test-mapping-main.yaml`:
- Around line 12-16: The global defaults under resources.'*' still use 200Mi
while you only updated resources.root; update resources.'*' to set
requests.memory: 1Gi and limits.memory: 3Gi (or remove/adjust resources.root if
it was intended as a subset override) so that the global/default resource values
match the intended 1Gi/3Gi; ensure you edit the resources.'*' block (and keep or
change resources.root accordingly) so defaults apply to all tests.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 9a7bce04-54c5-4015-94f5-0259739bd3c3
📒 Files selected for processing (1)
ci-operator/config/openshift-eng/ci-test-mapping/openshift-eng-ci-test-mapping-main.yaml
| root: | ||
| requests: | ||
| memory: 1Gi | ||
| limits: | ||
| memory: 3Gi |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
f='ci-operator/config/openshift-eng/ci-test-mapping/openshift-eng-ci-test-mapping-main.yaml'
echo "== Target file resources block =="
awk 'NR>=4 && NR<=17 {printf "%4d %s\n", NR, $0}' "$f"
echo
echo "== Search for resources key named \"root\" across ci-operator/config =="
rg -n --type=yaml '^\s{2}root:\s*$' ci-operator/config || true
echo
echo "== Search for default resources key \"*\" usage across ci-operator/config =="
rg -n --type=yaml "^\s{2}'\*':\s*$" ci-operator/config | head -n 40Repository: openshift/release
Length of output: 5320
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Search for how the ci-test-mapping "resources" block is modeled/consumed.
rg -n --hidden -S "ci-test-mapping|test-mapping" . \
| head -n 50
echo "----"
rg -n --hidden -S "resources\.\*|resources\s*:\s*\{|\bresources\b" . \
| head -n 80
echo "----"
rg -n --hidden -S "resources\.root|\broot:\s*$|\bresources\b.*root|root.*resources" . \
| head -n 80
echo "----"
# Look for schema/validation mentioning resources keys (e.g., '*' or root).
rg -n --hidden -S "resources.*\\*|\\*.*resources|resources.*root|root.*resources" . \
| head -n 120Repository: openshift/release
Length of output: 35935
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Find where ci-test-mapping YAML is parsed/applied (Go code + schema)
rg -n --hidden -S "ci-test-mapping" ci-operator | head -n 80
echo "----"
# Look for structs/logic referencing resources keys like '*' or root
rg -n --hidden -S "resources.*root|root.*resources|Resources.*Root|resource.*map\\[string\\].*\\*|map\\[string\\]struct\\{.*resources" ci-operator | head -n 120
echo "----"
# Broader search across repo for "resources" mapping with literal "*"
rg -n --hidden -S "resources.*\\*\\b|\\b\\*\\b.*resources|\"\\*\".*resources|\"resources\".*root|root.*\"resources\"" . | head -n 120Repository: openshift/release
Length of output: 3188
🏁 Script executed:
#!/bin/bash
set -euo pipefail
f='ci-operator/config/openshift-eng/ci-test-mapping/openshift-eng-ci-test-mapping-main.yaml'
echo "== File head (1-120) =="
nl -ba "$f" | sed -n '1,120p'
echo "----"
echo "== Find all occurrences of 'resources:' blocks in this file =="
rg -n --context 2 "^\s*resources\s*:" "$f" || true
echo "----"
echo "== Search for uses of 'resources:' under 'tests:' in this file =="
rg -n --context 4 "^\s*tests\s*:|^\s*resources\s*:|resources\." "$f"
echo "----"
echo "== Search repo for ci-test-mapping configs that define 'resources:' with 'root:' =="
rg -n --hidden -S "resources:\n(\s+.*\n)*?\s+root:" ci-operator/config | head -n 50
echo "----"
echo "== Search repo for exact 'root:' key at resources level (2-space root:) =="
rg -n --hidden -S "^\s{2}root:\s*$" ci-operator/config | head -n 50Repository: openshift/release
Length of output: 146
🏁 Script executed:
#!/bin/bash
set -euo pipefail
f='ci-operator/config/openshift-eng/ci-test-mapping/openshift-eng-ci-test-mapping-main.yaml'
echo "== File lines 1-120 =="
cat -n "$f" | sed -n '1,120p'
echo "----"
echo "== Find all occurrences of 'resources:' blocks in this file =="
rg -n --context 2 "^\s*resources\s*:" "$f" || true
echo "----"
echo "== Search for uses of resources/tests/resources keys in this file =="
rg -n --context 4 "^\s*tests\s*:|^\s*resources\s*:|resources\." "$f" || true
echo "----"
echo "== Search repo for resources: with root subkey (2nd-level) in ci-test-mapping configs =="
rg -n --hidden -S "resources:\n([[:space:]]+.*\n)*[[:space:]]+root:" ci-operator/config | head -n 50
echo "----"
echo "== Search repo for exact 'root:' key at resources level (2-space root:) =="
rg -n --hidden -S "^\s{2}root:\s*$" ci-operator/config | head -n 50Repository: openshift/release
Length of output: 1845
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Search ci-operator code for parsing/usage of the "resources" stanza and special keys like "*"/"root"
rg -n --hidden -S "resources\s*:\s*|type .*Resources|struct .*Resources|\bResources\b|map\\[string\\].*memory|resources\\.[*]|resources\\W*root|\broot\\W*requests|\broot\\W*limits" ci-operator | head -n 200
echo "----"
# Look specifically for a struct field/tag named "Resources" and YAML unmarshalling
rg -n --hidden -S "yaml:\"resources\"|json:\"resources\"|Unmarshal.*resources|resources\\\"\\s*:" ci-operator | head -n 200
echo "----"
# Look for any logic that treats "root" as a resource selector key
rg -n --hidden -S "resources.*root|root.*resources|case ['\"]root['\"]:|\"root\".*resources" ci-operator | head -n 200Repository: openshift/release
Length of output: 24828
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Go files under ci-operator (head) =="
find ci-operator -name '*.go' | head
echo "----"
echo "== Search for ci-test-mapping in Go =="
rg -n --type go --hidden -S "ci-test-mapping" ci-operator | head -n 80 || true
echo "----"
echo "== Search for resources parsing in Go (simple patterns) =="
rg -n --type go --hidden -S "type .*Resource|struct .*Resource|BuildRoot|project_image|resources" ci-operator | head -n 120 || true
echo "----"
echo "== Search for 'root' key handling near 'resources' in Go =="
rg -n --type go --hidden -S "(resources.*root|root.*resources|\\bRoot\\b.*\\bResources\\b)" ci-operator | head -n 120 || true
echo "----"
echo "== Search for YAML tag resources in Go =="
rg -n --type go --hidden -S "yaml:\"resources\"|json:\"resources\"" ci-operator | head -n 80 || trueRepository: openshift/release
Length of output: 381
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Find ci-test-mapping config YAMLs (repo: ci-test-mapping) =="
rg -n --type yaml "repo:\s*ci-test-mapping" ci-operator/config | head -n 50
echo "----"
echo "== Find files with build_root and top-level resources in ci-operator/config =="
rg -n --type yaml "build_root:" ci-operator/config | head -n 50
echo "----"
echo "== In ci-operator/config, find top-level resources blocks that include both '*' and 'root' =="
rg -n --type yaml "resources:\n([[:space:]]+|)*'\*':|resources:\n([[:space:]]+|)*\\*:\n|resources:\n([[:space:]]+|)*root:" ci-operator/config --context 3 || true
echo "----"
echo "== Search within ci-operator/config for exact YAML path pattern resources: '*' and then root later (heuristic) =="
rg -n --type yaml "resources:\n[[:space:]]+'\\*':[\s\S]*?\n[[:space:]]+root:" ci-operator/config -U --max-count 20 || true
echo "----"
echo "== Check if any ci-test-mapping configs reference resources.root elsewhere (string match) =="
rg -n --type yaml "resources:\s*|root:\s*$|'\*':\s*$" ci-operator/config/openshift-eng/ci-test-mapping/ -n || trueRepository: openshift/release
Length of output: 7848
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Search docs/scripts/templates for references to resources keys and build_root
rg -n --hidden -S "build_root:|ci-test-mapping|openshift-eng/ci-test-mapping|resources:\s*['\"*']|resources\.root|root:\s*\n\s*requests:\s*\n\s*memory:|resources:\s*$" . \
--glob '!**/*.yaml' --glob '!**/*.yml' | head -n 200
echo "----"
# Search for any jsonschema/yaml schema files
fd -t f --hidden "schema|jsonschema|cue|cue.mod|definitions|validate|validation" . | head -n 50
echo "----"
# Search for exact string patterns used by ci-operator templates
rg -n --hidden -S "resources.*\\*|resources.*root|requests:\s*\\{\\s*memory|memory:\s*200Mi|memory:\s*1Gi" . \
--glob '!**/*.yaml' --glob '!**/*.yml' | head -n 200Repository: openshift/release
Length of output: 253
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Search non-YAML files for resources.root / resources.'*' strings =="
rg -n --hidden -S "resources\.root|resources:\s*'\*'|resources:\s*\"\*\"|resources\.\*|resources.*root|root.*resources" . \
--glob '!**/*.yaml' --glob '!**/*.yml' | head -n 200
echo "----"
echo "== Search for ci-test-mapping config processing scripts/templates =="
rg -n --hidden -S "openshift-eng/ci-test-mapping|ci-test-mapping-main\.yaml|ci-test-mapping" . \
--glob '!**/*.yaml' --glob '!**/*.yml' | head -n 200
echo "----"
echo "== Search for references to top-level resources/build_root semantics in code =="
rg -n --hidden -S "build_root|project_image|zz_generated_metadata|resources" ci-operator | head -n 200Repository: openshift/release
Length of output: 26945
Global test resource defaults still at 200Mi (only resources.root changed)
In ci-operator/config/openshift-eng/ci-test-mapping/openshift-eng-ci-test-mapping-main.yaml, resources.'*' keeps requests.memory: 200Mi and limits.memory: 200Mi (lines 5-11), while only resources.root is set to requests.memory: 1Gi / limits.memory: 3Gi (lines 12-16). If the goal is global/default resources, update resources.'*' to 1Gi/3Gi (and adjust/remove root only if it’s intended for a separate subset).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/config/openshift-eng/ci-test-mapping/openshift-eng-ci-test-mapping-main.yaml`
around lines 12 - 16, The global defaults under resources.'*' still use 200Mi
while you only updated resources.root; update resources.'*' to set
requests.memory: 1Gi and limits.memory: 3Gi (or remove/adjust resources.root if
it was intended as a subset override) so that the global/default resource values
match the intended 1Gi/3Gi; ensure you edit the resources.'*' block (and keep or
change resources.root accordingly) so defaults apply to all tests.
There was a problem hiding this comment.
it's easy to just set high limits everywhere but i'm just looking for the minimal change. the root build is failing to clone and i can't make git do any better with memory so that has to be given more. but golang can be instructed to restrict memory, and it was able to do it with this limit in my local testing, there's so no need to hold open that much memory.
There was a problem hiding this comment.
@sosiouxme, that makes perfect sense — the resources.root block targets the build/clone step where git needs the headroom, while GOMEMLIMIT=200000000 in each test command instructs the Go runtime to stay within the lower resources.'*' limit. The split is clearly intentional and well-reasoned. Thanks for the explanation!
✏️ Learnings added
Learnt from: sosiouxme
Repo: openshift/release PR: 79729
File: ci-operator/config/openshift-eng/ci-test-mapping/openshift-eng-ci-test-mapping-main.yaml:12-16
Timestamp: 2026-05-26T22:28:41.551Z
Learning: In openshift/release ci-operator config YAMLs (e.g., ci-operator/config/openshift-eng/ci-test-mapping/openshift-eng-ci-test-mapping-main.yaml), a `resources.root` block (separate from `resources.'*'`) is used to set higher memory limits specifically for the build/clone (build root) container step, because git cannot be memory-constrained programmatically. The `resources.'*'` block intentionally stays at the lower default (e.g., 200Mi) for Go test steps, which are separately governed by `GOMEMLIMIT` env var in their `commands`. Do not flag this split as a misconfiguration — it is a deliberate minimal-change pattern to only raise limits where truly needed.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
🧠 Learnings used
Learnt from: rdiscala
Repo: openshift/release PR: 79357
File: ci-operator/config/openshift-kni/eco-ci-cd/openshift-kni-eco-ci-cd-main__cnf-ran-sno-day2-worker-4.18.yaml:23-23
Timestamp: 2026-05-20T11:54:20.424Z
Learning: In the openshift/release repository CI operator configuration YAMLs, do not treat an intentionally impossible cron schedule (e.g., `0 0 31 2 *` for “Feb 31”) as a misconfiguration. This pattern is an intentional mechanism used to disable/park a periodic CI job temporarily (e.g., while an upstream bug is being fixed). Only flag cron syntax errors when the schedule is not an intentional “impossible date” disable pattern used for job parking.
Learnt from: dhaiducek
Repo: openshift/release PR: 79563
File: ci-operator/config/stolostron/cluster-proxy/stolostron-cluster-proxy-main.yaml:41-43
Timestamp: 2026-05-20T19:31:57.286Z
Learning: In the openshift/release repo, for ci-operator container test step config YAML files under ci-operator/config/, remember that the `commands` field is read by ci-operator at runtime from the config YAML and is not embedded into the generated ProwJob YAML under `ci-operator/jobs/`. Therefore, if a PR changes only `commands` (and not any ProwJob-spec affecting structural fields such as test/job names, job types/spec, or metadata that changes the resulting ProwJob spec), you generally do NOT need to run `make update` and regenerating artifacts should be a no-op diff. Run `make update` only when the changes affect the generated ProwJob spec (e.g., new/renamed tests, new job types, or other spec/metadata changes that alter `ci-operator/jobs/` output).
Learnt from: tthvo
Repo: openshift/release PR: 79564
File: ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.20-upgrade-from-nightly-4.19.yaml:94-94
Timestamp: 2026-05-20T20:01:14.481Z
Learning: In the openshift/release repo, when editing ci-operator config YAMLs under ci-operator/config/, changes that only modify the *values* of environment variables inside test step `env` blocks (e.g., COMPUTE_NODE_TYPE, ADDITIONAL_WORKER_VM_TYPE, MIGRATION_INFRA_MACHINE_TYPE) do not change the generated ProwJob specs under ci-operator/jobs/. For these value-only env tweaks, reviewers should not require running `make update` because it should produce no diff. Require `make update` only for structural/semantic changes that can affect the generated ProwJob spec (e.g., adding/renaming tests, changing job types, or altering metadata/fields that feed into the ProwJob spec).
Learnt from: karampok
Repo: openshift/release PR: 79537
File: ci-operator/config/openshift/bond-cni/openshift-bond-cni-main.yaml:32-33
Timestamp: 2026-05-21T09:04:21.025Z
Learning: In openshift/release, when modifying a ci-operator config YAML under ci-operator/config/ by adding an `env` block (key/value environment variables such as `TEST_TYPE`, `TEST_SUITE`, `TEST_ARGS`) to an existing test step, this change does not affect the generated ProwJob specs under ci-operator/jobs/. In this case, `make update` is a no-op—do not request or flag that `make update` was (or needs to be) run for these env-only edits.
|
/pj-rehearse pull-ci-openshift-eng-ci-test-mapping-main-lint |
|
@sosiouxme: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
i can't even tell wtf happened here /pj-rehearse pull-ci-openshift-eng-ci-test-mapping-main-lint ...and if that doesn't look better, we'll just bump the limits |
|
@sosiouxme: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
sosiouxme
force-pushed
the
20260526-ctm-memory-limits
branch
2 times, most recently
from
9a88d02 to
1c09a5a
Compare
|
/pj-rehearse pull-ci-openshift-eng-ci-test-mapping-main-lint |
|
@sosiouxme: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@sosiouxme: requesting more than one rehearsal in one comment is not supported. If you would like to rehearse multiple specific jobs, please separate the job names by a space in a single command. |
1 similar comment
|
@sosiouxme: requesting more than one rehearsal in one comment is not supported. If you would like to rehearse multiple specific jobs, please separate the job names by a space in a single command. |
|
/pj-rehearse pull-ci-openshift-eng-ci-test-mapping-main-lint pull-ci-openshift-eng-ci-test-mapping-main-unit pull-ci-openshift-eng-ci-test-mapping-main-mapping |
|
@sosiouxme: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
huh... |
|
@sosiouxme: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse pull-ci-openshift-eng-ci-test-mapping-main-lint |
|
@sosiouxme: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse pull-ci-openshift-eng-ci-test-mapping-main-mapping |
|
@sosiouxme: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
with no memory limits, containers have tended to get scheduled to nodes where they get evicted or killed for their memory usage. root and src containers both perform git clones; with this repository, that by itself takes up around 2GB, although short-lived. and git gives us no real way to limit its memory usage. the actual golang executables running in tests can be limited easily with an env var. so we can use a much lower limit for those.
sosiouxme
force-pushed
the
20260526-ctm-memory-limits
branch
from
1c09a5a to
7f4eb0f
Compare
|
/pj-rehearse pull-ci-openshift-eng-ci-test-mapping-main-lint pull-ci-openshift-eng-ci-test-mapping-main-unit pull-ci-openshift-eng-ci-test-mapping-main-mapping |
|
@sosiouxme: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
/pj-rehearse ack |
|
@sosiouxme: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
the
rehearsals-ack
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: deepsm007, neisw, sosiouxme The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@sosiouxme: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
3 participants
Summary by CodeRabbit
This PR updates OpenShift CI configuration for the openshift-eng/ci-test-mapping repository to increase memory available to build containers and to constrain Go's memory usage during tests.
Practical impact:
Validation notes:
Files changed (high-level): ci-operator/config/openshift-eng/ci-test-mapping/openshift-eng-ci-test-mapping-main.yaml — adds per-container memory requests/limits and prefixes test commands with GOMEMLIMIT/HOME.