fix(DOC-1940): clarify rpk topic consume keeps connection open

[mfernest]

Summary

Resolves DOC-1940. The ticket reports that AI tools and users don't realize rpk topic consume opens a persistent connection and waits indefinitely for new records.

This PR was originally opened by @mfernest, who is no longer with the team. @Feediver1 is taking it over to finalize after @micheleRP's review.

What changed

All edits land below the Flags table so they survive the next gen-rpk-ascii.py regeneration:

• New == Connection behavior H2 (manual zone) explaining that rpk topic consume runs continuously by default, doesn't exit after consuming existing records, and listing three ways to stop it: kbd:[Ctrl+C], --num, or --offset (for example, -o :end).
• Default output-format clarification moved into the same == Connection behavior section: "Records are formatted according to --format. The default, --format json, prints each record as a JSON object." This was originally in the auto-gen description zone; relocating it preserves the cleaner wording without risking it being overwritten by the next rpk doc regeneration.
• Two pre-existing bug fixes in manual prose: "prints the number in as ASCII" → "prints the number as ASCII", and "will have their values printed" → "have their values printed".

Review history

Reviewer	Status
@daisukebe	Asked for -o :end instead of -o start:end — applied. Approved 2026-03-24
@r-vasquez	Approved 2026-03-24
@micheleRP	CHANGES_REQUESTED 2026-03-24 — asked to move the new content out of the auto-gen description zone and applied two style nits. Both substantively addressed: PATCH 3 (3-24) moved the content to a new manual-zone H2, and this PR's latest commit completes the cleanup by reverting the leftover description-zone wording change. Re-requesting review
CodeRabbit	Offset-range clarification nit — addressed

Note on rpk overrides

While reviewing this PR, I noted that the docs repo has no overrides mechanism for rpk reference pages — unlike properties (property-overrides.json) and connectors (overrides.json). Any manual edit in the auto-gen description / usage / flags zones of rpk pages is silently lost on the next gen-rpk-ascii.py run. The right long-term fix is either (a) a new rpk-overrides.json tooling feature in docs-extensions-and-macros, or (b) upstreaming descriptive prose into the rpk Go source. This PR works around the constraint by keeping every edit in manual zones; the broader tooling gap is a follow-up.

Preview pages

• rpk topic consume — new == Connection behavior section at the bottom of the page

Test plan

• Netlify preview builds without errors
• == Connection behavior section renders below the Flags table, with the three stop-mechanisms enumerated
• Default output-format sentence renders as a second paragraph under == Connection behavior
• No content remains in the auto-generated description zone that would be lost on regeneration

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for redpanda-docs-preview ready!

Name	Link
🔨 Latest commit	478809c
🔍 Latest deploy log	https://app.netlify.com/projects/redpanda-docs-preview/deploys/6a18665004247f0007e95ce7
😎 Deploy Preview	https://deploy-preview-1612--redpanda-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 567ccc13-69b8-4308-bbc9-cf135e70757c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This pull request adds documentation to the rpk topic consume command page, clarifying its default behavior. The changes explain that the command maintains a persistent connection and waits indefinitely for new records unless interrupted or controlled by the --num or --offset options. The documentation also specifies the stopping conditions and includes an example demonstrating the --offset option usage.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~5 minutes

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately and concisely describes the main change: clarifying that rpk topic consume keeps a persistent connection open.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description check	✅ Passed	The pull request description is comprehensive and well-structured, providing detailed context about the change, review history, and preview links.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/doc-1940-rpk-topic-consume-connection

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@modules/reference/pages/rpk/rpk-topic/rpk-topic-consume.adoc`:
- Around line 10-14: The doc text for rpk topic consume should clarify the range
offset pattern and either replace or augment `-o start:end` with explicit,
documented examples and a note about `end`'s meaning in ranges: update the
sentence referencing `--offset`/`-o` to show a clear example like `-o :end`
(consume until current end) and/or `-o `@-1m`:end` (consume from one minute ago to
current end), and add a short parenthetical explaining that `end` inside a range
denotes the current end offset (different from using `end` alone); edit the
offset table and examples section to include this explicit `:end`/`@-1m:end`
pattern and the clarifying note so readers aren’t confused by the standalone
`end` meaning.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 47393d7b-97a3-4f2f-9010-2d4e19f43fda

📥 Commits

Reviewing files that changed from the base of the PR and between fe8a357 and 1128095.

📒 Files selected for processing (1)

• modules/reference/pages/rpk/rpk-topic/rpk-topic-consume.adoc

[daisukebe]

-o start:end

-o :end is enough.

[mfernest]

Good call — fixed to -o :end. Also took the opportunity to clean up a few other prose issues on the page (kbd macro for Ctrl+C, simplify the --format json sentence, fix a tense issue, and a grammatical error).

[Feediver1]

Thanks — applied as -o :end in PATCH 2.

[daisukebe]

LGTM

[micheleRP]

Review Summary

The core issue (DOC-1940)

DOC-1940 reports that AI tools (and users) don't realize rpk topic consume opens a persistent connection and waits indefinitely. The description says: "we noticed that some AI tools fail to realize that rpk topic consume opens a connection and stays waiting for new messages to arrive."

RPK automation impact

Like PR #1618, this file is an rpk reference page where the generator (gen-rpk-ascii.py) produces:

• The title and :description: attribute
• The opening description paragraph (from rpk topic consume --help output)
• The == Usage section
• The == Flags table

The sections below the flags table (Attributes, Text, Headers, Values, Offsets, Examples) are manual additions that survive regeneration because the generator doesn't produce them.

The problem: The new paragraph about persistent connection behavior is inserted in the description zone (between the intro paragraph and the include:: directive). This zone is overwritten by whatever rpk topic consume --help outputs. So this addition will be lost on the next rpk doc regeneration.

The two bug fixes ("in as ASCII" → "as ASCII" on line 85, and "will have their values printed" → "have their values printed" on line 106) are in the manual sections and will survive regeneration.

Recommendation

The core value of this PR — documenting that rpk topic consume waits indefinitely — should be placed somewhere that survives regeneration. Two options:

1. Move the new paragraph below the == Flags table (e.g., as a new == Behavior or == Connection behavior section before == Attributes). Manual sections after the flags table are not overwritten by the generator. This is the simplest fix that preserves the intent.

2. Update the rpk topic consume --help text upstream in the Redpanda Go source so it includes the persistent-wait behavior in the command description itself, and then regenerate.

Style notes

The content is well-written. A few suggestions:

1. "opens a persistent connection" is imprecise for Kafka protocol semantics. Consider: "runs continuously, waiting for new records to arrive" — this describes the observable behavior without implying a specific connection model.

2. The three-clause "To stop consuming, press Ctrl+C, or use --num... or specify an end offset..." is long. Split into two sentences:

   To stop consuming, press kbd:[Ctrl+C]. You can also use --num to exit after a fixed number of records, or use --offset (for example, -o :end) to stop at a specific offset.

3. The grammar fix on line 85 ("in as ASCII" → "as ASCII") and the tense fix on line 106 ("will have" → "have") are both correct and good catches.

[micheleRP]

@mfernest please see Claude's review

[Feediver1]

Ran this through Claude:

Does it address DOC-1940? ✅ Yes

 The new == Connection behavior section directly addresses the ticket — it explains
 that rpk topic consume runs persistently, doesn't exit after consuming existing
 records, and documents the three ways to stop it (Ctrl+C, --num, --offset). The
 minor editorial changes (passive → active voice, typo fix) are also valid.

 ---
 Will changes be overwritten? ⚠️ Yes — this is a problem

 The rpk docs are fully regenerated from rpk --help output by gen-rpk-ascii.py. The
 script explicitly deletes and rewrites each file from scratch:

 # If the file exists, delete it.
 if os.path.exists(filename):
   os.remove(filename)

 with open(filename, "w", encoding="utf-8") as filetowrite:
   filetowrite.write(ascii_result) # ← entirely from rpk --help

 There is no overrides mechanism for rpk docs (unlike properties →
 property-overrides.json, connectors → overrides.json). The == Connection behavior
 section does not come from rpk --help output, so it will be silently lost the next
 time someone runs generate_rpk_docs.

 The other editorial changes (rewording --format json, removing "in as ASCII",
 passive→active voice) would also be reverted.

 ---
 Recommended fix

 The content needs to be added to the rpk source code help text in the redpanda repo
  (the rpk topic consume command's long description), so it gets generated
 automatically. That's the only sustainable path for this kind of contextual
 clarification.

 If upstreaming to rpk isn't feasible right now, this should at minimum be flagged
 with a comment in the file so it doesn't get silently dropped:

 // NOTE: The section below is manually maintained and must be re-added after each
 rpk docs regeneration.

---

asked Claude how this can be done without impacting automated content:

There is no overrides mechanism for rpk docs — it doesn't exist yet. Unlike
 properties (property-overrides.json) and connectors (overrides.json), the
 generate_rpk_docs tooling has no --overrides option and the generation script has
 no merge/append logic.

 So the options are:

 1. Keep the PR as-is with a clear comment warning it will be overwritten on
 regeneration — fragile but immediate
 2. Build an rpk overrides file (e.g., docs-data/rpk-overrides.json) and update
 gen-rpk-ascii.py to support appending custom sections — the right long-term fix,
 but requires tooling work in docs-extensions-and-macros
 3. Upstream to rpk source — the cleanest solution but requires a Go code change in
 the redpanda repo

 Given that the overrides pattern is already established for properties and
 connectors, option 2 would be the most consistent approach for the docs tooling.
 But that's a feature request for docs-extensions-and-macros, not something that can
  be done in this PR alone.

 My recommendation: request changes on the PR noting that directly editing a
 generated file isn't sustainable, and that the right fix is either upstreaming to
 rpk source or a tooling enhancement. In the meantime, if the team wants to merge it
  as a stopgap, at minimum add a prominent comment.

[Feediver1]

@JakeSCahill Curious to know why we don't have an overrides file for rpk like we do for props?

[Feediver1]

@micheleRP — picking this up from @mfernest (no longer with the team). All three items from your 2026-03-24 review are now addressed:

1. Moved the new content out of the auto-gen description zone — PATCH 3 (@mfernest) created the new == Connection behavior H2 below the Flags table. My follow-up commit 7336c040 cleans up a leftover wording edit on line 10 of the description zone (reverted that line; moved the "default output format" sentence into the manual zone as a second paragraph under == Connection behavior). Net diff vs main is now entirely in the manual zone, so nothing is at risk of being overwritten by the next gen-rpk-ascii.py run.

2. "opens a persistent connection" → "runs continuously, waiting for new records to arrive" — applied.

3. Split the long stop-mechanisms sentence into two — applied with the kbd:[Ctrl+C] macro.

Could you clear the CHANGES_REQUESTED state when you have a moment? Re-requesting your review.

Separately: the underlying gap your review surfaced — that rpk reference pages have no overrides mechanism analogous to property-overrides.json — is real and likely worth a follow-up issue against docs-extensions-and-macros. Open to chatting about whether that's a sensible scope or whether upstreaming to rpk Go source is the better path.

[micheleRP]

Approving — all the change requests from my earlier review are addressed:

• New == Connection behavior section now sits below the Flags table (survives gen-rpk-ascii.py regeneration), and the description-zone edit was reverted in PATCH 4.
• "opens a persistent connection" reworded to "runs continuously, waiting for new records to arrive."
• The long stop-mechanisms sentence was split as suggested.
• The two prose fixes (in as ASCII → as ASCII, will have → have) are correct and consistent with the current upstream rpk source.

One optional, non-blocking nit: the second paragraph under == Connection behavior ("Records are formatted according to --format...") is about output formatting rather than connection behavior, and slightly overlaps the auto-gen description zone. Fine to leave as-is given the regeneration-durability tradeoff. Thanks for landing this!