header_rewrite: fix set-body origin replacement across hooks and transforms

[bryancall]

Summary

• make set-body robust for origin responses by correctly handling internal body replacement before tunneling, including transform-bypass safety in HttpSM
• add READ_RESPONSE_HDR_HOOK as an allowed set-body hook while preserving SEND_RESPONSE_HDR_HOOK, and fix API state interaction in TSHttpTxnErrorBodySet()
• consolidate AuTests into one replay-driven test covering read/send hooks with and without transforms, with cache disabled and cache-bypass probes; update docs accordingly

Production validation

Tested with live traffic on a single production edge/cache host using a build from this change set: staged header_rewrite, then traffic_server, with drain/restart and post-restart checks; exercised set-body on READ_RESPONSE_HDR for origin 403 responses via config reload. Behavior matched expectations during rollout observation; the host was returned to rotation and informal bake/monitoring continues.

Ready for review / merge from the author side.

Test plan

• AUTEST_PORT_OFFSET=13000 ./autest.sh --ats-bin /tmp/ts-autest/bin --filter=header_rewrite_set_body_origin (eris)
• AUTEST_PORT_OFFSET=15000 ./autest.sh --ats-bin /tmp/ts-autest/bin --filter=header_rewrite_bundle (eris)

Supersedes #12879.

[bryancall]

/copilot review

[Copilot]

Pull request overview

This PR extends header_rewrite’s set-body operator to robustly replace origin response bodies (including across hooks and in the presence of response transforms), updates core HttpSM handling to honor the internal message buffer before tunneling, and adds/updates AuTests and documentation to cover the new behavior.

Changes:

• Update HttpSM to divert origin responses to setup_internal_transfer() when a plugin has set internal_msg_buffer (including transform-related paths).
• Allow set-body to run at TS_HTTP_READ_RESPONSE_HDR_HOOK (in addition to TS_HTTP_SEND_RESPONSE_HDR_HOOK).
• Add a replay-driven gold test matrix (read/send hooks; with/without transform) plus documentation updates.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/proxy/http/HttpSM.cc	Adds internal-body override handling in SERVER_READ / TRANSFORM_READ and transform-bypass logic in set_next_state().
src/api/InkAPI.cc	Ensures TSHttpTxnErrorBodySet() clears request-body override mode when reusing the shared buffer.
plugins/header_rewrite/operators.cc	Allows OperatorSetBody to run on TS_HTTP_READ_RESPONSE_HDR_HOOK.
tests/gold_tests/pluginTest/header_rewrite/header_rewrite_set_body_origin.test.py	Adds a new replay-driven test wrapper for origin replacement scenarios.
tests/gold_tests/pluginTest/header_rewrite/header_rewrite_set_body_origin.replay.yaml	Adds replay matrix for read/send hooks and transform/no-transform scenarios.
tests/gold_tests/pluginTest/header_rewrite/rules/rule_set_body_origin_read_resp.conf	New rule file to apply set-body at READ_RESPONSE_HDR_HOOK.
tests/gold_tests/pluginTest/header_rewrite/rules/rule_set_body_origin_send_resp.conf	New rule file to apply set-body at SEND_RESPONSE_HDR_HOOK.
doc/admin-guide/plugins/header_rewrite.en.rst	Documents origin replacement support and related caveats for set-body.
tests/prepare_proxy_verifier.sh	Updates expected Proxy Verifier tarball SHA1.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 10 comments.

[zwoop]

I think this looks good, complex code, so love the tests. Other than all the comment slop, my only question really is around that hook check. That seems unnecessary, albeit I'm not 100% sure. Possibly they should be assertions, but I don't see how you'd get there, if you don't have those plugin hooks.

[bryancall]

Posting a summary of the design discussion plus the production validation we just ran.

Why the SERVER_READ divert and not a response transform

TSHttpTxnErrorBodySet() from a response-stage hook only worked end-to-end when the origin returned an error path; against a valid origin response the SM took the server-transfer branch and the plugin's body was discarded. The fix in this PR is a precise divert in HttpSM::handle_api_return() when the plugin set an internal body, the origin response is valid, no plugin_tunnel is in play, and the recorded hook id at TSHttpTxnErrorBodySet() time was READ_RESPONSE_HDR or SEND_RESPONSE_HDR.

The natural alternative is to do what txn_box's upstream-rsp-body does — install a TS_HTTP_RESPONSE_TRANSFORM_HOOK and rewrite body bytes inline. We considered that and chose the SERVER_READ divert for these reasons.

Why the SERVER_READ divert

• Fixes the API, not just one plugin. Any plugin calling TSHttpTxnErrorBodySet() from a response hook gets correct behavior, not only header_rewrite.
• Smaller, contained patch. The change is localized to handle_api_return() plus a captured-hook-id field for scoping. No new plugin code, no new vconn lifecycle.
• Response headers are not perturbed by the body swap. Only the body bytes are substituted; status code, Content-Type, Cache-Control, and any other plugin-modified headers flow through the normal response path. A transform forces the plugin to recompute Content-Length and decide chunked vs. not-chunked encoding; the divert avoids that.
• Origin connection is closed for the substituted transaction. The server session is released before the body is consumed, so the replacement body goes to the client and the origin body is discarded. Drain-and-pool (so the connection can be reused) is a possible follow-up enhancement; this PR takes the simpler close path.

Trade-offs we accept

• Touches HttpSM core, which is what made review careful. The divert is gated on five conditions (internal_msg_buffer, !api_server_request_body_set, server_response.valid(), plugin_tunnel == nullptr, captured-hook-id in the response stage) to keep the blast radius narrow.
• plugin_tunnel == nullptr is the explicit TSHttpConnect intercept guard, replacing the indirect api_hooks.get() check.
• The captured-hook-id check (internal_msg_buffer_set_on) scopes the divert to plugin-driven response replacement.

Why not a response transform

• It is a substantially larger plugin rewrite: transform vconn lifecycle, chunked-encoding handling, Content-Length recomputation, abort cleanup.
• It does not address the underlying gap in TSHttpTxnErrorBodySet() behavior, so the next plugin to hit this learns the same lesson.
• Cache-write semantics differ (transformed vs untransformed body), which is configuration users have to think about explicitly.

Production validation

Validated on an internal production edge host running a build that contains this PR's handle_api_return() change (the same captured-hook-id divert logic). The host had been running this build for 65+ hours stably prior to the test.

Method: appended three header-gated header_rewrite rules to an existing per-remap header_rewrite config (rules only fire when the client sends a magic X-CDN-Set-Body-Test: 1 header, so real user traffic is unaffected), bumped remap.config mtime, ran traffic_ctl config reload. Drove ten curl requests with the magic header against a test endpoint where the origin returns a real response body. Reverted the config and reloaded again.

Result:

• All ten test requests returned HTTP 200 with the header_rewrite replacement body (41 bytes, the literal test string) instead of the origin body. Status code, response headers, and connection lifecycle were the expected ones for the divert path: the server session for each transaction was released, no transform was installed, and no client-side errors were observed.
• One verification request sent without the magic header returned the original origin body unchanged (full size), confirming the rule only fires on the gated condition.
• After reverting the rule and waiting for traffic_ctl config reload to propagate, the same test request returned the original origin body. No lingering side effects from the experiment.
• Background traffic on the host (not carrying the magic header) was unaffected throughout.

Automated test coverage

• header_rewrite_set_body_origin AuTest passes, including the null_transform interaction case (TRANSFORM_READ branch).
• Full SDK regression suite passes: SDK_API_TSHttpConnectIntercept, SDK_API_TSHttpConnectServerIntercept, SDK_API_HttpAltInfo, SDK_API_HttpTxnTransform, SDK_API_HttpTxnCache, SDK_API_HttpSsn, SDK_API_HttpHookAdd.
• Exported symbols of libtsapi / libtscore / libtsutil / libtscppapi are unchanged (0 added, 0 removed).