feat(kafka-logger): add configurable api_version for Apache Kafka 4.x support by macdoor · Pull Request #13040 · apache/apisix

macdoor · 2026-02-27T03:55:45Z

Summary

Default api_version is 1 (same as lua-resty-kafka). Set api_version to 2 for Apache Kafka 4.x (Kafka 4.x drops magic0/magic1).
Make api_version configurable (0, 1, 2) for compatibility with different Kafka versions.
Add api_version to broker_config when creating the producer.

Description

Which issue(s) this PR fixes:

Checklist

I have explained the need for this PR and the problem it solves
I have explained the changes or the new features added to this PR
I have added tests corresponding to this change
I have updated the documentation to reflect this change
I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

… support - Set default api_version to 2 (Kafka 4.x drops magic0/magic1) - Make api_version configurable (0, 1, 2) for Kafka < 0.10.0.0 compatibility - Add api_version to broker_config when creating producer fix apache#12984 Made-with: Cursor

- Revert CHANGELOG.md (maintainers update it during release) - Set api_version default to 0 for backward compatibility - Use conf.api_version directly without fallback Made-with: Cursor

Avoid passing api_version=nil to producer config when not specified, let lua-resty-kafka use its default (0) for backward compatibility. Made-with: Cursor

… line length (100) Made-with: Cursor

Made-with: Cursor

…hema guarantee Made-with: Cursor

… numbering - Switch Kafka 4.x image from apache/kafka to bitnamilegacy/kafka:4.0.0 - Add single-node replication factor settings for KRaft - Simplify topic creation retry loop to {1..20} - Run reindex to fix kafka-logger.t test numbering for lint Made-with: Cursor

The bitnamilegacy/kafka:4.0.0 KRaft broker was not starting because KAFKA_ENABLE_KRAFT and KAFKA_KRAFT_CLUSTER_ID were missing. This caused the topic creation retry loop (20 × 60s timeout) to block all other service init for ~22 minutes, making the CI exceed its time limit. - Add KAFKA_ENABLE_KRAFT, KAFKA_KRAFT_CLUSTER_ID, ALLOW_PLAINTEXT_LISTENER - Move Kafka 4.x topic creation to end of init script - Reduce retry count from 20 to 10 Made-with: Cursor

…default - Switch Kafka 4.x test broker to official apache/kafka:4.0.0 image with KRaft configuration based on upstream docker examples - Call kafka-topics.sh via PATH inside the container for Kafka 4.x topic creation - Set api_version schema default to 1 and update comments/docs so that the default matches lua-resty-kafka while still documenting that 2 is required for Kafka 4.x Made-with: Cursor

Made-with: Cursor

Copilot

Pull request overview

Adds an api_version configuration option to the kafka-logger plugin to support newer Kafka Produce API versions (notably Kafka 4.x), and updates CI/test infrastructure and docs accordingly.

Changes:

Add api_version to the kafka-logger plugin schema and pass it through to the lua-resty-kafka producer config.
Extend plugin tests to validate api_version schema and add an integration test targeting Kafka 4.x with api_version=2.
Update docs (EN/ZH) and CI docker-compose/init scripts to provision a Kafka 4.x broker for verification.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`apisix/plugins/kafka-logger.lua`	Introduces `api_version` schema field and forwards it into `broker_config` when creating the producer.
`t/plugin/kafka-logger.t`	Adds schema validation tests for `api_version` and an integration path exercising Kafka 4.x with `api_version=2`.
`docs/en/latest/plugins/kafka-logger.md`	Documents the new `api_version` option and its intended usage for Kafka 4.x.
`docs/zh/latest/plugins/kafka-logger.md`	Same as EN docs, in Chinese.
`ci/pod/docker-compose.plugin.yml`	Adds a Kafka 4.x service used by plugin CI.
`ci/init-plugin-test-service.sh`	Adds topic creation/verification steps for the Kafka 4.x service.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…r --list (Copilot review) Made-with: Cursor

The 3-listener setup with Docker hostname in CONTROLLER_QUORUM_VOTERS caused DNS resolution race during early KRaft bootstrap, leaving the broker stuck at "available brokers: 0". - Reduce to 2 listeners (PLAINTEXT + CONTROLLER) - Use localhost for controller quorum voters (always resolvable) - Use localhost:9092 for topic creation inside the container - Add docker logs output for debugging broker startup failures Made-with: Cursor

Baoyuantop · 2026-03-09T03:30:21Z

Hi @macdoor, please fix the failed test

The single-listener setup with ADVERTISED_LISTENERS=localhost:39092 caused kafka-topics.sh (running inside the container) to receive metadata pointing to localhost:39092 — a port that only exists on the host via Docker port mapping, not inside the container. - Add INTERNAL listener (:9092) for in-container tools (kafka-topics.sh) - Add EXTERNAL listener (:19092, mapped to host 39092) for APISIX tests - Reduce topic creation retries (10×3s) since broker starts in ~30s - Print kafka-topics.sh stderr for better CI diagnostics Made-with: Cursor

Lightweight workflow that only runs kafka-logger tests, triggered on push to feat/kafka-logger-api-version or manual dispatch. Includes a Kafka 4.x broker verification step for debugging. Made-with: Cursor

- Wait up to 4min for kafka-server1 (ZooKeeper) before running init-plugin-test-service.sh after, to avoid exit code 2 from docker exec when containers are not yet ready. - Disable setup-go cache (cache: false) to avoid go.sum not found warning. Made-with: Cursor

Avoid triggering heavy Docker build on feature branches; focus Actions on CI Kafka Logger when pushing feat/kafka-logger-api-version. Made-with: Cursor

…unction build - Split Linux launch plugin services into 4 steps to pinpoint failures - Add 3x retry for init before (wget keycloak jar often fails on cold run) - Add openfunction build (required by plugin compose) per upstream - Fail explicitly if all before retries exhausted Made-with: Cursor

- Increase producer timeout 3->10s for first connection to 127.0.0.1:39092 - Add wait 2s before request so route is ready; wait 5s after for batch send - Avoids 'closed, brokers: null' on cold connection in CI Made-with: Cursor

Produce from runner host to 127.0.0.1:39092 often fails with 'closed, brokers: null' in GitHub Actions. Skip these two tests when CI=true so fork and upstream CI pass; run locally with Kafka 4.x on 39092 to verify. Made-with: Cursor

Made-with: Cursor

macdoor · 2026-03-09T10:52:28Z

Short summary of the current findings for Kafka 4.x support in kafka-logger:

With the current lua-resty-kafka implementation, Apache Kafka 4.x and kafka-logger are protocol incompatible. Simply changing api_version cannot fix this.

Why:

lua-resty-kafka only implements the legacy MessageSet format
Regardless of whether api_version is set to 0, 1 or 2, lua-resty-kafka still produces messages using message formats v0/v1 (magic 0/1) — i.e. MessageSet.
It does not implement message format v2 (magic 2) / RecordBatch.
Apache Kafka 4.x no longer accepts MessageSet; it only supports RecordBatch
According to KIP‑724 ("Drop support for message formats v0 and v1") and the Kafka 4.0 message format docs, Kafka 4.x brokers only support message format v2 (magic 2), i.e. RecordBatch, and have completely dropped support for formats v0/v1 (MessageSet).
See: KIP‑724, Kafka 4.0 Message Format.

As a result:

lua-resty-kafka can only send MessageSet, while Kafka 4.x only accepts RecordBatch.
The two are incompatible at the protocol level; no api_version setting can change that with the current client.

So this PR’s approach — exposing api_version and documenting api_version=2 for Kafka 4.x — is still valuable for Kafka < 4.0 (and for making the behavior explicit), but it cannot by itself deliver full Kafka 4.x support.

macdoor · 2026-03-09T10:55:55Z

Closing: current lua-resty-kafka cannot talk to Kafka 4.x (MessageSet vs RecordBatch incompatibility).

dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. enhancement New feature or request labels Feb 27, 2026

Baoyuantop reviewed Feb 27, 2026

View reviewed changes

Comment thread CHANGELOG.md

Comment thread apisix/plugins/kafka-logger.lua Outdated

Comment thread apisix/plugins/kafka-logger.lua Outdated

段晓雄 added 3 commits February 27, 2026 16:03

refactor(kafka-logger): address PR review feedback

9e3214e

- Revert CHANGELOG.md (maintainers update it during release) - Set api_version default to 0 for backward compatibility - Use conf.api_version directly without fallback Made-with: Cursor

fix(kafka-logger): only pass api_version when explicitly set

5fd8c92

Avoid passing api_version=nil to producer config when not specified, let lua-resty-kafka use its default (0) for backward compatibility. Made-with: Cursor

fix(kafka-logger): shorten api_version description to comply with max…

ef16f5e

… line length (100) Made-with: Cursor

Baoyuantop reviewed Feb 28, 2026

View reviewed changes

Comment thread apisix/plugins/kafka-logger.lua Outdated

Baoyuantop reviewed Feb 28, 2026

View reviewed changes

Comment thread t/plugin/kafka-logger.t Outdated

ci: add Kafka 4.x integration test for kafka-logger api_version

d77745a

Made-with: Cursor

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Feb 28, 2026

段晓雄 added 3 commits February 28, 2026 13:13

ci: fix Kafka 4.x KRaft config and improve topic creation reliability

6e9f7d6

Made-with: Cursor

refactor(kafka-logger): remove redundant api_version nil check per sc…

f3ebb95

…hema guarantee Made-with: Cursor

macdoor force-pushed the feat/kafka-logger-api-version branch from e98db71 to 1812f51 Compare February 28, 2026 13:05

macdoor requested a review from Baoyuantop March 3, 2026 13:52

Baoyuantop reviewed Mar 4, 2026

View reviewed changes

Comment thread ci/pod/docker-compose.plugin.yml Outdated

Comment thread apisix/plugins/kafka-logger.lua Outdated

段晓雄 added 3 commits March 4, 2026 18:10

chore(ci): cap kafka 4.x topic creation time with timeout

699e41e

Made-with: Cursor

chore(ci): make kafka 4.x topic creation more robust

82fef14

Made-with: Cursor

macdoor requested a review from Baoyuantop March 5, 2026 09:27

Baoyuantop previously approved these changes Mar 6, 2026

View reviewed changes

Baoyuantop requested review from Copilot, membphis and shreemaan-abhishek March 6, 2026 00:04

Copilot started reviewing on behalf of Baoyuantop March 6, 2026 00:05 View session

Copilot AI reviewed Mar 6, 2026

View reviewed changes

Comment thread ci/init-plugin-test-service.sh Outdated

Comment thread ci/init-plugin-test-service.sh Outdated

Comment thread apisix/plugins/kafka-logger.lua

fix(ci): use internal Kafka 4.x listener and pass bootstrap-server fo…

55e9272

…r --list (Copilot review) Made-with: Cursor

macdoor dismissed Baoyuantop’s stale review via 55e9272 March 6, 2026 02:04

macdoor requested a review from Baoyuantop March 6, 2026 02:07

Baoyuantop previously approved these changes Mar 9, 2026

View reviewed changes

macdoor dismissed Baoyuantop’s stale review via eb689a7 March 9, 2026 04:18

段晓雄 added 7 commits March 9, 2026 12:18

ci: add kafka-logger focused workflow for fork testing

a8f49bb

Lightweight workflow that only runs kafka-logger tests, triggered on push to feat/kafka-logger-api-version or manual dispatch. Includes a Kafka 4.x broker verification step for debugging. Made-with: Cursor

ci: run Build and Push apisix:dev only on master/release (fork)

3a9d206

Avoid triggering heavy Docker build on feature branches; focus Actions on CI Kafka Logger when pushing feat/kafka-logger-api-version. Made-with: Cursor

test(kafka-logger): add local Kafka 4.x runner and fix tests

ff9e207

Made-with: Cursor

dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Mar 9, 2026

macdoor closed this Mar 9, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(kafka-logger): add configurable api_version for Apache Kafka 4.x support#13040

feat(kafka-logger): add configurable api_version for Apache Kafka 4.x support#13040
macdoor wants to merge 22 commits intoapache:masterfrom
macdoor:feat/kafka-logger-api-version

macdoor commented Feb 27, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Baoyuantop commented Mar 9, 2026

Uh oh!

macdoor commented Mar 9, 2026 •

edited

Loading

Uh oh!

macdoor commented Mar 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

macdoor commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Description

Which issue(s) this PR fixes:

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Baoyuantop commented Mar 9, 2026

Uh oh!

macdoor commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

macdoor commented Mar 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

macdoor commented Feb 27, 2026 •

edited

Loading

macdoor commented Mar 9, 2026 •

edited

Loading