Add kafka_input_otel package for Kafka receiver#17438
Conversation
Create new OpenTelemetry input package for consuming telemetry data (logs, metrics, traces) from Kafka using the Kafka receiver. Key features: - Multi-signal support with dynamic_signal_types: true - Signal-specific topics and encoding configuration - SASL, Kerberos, and TLS authentication options - Consumer settings (offset, session timeout, heartbeat) - Metadata and auto-commit configuration - Header extraction support - Telemetry and error backoff configuration Requires Kibana ^9.4.0 and format_version 3.6.0 for multi-signal support. Closes #17139 Co-authored-by: Mario Rodriguez Molins <mrodm@users.noreply.github.com>
✅ Vale Linting ResultsNo issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
The template has a structural issue where SASL and Kerberos authentication blocks could both generate separate auth: YAML keys, creating invalid YAML. See inline comment for the fix.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
andrewkroh
added
New Integration
There was a problem hiding this comment.
The duplicate auth: block issue (SASL vs Kerberos generating separate YAML keys) identified in the prior review remains the blocking issue. Once fixed using {{else if kerberos_service_name}}, the package should be ready.
Consider adding policy tests for SASL and Kerberos authentication branches once the template is fixed, to catch this class of issue in the future.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
There was a problem hiding this comment.
The duplicate auth: block issue identified in prior reviews (SASL vs Kerberos generating separate YAML keys) still needs to be fixed before this can be approved. No additional issues found.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
Ensure that auth.sasl and auth.kerberos are not set at the same time. Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
There was a problem hiding this comment.
The authentication template logic at lines 96-167 in input.yml.hbs is still broken. The attempted fix from prior reviews created a new problem: when kerberos_service_name is set without sasl_mechanism, both the kerberos block at lines 127-146 AND the duplicate kerberos block at lines 148-166 are output, creating invalid YAML with duplicate kerberos: keys.
The entire auth section (lines 96-167) needs to be replaced with the single, clean if/else-if structure provided in the inline comment.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
Removed redundant SASL and Kerberos authentication options to streamline the configuration process. Ensured that only relevant authentication settings are retained for clarity and maintainability.
Streamlined the Kerberos authentication configuration by removing redundant checks and ensuring that the keytab file is only referenced when the keytab option is enabled. This improves clarity and maintainability of the authentication settings.
|
Working on adding system tests to validate that it works as expected. |
There was a problem hiding this comment.
The auth block issues from prior reviews appear to be fixed with proper {{else if kerberos_service_name}} syntax. See inline comment for remaining issue with the profiles block placement.
Why is Claude responding? | Type @claude to interact further
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
…, and otelcol - Replace multi-variant (logs/metrics/traces/java8/zookeeper) setup with unified telemetrygen + otelcol + Kafka (KRaft) architecture - Add _dev/build/docs/README.md template; docs rendered by elastic-package build - Add otelcol config, telemetrygen Dockerfile/entrypoint, kraft/ directory - Consolidate system tests to test-default-config.yml - Update docker-compose, variants, input template, manifest Made-with: Cursor
- Update Documentation section to reference _dev/build/docs/README.md template - Update checklist: template rendered by elastic-package build Made-with: Cursor
Add three new system test configurations to validate each signal type (logs, metrics, traces) independently using otlp_proto encoding, along with a test config.yml setting parallel execution to false for easier debugging. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add {{else}} fallback to logs, metrics, and traces topic blocks in
input.yml.hbs so that explicitly setting an empty topics list emits
`topics: []` instead of falling back to receiver defaults. Also move
the unconditional `profiles: topics: []` outside the traces block so it
is always rendered regardless of traces configuration.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Set empty arrays for unused signal topics (logs_topics, metrics_topics, traces_topics) in per-signal test configs so the kafka receiver emits `topics: []` for those signals rather than falling back to receiver defaults. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the telemetrygen shell script container with a custom Go generator (modelled on otlp_input_otel) that sends 1000 logs, metrics, and traces in parallel via gRPC to otelcol:4317. Instead of waiting for a SIGHUP signal, the generator polls the otelcol health endpoint (otelcol:13133) and starts sending once it responds healthy. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Update architecture description to replace telemetrygen references with the custom Go generator, document the dual Kafka exporter topics and encodings, and note the batch processor added to otelcol. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
mrodm
added
the
New Integration
- Update secret handling: password fields now use `secret: true`, supported from Kibana v9.2.7, v9.3.2, and v9.4.0+ (fleet-server#6277) - Add Kibana version requirements section: logs/metrics require ^9.2.0, traces and multi-signal packages (dynamic_signal_types: true) require ^9.4.0 - Clarify README approach: docs/README.md can be written directly or generated from _dev/build/docs/README.md template (optional) - Update submission checklist to reflect the above changes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add a Multi-Signal Packages section explaining how to use dynamic_signal_types: true for receivers that emit multiple signal types, including a manifest snippet, service.pipelines template example, and a note on the Kibana 9.4.0+ requirement. Also update the kafka_input_otel entry in the examples table to reflect its multi-signal nature. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
andrewkroh
removed
the
New Integration
…Interval - Replace wg.Add/defer wg.Done goroutine boilerplate with wg.Go (Go 1.25+) - Extract sendSignal helper to eliminate three near-identical goroutine blocks - Name the 100ms periodic reader interval as metricInterval constant Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…ages docs The type-mapping table now links readers directly to the Variable Types section where the Kibana version requirement for secret: true is explained. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…eference The batch processor was removed from the otelcol pipeline but the README still described it. Also fix a stale telemetrygen reference in the otelcol config comment. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace 'via' with 'using' and 'e.g.' with 'for example' to comply with the Elastic Docs style guide (Elastic.Latinisms rule). Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
💚 Build Succeeded
History
|
4 participants
Proposed commit message
Create new OpenTelemetry input package for consuming telemetry data (logs, metrics, traces) from Kafka using the Kafka receiver.
Key features:
dynamic_signal_types: trueRequires Kibana ^9.4.0 and format_version 3.6.0.
Checklist
changelog.ymlfile.I have verified that any added dashboard complies with Kibana's Dashboard good practicesAuthor's Checklist
How to test this PR locally
Related issues
use_apmvariable in policy and system tests in input packages elastic-package#3295Screenshots
Generated with Claude Code
Logo generated also with the assistance of Cursor.