OTEL Kafka Exporter

[jackburridge]

Is your feature request related to a problem?

In some cases it is good to export spans, logs, and metrics direct to Kafka, rather than elsewhere. For example, all your applications could publish to a Kafka cluster/s, with an OpenTelemetry Collector configured to export elsewhere. If you then wanted to change where you wanted to export your telemetry data you would only need to update the configuration for a single deployment

Describe the solution you'd like

Exporters that publish to Kafka. Defaults should match those of the Kafka Receiver from the opentelemetry-collector. For example the default topic for traces would be otlp_spans

Describe alternatives you've considered

No response

Additional Context

No response

Would you like to implement a fix?

Yes

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[xrmx]

@jackburridge What about having an OTel collector on the side re-exporting to kafka with the kafka exporter?

[jackburridge]

@xrmx agreed you could, but this would add additional compute (not a huge amount). The implementation of a Kafka exporter should be relatively simple, most of the protobuf encoding code is in opentelemetry-exporter-otlp-proto-common already.

If you were to use it in something like AWS Lambda for example you could use https://github.com/open-telemetry/opentelemetry-lambda but this adds a collector to the Lambda, why increase the size if you don't have to