feat: add Apache Kafka destination provider

[alexluong]

Summary

• Add Apache Kafka as a destination provider with support for PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512 SASL authentication
• Add Kafka test infrastructure for local development/QA
• Fix portal destination type list scroll and config form overlap
• Add select field type to portal destination config, used for Kafka SASL Mechanism dropdown

Closes #141

Test plan

• Create Kafka destination with brokers, topic, and each SASL mechanism — destination created successfully
• Publish events to Kafka destination — events delivered to correct topic
• Portal renders SASL Mechanism as a dropdown with correct options
• Edit existing Kafka destination — config values reflected correctly
• TLS toggle works as expected
• Partition key template extracts key from event payload
• go build passes

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
outpost-docs	Ready	Preview, Comment	Mar 23, 2026 8:48am
outpost-website	Ready	Preview, Comment	Mar 23, 2026 8:48am

[alexluong]

Local testing guide

1. Start Kafka UI

docker run -d --name outpost-kafka-ui \
  -p 38080:8080 \
  -e KAFKA_CLUSTERS_0_NAME=outpost-test \
  --network outpost \
  -e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=test-kafka:29092 \
  provectuslabs/kafka-ui:latest

Open http://localhost:38080

2. Create a topic

In Kafka UI, create a topic (e.g. test-events).

3. Create a Kafka destination via Portal

Open the portal at http://localhost:3334 and create a new destination:

• Type: Apache Kafka
• Brokers: test-kafka:29092
• Topic: test-events
• SASL Mechanism: None
• TLS: unchecked

4. Publish an event

curl -X POST http://localhost:3333/api/v1/publish \
  -H "Authorization: Bearer apikey" \
  -H "Content-Type: application/json" \
  -d '{
    "tenant_id": "tenant_1",
    "topic": "user.created",
    "data": {"user_id": "u_123", "email": "test@example.com"},
    "metadata": {"event-id": "evt_abc"}
  }'

5. Verify delivery

• Check the Outpost portal for the delivery status
• Check Kafka UI (http://localhost:38080) → topic test-events:
  • Payload matches the published event data
  • Partition key is correct (based on partition_key_template if configured)
  • Headers include expected metadata

6. Cleanup

docker rm -f outpost-kafka-ui