update: removed all schema registry stuff

[HardMax71]

---

Summary by cubic

Removed Confluent Schema Registry and Avro from the stack and switched Kafka events to JSON via FastStream with Pydantic models. This simplifies infra across local, CI, and Helm, and updates docs to reflect the new approach.

• Refactors

  • Replaced Avro/Schema Registry with FastStream JSON; domain events are plain Pydantic BaseModel (docs updated across architecture and components).
  • Removed schema-registry service and references from docker-compose, deploy.sh, Kafdrop config, e2e boot action, and stack-tests workflow.
  • Dropped Helm Schema Registry templates, helpers, values, and env vars; kafka-init job now waits only for Kafka.
  • Cleaned configuration docs by removing SCHEMA_REGISTRY_URL and updating diagrams and examples.

• Migration

  • Helm: upgrade the chart; remove any schemaRegistry values and delete existing schema-registry resources if present.
  • Config: remove SCHEMA_REGISTRY_URL from environment/Secrets; not used anymore.
  • No app code changes required for producers/consumers using FastStream; redeploy to apply infra changes.

^{Written for commit c8d0a90. Summary will update on new commits.}

Summary by CodeRabbit

• Refactor

  • Removed Schema Registry; events now use JSON serialization with Pydantic and FastStream, and producer/subscriber flow simplified.

• Chores

  • Updated infrastructure, deployment, Docker Compose, Helm charts, and startup flows to omit Schema Registry and related readiness steps.

• Documentation

  • Revised architecture, getting-started, operations, and component docs to reflect JSON-based events and the new event lifecycle.

• Tests

  • Adjusted test guidance and fixtures to align with the simplified producer lifecycle and broker-managed lifecycle.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR removes the Confluent Schema Registry from infra, CI, Helm charts, and docs; it migrates domain events and producer wiring from Avro/Schema-Registry-based serialization to Pydantic (JSON) with FastStream and relocates UnifiedProducer to backend/app/events/core/producer.py.

Changes

Cohort / File(s)	Summary
Infrastructure & Orchestration docker-compose.yaml, deploy.sh	Removed the schema-registry service and all runtime references; infra startup commands no longer include schema-registry.
CI/CD & GitHub Actions .github/actions/e2e-boot/action.yml, .github/workflows/stack-tests.yml	Removed schema-registry from E2E pre-warm/start commands, log collection loops, and removed related env var.
Helm Chart Values & Templates helm/integr8scode/values.yaml, helm/integr8scode/values-prod.yaml, helm/integr8scode/templates/_helpers.tpl, helm/integr8scode/templates/infrastructure/schema-registry.yaml, helm/integr8scode/templates/jobs/kafka-init.yaml, helm/integr8scode/templates/secrets/env-secret.yaml	Deleted Schema Registry config/values, helper template, Helm template for deployment/service, init-container wait step, and secret env var. Kafka init now waits only for Kafka.
Event System Code & Docs backend/app/domain/events/typed.py, backend/app/events/core/producer.py, docs/architecture/..., docs/components/schema-manager.md, docs/architecture/event-system-design.md, docs/architecture/kafka-topic-architecture.md	Domain events now subclass Pydantic BaseModel instead of AvroBase; UnifiedProducer moved to events/core/producer.py; serialization moved from Avro/Schema Registry to FastStream JSON; documentation updated accordingly.
Operational & Reference Documentation docs/getting-started.md, docs/operations/*, docs/reference/configuration.md, docs/testing/kafka-test-stability.md	Removed Schema Registry mentions, updated deployment/healthcheck/docs and test fixture guidance (producer construction/yield usage).

Sequence Diagram(s)

sequenceDiagram
    participant Producer as UnifiedProducer<br/>(events/core/producer.py)
    participant Broker as FastStream Broker
    participant Kafka as Kafka Topic
    participant Subscriber as FastStream Subscriber<br/>(events/handlers.py)
    participant Handler as Event Handler

    Producer->>Broker: publish(pydantic_model)
    Note over Broker: Serialize to JSON
    Broker->>Kafka: Send JSON message
    Kafka->>Subscriber: Deliver JSON message
    Note over Subscriber: Deserialize to Pydantic<br/>model (validation)
    Subscriber->>Handler: Call with typed model
    Handler->>Handler: Process event

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• dlq manager direct calls of broker #137 — Similar removal of Schema Registry and migration to FastStream/Pydantic JSON across infra and event flow.
• fix: producer to DI #132 — Changes to UnifiedProducer lifecycle, produce signature, and header handling that intersect producer relocation and behavior.
• using faststream broker directly #138 — Removes Schema Registry and updates infra/config references; strongly related at infra and serialization layers.

Poem

🐰 No registry hops upon our stack today,
Pydantic JSON takes Avro's old ballet,
FastStream hums and producers send away,
Events arrive typed, validated, and gay,
Hoppity-hop — new flows for a brighter day!

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check	❓ Inconclusive	The title 'update: removed all schema registry stuff' is vague and uses the non-descriptive term 'stuff', failing to clearly communicate the scope and significance of the changes.	Replace with a more specific and descriptive title such as 'Remove Confluent Schema Registry and migrate to FastStream JSON serialization' or 'Replace Avro/Schema Registry with FastStream Pydantic models'.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch fix/fastream-docs

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[cubic-dev-ai]

No issues found across 20 files

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@docs/testing/kafka-test-stability.md`:
- Around line 63-67: Update the documentation to remove or reframe the
confluent-kafka/librdkafka race-condition explanation: state that this project
uses aiokafka (aiokafka==0.12.0) with the FastStream backend, and that
UnifiedProducer (constructed with broker, event_repository, logger, settings,
event_metrics) delegates to await self._broker.publish() so the confluent-kafka
producer-init lock is not present nor required; either delete the
confluent-kafka-specific paragraphs or replace them with a short note that
aiokafka/FastStream avoids that race and broker lifecycle is managed externally
so explicit start()/stop() on UnifiedProducer is unnecessary.

🧹 Nitpick comments (1)

docs/operations/cicd.md (1)

180-181: Minor doc accuracy nit: the actual service list differs slightly.

The narrative lists (mongo, redis, kafka, zookeeper), but the actual e2e-boot/action.yml (line 31) also starts shared-ca and zookeeper-certgen. Consider mentioning these for completeness, or adding a brief note that cert-generation services are also included.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud