Skip to content

README.md

Latest commit

 

History

History
285 lines (193 loc) · 13.4 KB

README.md

File metadata and controls

285 lines (193 loc) · 13.4 KB

🚀 Distributed Streaming Platform

A production-grade distributed microservices streaming platform built using Spring Boot, Kafka, and Kubernetes.

This project demonstrates how modern backend systems are designed using:

  • 🧩 Microservices Architecture
  • ⚡ Event-Driven Communication (Kafka)
  • 🔍 Service Discovery (Eureka)
  • 🌐 API Gateway Routing
  • ☸️ Kubernetes Deployment
  • 🐳 Containerization with Docker

🎯 What This Project Solves

This system simulates a real-world video streaming backend where:

  • Users upload content
  • Content is processed asynchronously
  • Streaming is handled efficiently
  • Analytics are generated from events

💡 Why This Project Matters

This project is designed to reflect real industry-level system design, covering:

  • Scalable microservices
  • Distributed communication
  • Fault tolerance challenges
  • Kubernetes orchestration

🏗️ Architecture Overview

The system follows a microservices-based distributed architecture deployed on Kubernetes.

It is designed to handle high-throughput streaming workloads using asynchronous event-driven communication.

📊 High-Level System Architecture

system_architecture.png

🔍 Microservices Architecture

microservices-architectur.jpg

🧠 Core Architectural Concepts

1. API Gateway Pattern

All incoming requests are routed through a centralized API Gateway, which:

  • Handles routing
  • Provides a single entry point
  • Simplifies client interaction

2. Service Discovery (Eureka)

Each microservice registers with the Discovery Server, enabling:

  • Dynamic service lookup
  • Loose coupling between services
  • Resilience in distributed communication

3. Event-Driven Architecture (Kafka)

The system uses Apache Kafka for asynchronous communication:

  • Content ingestion → produces events
  • Streaming & analytics → consume events
  • Decouples services for scalability

4. Database per Service Pattern

Each service manages its own data:

  • MySQL → persistent storage
  • Redis → caching layer
  • MinIO → object storage (videos/files)

5. Kubernetes Orchestration

All components are deployed using Kubernetes:

  • Deployments → stateless services
  • StatefulSets → Kafka
  • Services → internal networking
  • Ingress → external routing

🔄 Request Flow (Simplified)

  1. User hits streaming.local via Ingress
  2. Request goes to API Gateway
  3. Routed to respective microservice
  4. Services communicate via:
    • REST (synchronous)
    • Kafka (asynchronous)
  5. Data stored and processed across services

📁 Project Structure

distributed-streaming-platform/
│
├── api-gateway/              # API Gateway (Routing layer)
├── discovery-server/         # Eureka Service Registry
├── auth-service/             # Authentication & JWT
├── user-service/             # User management
├── content-service/          # Content metadata
├── content-ingestion-service/# Content upload pipeline
├── streaming-service/        # Streaming logic
├── video-processing-service/ # Video processing pipeline
├── analytic-service/         # Analytics & event processing
│
├── common-lib/               # Shared DTOs & utilities
├── kafka-kraft/              # Kafka setup (KRaft mode)
│
├── k8s/
│   ├── infrastructure/       # Kafka, MySQL, Redis, MinIO
│   ├── services/             # Microservice deployments
│   ├── ingress/              # Ingress configuration
│   └── namespace/            # Namespace setup
│
├── docker-compose.yaml       # Local development setup
├── init-db.sql               # Database initialization
├── images.txt                # List of Docker images
├── prometheus.yml           # Monitoring config
├── pom.xml                  # Parent Maven configuration
└── README.md

🧠 Structure Highlights

🔹 Modular Microservices

Each service is independently developed and deployed, enabling:

  • Independent scaling
  • Fault isolation
  • Faster development cycles

🔹 Shared Library (common-lib)

Contains:

  • DTOs
  • Event classes
  • Common utilities

👉 Ensures consistency across services

🔹 Infrastructure as Code (Kubernetes)

All infrastructure is defined declaratively under k8s/:

  • Infrastructure layer (Kafka, DBs)
  • Service deployments
  • Networking (Ingress)

🔹 Hybrid Setup

  • Docker Compose → Local development
  • Kubernetes → Production-like deployment

⚙️ Tech Stack

🧠 Backend

  • Java (Spring Boot) — Core microservices framework
  • Spring Cloud — Service discovery & gateway
  • Spring Security + JWT — Authentication & authorization

⚡ Messaging & Streaming

  • Apache Kafka — Event-driven communication
  • Kafka KRaft Mode — Zookeeper-less setup

🗄️ Databases & Storage

  • MySQL — Relational database
  • Redis — Caching layer
  • MinIO — Object storage (video/files)

☸️ DevOps & Infrastructure

  • Docker — Containerization
  • Kubernetes (Minikube) — Container orchestration
  • Ingress Controller (NGINX) — External routing

📊 Monitoring & Observability

  • Prometheus — Metrics collection

🛠️ Build & Tools

  • Maven — Build tool
  • Docker Compose — Local setup

🧩 Microservices Overview

Service Responsibility
API Gateway Routes incoming requests
Discovery Server Service registry (Eureka)
Auth Service Authentication & JWT
User Service User management
Content Service Content metadata
Content Ingestion Service Upload pipeline
Streaming Service Streaming logic
Video Processing Service Video processing
Analytics Service Event analytics

📸 Screenshots

🟢 Kubernetes Pods (All Services Running)

!3.jpg

🔌 Services (Internal Networking)

!5.jpg

🚀 Key Highlights

  • Built a distributed microservices system from scratch
  • Implemented event-driven architecture using Kafka
  • Deployed complete system on Kubernetes (Minikube)
  • Integrated API Gateway + Service Discovery (Eureka)
  • Designed for scalability, modularity, and fault isolation

🔮 Future Improvements

  • Add CI/CD pipeline (GitHub Actions)
  • Enable autoscaling (HPA)

🧑‍💻 Author

Yash chauhan
Backend Engineer | Distributed Systems Enthusiast

⭐ Support

If you found this project useful, consider giving it a ⭐ on GitHub.