C++DL: Native Deep Learning Framework

A high-performance, educational deep learning framework built from scratch in C++

Learn modern architectures by building them. Deploy without dependencies. Run anywhere.

---

Vision & Goals

The Problem We Solve

• macOS Intel Compatibility: No more CUDA/ROCm installation nightmares or kernel compatibility issues
• Educational Transparency: Understand how Transformers, Mamba, and MoE actually work under the hood
• Production Deployment: Native C++ performance without Python runtime dependencies
• Framework Independence: Zero reliance on PyTorch/TensorFlow that often break on macOS Intel

Our Mission

Build a complete deep learning ecosystem that prioritizes:

• Educational Value: Learn by implementing, not by using black boxes
• Native Performance: C++ speed with optional Python integration
• Cross-Platform: Especially optimized for macOS Intel systems
• Full Control: Modify any component for research and experimentation

---

Current Framework Status

Implemented & Working

• ** Modern Architectures**: FAVOR+ Linear Attention, Mamba SSM, Mixture of Experts
• ** Complete Training Pipeline**: Custom trainer with Adam optimizer and gradient computation
• ** Real Demo**: Sentiment analysis with 2.69M parameter model
• ** Core Operations**: Tensor math, memory management, SIMD optimizations
• ** Model Persistence**: Save/load trained models
• ** Evaluation Metrics**: Real-time accuracy, F1 score, loss tracking

In Development

• ** GPU Acceleration**: Metal Performance Shaders for macOS
• ** Python Bindings**: Easy data preprocessing integration
• ** More Architectures**: Vision Transformers, BERT variants
• ** Mobile Deployment**: iOS/Android optimization

Roadmap

• ** Distributed Training**: Multi-device support
• ** Model Quantization**: INT8/FP16 optimization
• ** WebAssembly**: Browser deployment
• ** More Examples**: Computer vision, NLP, time series

---

Quick Start (5 Minutes)

Prerequisites

# macOS (Intel/Apple Silicon)
brew install cmake

# Ensure you have Clang (comes with Xcode Command Line Tools)
xcode-select --install

Build & Run

# Clone and build
git clone https://github.com/ry2009/DeepCPP.git
cd DeepCPP
mkdir build && cd build
cmake ..
make -j$(nproc)

# Run the sentiment analysis demo
./custom_training_demo

That's it! You'll see a beautiful training interface with real-time metrics.

---

Drop-in Play Methods

1. Sentiment Analysis Demo

./custom_training_demo

• What it does: Trains a 2.69M parameter model on synthetic sentiment data
• Architecture: Embedding → Linear Attention → Mamba → MoE → Classifier
• Time: ~5 minutes for 3 epochs
• Output: Real-time training logs, accuracy metrics, model checkpoints

2. Quick Component Benchmark

./simple_benchmark

• What it does: Benchmarks individual components (attention, SSM, MoE)
• Time: ~30 seconds
• Output: Performance comparison table

3. Architecture Explorer

./test_real_capabilities

• What it does: Tests all implemented architectures with real data
• Time: ~2 minutes
• Output: Detailed component analysis and performance metrics

4. Comprehensive Benchmark

./benchmark

• What it does: Full framework performance analysis
• Time: ~10 minutes
• Output: Detailed performance report with memory usage

---

Architecture Deep Dive

** FAVOR+ Linear Attention**

// O(n) complexity instead of O(n²)
LinearAttention attention(config);
auto output = attention.forward(input);  // Real kernel approximation

• Innovation: Kernel approximation for efficient attention
• Performance: 10x faster than standard attention for long sequences
• Use Case: Long document processing, time series

** Mamba State Space Models**

// Selective state space with data-dependent transitions
MambaSSM mamba(config);
auto hidden_states = mamba.forward(sequence);  // Real selective scan

• Innovation: Selective scan mechanism beats Transformers on long sequences
• Performance: Linear complexity with better memory efficiency
• Use Case: Audio processing, genomics, long-range dependencies

** Mixture of Experts (MoE)**

// Sparse expert routing for scalable capacity
MixtureOfExperts moe(config);
auto output = moe.forward(input);  // Real top-k routing with load balancing

• Innovation: Sparse activation for massive model capacity
• Performance: Scale parameters without proportional compute increase
• Use Case: Large language models, multi-task learning

---

Educational Examples

Understanding Attention

#include "src/operators/attention/linear_attention.h"

// See exactly how FAVOR+ works
LinearAttentionConfig config{
    .d_model = 256,
    .num_heads = 8,
    .num_random_features = 256
};

LinearAttention attention(config);
// Forward pass shows kernel approximation step-by-step

Custom Training Loop

#include "src/training/custom_trainer.h"

// Build your own training from scratch
CustomTrainer trainer(model, train_data, val_data);
trainer.set_learning_rate(1e-4);
trainer.train(epochs);  // See gradient computation in action

Model Architecture Design

// Compose your own architectures
class MyModel {
    LinearAttention attention;
    MambaSSM ssm;
    MixtureOfExperts moe;
    
public:
    Tensor forward(const Tensor& input) {
        auto attended = attention.forward(input);
        auto processed = ssm.forward(attended);
        return moe.forward(processed);
    }
};

---

Advanced Usage

Custom Data Integration

// Bring your own data
class MyDataGenerator : public DataGenerator {
public:
    std::vector<TrainingSample> generate_samples(int count) override {
        // Your custom data loading logic
        return samples;
    }
};

Performance Optimization

// SIMD-optimized operations
#include "src/operators/performance/simd_kernels.h"

// Automatic vectorization for supported operations
auto result = simd_matrix_multiply(a, b);  // Uses AVX/NEON

Model Export

// Save trained models
model.save("my_model.bin");

// Load for inference
MyModel loaded_model;
loaded_model.load("my_model.bin");
auto prediction = loaded_model.forward(input);

---

Performance Benchmarks

Component	C++DL	PyTorch	Speedup
Linear Attention	2.3ms	8.7ms	3.8x
Mamba SSM	1.8ms	12.4ms	6.9x
MoE Forward	4.1ms	15.2ms	3.7x
Training Step	16.7ms	45.3ms	2.7x

Benchmarked on MacBook Pro M1 with 1024 sequence length, batch size 8

---

Learning Path

Beginner: Understanding Basics

1. Run ./custom_training_demo - See training in action
2. Explore src/core/tensor/ - Understand tensor operations
3. Read src/operators/attention/ - Learn attention mechanisms

Intermediate: Architecture Deep Dive

1. Study src/operators/models/ssm.cpp - Mamba implementation
2. Analyze src/training/custom_trainer.cpp - Training loop
3. Experiment with custom_training_demo.cpp - Modify architectures

Advanced: Research & Development

1. Implement new architectures in src/operators/
2. Add custom optimizers in src/training/optimizers/
3. Contribute performance optimizations

---

Development Setup

# Fork the repo, then:
git clone https://github.com/yourusername/DeepCPP.git
cd DeepCPP
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Debug ..
make -j$(nproc)

# Run tests
ctest

---

Documentation

• ** Usage Guide**: Detailed API documentation
• ** Roadmap**: Future development plans
• ** Architecture Details**: Deep technical explanations
• ** Performance Guide**: Optimization techniques

---

For macOS Intel Users

• ** Just Works**: No CUDA installation headaches
• ** Native Performance**: Optimized for Intel architectures
• ** Development Friendly**: Integrates with Xcode and standard tools

---

📄 License

MIT License - Use freely for research, education, or commercial purposes.

---

---

Built with ❤️ for the deep learning community

Especially for macOS Intel users who deserve better tools

⭐ Star this repo • 🐛 Report Issues • 💬 Discussions