🌐 DaNetworkCode Streamer Scraper

A high-performance, multi-platform livestream monitoring service.

Actively tracks live status, viewer counts, metadata, and VODs across Kick, Twitch, YouTube, Vaughn, Parti, and Pump.fun.

---

✨ Key Features

• Multi-Platform Integration: Connects to Kick, Twitch, YouTube, Vaughn, Parti, and Pump.fun via API and web scraping to build a unified streamer database.
• Pump.fun Crypto Streaming: Tracks streams tied to Solana mint addresses, complete with intelligent WebSocket/React rendering bypass using Puppeteer to scrape live viewer counts.
• Advanced YouTube Quota Management: Features a built-in admin dashboard (/youtube-dashboard) to track daily/monthly usage, archive history, and prevent API exhaustion. Operates on a strict budget target (default 85%).
• Kick OAuth2 & PKCE: Fully automated Kick token generation, client credentials usage, and refreshing for reliable data extraction.
• Intelligent Puppeteer Fallback: Uses headless browser scraping for platforms that require JS rendering, specifically tuned for low-core VPS environments (auto-restarts to prevent memory leaks).
• Crash-Proof Data: Utilizes better-sqlite3 in WAL mode for persistent, lightning-fast reads and writes.
• Security Hardened: Routes are protected using helmet and express-rate-limit.
• Interactive Frontend: Serves a fully responsive UI displaying real-time live status updates, avatars, premiere badges, and viewer counts.

🚀 Installation & Setup

Ensure you have Node.js version 18 or higher installed on your system, or use Docker for a containerized deployment.

Option 1: Docker Compose (Recommended)

# 1. Clone and navigate to the directory
  
git clone https://github.com/Riotcoke123/danetworkcode.git
  
cd danetworkcode

# 2. Configure environment variables (see below)
cp .env.example .env

# 3. Build and run the container

docker-compose up -d --build

Option 2: Standard Node.js / PM2

# 1. Clone the repository
  
git clone https://github.com/Riotcoke123/danetworkcode.git
  
cd danetworkcode

# 2. Install dependencies
  
npm install
  
# 3. Configure environment variables
  
cp .env.example .env
  
# 4. Start the server
  
npm start
  
# OR start with PM2 for production deployments
  
npm run pm2-start
  

⚙️ Environment Variables (.env)

You will need to create a .env file in the root directory. Below are the required configurations:

# Server Settings
PORT=3000
CHECK_INTERVAL_SECONDS=300
ADMIN_TOKEN=your_secure_random_string_here
TRUST_PROXY_HOPS=1
USER_AGENT=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36

# Concurrency Limits (Tune based on your VPS cores/RAM)
CONCURRENT_LIMIT=4
PUPPETEER_CONCURRENT_LIMIT=2
MAX_SCRAPES_BEFORE_RESTART=20
SCRAPE_TIMEOUT_MS=240000

# Streamer Lists (Comma separated)
KICK_USERNAMES=streamer1,streamer2
TWITCH_USERNAMES=streamer1,streamer2
YOUTUBE_USERNAMES=streamer1,streamer2
VAUGHN_USERNAMES=streamer1,streamer2
PARTI_USER_IDS=12345,67890
PUMPFUN_MINTS=21rKrtBzibPAZHAHQRzGiGDSh7XimCKB2a8VgsjZpump,AnotherMint

# API Keys & Auth Setup
YOUTUBE_API_KEY=your_google_api_key
YOUTUBE_QUOTA_DAILY_LIMIT=1010000
YOUTUBE_MONTHLY_BUDGET_PERCENT=0.85

TWITCH_CLIENT_ID=your_twitch_client_id
TWITCH_CLIENT_SECRET=your_twitch_client_secret

KICK_CLIENT_ID=your_kick_client_id
KICK_CLIENT_SECRET=your_kick_client_secret
KICK_REDIRECT_URI=http://localhost:3000/auth/kick/callback

# Optional Parti API Token
PARTI_AUTH_TOKEN=your_parti_token

# Optional Pump.fun API Key (if required by frontend API)
PUMPFUN_API_KEY=your_pumpfun_api_key

🛡️ Admin Dashboard & Endpoints

This project exposes several endpoints. Admin routes require your ADMIN_TOKEN passed as a Bearer token or as a query parameter (?token=YOUR_TOKEN).

Endpoint	Description	Access
/api/streamers	Returns JSON list of all tracked streamers & their current live status.	Public
/api/stats	Returns aggregated viewer counts and total active streams.	Public
/login/kick	OAuth entry point to authorize the Kick API integration.	Public (Rate Limited)
/youtube-dashboard	Visual HTML dashboard showing YouTube API quota consumption.	Admin Only
/api/youtube/quota	Returns current YouTube quota usage and budget status data.	Admin Only
/api/youtube/audit	Generates JSON audit history for API compliance checks.	Admin Only
/healthz	System health, active scrapers, quota snapshots, and error logs.	Admin Only

---

Developed & Maintained by Riotcoke123. Released under the GNU General Public License v3.0 (GPL-3.0).