From 78e4a4097ab7d14cb0b75d2545196d2cc145a112 Mon Sep 17 00:00:00 2001 From: Pulkit Chauhan Date: Mon, 16 Feb 2026 22:47:06 +0530 Subject: [PATCH 01/11] Introduction of Docker --- .dockerignore | 61 ++++++ .gitignore | 5 + DOCKER.md | 468 +++++++++++++++++++++++++++++++++++++++++++ Dockerfile | 70 +++++++ docker-compose.yml | 63 ++++++ docker-entrypoint.sh | 289 ++++++++++++++++++++++++++ env.example | 63 ++++++ utility.py | 34 +++- 8 files changed, 1043 insertions(+), 10 deletions(-) create mode 100644 .dockerignore create mode 100644 DOCKER.md create mode 100644 Dockerfile create mode 100644 docker-compose.yml create mode 100644 docker-entrypoint.sh create mode 100644 env.example diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 00000000..d5477c63 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,61 @@ +# Git +.git +.gitignore +.gitattributes + +# Python +__pycache__/ +*.py[cod] +*$py.class +*.so +.Python +env/ +venv/ +ENV/ +.venv +*.egg-info/ +dist/ +build/ +*.egg + +# IDE +.vscode/ +.idea/ +*.swp +*.swo +*~ + +# OS +.DS_Store +Thumbs.db + +# Docker (don't copy docker files into the image) +Dockerfile +docker-compose*.yml +.dockerignore + +# Database lock files +*.db +*.sqlite +*.sqlite3 +.db_initialized + +# Logs +logs/ +*.log + +# Test files +.coverage +htmlcov/ +.pytest_cache/ +.tox/ + +# CI/CD +.github/ +.travis.yml +.circleci/ + +# Temporary files +tmp/ +temp/ +*.tmp \ No newline at end of file diff --git a/.gitignore b/.gitignore index 4f8ca2d8..4a42b168 100644 --- a/.gitignore +++ b/.gitignore @@ -33,6 +33,11 @@ static/img/status/build-windows.svg # Gunicorn gunicorn.pid +gunicorn.ctl + +# Docker-generated files (leak via .:/app volume mount) +.db_initialized +migrations/versions/*docker_auto_migration* # OS Generated Files .DS_Store diff --git a/DOCKER.md b/DOCKER.md new file mode 100644 index 00000000..f33f4bc9 --- /dev/null +++ b/DOCKER.md @@ -0,0 +1,468 @@ +# ๐Ÿณ Sample Platform โ€” Docker Setup Guide + +> One-command local development environment for the CCExtractor Sample Platform. + +--- + +## Table of Contents + +- [Prerequisites](#prerequisites) +- [Quick Start](#quick-start) +- [Architecture](#architecture) +- [Configuration](#configuration) + - [Environment Variables Reference](#environment-variables-reference) + - [Google Cloud Storage](#google-cloud-storage) + - [GitHub Integration](#github-integration) +- [Usage](#usage) + - [Starting the Platform](#starting-the-platform) + - [Stopping the Platform](#stopping-the-platform) + - [Viewing Logs](#viewing-logs) + - [Live Code Reloading (Development)](#live-code-reloading-development) + - [Full Reset (Clean Slate)](#full-reset-clean-slate) +- [Database](#database) + - [Connecting Directly](#connecting-directly) + - [Migrations](#migrations) + - [Re-seeding](#re-seeding) +- [Design Decisions](#design-decisions) +- [Troubleshooting](#troubleshooting) +- [File Overview](#file-overview) + +--- + +## Prerequisites + +| Tool | Minimum Version | Check Command | +| ------------------ | --------------- | ------------------------ | +| **Docker Engine** | 20.10+ | `docker --version` | +| **Docker Compose** | 2.0+ (V2) | `docker compose version` | + +> **Windows / macOS**: Install [Docker Desktop](https://www.docker.com/products/docker-desktop/) โ€” it bundles both. +> +> **Linux**: Install Docker Engine + the Compose plugin via [the official docs](https://docs.docker.com/engine/install/). + +--- + +## Quick Start + +```bash +# 1. Clone the repository (if you haven't already) +git clone https://github.com/CCExtractor/sample-platform.git +cd sample-platform + +# 2. Create your environment file from the template +cp env.example .env +# โ†’ Edit .env with your own values (see Configuration below) + +# 3. (Optional) Place your GCP service-account key +# If you don't have one, the app will still start but GCS features won't work. +# See "Google Cloud Storage" section below. + +# 4. Build and start everything +docker compose up -d --build + +# 5. Wait ~20 seconds for MySQL to initialize, then open: +# http://localhost:5000 +``` + +**Default admin credentials** (set in `.env`): + +| Field | Value | +| -------- | ------------------- | +| Email | `admin@example.com` | +| Password | `admin` | + +--- + +## Architecture + +``` +โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” +โ”‚ Docker Network โ”‚ +โ”‚ sample_platform_network โ”‚ +โ”‚ โ”‚ +โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ +โ”‚ โ”‚ MySQL 8.0 โ”‚โ—„โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ โ”‚ Flask Backend (Py 3.11) โ”‚ โ”‚ +โ”‚ โ”‚ โ”‚ :3306 โ”‚ โ”‚ โ”‚ +โ”‚ โ”‚ db_data vol โ”‚ โ”‚ Gunicorn (4 workers) โ”‚ โ”‚ +โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ :5000 โ”‚ โ”‚ +โ”‚ โ”‚ โ”‚ โ”‚ +โ”‚ โ”‚ .:/app (live mount) โ”‚ โ”‚ +โ”‚ โ”‚ repository_data vol โ”‚ โ”‚ +โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ”‚ +โ”‚ โ”‚ โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”‚โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ + โ”‚ + Host :5000 + http://localhost:5000 +``` + +| Service | Container Name | Image | Exposed Port | +| ----------- | ------------------------- | ----------------------- | ------------ | +| **db** | `sample_platform_db` | `mysql:8.0` | `3306` | +| **backend** | `sample_platform_backend` | Built from `Dockerfile` | `5000` | + +--- + +## Configuration + +All configuration is driven by the **`.env`** file. Docker Compose reads it via `env_file:`. + +> **Single source of truth**: The `.env` file is the only place you set values. There is no +> duplicate `environment:` block in `docker-compose.yml` โ€” this avoids the common problem of +> environment overriding env_file silently. + +### Environment Variables Reference + +#### MySQL + +| Variable | Description | Default | +| ---------------------- | ------------------------------------------------ | ------------------ | +| `MYSQL_ROOT_PASSWORD` | MySQL root password | `root` | +| `MYSQL_USER` | Application-level DB user (auto-created by MySQL)| `sample_platform` | +| `MYSQL_PASSWORD` | Password for the application DB user | `sample_platform` | +| `MYSQL_DATABASE` | Database name | `sample_platform` | + +> MySQL auto-creates `MYSQL_USER` with full grants on `MYSQL_DATABASE`. The root user +> is only used for the healthcheck and initial bootstrap โ€” the application connects as +> the dedicated user. + +#### Database URI + +| Variable | Description | Default | +| -------------------------- | ---------------------------------------- | ------- | +| `SQLALCHEMY_DATABASE_URI` | Full SQLAlchemy connection string | *(see env.example)* | + +> Must be kept consistent with the MySQL variables above. Format: +> `mysql+pymysql://:@db/?charset=utf8mb4` + +#### Networking + +| Variable | Description | Default | +| ------------------ | --------------------------------- | ------- | +| `APP_PORT` | Host port for the Flask app | `5000` | +| `DB_EXTERNAL_PORT` | Host port for direct MySQL access | `3306` | + +#### Security + +| Variable | Description | Default | +| ------------ | ---------------------------------------------------- | ------------------------- | +| `SECRET_KEY` | Flask session secret (fallback; file overrides it) | `change-me-in-production` | +| `HMAC_KEY` | Used by `mod_auth` for email verification tokens | `change-me-in-production` | + +> **How secret keys actually work**: `run.py` reads binary files `/app/secret_key` and +> `/app/secret_csrf` at startup and uses their contents as the real `SECRET_KEY` and +> `CSRF_SESSION_KEY`. These files are auto-generated by the entrypoint on first run. +> The `SECRET_KEY` env var in `.env` is only a config-level fallback used briefly +> before the file-based keys overwrite it. + +โš ๏ธ **Change `SECRET_KEY` and `HMAC_KEY`** before deploying to any shared environment. + +#### Google Cloud Storage + +| Variable | Description | Default | +| ---------------------- | ------------------------------------------------ | --------------------- | +| `GCS_BUCKET_NAME` | GCS bucket for sample file storage | `sample-platform-dev` | +| `SERVICE_ACCOUNT_FILE` | Filename of the GCP key (relative to project root)| `service-account.json`| + +> โš ๏ธ **Must be non-empty**: `run.py` calls `client.bucket(name)` at import time. Use your real bucket name or +> keep the default placeholder. + +#### GitHub + +| Variable | Description | Default | +| ------------------- | ---------------------------------- | ------------- | +| `GITHUB_TOKEN` | Personal access token for API | *(empty)* | +| `GITHUB_OWNER` | GitHub org/user owning the repo | `CCExtractor` | +| `GITHUB_REPOSITORY` | Repository name | `ccextractor` | + +#### Admin Bootstrap + +| Variable | Description | Default | +| ---------------- | ----------------------------------- | ------------------- | +| `ADMIN_USERNAME` | Username for auto-created admin | `admin` | +| `ADMIN_EMAIL` | Email for auto-created admin | `admin@example.com` | +| `ADMIN_PASSWORD` | Password for auto-created admin | `admin` | + +#### Email (Mailgun) + +| Variable | Description | Default | +| --------------- | ---------------------- | --------- | +| `EMAIL_DOMAIN` | Mailgun sending domain | *(empty)* | +| `EMAIL_API_KEY` | Mailgun API key | *(empty)* | + +#### Feature Flags + +| Variable | Description | Default | +| --------------------- | ------------------------------------------------- | ------------- | +| `INSTALL_SAMPLE_DATA` | Seed DB with sample categories & regression tests | `false` | +| `MAINTENANCE` | Enable maintenance mode | `false` | +| `FLASK_ENV` | Flask environment (`development` / `production`) | `development` | + +--- + +### Google Cloud Storage + +The platform uses GCS to store sample files. To enable this: + +1. Create a GCP service account with **Storage Object Admin** permissions. +2. Download the JSON key file. +3. Save it as `./service-account.json` in the project root. +4. Set `GCS_BUCKET_NAME` in your `.env`. + +**Mounting Strategy**: +- **Enabled**: If `GCS_BUCKET_NAME` is set, the container mounts the bucket to `/mnt/gcs_repository` and updates `SAMPLE_REPOSITORY` to point there. This ensures the GCS mount doesn't conflict with your local code volume. +- **Disabled**: If not set, `SAMPLE_REPOSITORY` defaults to `/repository`, which is a standard Docker volume persisting data to your local machine. + +--- + +### GitHub Integration + +GitHub features (CI webhooks, PR testing) require a **Personal Access Token** with `public_repo` scope: + +1. Generate a token at [github.com/settings/tokens](https://github.com/settings/tokens). +2. Set `GITHUB_TOKEN` in your `.env`. + +--- + +## Usage + +### Starting the Platform + +```bash +docker compose up -d --build +``` + +- `-d` runs in detached mode (background). +- `--build` rebuilds the image if `Dockerfile` or `requirements.txt` changed. + +**Startup sequence** (handled by `docker-entrypoint.sh`): + +1. MySQL starts and becomes healthy (~10 s). +2. Backend container starts and the entrypoint: + 1. **Generates secret key files** (`/app/secret_key`, `/app/secret_csrf`) if they don't exist. + 2. **Initializes a git repo** at `/app` (required by GitPython for build-commit display). + 3. **Creates directories** mirroring `install/install.sh` (including `TempFiles/`, `TestFiles/media/`, `TestData/ci-linux/`, `TestData/ci-windows/`, etc.). + 4. **Copies sample files** (`sample1.ts`, `sample2.ts`) to `TestFiles/` and CI scripts to `TestData/`. + 5. **Waits for MySQL** to accept connections. + 6. **Runs database migrations** (stamps HEAD on fresh databases to avoid conflicts). + 7. **Creates the admin user** if no admin exists in the DB (checked via SQL query). + 8. **Seeds sample data** if `INSTALL_SAMPLE_DATA=true` and no admin existed. + 9. **Starts Gunicorn** on port 5000. + +### Stopping the Platform + +```bash +docker compose down +``` + +> Data is persisted in Docker volumes (`db_data`, `repository_data`), so nothing is lost. + +### Viewing Logs + +```bash +# All services +docker compose logs -f + +# Backend only +docker logs -f sample_platform_backend + +# MySQL only +docker logs -f sample_platform_db +``` + +### Live Code Reloading (Development) + +The project root is mounted into the container at `/app`, so **any code change on your host +is immediately visible inside the container**. However, Gunicorn doesn't auto-reload by default. + +To pick up changes without rebuilding: + +```bash +# Restart just the backend (fast, no rebuild) +docker compose restart backend +``` + +If you changed `requirements.txt` or `Dockerfile`, you need a full rebuild: + +```bash +docker compose up -d --build +``` + +### Full Reset (Clean Slate) + +```bash +# Remove containers AND volumes (wipes DB + repository data + secret keys) +docker compose down -v + +# Rebuild from scratch +docker compose up -d --build +``` + +--- + +## Database + +### Connecting Directly + +From your host, using the application user: + +```bash +mysql -h 127.0.0.1 -P 3306 -u sample_platform -psample_platform sample_platform +``` + +Or via Docker: + +```bash +docker exec -it sample_platform_db mysql -u sample_platform -psample_platform sample_platform +``` + +### Migrations + +The entrypoint handles migrations automatically. To run them manually: + +```bash +# Generate a new migration +docker exec -it sample_platform_backend flask db migrate -m "Description" + +# Apply pending migrations +docker exec -it sample_platform_backend flask db upgrade + +# View migration history +docker exec -it sample_platform_backend flask db history + +# Check current migration state +docker exec -it sample_platform_backend flask db current +``` + +### Re-seeding + +The entrypoint checks the database directly for an existing admin user โ€” there is no +file-based sentinel. To re-seed from scratch: + +```bash +# Full reset โ€” wipes the DB volume and restarts +docker compose down -v +docker compose up -d --build +``` + +Or to re-seed without wiping: + +```bash +# Delete the admin user, then restart +docker exec sample_platform_db mysql -u root -proot sample_platform -e "DELETE FROM user;" +docker compose restart backend +``` + +--- + +## Design Decisions + +### Why mount GCS to `/mnt/gcs_repository`? + +We previously mounted GCS directly to `/repository`. However, `docker-compose.yml` mounts a local volume to `/repository` for persistence. GCS FUSE requires an empty directory (or specific flags) and mounting it *over* a Docker volume hides the volume's contents and causes "non-empty directory" errors. + +**Solution**: We mount GCS to a dedicated, clean path (`/mnt/gcs_repository`) and export `SAMPLE_REPOSITORY` to point to it. The application respects this variable, seamlessly switching between local storage and Cloud Storage without code changes. + +### Why are secret keys generated at runtime, not in the Dockerfile? + +`run.py` reads two binary files (`secret_key`, `secret_csrf`) to set Flask's `SECRET_KEY` and +`CSRF_SESSION_KEY`. We generate these files in the **entrypoint** (runtime) rather than the +**Dockerfile** (build time) because: + +- **Security**: Build-time files are baked into image layers and visible via `docker history`. +- **Uniqueness**: Each container gets its own keys instead of sharing from a single image. +- **Dev compatibility**: The `.:/app` volume mount would overwrite build-time files anyway. + +### Why does the container need a git repo? + +`run.py` line 69-70 uses GitPython to read `repo.head.object.hexsha` for build-commit display +in the UI. It crashes at import time if no `.git` directory exists. The `.dockerignore` (correctly) +excludes `.git`, so the entrypoint creates a minimal repo at runtime. + +### Why use `flask db stamp head` instead of `flask db upgrade`? + +On a fresh database, `create_all()` (called by the application startup or `init_db.py`) builds the full schema. If we were to run `flask db upgrade` instead, it might try to create tables but could conflict with `create_all` logic or require a linear migration history that matches the current models exactly. Instead, we let `create_all` or the application manage table creation, and stamp the database as "already at HEAD" so Alembic knows the schema is current. + +### Why a dedicated MySQL user? + +MySQL's `MYSQL_USER` + `MYSQL_PASSWORD` env vars auto-create a user with grants only on +`MYSQL_DATABASE`. The root user is used only for the healthcheck probe. This follows the +principle of least privilege. + +### Why `env_file` without a duplicate `environment:` block? + +Docker Compose's `environment:` section **overrides** `env_file` for any duplicate keys. +Having both is confusing โ€” you think you're editing `.env` but the compose file silently +overrides your changes. Using `env_file` alone keeps `.env` as the single source of truth. + +--- + +## Troubleshooting + +### Container exits immediately + +```bash +docker logs sample_platform_backend +``` + +| Symptom | Cause & Fix | +| ----------------------------------------- | ------------------------------------------------------------------------------ | +| `SecretKeyInstallationException` | `/app/secret_key` or `/app/secret_csrf` missing. Entrypoint should create them. Rebuild image. | +| `git.exc.InvalidGitRepositoryError` | `.git` directory missing. Entrypoint should init one. Rebuild image. | +| `No module named 'config'` | `config.py` not in build context. Check `.dockerignore`. | +| `MySQL connection timeout` | MySQL isn't ready. Increase `retries` in healthcheck or `sleep` in entrypoint. | +| `ModuleNotFoundError: No module named 'X'` | Missing pip dependency. Add to `requirements.txt` and rebuild. | +| `OperationalError: (1045, "Access denied")`| `SQLALCHEMY_DATABASE_URI` user/password doesn't match `MYSQL_USER`/`MYSQL_PASSWORD` in `.env`. | + +### Port conflict + +If port 5000 or 3306 is already in use, change in `.env`: + +```env +APP_PORT=8080 +DB_EXTERNAL_PORT=3307 +``` + +### Database migration errors on fresh DB + +The entrypoint stamps the database at HEAD on first run to avoid conflicts between +`create_all()` and Alembic. If you still see errors, do a full reset: + +```bash +docker compose down -v +docker compose up -d --build +``` + +### Windows line-ending issues + +If you see `/bin/bash^M: bad interpreter`, the entrypoint has Windows-style line endings. The +Dockerfile runs `sed -i 's/\r$//'` to fix this automatically. If you've volume-mounted and +edited the file on Windows, convert manually: + +```bash +# Git Bash +sed -i 's/\r$//' docker-entrypoint.sh + +# Or configure git globally +git config core.autocrlf input +``` + +--- + +## File Overview + +``` +. +โ”œโ”€โ”€ .dockerignore # Files excluded from Docker build context +โ”œโ”€โ”€ .env # Your local config (git-ignored, single source of truth) +โ”œโ”€โ”€ Dockerfile # Image: Python 3.11 + system deps + pip install +โ”œโ”€โ”€ DOCKER.md # โ† You are here +โ”œโ”€โ”€ docker-compose.yml # Orchestration (db + backend), reads .env +โ”œโ”€โ”€ docker-entrypoint.sh # Runtime: secrets โ†’ git โ†’ dirs โ†’ DB wait โ†’ migrate โ†’ gunicorn +โ”œโ”€โ”€ env.example # Template for .env (committed to git) +โ”œโ”€โ”€ config.py # Flask config (reads env vars, no hardcoded project values) +โ”œโ”€โ”€ utility.py # Helper functions (GCS download fallback, etc.) +โ”œโ”€โ”€ requirements.txt # Python dependencies +โ”œโ”€โ”€ run.py # Flask application entry point +โ””โ”€โ”€ service-account.json # GCP key (If missing, Docker might create a directory here; entrypoint handles this) +``` diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 00000000..5c239558 --- /dev/null +++ b/Dockerfile @@ -0,0 +1,70 @@ +# Use Python 3.11-slim for compatibility with modern dependencies +FROM python:3.11-slim-bullseye + +# Environment variables to optimize Python for Docker +ENV PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 \ + DEBIAN_FRONTEND=noninteractive \ + FLASK_APP=run.py + +# 1. Install System Dependencies +RUN apt-get update && apt-get install -y \ + build-essential \ + pkg-config \ + default-libmysqlclient-dev \ + default-mysql-client \ + libxml2-dev \ + libxslt-dev \ + libmagic1 \ + mediainfo \ + git \ + lsb-release \ + curl \ + netcat-openbsd \ + gnupg2 \ + fuse \ + && export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s` \ + && echo "deb [signed-by=/usr/share/keyrings/cloud.google.asc] https://packages.cloud.google.com/apt $GCSFUSE_REPO main" | tee /etc/apt/sources.list.d/gcsfuse.list \ + && curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | tee /usr/share/keyrings/cloud.google.asc \ + && apt-get update \ + && apt-get install -y gcsfuse \ + && rm -rf /var/lib/apt/lists/* + +# 2. Setup Workspace +WORKDIR /app + +# 3. Upgrade Pip & Build Tools +RUN pip install --upgrade pip wheel setuptools + +# 4. Install heavy C-extension packages first (Caching Layer) +RUN pip install --no-cache-dir mysqlclient lxml cryptography + +# 5. Install Project Dependencies +COPY requirements.txt . +RUN pip install --no-cache-dir --default-timeout=100 -r requirements.txt + +# 6. Install gunicorn (not in requirements.txt) +RUN pip install --no-cache-dir gunicorn + +# 7. Copy Application Code +COPY . . + +# 8. Create logs directory (the only build-time prep needed) +RUN mkdir -p logs + +# NOTE: Secret keys and git repo are created at RUNTIME in docker-entrypoint.sh, +# NOT here. Baking them into the image would: +# - Expose secrets in image layers (docker history) +# - Share the same keys across all containers from this image +# - Conflict with the dev volume mount (.:/app) + +# 9. Setup Entrypoint Script +COPY docker-entrypoint.sh /usr/local/bin/ +RUN sed -i 's/\r$//' /usr/local/bin/docker-entrypoint.sh && \ + chmod +x /usr/local/bin/docker-entrypoint.sh + +# 10. Expose the Flask Port +EXPOSE 5000 + +# 11. Define the runtime command +ENTRYPOINT ["docker-entrypoint.sh"] \ No newline at end of file diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 00000000..febd0309 --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,63 @@ +services: + # --- 1. Database Service --- + db: + image: mysql:8.0 + container_name: sample_platform_db + environment: + MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD} + MYSQL_DATABASE: ${MYSQL_DATABASE} + MYSQL_USER: ${MYSQL_USER} + MYSQL_PASSWORD: ${MYSQL_PASSWORD} + MYSQL_CHARSET: utf8mb4 + MYSQL_COLLATION: utf8mb4_unicode_ci + ports: + - "${DB_EXTERNAL_PORT:-3306}:3306" + volumes: + - db_data:/var/lib/mysql + command: --default-authentication-plugin=mysql_native_password --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci + healthcheck: + test: [ "CMD", "mysqladmin", "ping", "-h", "localhost", "-u", "root", "-p${MYSQL_ROOT_PASSWORD}" ] + interval: 5s + timeout: 5s + retries: 20 + networks: + - sample_platform_network + + # --- 2. Backend Service (Flask) --- + backend: + build: . + container_name: sample_platform_backend + ports: + - "${APP_PORT:-5000}:5000" + depends_on: + db: + condition: service_healthy + volumes: + # Live-reload: mount source code for development + - .:/app + # Prevent host pollution from Python caches + - /app/__pycache__ + - /app/logs + # Mount the Service Account Key (read-only) + - ./service-account.json:/app/service-account.json:ro + # Persistent storage for sample files + - repository_data:/repository + env_file: + - .env + networks: + - sample_platform_network + restart: unless-stopped + cap_add: + - SYS_ADMIN + devices: + - /dev/fuse + +networks: + sample_platform_network: + driver: bridge + +volumes: + db_data: + driver: local + repository_data: + driver: local diff --git a/docker-entrypoint.sh b/docker-entrypoint.sh new file mode 100644 index 00000000..f8d05fea --- /dev/null +++ b/docker-entrypoint.sh @@ -0,0 +1,289 @@ +#!/bin/bash +set -e + +# Professional Logging Function +log() { + echo -e "\033[1;34m[Platform]\033[0m $1" +} + +# --- 1. Ensure Secret Key Files Exist --- + +if [ ! -f "/app/secret_key" ]; then + log "Generating secret_key file..." + head -c 24 /dev/urandom > /app/secret_key +fi +if [ ! -f "/app/secret_csrf" ]; then + log "Generating secret_csrf file..." + head -c 24 /dev/urandom > /app/secret_csrf +fi + +# --- 2. Ensure Git Repo Exists --- +if [ ! -d "/app/.git" ]; then + log "Initializing git repository (required by GitPython for build commit display)..." + git init /app > /dev/null 2>&1 + git -C /app config user.email "docker@sample-platform.local" + git -C /app config user.name "Docker" + git -C /app add -A > /dev/null 2>&1 + git -C /app commit -m "Docker build" --allow-empty > /dev/null 2>&1 +fi + +# --- 3. Ensure GCP Service Account File Exists --- +SA_PATH="/app/service-account.json" +REAL_SA_PATH="$SA_PATH" + +# Docker mounts a directory if the host file doesn't exist. +if [ -d "$SA_PATH" ]; then + log "WARNING: $SA_PATH is a directory (likely because ./service-account.json is missing on host)." + log "Using internal path for generated credentials..." + REAL_SA_PATH="/app/generated-service-account.json" + export GOOGLE_APPLICATION_CREDENTIALS="$REAL_SA_PATH" + export SERVICE_ACCOUNT_FILE="generated-service-account.json" +fi + +if [ ! -f "$REAL_SA_PATH" ]; then + log "Generating dummy service-account.json at $REAL_SA_PATH (GCS will use local fallback)..." + python3 -c " +import json +from cryptography.hazmat.primitives.asymmetric import rsa +from cryptography.hazmat.primitives import serialization + +try: + key = rsa.generate_private_key(public_exponent=65537, key_size=2048) + pem = key.private_bytes(serialization.Encoding.PEM, + serialization.PrivateFormat.TraditionalOpenSSL, + serialization.NoEncryption()).decode() +except Exception as e: + print(f'WARNING: Key generation failed: {e}') + pem = 'DUMMY_KEY' + +sa = { + 'type': 'service_account', + 'project_id': 'docker-dev', + 'private_key_id': 'docker-dev-key', + 'private_key': pem, + 'client_email': 'docker-dev@docker-dev.iam.gserviceaccount.com', + 'client_id': '000000000000', + 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', + 'token_uri': 'https://oauth2.googleapis.com/token', +} +with open('$REAL_SA_PATH', 'w') as f: + json.dump(sa, f, indent=2) +" +fi + +# Ensure logs directory exists (critical for gunicorn) +mkdir -p logs + +# --- 4. Configure & Mount Storage --- +# Determine where the repository is located. +# If GCS_BUCKET_NAME is set, we mount it to a clean path and use that. +# Otherwise, we use the default volume mount at /repository. + +if [ -n "$GCS_BUCKET_NAME" ] && [ -f "$REAL_SA_PATH" ]; then + log "GCS_BUCKET_NAME is set to '$GCS_BUCKET_NAME'. Configuring GCS mount..." + + # Use a separate mount point to avoid conflict with local volume at /repository + GCS_MOUNT_POINT="/mnt/gcs_repository" + mkdir -p "$GCS_MOUNT_POINT" + + log "Mounting '$GCS_BUCKET_NAME' to '$GCS_MOUNT_POINT'..." + set +e + gcsfuse --key-file "$REAL_SA_PATH" \ + --implicit-dirs \ + --debug_gcs \ + --debug_fuse \ + --log-file /tmp/gcsfuse_debug.log \ + --log-format text \ + "$GCS_BUCKET_NAME" "$GCS_MOUNT_POINT" > /tmp/gcsfuse.log 2>&1 + MOUNT_STATUS=$? + set -e + + if [ $MOUNT_STATUS -eq 0 ]; then + log "SUCCESS: GCS bucket mounted at $GCS_MOUNT_POINT" + export SAMPLE_REPOSITORY="$GCS_MOUNT_POINT" + else + log "CRITICAL ERROR: Failed to mount GCS bucket." + log "--- gcsfuse stderr ---" + cat /tmp/gcsfuse.log + log "----------------------" + if [ -f "/tmp/gcsfuse_debug.log" ]; then + log "--- gcsfuse debug log (last 20 lines) ---" + tail -n 20 /tmp/gcsfuse_debug.log + log "----------------------" + fi + exit 1 + fi +else + log "GCS not configured. Using local storage." + # Default to /repository if not set + export SAMPLE_REPOSITORY="${SAMPLE_REPOSITORY:-/repository}" +fi + +# --- 5. Setup Repository Structure --- +REPO="$SAMPLE_REPOSITORY" +log "Ensuring repository structure exists in: $REPO" + +mkdir -p "${REPO}/ci-tests" +mkdir -p "${REPO}/unsafe-ccextractor" +mkdir -p "${REPO}/TempFiles" +mkdir -p "${REPO}/LogFiles" +mkdir -p "${REPO}/TestResults" +mkdir -p "${REPO}/TestFiles" +mkdir -p "${REPO}/TestFiles/media" +mkdir -p "${REPO}/QueuedFiles" +mkdir -p "${REPO}/TestData/ci-linux" +mkdir -p "${REPO}/TestData/ci-windows" +mkdir -p "${REPO}/vm_data" + +# --- 6. Install Initial Data (if requested) --- +if [ "${INSTALL_SAMPLE_DATA}" = "true" ]; then + if [ -d "/app/install/sample_files" ]; then + log "Copying sample files to ${REPO}/TestFiles/..." + # Copy without overwriting if unnecessary, but -n might be safer? + # Using -rn to not overwrite existing + cp -rn /app/install/sample_files/* "${REPO}/TestFiles/" 2>/dev/null || true + fi +fi + +if [ -d "/app/install/ci-vm/ci-windows/ci" ]; then + cp -rn /app/install/ci-vm/ci-windows/ci/* "${REPO}/TestData/ci-windows/" 2>/dev/null || true +fi +if [ -d "/app/install/ci-vm/ci-linux/ci" ]; then + cp -rn /app/install/ci-vm/ci-linux/ci/* "${REPO}/TestData/ci-linux/" 2>/dev/null || true +fi + +# --- 7. Wait for Database Service --- +log "Waiting for MySQL at ${DB_HOST:-db}:${DB_PORT:-3306}..." +timeout=60 +counter=0 +DB_HOST="${DB_HOST:-db}" +DB_PORT="${DB_PORT:-3306}" +while ! nc -z "$DB_HOST" "$DB_PORT"; do + sleep 1 + counter=$((counter + 1)) + if [ $counter -ge $timeout ]; then + log "ERROR: MySQL connection timeout after ${timeout} seconds" + exit 1 + fi +done +log "MySQL is up and reachable." + +# Give MySQL extra time to finish initialization +sleep 3 + +# --- 8. Database Schema Setup --- +# Two schema mechanisms exist in this codebase: +# a) database.py โ†’ create_session() calls Base.metadata.create_all() when the app is imported +# b) Flask-Migrate (Alembic) for versioned migrations +# +# Strategy: +# - Fresh DB: Let create_all() build schema, then stamp as HEAD so Alembic doesn't re-apply +# - Existing DB: Run migrate + upgrade for any new changes + +ALEMBIC_EXISTS=$(python3 -c " +import pymysql, os +try: + conn = pymysql.connect(host='${DB_HOST}', port=${DB_PORT}, + user='${MYSQL_USER:-root}', password='${MYSQL_ROOT_PASSWORD:-root}', + database='${MYSQL_DATABASE:-sample_platform}') + cursor = conn.cursor() + cursor.execute(\"SHOW TABLES LIKE 'alembic_version'\") + result = cursor.fetchone() + conn.close() + print('yes' if result else 'no') +except Exception: + print('no') +" 2>/dev/null) + +if [ "$ALEMBIC_EXISTS" = "no" ]; then + log "Fresh database detected. Setting up schema..." + + # Ensure migrations directory is properly initialized + if [ ! -d "migrations/versions" ]; then + log "Initializing fresh migrations folder..." + rm -rf migrations + flask db init || { + log "ERROR: Failed to initialize migrations" + exit 1 + } + fi + + # Import the app (triggers create_all via create_session), then stamp HEAD. + log "Creating tables and stamping migration head..." + flask db stamp head || { + log "ERROR: Could not stamp migration head. Cannot proceed." + exit 1 + } + + flask db migrate -m "Docker auto-migration" 2>/dev/null || log "No new migrations needed" + flask db upgrade 2>/dev/null || log "No upgrades needed" +else + log "Existing database detected. Applying any pending migrations..." + + if [ ! -d "migrations/versions" ]; then + rm -rf migrations + flask db init || { + log "ERROR: Failed to initialize migrations" + exit 1 + } + flask db stamp head || { + log "ERROR: Could not stamp migration head." + exit 1 + } + fi + + flask db migrate -m "Docker auto-migration" 2>/dev/null || log "No new migrations detected" + flask db upgrade 2>/dev/null || log "No upgrades to apply" +fi + +log "Database schema is ready." + +# --- 9. Initialize Admin User and Sample Data --- +ADMIN_EXISTS=$(python3 -c " +import pymysql, os +try: + conn = pymysql.connect(host='${DB_HOST}', port=${DB_PORT}, + user='${MYSQL_USER:-root}', password='${MYSQL_ROOT_PASSWORD:-root}', + database='${MYSQL_DATABASE:-sample_platform}') + cursor = conn.cursor() + cursor.execute(\"SELECT COUNT(*) FROM user WHERE role = 'admin'\") + result = cursor.fetchone() + conn.close() + print('yes' if result and result[0] > 0 else 'no') +except Exception: + print('no') +" 2>/dev/null) + +if [ -f "install/init_db.py" ]; then + if [ "$ADMIN_EXISTS" = "no" ]; then + log "Creating Admin User..." + + python3 install/init_db.py \ + "$SQLALCHEMY_DATABASE_URI" \ + "${ADMIN_USERNAME:-admin}" \ + "${ADMIN_EMAIL:-admin@example.com}" \ + "${ADMIN_PASSWORD:-admin}" || log "Admin creation skipped (may already exist)" + + if [ "$INSTALL_SAMPLE_DATA" = "true" ] && [ -f "install/sample_db.py" ]; then + log "Populating sample data..." + python3 install/sample_db.py "$SQLALCHEMY_DATABASE_URI" || log "Sample data population skipped" + fi + else + log "Admin user already exists โ€” skipping initialization" + fi +else + log "WARNING: install/init_db.py not found โ€” skipping admin user creation" +fi + +# --- 10. Start Server --- +log "Starting Gunicorn on 0.0.0.0:5000..." +log "Application accessible at http://localhost:${APP_PORT:-5000}" + +exec gunicorn \ + --workers 4 \ + --bind 0.0.0.0:5000 \ + --timeout 120 \ + --access-logfile - \ + --error-logfile - \ + --log-level info \ + run:app \ No newline at end of file diff --git a/env.example b/env.example new file mode 100644 index 00000000..25faf19c --- /dev/null +++ b/env.example @@ -0,0 +1,63 @@ +# ============================================================ +# Sample Platform โ€“ Docker Environment Variables +# Copy env.example โ†’ .env, then edit the values below. +# ============================================================ + +# ---------- MySQL ---------- +MYSQL_ROOT_PASSWORD=root +# Application-level DB user (MySQL auto-creates this with access to MYSQL_DATABASE) +MYSQL_USER=sample_platform +MYSQL_PASSWORD=sample_platform +MYSQL_DATABASE=sample_platform + +# ---------- Networking ---------- +# Port exposed on the HOST for the Flask app (container always listens on 5000) +APP_PORT=5000 +# Port exposed on the HOST for direct MySQL access (optional, for debugging) +DB_EXTERNAL_PORT=3306 + +# ---------- Flask ---------- +FLASK_APP=run.py +FLASK_ENV=development + +# ---------- Database URI ---------- +# Constructed from the MySQL vars above. Uses the container hostname 'db'. +SQLALCHEMY_DATABASE_URI=mysql+pymysql://sample_platform:sample_platform@db/sample_platform?charset=utf8mb4 + +# ---------- Internal Container Paths ---------- +DB_HOST=db +DB_PORT=3306 +SAMPLE_REPOSITORY=/repository +INSTALL_FOLDER=/app +GOOGLE_APPLICATION_CREDENTIALS=/app/service-account.json + +# ---------- Security ---------- +SECRET_KEY=change-me-in-production +HMAC_KEY=change-me-in-production + +# ---------- Google Cloud Storage (Optional for Docker) ---------- +# For local Docker dev: leave as-is. The entrypoint auto-generates a dummy +# service-account.json, and downloads fall back to local file serving. +# For production: set your real bucket name. +# NOTE: Must be non-empty โ€” run.py crashes on empty bucket name at import time. +GCS_BUCKET_NAME=sample-platform-dev + +# ---------- Admin Bootstrap ---------- +# Created automatically on first run +ADMIN_USERNAME=admin +ADMIN_EMAIL=admin@example.com +ADMIN_PASSWORD=admin + +# ---------- GitHub Integration (Optional) ---------- +GITHUB_TOKEN= +GITHUB_OWNER=CCExtractor +GITHUB_REPOSITORY=ccextractor + +# ---------- Email / Mailgun (Optional) ---------- +EMAIL_DOMAIN= +EMAIL_API_KEY= + +# ---------- Feature Flags ---------- +# Populate the DB with sample categories, samples, and regression tests +INSTALL_SAMPLE_DATA=true +MAINTENANCE=false diff --git a/utility.py b/utility.py index 96308e41..b0a58cab 100644 --- a/utility.py +++ b/utility.py @@ -10,7 +10,7 @@ import requests import werkzeug -from flask import abort, g, redirect, request +from flask import abort, g, redirect, request, send_file ROOT_DIR = path.dirname(path.abspath(__file__)) @@ -19,6 +19,10 @@ def serve_file_download(file_name, file_folder, file_sub_folder='') -> werkzeug. """ Serve file download by redirecting using Signed Download URLs. + Falls back to serving from local filesystem if GCS returns NotFound. + This enables Docker development environments where /repository is a plain + volume rather than a gcsfuse mount backed by GCS. + :param file_name: name of the file :type file_name: str :param file_folder: name of the folder @@ -31,15 +35,25 @@ def serve_file_download(file_name, file_folder, file_sub_folder='') -> werkzeug. from run import config, storage_client_bucket file_path = path.join(file_folder, file_sub_folder, file_name) - blob = storage_client_bucket.blob(file_path) - blob.content_disposition = f'attachment; filename="{file_name}"' - blob.patch() - url = blob.generate_signed_url( - version="v4", - expiration=timedelta(minutes=config.get('GCS_SIGNED_URL_EXPIRY_LIMIT', '')), - method="GET", - ) - return redirect(url) + + # Try GCS first (production path โ€” /repository is a gcsfuse mount) + try: + blob = storage_client_bucket.blob(file_path) + blob.content_disposition = f'attachment; filename="{file_name}"' + blob.patch() + url = blob.generate_signed_url( + version="v4", + expiration=timedelta(minutes=config.get('GCS_SIGNED_URL_EXPIRY_LIMIT', '')), + method="GET", + ) + return redirect(url) + except Exception: + # GCS failed โ€” fall back to local file serving (Docker dev environment) + local_path = path.join(config.get('SAMPLE_REPOSITORY', ''), file_path) + if path.isfile(local_path): + return send_file(local_path, as_attachment=True, download_name=file_name) + # File doesn't exist locally either โ€” re-raise + raise def request_from_github(abort_code: int = 418) -> Callable: From 31c63d942d7b424995342d5da3f8ad2657d1f3c7 Mon Sep 17 00:00:00 2001 From: Pulkit Chauhan Date: Mon, 16 Feb 2026 22:53:07 +0530 Subject: [PATCH 02/11] Dockerfile --- Dockerfile | 9 +-------- PR_CONTENT.md | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+), 8 deletions(-) create mode 100644 PR_CONTENT.md diff --git a/Dockerfile b/Dockerfile index 5c239558..f88554cc 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,4 +1,3 @@ -# Use Python 3.11-slim for compatibility with modern dependencies FROM python:3.11-slim-bullseye # Environment variables to optimize Python for Docker @@ -43,7 +42,7 @@ RUN pip install --no-cache-dir mysqlclient lxml cryptography COPY requirements.txt . RUN pip install --no-cache-dir --default-timeout=100 -r requirements.txt -# 6. Install gunicorn (not in requirements.txt) +# 6. Install gunicorn RUN pip install --no-cache-dir gunicorn # 7. Copy Application Code @@ -52,12 +51,6 @@ COPY . . # 8. Create logs directory (the only build-time prep needed) RUN mkdir -p logs -# NOTE: Secret keys and git repo are created at RUNTIME in docker-entrypoint.sh, -# NOT here. Baking them into the image would: -# - Expose secrets in image layers (docker history) -# - Share the same keys across all containers from this image -# - Conflict with the dev volume mount (.:/app) - # 9. Setup Entrypoint Script COPY docker-entrypoint.sh /usr/local/bin/ RUN sed -i 's/\r$//' /usr/local/bin/docker-entrypoint.sh && \ diff --git a/PR_CONTENT.md b/PR_CONTENT.md new file mode 100644 index 00000000..fafe5fc4 --- /dev/null +++ b/PR_CONTENT.md @@ -0,0 +1,56 @@ +# Docker Development Environment Setup + +## ๐Ÿš€ Summary +This PR introduces a complete Docker-based development environment for the Sample Platform. It allows developers to spin up the entire application stack (Flask + MySQL) with a **single command**, eliminating manual dependency installation and configuration headaches. + +## โœจ Key Features +- **One-Command Setup**: `docker compose up --build` handles everything from database creation to dependency installation. +- **Live Code Reloading**: The local source code is mounted into the container, so changes are reflected immediately without rebuilding. +- **Automated Database Management**: + - Automatically waits for MySQL to be healthy. + - Handles schema creation and migration stamping (`flask db stamp head`) on the first run. + - Seeds the database with an admin user and sample data (configurable via `.env`). +- **Google Cloud Storage (GCS) Emulation**: + - Automatically generates a dummy `service-account.json` if one is missing, preventing startup crashes. + - Supports mounting a real GCS bucket via `gcsfuse` if credentials are provided. +- **Security Best Practices**: + - `SECRET_KEY` and `CSRF_SESSION_KEY` are generated **at runtime** (not baked into the image) to ensure unique secrets per container. + - Runs with a non-root user (Gunicorn). + +## ๐Ÿ“‚ File Overview +- `Dockerfile`: Multi-stage build based on Python 3.11-slim. +- `docker-compose.yml`: Orchestrates the Flask backend and MySQL 8.0 database. +- `docker-entrypoint.sh`: A robust startup script that handles: + - Secret key generation. + - Git repository initialization (required for build commit display). + - Database waiting and migration strategies. + - Gunicorn execution. +- `DOCKER.md`: Comprehensive documentation on usage, architecture, and troubleshooting. +- `env.example`: A template for environment variables tailored for Docker. +- `.dockerignore`: Optimizes build context by excluding unnecessary files. + +## ๐Ÿงช How to Test +1. **Checkout the branch**: + ```bash + git checkout feature/Docker + ``` +2. **Setup Environment**: + ```bash + cp env.example .env + ``` +3. **Start the Platform**: + ```bash + docker compose up --build + ``` +4. **Verify**: + - Access the app at [http://localhost:5000](http://localhost:5000). + - Log in with default credentials (`admin@example.com` / `admin`). + - Standard logs should appear in your terminal. + +## โš ๏ธ Design Decisions & Trade-offs +- **Runtime Secrets**: We generate secrets in the entrypoint instead of the Dockerfile to prevent leaking them in image layers and to ensure every container has unique keys. +- **GCS Mounting**: We mount GCS buckets to `/mnt/gcs_repository` instead of directly to `/repository` to avoid conflicts with Docker's volume mounting behavior. +- **Database Stamping**: On a fresh DB, we use `flask db stamp head` because `create_all()` builds the schema faster than running 50+ migrations sequentially. + +--- +**Documentation**: See `DOCKER.md` for full details. From 4b5ef3e750868035bffedfe394caf4dcd7e2a1a8 Mon Sep 17 00:00:00 2001 From: Pulkit Chauhan Date: Mon, 16 Feb 2026 22:58:36 +0530 Subject: [PATCH 03/11] Deleted useless file --- PR_CONTENT.md | 56 --------------------------------------------------- 1 file changed, 56 deletions(-) delete mode 100644 PR_CONTENT.md diff --git a/PR_CONTENT.md b/PR_CONTENT.md deleted file mode 100644 index fafe5fc4..00000000 --- a/PR_CONTENT.md +++ /dev/null @@ -1,56 +0,0 @@ -# Docker Development Environment Setup - -## ๐Ÿš€ Summary -This PR introduces a complete Docker-based development environment for the Sample Platform. It allows developers to spin up the entire application stack (Flask + MySQL) with a **single command**, eliminating manual dependency installation and configuration headaches. - -## โœจ Key Features -- **One-Command Setup**: `docker compose up --build` handles everything from database creation to dependency installation. -- **Live Code Reloading**: The local source code is mounted into the container, so changes are reflected immediately without rebuilding. -- **Automated Database Management**: - - Automatically waits for MySQL to be healthy. - - Handles schema creation and migration stamping (`flask db stamp head`) on the first run. - - Seeds the database with an admin user and sample data (configurable via `.env`). -- **Google Cloud Storage (GCS) Emulation**: - - Automatically generates a dummy `service-account.json` if one is missing, preventing startup crashes. - - Supports mounting a real GCS bucket via `gcsfuse` if credentials are provided. -- **Security Best Practices**: - - `SECRET_KEY` and `CSRF_SESSION_KEY` are generated **at runtime** (not baked into the image) to ensure unique secrets per container. - - Runs with a non-root user (Gunicorn). - -## ๐Ÿ“‚ File Overview -- `Dockerfile`: Multi-stage build based on Python 3.11-slim. -- `docker-compose.yml`: Orchestrates the Flask backend and MySQL 8.0 database. -- `docker-entrypoint.sh`: A robust startup script that handles: - - Secret key generation. - - Git repository initialization (required for build commit display). - - Database waiting and migration strategies. - - Gunicorn execution. -- `DOCKER.md`: Comprehensive documentation on usage, architecture, and troubleshooting. -- `env.example`: A template for environment variables tailored for Docker. -- `.dockerignore`: Optimizes build context by excluding unnecessary files. - -## ๐Ÿงช How to Test -1. **Checkout the branch**: - ```bash - git checkout feature/Docker - ``` -2. **Setup Environment**: - ```bash - cp env.example .env - ``` -3. **Start the Platform**: - ```bash - docker compose up --build - ``` -4. **Verify**: - - Access the app at [http://localhost:5000](http://localhost:5000). - - Log in with default credentials (`admin@example.com` / `admin`). - - Standard logs should appear in your terminal. - -## โš ๏ธ Design Decisions & Trade-offs -- **Runtime Secrets**: We generate secrets in the entrypoint instead of the Dockerfile to prevent leaking them in image layers and to ensure every container has unique keys. -- **GCS Mounting**: We mount GCS buckets to `/mnt/gcs_repository` instead of directly to `/repository` to avoid conflicts with Docker's volume mounting behavior. -- **Database Stamping**: On a fresh DB, we use `flask db stamp head` because `create_all()` builds the schema faster than running 50+ migrations sequentially. - ---- -**Documentation**: See `DOCKER.md` for full details. From 4633359552f5dfd1fbc12c830f493d7c38de69bc Mon Sep 17 00:00:00 2001 From: Pulkit Chauhan Date: Thu, 19 Feb 2026 15:30:23 +0530 Subject: [PATCH 04/11] Fixed security issues --- .dockerignore | 10 +++++++++- Dockerfile | 36 +++++++++++++++++------------------- docker-entrypoint.sh | 42 ++++++++++++++++++++++-------------------- 3 files changed, 48 insertions(+), 40 deletions(-) diff --git a/.dockerignore b/.dockerignore index d5477c63..3fe5b51c 100644 --- a/.dockerignore +++ b/.dockerignore @@ -58,4 +58,12 @@ htmlcov/ # Temporary files tmp/ temp/ -*.tmp \ No newline at end of file +*.tmp + +# Secrets and credentials (prevent accidental inclusion) +*.pem +*.key +service-account.json +gcp-key.json +secret_key +secret_csrf \ No newline at end of file diff --git a/Dockerfile b/Dockerfile index f88554cc..7661d202 100644 --- a/Dockerfile +++ b/Dockerfile @@ -32,32 +32,30 @@ RUN apt-get update && apt-get install -y \ # 2. Setup Workspace WORKDIR /app -# 3. Upgrade Pip & Build Tools -RUN pip install --upgrade pip wheel setuptools - -# 4. Install heavy C-extension packages first (Caching Layer) -RUN pip install --no-cache-dir mysqlclient lxml cryptography - -# 5. Install Project Dependencies +# 3. Install all Python dependencies in a single layer COPY requirements.txt . -RUN pip install --no-cache-dir --default-timeout=100 -r requirements.txt - -# 6. Install gunicorn -RUN pip install --no-cache-dir gunicorn +RUN pip install --no-cache-dir --upgrade pip wheel setuptools && \ + pip install --no-cache-dir mysqlclient lxml cryptography && \ + pip install --no-cache-dir --default-timeout=100 -r requirements.txt && \ + pip install --no-cache-dir gunicorn -# 7. Copy Application Code +# 4. Copy Application Code COPY . . -# 8. Create logs directory (the only build-time prep needed) -RUN mkdir -p logs - -# 9. Setup Entrypoint Script +# 5. Create logs directory & setup entrypoint COPY docker-entrypoint.sh /usr/local/bin/ -RUN sed -i 's/\r$//' /usr/local/bin/docker-entrypoint.sh && \ +RUN mkdir -p logs && \ + sed -i 's/\r$//' /usr/local/bin/docker-entrypoint.sh && \ chmod +x /usr/local/bin/docker-entrypoint.sh -# 10. Expose the Flask Port +# 6. Create a non-root user and set ownership +RUN groupadd --gid 1001 appuser && \ + useradd --uid 1001 --gid appuser --shell /bin/bash --create-home appuser && \ + chown -R appuser:appuser /app +USER appuser + +# 7. Expose the Flask Port EXPOSE 5000 -# 11. Define the runtime command +# 8. Define the runtime command ENTRYPOINT ["docker-entrypoint.sh"] \ No newline at end of file diff --git a/docker-entrypoint.sh b/docker-entrypoint.sh index f8d05fea..7f7bb36e 100644 --- a/docker-entrypoint.sh +++ b/docker-entrypoint.sh @@ -3,22 +3,24 @@ set -e # Professional Logging Function log() { - echo -e "\033[1;34m[Platform]\033[0m $1" + local message="$1" + echo -e "\033[1;34m[Platform]\033[0m ${message}" + return 0 } # --- 1. Ensure Secret Key Files Exist --- -if [ ! -f "/app/secret_key" ]; then +if [[ ! -f "/app/secret_key" ]]; then log "Generating secret_key file..." head -c 24 /dev/urandom > /app/secret_key fi -if [ ! -f "/app/secret_csrf" ]; then +if [[ ! -f "/app/secret_csrf" ]]; then log "Generating secret_csrf file..." head -c 24 /dev/urandom > /app/secret_csrf fi # --- 2. Ensure Git Repo Exists --- -if [ ! -d "/app/.git" ]; then +if [[ ! -d "/app/.git" ]]; then log "Initializing git repository (required by GitPython for build commit display)..." git init /app > /dev/null 2>&1 git -C /app config user.email "docker@sample-platform.local" @@ -32,7 +34,7 @@ SA_PATH="/app/service-account.json" REAL_SA_PATH="$SA_PATH" # Docker mounts a directory if the host file doesn't exist. -if [ -d "$SA_PATH" ]; then +if [[ -d "$SA_PATH" ]]; then log "WARNING: $SA_PATH is a directory (likely because ./service-account.json is missing on host)." log "Using internal path for generated credentials..." REAL_SA_PATH="/app/generated-service-account.json" @@ -40,7 +42,7 @@ if [ -d "$SA_PATH" ]; then export SERVICE_ACCOUNT_FILE="generated-service-account.json" fi -if [ ! -f "$REAL_SA_PATH" ]; then +if [[ ! -f "$REAL_SA_PATH" ]]; then log "Generating dummy service-account.json at $REAL_SA_PATH (GCS will use local fallback)..." python3 -c " import json @@ -79,7 +81,7 @@ mkdir -p logs # If GCS_BUCKET_NAME is set, we mount it to a clean path and use that. # Otherwise, we use the default volume mount at /repository. -if [ -n "$GCS_BUCKET_NAME" ] && [ -f "$REAL_SA_PATH" ]; then +if [[ -n "$GCS_BUCKET_NAME" ]] && [[ -f "$REAL_SA_PATH" ]]; then log "GCS_BUCKET_NAME is set to '$GCS_BUCKET_NAME'. Configuring GCS mount..." # Use a separate mount point to avoid conflict with local volume at /repository @@ -98,7 +100,7 @@ if [ -n "$GCS_BUCKET_NAME" ] && [ -f "$REAL_SA_PATH" ]; then MOUNT_STATUS=$? set -e - if [ $MOUNT_STATUS -eq 0 ]; then + if [[ $MOUNT_STATUS -eq 0 ]]; then log "SUCCESS: GCS bucket mounted at $GCS_MOUNT_POINT" export SAMPLE_REPOSITORY="$GCS_MOUNT_POINT" else @@ -106,7 +108,7 @@ if [ -n "$GCS_BUCKET_NAME" ] && [ -f "$REAL_SA_PATH" ]; then log "--- gcsfuse stderr ---" cat /tmp/gcsfuse.log log "----------------------" - if [ -f "/tmp/gcsfuse_debug.log" ]; then + if [[ -f "/tmp/gcsfuse_debug.log" ]]; then log "--- gcsfuse debug log (last 20 lines) ---" tail -n 20 /tmp/gcsfuse_debug.log log "----------------------" @@ -136,8 +138,8 @@ mkdir -p "${REPO}/TestData/ci-windows" mkdir -p "${REPO}/vm_data" # --- 6. Install Initial Data (if requested) --- -if [ "${INSTALL_SAMPLE_DATA}" = "true" ]; then - if [ -d "/app/install/sample_files" ]; then +if [[ "${INSTALL_SAMPLE_DATA}" = "true" ]]; then + if [[ -d "/app/install/sample_files" ]]; then log "Copying sample files to ${REPO}/TestFiles/..." # Copy without overwriting if unnecessary, but -n might be safer? # Using -rn to not overwrite existing @@ -145,10 +147,10 @@ if [ "${INSTALL_SAMPLE_DATA}" = "true" ]; then fi fi -if [ -d "/app/install/ci-vm/ci-windows/ci" ]; then +if [[ -d "/app/install/ci-vm/ci-windows/ci" ]]; then cp -rn /app/install/ci-vm/ci-windows/ci/* "${REPO}/TestData/ci-windows/" 2>/dev/null || true fi -if [ -d "/app/install/ci-vm/ci-linux/ci" ]; then +if [[ -d "/app/install/ci-vm/ci-linux/ci" ]]; then cp -rn /app/install/ci-vm/ci-linux/ci/* "${REPO}/TestData/ci-linux/" 2>/dev/null || true fi @@ -161,7 +163,7 @@ DB_PORT="${DB_PORT:-3306}" while ! nc -z "$DB_HOST" "$DB_PORT"; do sleep 1 counter=$((counter + 1)) - if [ $counter -ge $timeout ]; then + if [[ $counter -ge $timeout ]]; then log "ERROR: MySQL connection timeout after ${timeout} seconds" exit 1 fi @@ -195,11 +197,11 @@ except Exception: print('no') " 2>/dev/null) -if [ "$ALEMBIC_EXISTS" = "no" ]; then +if [[ "$ALEMBIC_EXISTS" = "no" ]]; then log "Fresh database detected. Setting up schema..." # Ensure migrations directory is properly initialized - if [ ! -d "migrations/versions" ]; then + if [[ ! -d "migrations/versions" ]]; then log "Initializing fresh migrations folder..." rm -rf migrations flask db init || { @@ -220,7 +222,7 @@ if [ "$ALEMBIC_EXISTS" = "no" ]; then else log "Existing database detected. Applying any pending migrations..." - if [ ! -d "migrations/versions" ]; then + if [[ ! -d "migrations/versions" ]]; then rm -rf migrations flask db init || { log "ERROR: Failed to initialize migrations" @@ -254,8 +256,8 @@ except Exception: print('no') " 2>/dev/null) -if [ -f "install/init_db.py" ]; then - if [ "$ADMIN_EXISTS" = "no" ]; then +if [[ -f "install/init_db.py" ]]; then + if [[ "$ADMIN_EXISTS" = "no" ]]; then log "Creating Admin User..." python3 install/init_db.py \ @@ -264,7 +266,7 @@ if [ -f "install/init_db.py" ]; then "${ADMIN_EMAIL:-admin@example.com}" \ "${ADMIN_PASSWORD:-admin}" || log "Admin creation skipped (may already exist)" - if [ "$INSTALL_SAMPLE_DATA" = "true" ] && [ -f "install/sample_db.py" ]; then + if [[ "$INSTALL_SAMPLE_DATA" = "true" ]] && [[ -f "install/sample_db.py" ]]; then log "Populating sample data..." python3 install/sample_db.py "$SQLALCHEMY_DATABASE_URI" || log "Sample data population skipped" fi From 18f57880a83f10938c27cfdc1e349ec934dc1839 Mon Sep 17 00:00:00 2001 From: Pulkit Chauhan Date: Thu, 19 Feb 2026 15:34:48 +0530 Subject: [PATCH 05/11] Fixed docker copy issue --- Dockerfile | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/Dockerfile b/Dockerfile index 7661d202..7dca73f1 100644 --- a/Dockerfile +++ b/Dockerfile @@ -39,8 +39,24 @@ RUN pip install --no-cache-dir --upgrade pip wheel setuptools && \ pip install --no-cache-dir --default-timeout=100 -r requirements.txt && \ pip install --no-cache-dir gunicorn -# 4. Copy Application Code -COPY . . +# 4. Copy Application Code +COPY run.py manage.py config.py config_parser.py config_sample.py database.py \ + decorators.py exceptions.py log_configuration.py mailer.py utility.py \ + bootstrap_gunicorn.py ./ +COPY mod_auth/ mod_auth/ +COPY mod_ci/ mod_ci/ +COPY mod_customized/ mod_customized/ +COPY mod_health/ mod_health/ +COPY mod_home/ mod_home/ +COPY mod_regression/ mod_regression/ +COPY mod_sample/ mod_sample/ +COPY mod_test/ mod_test/ +COPY mod_upload/ mod_upload/ +COPY templates/ templates/ +COPY static/ static/ +COPY install/ install/ +COPY migrations/ migrations/ +COPY tests/ tests/ # 5. Create logs directory & setup entrypoint COPY docker-entrypoint.sh /usr/local/bin/ From 12f477833204f582b6b4f8934504696a07d4c8b9 Mon Sep 17 00:00:00 2001 From: Pulkit Chauhan Date: Thu, 19 Feb 2026 16:31:48 +0530 Subject: [PATCH 06/11] Comment fixes --- Dockerfile | 7 ++++--- docker-entrypoint.sh | 19 +++++++++++-------- mod_sample/controllers.py | 18 ++++++++++-------- templates/sample/sample_info.html | 26 +++++++++++++++----------- 4 files changed, 40 insertions(+), 30 deletions(-) diff --git a/Dockerfile b/Dockerfile index 7dca73f1..e4e2c1a3 100644 --- a/Dockerfile +++ b/Dockerfile @@ -64,11 +64,12 @@ RUN mkdir -p logs && \ sed -i 's/\r$//' /usr/local/bin/docker-entrypoint.sh && \ chmod +x /usr/local/bin/docker-entrypoint.sh -# 6. Create a non-root user and set ownership -RUN groupadd --gid 1001 appuser && \ +# 6. Create a non-root user for running the application server +RUN apt-get update && apt-get install -y --no-install-recommends gosu && \ + rm -rf /var/lib/apt/lists/* && \ + groupadd --gid 1001 appuser && \ useradd --uid 1001 --gid appuser --shell /bin/bash --create-home appuser && \ chown -R appuser:appuser /app -USER appuser # 7. Expose the Flask Port EXPOSE 5000 diff --git a/docker-entrypoint.sh b/docker-entrypoint.sh index 7f7bb36e..2d7ced1f 100644 --- a/docker-entrypoint.sh +++ b/docker-entrypoint.sh @@ -20,6 +20,7 @@ if [[ ! -f "/app/secret_csrf" ]]; then fi # --- 2. Ensure Git Repo Exists --- +git config --global --add safe.directory /app if [[ ! -d "/app/.git" ]]; then log "Initializing git repository (required by GitPython for build commit display)..." git init /app > /dev/null 2>&1 @@ -92,6 +93,9 @@ if [[ -n "$GCS_BUCKET_NAME" ]] && [[ -f "$REAL_SA_PATH" ]]; then set +e gcsfuse --key-file "$REAL_SA_PATH" \ --implicit-dirs \ + --uid 1001 --gid 1001 \ + --file-mode 666 --dir-mode 777 \ + -o allow_other \ --debug_gcs \ --debug_fuse \ --log-file /tmp/gcsfuse_debug.log \ @@ -137,12 +141,13 @@ mkdir -p "${REPO}/TestData/ci-linux" mkdir -p "${REPO}/TestData/ci-windows" mkdir -p "${REPO}/vm_data" +# Ensure appuser has write access to the repository +chown -R appuser:appuser "$REPO" 2>/dev/null || true + # --- 6. Install Initial Data (if requested) --- if [[ "${INSTALL_SAMPLE_DATA}" = "true" ]]; then if [[ -d "/app/install/sample_files" ]]; then log "Copying sample files to ${REPO}/TestFiles/..." - # Copy without overwriting if unnecessary, but -n might be safer? - # Using -rn to not overwrite existing cp -rn /app/install/sample_files/* "${REPO}/TestFiles/" 2>/dev/null || true fi fi @@ -177,10 +182,6 @@ sleep 3 # Two schema mechanisms exist in this codebase: # a) database.py โ†’ create_session() calls Base.metadata.create_all() when the app is imported # b) Flask-Migrate (Alembic) for versioned migrations -# -# Strategy: -# - Fresh DB: Let create_all() build schema, then stamp as HEAD so Alembic doesn't re-apply -# - Existing DB: Run migrate + upgrade for any new changes ALEMBIC_EXISTS=$(python3 -c " import pymysql, os @@ -278,10 +279,12 @@ else fi # --- 10. Start Server --- -log "Starting Gunicorn on 0.0.0.0:5000..." +chown -R appuser:appuser /app 2>/dev/null || true + +log "Starting Gunicorn on 0.0.0.0:5000 as appuser..." log "Application accessible at http://localhost:${APP_PORT:-5000}" -exec gunicorn \ +exec gosu appuser gunicorn \ --workers 4 \ --bind 0.0.0.0:5000 \ --timeout 120 \ diff --git a/mod_sample/controllers.py b/mod_sample/controllers.py index 5d315f5c..328a4da8 100755 --- a/mod_sample/controllers.py +++ b/mod_sample/controllers.py @@ -297,10 +297,11 @@ def edit_sample(sample_id): if form.validate_on_submit(): # Store values upload = sample.upload - upload.notes = form.notes.data - upload.version_id = form.version.data - upload.platform = Platform.from_string(form.platform.data) - upload.parameters = form.parameters.data + if upload is not None: + upload.notes = form.notes.data + upload.version_id = form.version.data + upload.platform = Platform.from_string(form.platform.data) + upload.parameters = form.parameters.data sample.tags = list(Tag.query.filter(Tag.id.in_(form.tags.data))) g.db.commit() g.log.info(f"sample with id: {sample_id} updated") @@ -308,10 +309,11 @@ def edit_sample(sample_id): if not form.is_submitted(): # Populate form with current set sample values - form.version.data = sample.upload.version.id - form.platform.data = sample.upload.platform.name - form.notes.data = sample.upload.notes - form.parameters.data = sample.upload.parameters + if sample.upload is not None: + form.version.data = sample.upload.version.id if sample.upload.version else None + form.platform.data = sample.upload.platform.name if sample.upload.platform else None + form.notes.data = sample.upload.notes + form.parameters.data = sample.upload.parameters form.tags.data = [tag.id for tag in sample.tags] return { diff --git a/templates/sample/sample_info.html b/templates/sample/sample_info.html index 8e181b30..9801ee84 100644 --- a/templates/sample/sample_info.html +++ b/templates/sample/sample_info.html @@ -26,16 +26,20 @@

Basic details

SHA256-hash: {{ sample.sha }}
Extension: {{ sample.extension }}
- {% if user.is_admin or user.id == sample.upload.user.id %} - Original name: {{ sample.original_name }}
- {% endif %} - {% if user.is_admin %} - Submitted by: {{ sample.upload.user.name }}
+ {% if sample.upload %} + {% if user.is_admin or user.id == sample.upload.user.id %} + Original name: {{ sample.original_name }}
+ {% endif %} + {% if user.is_admin %} + Submitted by: {{ sample.upload.user.name }}
+ {% endif %} + Submitted for CCExtractor version {{ sample.upload.version.version }} (released {{ sample.upload.version.released }})
+ Platform: {{ sample.upload.platform.description }}
+ Parameters: {{ sample.upload.parameters }}
+ Notes: {{ sample.upload.notes }}
+ {% else %} + Upload information: N/A
{% endif %} - Submitted for CCExtractor version {{ sample.upload.version.version }} (released {{ sample.upload.version.released }})
- Platform: {{ sample.upload.platform.description }}
- Parameters: {{ sample.upload.parameters }}
- Notes: {{ sample.upload.notes }}

{% if additional_files|length > 0 %} @@ -44,7 +48,7 @@

Additional files

File - {% if user.is_admin or user.id == sample.upload.user.id %} + {% if sample.upload and (user.is_admin or user.id == sample.upload.user.id) %} Original name {% endif %} Actions @@ -54,7 +58,7 @@

Additional files

{% for file in additional_files %} {{ file.short_name }} - {% if user.is_admin or user.id == sample.upload.user.id %} + {% if sample.upload and (user.is_admin or user.id == sample.upload.user.id) %} {{ file.original_name }} {% endif %} From bbec88ac7991120aa394804d38f7c02e1a4edd86 Mon Sep 17 00:00:00 2001 From: Pulkit Chauhan Date: Thu, 19 Feb 2026 16:38:40 +0530 Subject: [PATCH 07/11] merged the if statements for sonarqube lint --- docker-entrypoint.sh | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/docker-entrypoint.sh b/docker-entrypoint.sh index 2d7ced1f..db1ade29 100644 --- a/docker-entrypoint.sh +++ b/docker-entrypoint.sh @@ -145,11 +145,9 @@ mkdir -p "${REPO}/vm_data" chown -R appuser:appuser "$REPO" 2>/dev/null || true # --- 6. Install Initial Data (if requested) --- -if [[ "${INSTALL_SAMPLE_DATA}" = "true" ]]; then - if [[ -d "/app/install/sample_files" ]]; then - log "Copying sample files to ${REPO}/TestFiles/..." - cp -rn /app/install/sample_files/* "${REPO}/TestFiles/" 2>/dev/null || true - fi +if [[ "${INSTALL_SAMPLE_DATA}" = "true" ]] && [[ -d "/app/install/sample_files" ]]; then + log "Copying sample files to ${REPO}/TestFiles/..." + cp -rn /app/install/sample_files/* "${REPO}/TestFiles/" 2>/dev/null || true fi if [[ -d "/app/install/ci-vm/ci-windows/ci" ]]; then From 62e4f6cc5a5c280f158f2225887a78175078314a Mon Sep 17 00:00:00 2001 From: Pulkit Chauhan Date: Thu, 19 Feb 2026 16:43:38 +0530 Subject: [PATCH 08/11] changed root user --- Dockerfile | 5 ++++- docker-compose.yml | 1 + 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/Dockerfile b/Dockerfile index e4e2c1a3..bfc486de 100644 --- a/Dockerfile +++ b/Dockerfile @@ -71,7 +71,10 @@ RUN apt-get update && apt-get install -y --no-install-recommends gosu && \ useradd --uid 1001 --gid appuser --shell /bin/bash --create-home appuser && \ chown -R appuser:appuser /app -# 7. Expose the Flask Port +# 7. Switch to non-root user +USER appuser + +# 8. Expose the Flask Port EXPOSE 5000 # 8. Define the runtime command diff --git a/docker-compose.yml b/docker-compose.yml index febd0309..348fa958 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -27,6 +27,7 @@ services: backend: build: . container_name: sample_platform_backend + user: root ports: - "${APP_PORT:-5000}:5000" depends_on: From dc8b4998070ea6f8b6fce8fa45264c29785299e9 Mon Sep 17 00:00:00 2001 From: Pulkit Chauhan Date: Thu, 19 Feb 2026 17:01:08 +0530 Subject: [PATCH 09/11] removed the if else fix i provided to fix sample info as it is giving sonarqube errors --- mod_sample/controllers.py | 18 ++- templates/sample/sample_info.html | 217 ++++++++++++++++-------------- 2 files changed, 122 insertions(+), 113 deletions(-) diff --git a/mod_sample/controllers.py b/mod_sample/controllers.py index 328a4da8..6a138625 100755 --- a/mod_sample/controllers.py +++ b/mod_sample/controllers.py @@ -297,11 +297,10 @@ def edit_sample(sample_id): if form.validate_on_submit(): # Store values upload = sample.upload - if upload is not None: - upload.notes = form.notes.data - upload.version_id = form.version.data - upload.platform = Platform.from_string(form.platform.data) - upload.parameters = form.parameters.data + upload.notes = form.notes.data + upload.version_id = form.version.data + upload.platform = Platform.from_string(form.platform.data) + upload.parameters = form.parameters.data sample.tags = list(Tag.query.filter(Tag.id.in_(form.tags.data))) g.db.commit() g.log.info(f"sample with id: {sample_id} updated") @@ -309,11 +308,10 @@ def edit_sample(sample_id): if not form.is_submitted(): # Populate form with current set sample values - if sample.upload is not None: - form.version.data = sample.upload.version.id if sample.upload.version else None - form.platform.data = sample.upload.platform.name if sample.upload.platform else None - form.notes.data = sample.upload.notes - form.parameters.data = sample.upload.parameters + form.version.data = sample.upload.version.id if sample.upload.version else None + form.platform.data = sample.upload.platform.name if sample.upload.platform else None + form.notes.data = sample.upload.notes + form.parameters.data = sample.upload.parameters form.tags.data = [tag.id for tag in sample.tags] return { diff --git a/templates/sample/sample_info.html b/templates/sample/sample_info.html index 9801ee84..a439c7ea 100644 --- a/templates/sample/sample_info.html +++ b/templates/sample/sample_info.html @@ -2,124 +2,135 @@ {% block title %}Sample information {{ super() }}{% endblock %} {% block body %} - {{ super() }} -
+{{ super() }} +
+
-
-
- {% with messages = get_flashed_messages(with_categories=true) %} - {% if messages %} -
-
-
    - {% for category, message in messages %} -
  • {{ message }}
  • - {% endfor %} -
-
-
- {% endif %} - {% endwith %} -

Sample information

-

Basic details

-
-

- SHA256-hash: {{ sample.sha }}
- Extension: {{ sample.extension }}
- {% if sample.upload %} - {% if user.is_admin or user.id == sample.upload.user.id %} - Original name: {{ sample.original_name }}
- {% endif %} - {% if user.is_admin %} - Submitted by: {{ sample.upload.user.name }}
- {% endif %} - Submitted for CCExtractor version {{ sample.upload.version.version }} (released {{ sample.upload.version.released }})
- Platform: {{ sample.upload.platform.description }}
- Parameters: {{ sample.upload.parameters }}
- Notes: {{ sample.upload.notes }}
- {% else %} - Upload information: N/A
- {% endif %} -

-
- {% if additional_files|length > 0 %} -

Additional files

- - - - - {% if sample.upload and (user.is_admin or user.id == sample.upload.user.id) %} - - {% endif %} - - - - - {% for file in additional_files %} - - - {% if sample.upload and (user.is_admin or user.id == sample.upload.user.id) %} - - {% endif %} - - - {% endfor %} - -
FileOriginal nameActions
{{ file.short_name }}{{ file.original_name }} - - {% if user.is_admin %} -   - {% endif %} -
- {% endif %} -

- Tags: - {% if sample.tags|length %} - {% for tag in sample.tags %} - {{ tag.name }} +

+ {% with messages = get_flashed_messages(with_categories=true) %} + {% if messages %} +
+
+
    + {% for category, message in messages %} +
  • {{ message }}
  • {% endfor %} - {% else %} - No tags yet. +
+
+
+ {% endif %} + {% endwith %} +

Sample information

+

Basic details

+
+

+ SHA256-hash: {{ sample.sha }}
+ Extension: {{ sample.extension }}
+ + {% if user.is_admin or user.id == sample.upload.user.id %} + Original name: {{ sample.original_name }}
{% endif %} {% if user.is_admin %} -   + Submitted by: {{ + sample.upload.user.name }}
{% endif %} + Submitted for CCExtractor version {{ sample.upload.version.version }} (released {{ + sample.upload.version.released }})
+ Platform: {{ sample.upload.platform.description }}
+ Parameters: {{ sample.upload.parameters }}
+ Notes: {{ sample.upload.notes }}

-

Test status & regression tests

-

Repository: {{ latest_commit }}

-

Last release: {{ latest_release }}

-

See in which regression tests this sample has been used

- {% if media %} -

Media info

-

Full media info can be downloaded using the link at the right -->.

-
- {% if media and media|length > 0 %} - {{ macros.render_media_info(media) }} - {% else %} -

No media info available

+
+ {% if additional_files|length > 0 %} +

Additional files

+ + + + + {user.is_admin or user.id == sample.upload.user.id %} + {% endif %} - + + + + + {% for file in additional_files %} + + + {user.is_admin or user.id == sample.upload.user.id %} + + {% endif %} + + + {% endfor %} + +
FileOriginal nameActions
{{ file.short_name }}{{ file.original_name }} + + {% if user.is_admin %} +   + {% endif %} +
+ {% endif %} +

+ Tags: + {% if sample.tags|length %} + {% for tag in sample.tags %} + {{ tag.name }} + {% endfor %} + {% else %} + No tags yet. {% endif %} -

GitHub Issues

- {% include "sample/list_issues.html" %} -

Do you want to see more information displayed by default or did I forget something? Please make a request on GitHub!

-
-
+ {% if user.is_admin %} +   + {% endif %} +

+

Test status & regression tests

+

Repository: {{ latest_commit }}

+

Last release: {{ latest_release }}

+

See in which regression tests this + sample has been used

+ {% if media %} +

Media info

+

Full media info can be downloaded using the link at the right -->.

+
+ {% if media and media|length > 0 %} + {{ macros.render_media_info(media) }} + {% else %} +

No media info available

+ {% endif %} +
+ {% endif %} +

GitHub Issues

+ {% include "sample/list_issues.html" %} +

Do you want to see more information displayed by default or did I forget something? Please + make a request on GitHub!

+
+
Actions
-
+
{% endblock %} \ No newline at end of file From b8b66bc4520b4449a9dd6c82b954592881ea6814 Mon Sep 17 00:00:00 2001 From: Pulkit Chauhan Date: Thu, 19 Feb 2026 18:46:36 +0530 Subject: [PATCH 10/11] Revert files to original --- mod_sample/controllers.py | 6 +++--- templates/sample/sample_info.html | 5 ++--- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/mod_sample/controllers.py b/mod_sample/controllers.py index 6a138625..4ad216b0 100755 --- a/mod_sample/controllers.py +++ b/mod_sample/controllers.py @@ -308,8 +308,8 @@ def edit_sample(sample_id): if not form.is_submitted(): # Populate form with current set sample values - form.version.data = sample.upload.version.id if sample.upload.version else None - form.platform.data = sample.upload.platform.name if sample.upload.platform else None + form.version.data = sample.upload.version.id + form.platform.data = sample.upload.platform.name form.notes.data = sample.upload.notes form.parameters.data = sample.upload.parameters form.tags.data = [tag.id for tag in sample.tags] @@ -398,4 +398,4 @@ def delete_sample_additional(sample_id, additional_id): 'form': form } raise SampleNotFoundException(f"Extra file {additional_id} for sample {sample.id} not found") - raise SampleNotFoundException(f"Sample with id {sample_id} not found") + raise SampleNotFoundException(f"Sample with id {sample_id} not found") \ No newline at end of file diff --git a/templates/sample/sample_info.html b/templates/sample/sample_info.html index a439c7ea..8164405a 100644 --- a/templates/sample/sample_info.html +++ b/templates/sample/sample_info.html @@ -26,7 +26,6 @@

Basic details

SHA256-hash: {{ sample.sha }}
Extension: {{ sample.extension }}
- {% if user.is_admin or user.id == sample.upload.user.id %} Original name: {{ sample.original_name }}
{% endif %} @@ -47,7 +46,7 @@

Additional files

File - {user.is_admin or user.id == sample.upload.user.id %} + {% if user.is_admin or user.id == sample.upload.user.id %} Original name {% endif %} Actions @@ -57,7 +56,7 @@

Additional files

{% for file in additional_files %} {{ file.short_name }} - {user.is_admin or user.id == sample.upload.user.id %} + {% if user.is_admin or user.id == sample.upload.user.id %} {{ file.original_name }} {% endif %} From 9299ecd709848b7336d4d5306647112c934d67e3 Mon Sep 17 00:00:00 2001 From: Pulkit Chauhan Date: Thu, 19 Feb 2026 18:55:03 +0530 Subject: [PATCH 11/11] Revert back to orignal --- templates/sample/sample_info.html | 212 ++++++++++++++---------------- 1 file changed, 99 insertions(+), 113 deletions(-) diff --git a/templates/sample/sample_info.html b/templates/sample/sample_info.html index 8164405a..8e181b30 100644 --- a/templates/sample/sample_info.html +++ b/templates/sample/sample_info.html @@ -2,134 +2,120 @@ {% block title %}Sample information {{ super() }}{% endblock %} {% block body %} -{{ super() }} -
-
+ {{ super() }} +
-
- {% with messages = get_flashed_messages(with_categories=true) %} - {% if messages %} -
-
-
    - {% for category, message in messages %} -
  • {{ message }}
  • +
    +
    + {% with messages = get_flashed_messages(with_categories=true) %} + {% if messages %} +
    +
    +
      + {% for category, message in messages %} +
    • {{ message }}
    • + {% endfor %} +
    +
    +
    + {% endif %} + {% endwith %} +

    Sample information

    +

    Basic details

    +
    +

    + SHA256-hash: {{ sample.sha }}
    + Extension: {{ sample.extension }}
    + {% if user.is_admin or user.id == sample.upload.user.id %} + Original name: {{ sample.original_name }}
    + {% endif %} + {% if user.is_admin %} + Submitted by: {{ sample.upload.user.name }}
    + {% endif %} + Submitted for CCExtractor version {{ sample.upload.version.version }} (released {{ sample.upload.version.released }})
    + Platform: {{ sample.upload.platform.description }}
    + Parameters: {{ sample.upload.parameters }}
    + Notes: {{ sample.upload.notes }}
    +

    +
    + {% if additional_files|length > 0 %} +

    Additional files

    + + + + + {% if user.is_admin or user.id == sample.upload.user.id %} + + {% endif %} + + + + + {% for file in additional_files %} + + + {% if user.is_admin or user.id == sample.upload.user.id %} + + {% endif %} + + + {% endfor %} + +
    FileOriginal nameActions
    {{ file.short_name }}{{ file.original_name }} + + {% if user.is_admin %} +   + {% endif %} +
    + {% endif %} +

    + Tags: + {% if sample.tags|length %} + {% for tag in sample.tags %} + {{ tag.name }} {% endfor %} -

-
-
- {% endif %} - {% endwith %} -

Sample information

-

Basic details

-
-

- SHA256-hash: {{ sample.sha }}
- Extension: {{ sample.extension }}
- {% if user.is_admin or user.id == sample.upload.user.id %} - Original name: {{ sample.original_name }}
+ {% else %} + No tags yet. {% endif %} {% if user.is_admin %} - Submitted by: {{ - sample.upload.user.name }}
+   {% endif %} - Submitted for CCExtractor version {{ sample.upload.version.version }} (released {{ - sample.upload.version.released }})
- Platform: {{ sample.upload.platform.description }}
- Parameters: {{ sample.upload.parameters }}
- Notes: {{ sample.upload.notes }}

-
- {% if additional_files|length > 0 %} -

Additional files

- - - - - {% if user.is_admin or user.id == sample.upload.user.id %} - - {% endif %} - - - - - {% for file in additional_files %} - - - {% if user.is_admin or user.id == sample.upload.user.id %} - +

Test status & regression tests

+

Repository: {{ latest_commit }}

+

Last release: {{ latest_release }}

+

See in which regression tests this sample has been used

+ {% if media %} +

Media info

+

Full media info can be downloaded using the link at the right -->.

+
+ {% if media and media|length > 0 %} + {{ macros.render_media_info(media) }} + {% else %} +

No media info available

{% endif %} -
- - {% endfor %} - -
FileOriginal nameActions
{{ file.short_name }}{{ file.original_name }} - - {% if user.is_admin %} -   - {% endif %} -
- {% endif %} -

- Tags: - {% if sample.tags|length %} - {% for tag in sample.tags %} - {{ tag.name }} - {% endfor %} - {% else %} - No tags yet. - {% endif %} - {% if user.is_admin %} -   + {% endif %} -

-

Test status & regression tests

-

Repository: {{ latest_commit }}

-

Last release: {{ latest_release }}

-

See in which regression tests this - sample has been used

- {% if media %} -

Media info

-

Full media info can be downloaded using the link at the right -->.

-
- {% if media and media|length > 0 %} - {{ macros.render_media_info(media) }} - {% else %} -

No media info available

- {% endif %} -
- {% endif %} -

GitHub Issues

- {% include "sample/list_issues.html" %} -

Do you want to see more information displayed by default or did I forget something? Please - make a request on GitHub!

-
-
+

GitHub Issues

+ {% include "sample/list_issues.html" %} +

Do you want to see more information displayed by default or did I forget something? Please make a request on GitHub!

+
+
Actions
+
-
{% endblock %} \ No newline at end of file