PR to Study Prod Updates

.dockerignore

@@ -7,5 +7,13 @@
 .pre-commit-config.yaml
 .readthedocs.yml
 .travis.yml
-venv
 .git
+
+# ignore local python environments
+venv
+.venv
+
+# prevent large backup files from being copied into the image
+/backups
+*.sql
+*.gz

.envs/.local/.django

@@ -33,9 +33,17 @@ SINEQUA_CONFIGS_REPO_WEBAPP_PR_BRANCH='dummy_branch'
 # Slack Webhook
 # ------------------------------------------------------------------------------
 SLACK_WEBHOOK_URL=''
-LRM_USER=''
-LRM_PASSWORD=''
+
+#Server Credentials
+#--------------------------------------------------------------------------------
+LRM_DEV_USER=''
+LRM_DEV_PASSWORD=''
 XLI_USER=''
 XLI_PASSWORD=''
 LRM_QA_USER=''
 LRM_QA_PASSWORD=''
+
+#Server Tokens
+#--------------------------------------------------------------------------------
+LRM_DEV_TOKEN=''
+XLI_TOKEN=''

.gitignore

@@ -292,8 +292,7 @@ config_generation/config.py
 # Model's inference files
 Document_Classifier_inference/model.pt
 
-# Database backup
-backup.json
-
-# Prod backup
-prod_backup-20240423.json
+# Ignore Database Backup files
+/backups
+*.sql
+*.gz

.pre-commit-config.yaml

@@ -8,6 +8,8 @@ repos:
       - id: trailing-whitespace
       - id: end-of-file-fixer
       - id: check-yaml
+      - id: check-merge-conflict
+      - id: debug-statements
 
   - repo: https://github.com/asottile/pyupgrade
     rev: v3.17.0
@@ -37,14 +39,41 @@ repos:
     hooks:
       - id: mypy
         args: ["--strict"]
-        # ignoring everything for now
-        exclude: .
-        additional_dependencies: [django-stubs, celery,  django-environ, django-extensions, django-crispy-forms,
-        crispy-bootstrap5, django-allauth, django-celery-beat, djangorestframework, djangorestframework-datatables,
-        django-debug-toolbar, psycopg2-binary, python-slugify, xmltodict, PyGithub, boto3, scrapy, types-requests]
+        exclude: "."
+        additional_dependencies:
+          - django-stubs
+          - celery
+          - django-environ
+          - django-extensions
+          - django-crispy-forms
+          - crispy-bootstrap5
+          - django-allauth
+          - django-celery-beat
+          - djangorestframework
+          - djangorestframework-datatables
+          - django-debug-toolbar
+          - psycopg2-binary
+          - python-slugify
+          - xmltodict
+          - PyGithub
+          - boto3
+          - scrapy
+          - types-requests
+
+  - repo: https://github.com/PyCQA/bandit
+    rev: '1.7.0'
+    hooks:
+      - id: bandit
+        args: ['-r', '--configfile=bandit-config.yml']
+
+  - repo: https://github.com/zricethezav/gitleaks
+    rev: 'v8.0.4'
+    hooks:
+      - id: gitleaks
+        args: ['--config=gitleaks-config.toml']
+
 
 
-# sets up .pre-commit-ci.yaml to ensure pre-commit dependencies stay up to date
 ci:
   autoupdate_schedule: weekly
   skip: []

CODE_STANDARDS.md

@@ -0,0 +1,63 @@
+# Coding Standards and Conventions for COSMOS
+
+## Overview
+To maintain high-quality code and ensure consistency across the entire COSMOS project, we have established coding standards and conventions. This document outlines the key standards and practices that all contributors are expected to follow. Adhering to these guidelines helps us to achieve a codebase that appears as if it were written by a single entity, regardless of the number of contributors.
+
+## Coding Standards
+
+### Formatting Standards
+- **Line Length**: Maximum of 120 characters per line to ensure readability across various environments.
+- **Code Formatting**: Utilize tools like Black for Python code to ensure consistent formatting across the entire codebase.
+- **Import Ordering**: Follow a consistent import order:
+  - Standard library imports.
+  - Third-party imports.
+  - Application-specific imports.
+
+### Naming Conventions
+- **Variables and Functions**: Use `snake_case`.
+- **Classes and Exceptions**: Use `CamelCase`.
+- **Constants**: Use `UPPER_CASE`.
+
+### Commenting
+- Inline comments should be used sparingly and only when necessary to explain "why" something is done, not "what" is done.
+- All public methods, classes, and modules should include docstrings that follow the [Google style guide](https://google.github.io/styleguide/pyguide.html).
+
+### Error Handling
+- Explicit is better than implicit. Raise exceptions rather than returning None or any error codes.
+- Use custom exceptions over generic exceptions when possible to make error handling more predictive.
+
+## Tool Configurations and Pre-commit Hooks
+
+To automate and enforce these standards, the following tools are configured with pre-commit hooks in our development process:
+
+### Pre-commit Hooks Setup
+
+To ensure that these tools are run automatically on every commit, contributors must set up pre-commit hooks locally. Run the following commands to install and configure pre-commit hooks:
+
+```bash
+pip install pre-commit
+pre-commit install
+pre-commit run --all-files
+```
+
+The following pre-commit hooks are configured:
+
+- trailing-whitespace, end-of-file-fixer, check-yaml, check-merge-conflict, debug-statements: Checks for common formatting issues.
+- pyupgrade: Automatically upgrades syntax for newer versions of the language.
+- black: Formats Python code to ensure consistent styling.
+- isort: Sorts imports alphabetically and automatically separated into sections.
+- flake8: Lints code to catch styling errors and potential bugs.
+- mypy: Checks type annotations to catch potential bugs.
+- bandit: Scans code for common security issues.
+- gitleaks: Prevents secrets from being committed to the repository.
+- hadolint: Lints Dockerfiles to ensure best practices and common conventions are followed.
+
+## Continuous Integration (CI)
+When a commit is pushed to a branch that is part of a Pull Request, our Continuous Integration (CI) pipeline automatically runs specified tools to check code quality, style, security and other standards. If these checks fail, the PR cannot be merged until all issues are resolved.
+
+## Quality Standards Enforcement
+- PRs must pass all checks from the configured pre-commit hooks and CI pipeline to be eligible for merging.
+- Code reviews additionally focus on logical errors and code quality beyond what automated tools can detect.
+
+## Conclusion
+By adhering to these standards and utilizing the tools set up, we maintain the high quality and consistency of our codebase, making it easier for developers to collaborate effectively.

README.md

@@ -18,7 +18,6 @@ $ docker-compose -f local.yml build
 ```bash
 $ docker-compose -f local.yml up
 ```
-
 ### Non-Docker Local Setup
 
 If you prefer to run the project without Docker, follow these steps:
@@ -69,57 +68,103 @@ $ docker-compose -f local.yml run --rm django python manage.py createsuperuser
 #### Creating Additional Users
 
 Create additional users through the admin interface (/admin).
+## Database Backup and Restore
+
+COSMOS provides dedicated management commands for backing up and restoring your PostgreSQL database. These commands handle both compressed and uncompressed backups and work seamlessly in both local and production environments using Docker.
 
-### Loading Fixtures
+### Backup Directory Structure
 
-To load collections:
+All backups are stored in the `/backups` directory at the root of your project. This directory is mounted as a volume in both local and production Docker configurations, making it easy to manage backups across different environments.
 
+- Local development: `./backups/`
+- Production server: `/path/to/project/backups/`
+
+If the directory doesn't exist, create it:
 ```bash
-$ docker-compose -f local.yml run --rm django python manage.py loaddata sde_collections/fixtures/collections.json
+mkdir backups
 ```
 
-### Manually Creating and Loading a ContentTypeless Backup
-Navigate to the server running prod, then to the project folder. Run the following command to create a backup:
+### Creating a Database Backup
+
+To create a backup of your database:
 
 ```bash
-docker-compose -f production.yml run --rm --user root django python manage.py dumpdata --natural-foreign --natural-primary --exclude=contenttypes --exclude=auth.Permission --indent 2 --output /app/backups/prod_backup-20240812.json
+# Create a compressed backup (recommended)
+docker-compose -f local.yml run --rm django python manage.py database_backup
+
+# Create an uncompressed backup
+docker-compose -f local.yml run --rm django python manage.py database_backup --no-compress
+
+# Specify custom output location within backups directory
+docker-compose -f local.yml run --rm django python manage.py database_backup --output my_custom_backup.sql
 ```
-This will have saved the backup in a folder outside of the docker container. Now you can copy it to your local machine.
+
+The backup command will automatically:
+- Detect your server environment (Production/Staging/Local)
+- Use database credentials from your environment settings
+- Generate a dated filename if no output path is specified
+- Save the backup to the mounted `/backups` directory
+- Compress the backup by default (can be disabled with --no-compress)
+
+### Restoring from a Database Backup
+
+To restore your database from a backup, it will need to be in the `/backups` directory. You can then run the following command:
 
 ```bash
-mv ~/prod_backup-20240812.json <project_path>/prod_backup-20240812.json
-scp sde:/home/ec2-user/sde_indexing_helper/backups/prod_backup-20240812.json prod_backup-20240812.json
+# Restore from a backup (handles both .sql and .sql.gz files)
+docker-compose -f local.yml run --rm django python manage.py database_restore backups/backup_file_name.sql.gz
 ```
 
-Finally, load the backup into your local database:
+The restore command will:
+- Automatically detect if the backup is compressed (.gz)
+- Terminate existing database connections
+- Drop and recreate the database
+- Restore all data from the backup
+- Handle all database credentials from your environment settings
+
+### Working with Remote Servers
 
+When working with production or staging servers:
+
+1. First, SSH into the appropriate server:
 ```bash
-docker-compose -f local.yml run --rm django python manage.py loaddata prod_backup-20240812.json
+# For production
+ssh user@production-server
+cd /path/to/project
 ```
 
-### Loading the Database from an Arbitrary Backup
+2. Create a backup on the remote server:
+```bash
+docker-compose -f production.yml run --rm django python manage.py database_backup
+```
 
-1. Build the project and run the necessary containers (as documented above).
-2. Clear out content types using the Django shell:
+3. Copy the backup from the remote server's backup directory to your local machine:
+```bash
+scp user@remote-server:/path/to/project/backups/backup_name.sql.gz ./backups/
+```
 
+4. Restore locally:
 ```bash
-$ docker-compose -f local.yml run --rm django python manage.py shell
->>> from django.contrib.contenttypes.models import ContentType
->>> ContentType.objects.all().delete()
->>> exit()
+docker-compose -f local.yml run --rm django python manage.py database_restore backups/backup_name.sql.gz
 ```
 
-3. Load your backup database:
+### Alternative Methods
+
+While the database_backup and database_restore commands are the recommended approach, you can also use Django's built-in fixtures for smaller datasets:
 
 ```bash
-$ docker cp /path/to/your/backup.json container_name:/path/inside/container/backup.json
-$ docker-compose -f local.yml run --rm django python manage.py loaddata /path/inside/the/container/backup.json
-$ docker-compose -f local.yml run --rm django python manage.py migrate
+# Create a backup excluding content types
+docker-compose -f production.yml run --rm django python manage.py dumpdata \
+    --natural-foreign --natural-primary \
+    --exclude=contenttypes --exclude=auth.Permission \
+    --indent 2 \
+    --output backups/prod_backup-$(date +%Y%m%d).json
+
+# Restore from a fixture
+docker-compose -f local.yml run --rm django python manage.py loaddata backups/backup_name.json
 ```
-### Restoring the Database from a SQL Dump
-If the JSON file is particularly large (>1.5GB), Docker might struggle with this method. In such cases, you can use SQL dump and restore commands as an alternative, as described [here](./SQLDumpRestoration.md).
-
 
+Note: For large databases (>1.5GB), the database_backup and database_restore commands are strongly recommended over JSON fixtures as they handle large datasets more efficiently.
 
 ## Additional Commands
 
@@ -180,6 +225,7 @@ $ pip install pre-commit
 $ pre-commit install
 $ pre-commit run --all-files
 ```
+For detailed information on the coding standards and conventions we enforce, please see our [Coding Standards and Conventions](CODE_STANDARDS.md).
 
 ### Sentry Setup
 
@@ -208,3 +254,24 @@ Eventually, job creation will be done seamlessly by the webapp. Until then, edit
   - JavaScript: `/sde_indexing_helper/static/js`
   - CSS: `/sde_indexing_helper/static/css`
   - Images: `/sde_indexing_helper/static/images`
+
+
+## Running Long Scripts on the Server
+```shell
+tmux new -s docker_django
+```
+Once you are inside, you can run dmshell or for example a managment command:
+
+```shell
+docker-compose -f production.yml run --rm django python manage.py deduplicate_urls
+```
+
+Later, you can do this to get back in.
+```shell
+tmux attach -t docker_django
+```
+
+To delete the session:
+```shell
+tmux kill-session -t docker_django
+```