Dev to Main December 2025

ENV.md

@@ -57,19 +57,30 @@ Note that some tasks/subtasks are themselves enabled by other tasks.
 
 ### Scheduled Task Flags
 
-| Flag                                | Description                                                                   |
-|-------------------------------------|-------------------------------------------------------------------------------|
-| `SCHEDULED_TASKS_FLAG`              | All scheduled tasks. Disabling disables all other scheduled tasks.            |
-| `PUSH_TO_HUGGING_FACE_TASK_FLAG`    | Pushes data to HuggingFace.                                                   |
-| `POPULATE_BACKLOG_SNAPSHOT_TASK_FLAG` | Populates the backlog snapshot.                                               |
-| `DELETE_OLD_LOGS_TASK_FLAG`         | Deletes old logs.                                                             |
-| `RUN_URL_TASKS_TASK_FLAG`           | Runs URL tasks.                                                               |
-| `IA_PROBE_TASK_FLAG`            | Extracts and links Internet Archives metadata to URLs.                        |
-| `IA_SAVE_TASK_FLAG`             | Saves URLs to Internet Archives.                                              |
-| `MARK_TASK_NEVER_COMPLETED_TASK_FLAG` | Marks tasks that were started but never completed (usually due to a restart). |
-| `DELETE_STALE_SCREENSHOTS_TASK_FLAG` | Deletes stale screenshots for URLs already validated.                         |
-| `TASK_CLEANUP_TASK_FLAG` | Cleans up tasks that are no longer needed. |
-| `REFRESH_MATERIALIZED_VIEWS_TASK_FLAG` | Refreshes materialized views. |
+| Flag                                       | Description                                                                   |
+|--------------------------------------------|-------------------------------------------------------------------------------|
+| `SCHEDULED_TASKS_FLAG`                     | All scheduled tasks. Disabling disables all other scheduled tasks.            |
+| `PUSH_TO_HUGGING_FACE_TASK_FLAG`           | Pushes data to HuggingFace.                                                   |
+| `POPULATE_BACKLOG_SNAPSHOT_TASK_FLAG`      | Populates the backlog snapshot.                                               |
+| `DELETE_OLD_LOGS_TASK_FLAG`                | Deletes old logs.                                                             |
+| `RUN_URL_TASKS_TASK_FLAG`                  | Runs URL tasks.                                                               |
+| `IA_PROBE_TASK_FLAG`                       | Extracts and links Internet Archives metadata to URLs.                        |
+| `IA_SAVE_TASK_FLAG`                        | Saves URLs to Internet Archives.                                              |
+| `MARK_TASK_NEVER_COMPLETED_TASK_FLAG`      | Marks tasks that were started but never completed (usually due to a restart). |
+| `DELETE_STALE_SCREENSHOTS_TASK_FLAG`       | Deletes stale screenshots for URLs already validated.                         |
+| `TASK_CLEANUP_TASK_FLAG`                   | Cleans up tasks that are no longer needed. |
+| `REFRESH_MATERIALIZED_VIEWS_TASK_FLAG`     | Refreshes materialized views. |
+| `UPDATE_URL_STATUS_TASK_FLAG`              | Updates the status of URLs. |
+| `DS_APP_SYNC_AGENCY_ADD_TASK_FLAG`         | Adds new agencies to the Data Sources App|
+| `DS_APP_SYNC_AGENCY_UPDATE_TASK_FLAG`      | Updates existing agencies in the Data Sources App|
+| `DS_APP_SYNC_AGENCY_DELETE_TASK_FLAG`      | Deletes agencies in the Data Sources App|
+| `DS_APP_SYNC_DATA_SOURCE_ADD_TASK_FLAG`    | Adds new data sources to the Data Sources App|
+| `DS_APP_SYNC_DATA_SOURCE_UPDATE_TASK_FLAG` | Updates existing data sources in the Data Sources App|
+| `DS_APP_SYNC_DATA_SOURCE_DELETE_TASK_FLAG` | Deletes data sources in the Data Sources App|
+| `DS_APP_SYNC_META_URL_ADD_TASK_FLAG`       | Adds new meta URLs to the Data Sources App|
+| `DS_APP_SYNC_META_URL_UPDATE_TASK_FLAG`    | Updates existing meta URLs in the Data Sources App|
+| `DS_APP_SYNC_META_URL_DELETE_TASK_FLAG`    | Deletes meta URLs in the Data Sources App|
+| `INTEGRITY_MONITOR_TASK_FLAG`              | Runs integrity checks. |
 
 ### URL Task Flags
 
@@ -81,7 +92,6 @@ URL Task Flags are collectively controlled by the `RUN_URL_TASKS_TASK_FLAG` flag
 | `URL_HTML_TASK_FLAG`                | URL HTML scraping task.                               |
 | `URL_RECORD_TYPE_TASK_FLAG`         | Automatically assigns Record Types to URLs.           |
 | `URL_AGENCY_IDENTIFICATION_TASK_FLAG` | Automatically assigns and suggests Agencies for URLs. |
-| `URL_SUBMIT_APPROVED_TASK_FLAG`     | Submits approved URLs to the Data Sources App.        |
 | `URL_MISC_METADATA_TASK_FLAG`       | Adds misc metadata to URLs.                           |
 | `URL_AUTO_RELEVANCE_TASK_FLAG`      | Automatically assigns Relevances to URLs.             |
 | `URL_PROBE_TASK_FLAG`               | Probes URLs for web metadata.                         |
@@ -90,7 +100,6 @@ URL Task Flags are collectively controlled by the `RUN_URL_TASKS_TASK_FLAG` flag
 | `URL_AUTO_VALIDATE_TASK_FLAG`       | Automatically validates URLs.                         |
 | `URL_AUTO_NAME_TASK_FLAG`           | Automatically names URLs.                             |
 | `URL_SUSPEND_TASK_FLAG`             | Suspends URLs meeting suspension criteria.            |
-| `URL_SUBMIT_META_URLS_TASK_FLAG`    | Submits meta URLs to the Data Sources App.            |
 
 ### Agency ID Subtasks
 

README.md

@@ -156,3 +156,71 @@ if it detects any missing docstrings or type hints in files that you have modifi
 These will *not* block any Pull request, but exist primarily as advisory comments to encourage good coding standards.
 
 Note that `python_checks.yml` will only function on pull requests made from within the repo, not from a forked repo.
+
+# Syncing to Data Sources App
+
+The Source Manager (SM) is part of a two app system, with the other app being the Data Sources (DS) App.
+
+
+## Add, Update, and Delete
+
+These are the core synchronization actions.
+
+In order to propagate changes to DS, we synchronize additions, updates, and deletions of the following entities:
+- Agencies
+- Data Sources
+- Meta URLs
+
+Each action for each entity occurs through a separate task. At the moment, there are nine tasks total.
+
+Each task gathers requisite information from the SM database and sends a request to one of nine corresponding endpoints in the DS API.
+
+Each DS endpoint follows the following format:
+
+```text
+/v3/sync/{entity}/{action}
+```
+
+Synchronizations are designed to occur on an hourly basis.
+
+Here is a high-level description of how each action works:
+
+### Add
+
+Adds the given entities to DS.
+
+These are denoted with the `/{entity}/add` path in the DS API.
+
+When an entity is added, it returns a unique DS ID that is mapped to the internal SM database ID via the DS app link tables.
+
+For an entity to be added, it must meet preconditions which are distinct for each entity:
+- Agencies: Must have an agency entry in the database and be linked to a location.
+- Data Sources: Must be a URL that has been internally validated as a data source and linked to an agency.
+- Meta URLs: Must be a URL that has been internally validated as a meta URL and linked to an agency.
+
+### Update
+
+Updates the given entities in DS.
+
+These are denoted with the `/{entity}/update` path in the DS API.
+
+These consist of submitting the updated entities (in full) to the requisite endpoint, and updating the local app link to indicate that the update occurred. All updates are designed to be full overwrites of the entity.
+
+For an entity to be updated, it must meet preconditions which are distinct for each entity:
+- Agencies: Must have either an agency row updated or an agency/location link updated or deleted.
+- Data Sources: One of the following must be updated:
+  - The URL table
+  - The record type table
+  - The optional data sources metadata table
+  - The agency link table (either an addition or deletion)
+- Meta URLs: Must be a URL that has been internally validated as a meta URL and linked to an agency.  Either the URL table or the agency link table (addition or deletion) must be updated.
+
+### Delete
+
+Deletes the given entities from DS.
+
+These are denoted with the `/{entity}/delete` path in the DS API.
+
+This consists of submitting a set of DS IDs to the requisite endpoint, and removing the associated DS app link entry in the SM database.
+
+When an entity with a corresponding DS App Link is deleted from the Source Manager, the core data is removed but a deletion flag is appended to the DS App Link entry, indicating that the entry is not yet removed from the DS App. The deletion task uses this flag to identify entities to be deleted, submits the deletion request to the DS API, and removes both the flag and the DS App Link.

alembic/Jenkinsfile

@@ -0,0 +1,30 @@
+pipeline {
+        agent {
+        dockerfile {
+            filename 'Dockerfile'
+            args '-e POSTGRES_USER=POSTGRES_USER -e POSTGRES_PASSWORD=POSTGRES_PASSWORD -e POSTGRES_DB=POSTGRES_DB -e POSTGRES_HOST=POSTGRES_HOST -e POSTGRES_PORT=POSTGRES_PORT'
+        }
+    }
+
+    stages {
+        stage('Migrate using Alembic') {
+            steps {
+                echo 'Building..'
+                sh 'python apply_migrations.py'
+            }
+        }
+    }
+    post {
+        failure {
+            script {
+                def payload = """{
+                    "content": "🚨 Build Failed: ${env.JOB_NAME} #${env.BUILD_NUMBER}"
+                }"""
+
+                sh """
+                curl -X POST -H "Content-Type: application/json" -d '${payload}' ${env.WEBHOOK_URL}
+                """
+            }
+        }
+    }
+}