From 09f7a770d648c5b1a302d3fab00b00b7f54af21d Mon Sep 17 00:00:00 2001 From: Max Chis Date: Mon, 24 Nov 2025 18:19:38 -0500 Subject: [PATCH] Migrate sync information into main README --- README.md | 67 +++++++++++++++++++ .../tasks/scheduled/impl/sync_to_ds/README.md | 65 ------------------ 2 files changed, 67 insertions(+), 65 deletions(-) delete mode 100644 src/core/tasks/scheduled/impl/sync_to_ds/README.md diff --git a/README.md b/README.md index 4fa95b40..56e8182d 100644 --- a/README.md +++ b/README.md @@ -157,3 +157,70 @@ These will *not* block any Pull request, but exist primarily as advisory comment Note that `python_checks.yml` will only function on pull requests made from within the repo, not from a forked repo. +# Syncing to Data Sources App + +The Source Manager (SM) is part of a two app system, with the other app being the Data Sources (DS) App. + + +## Add, Update, and Delete + +These are the core synchronization actions. + +In order to propagate changes to DS, we synchronize additions, updates, and deletions of the following entities: +- Agencies +- Data Sources +- Meta URLs + +Each action for each entity occurs through a separate task. At the moment, there are nine tasks total. + +Each task gathers requisite information from the SM database and sends a request to one of nine corresponding endpoints in the DS API. + +Each DS endpoint follows the following format: + +```text +/v3/sync/{entity}/{action} +``` + +Synchronizations are designed to occur on an hourly basis. + +Here is a high-level description of how each action works: + +### Add + +Adds the given entities to DS. + +These are denoted with the `/{entity}/add` path in the DS API. + +When an entity is added, it returns a unique DS ID that is mapped to the internal SM database ID via the DS app link tables. + +For an entity to be added, it must meet preconditions which are distinct for each entity: +- Agencies: Must have an agency entry in the database and be linked to a location. +- Data Sources: Must be a URL that has been internally validated as a data source and linked to an agency. +- Meta URLs: Must be a URL that has been internally validated as a meta URL and linked to an agency. + +### Update + +Updates the given entities in DS. + +These are denoted with the `/{entity}/update` path in the DS API. + +These consist of submitting the updated entities (in full) to the requisite endpoint, and updating the local app link to indicate that the update occurred. All updates are designed to be full overwrites of the entity. + +For an entity to be updated, it must meet preconditions which are distinct for each entity: +- Agencies: Must have either an agency row updated or an agency/location link updated or deleted. +- Data Sources: One of the following must be updated: + - The URL table + - The record type table + - The optional data sources metadata table + - The agency link table (either an addition or deletion) +- Meta URLs: Must be a URL that has been internally validated as a meta URL and linked to an agency. Either the URL table or the agency link table (addition or deletion) must be updated. + +### Delete + +Deletes the given entities from DS. + +These are denoted with the `/{entity}/delete` path in the DS API. + +This consists of submitting a set of DS IDs to the requisite endpoint, and removing the associated DS app link entry in the SM database. + +When an entity with a corresponding DS App Link is deleted from the Source Manager, the core data is removed but a deletion flag is appended to the DS App Link entry, indicating that the entry is not yet removed from the DS App. The deletion task uses this flag to identify entities to be deleted, submits the deletion request to the DS API, and removes both the flag and the DS App Link. \ No newline at end of file diff --git a/src/core/tasks/scheduled/impl/sync_to_ds/README.md b/src/core/tasks/scheduled/impl/sync_to_ds/README.md deleted file mode 100644 index 3af42af8..00000000 --- a/src/core/tasks/scheduled/impl/sync_to_ds/README.md +++ /dev/null @@ -1,65 +0,0 @@ -The Source Manager (SM) is part of a two app system, with the other app being the Data Sources (DS) App. - - -# Add, Update, and Delete - -These are the core synchronization actions. - -In order to propagate changes to DS, we synchronize additions, updates, and deletions of the following entities: -- Agencies -- Data Sources -- Meta URLs - -Each action for each entity occurs through a separate task. At the moment, there are nine tasks total. - -Each task gathers requisite information from the SM database and sends a request to one of nine corresponding endpoints in the DS API. - -Each DS endpoint follows the following format: - -```text -/v3/sync/{entity}/{action} -``` - -Synchronizations are designed to occur on an hourly basis. - -Here is a high-level description of how each action works: - -## Add - -Adds the given entities to DS. - -These are denoted with the `/{entity}/add` path in the DS API. - -When an entity is added, it returns a unique DS ID that is mapped to the internal SM database ID via the DS app link tables. - -For an entity to be added, it must meet preconditions which are distinct for each entity: -- Agencies: Must have an agency entry in the database and be linked to a location. -- Data Sources: Must be a URL that has been internally validated as a data source and linked to an agency. -- Meta URLs: Must be a URL that has been internally validated as a meta URL and linked to an agency. - -## Update - -Updates the given entities in DS. - -These are denoted with the `/{entity}/update` path in the DS API. - -These consist of submitting the updated entities (in full) to the requisite endpoint, and updating the local app link to indicate that the update occurred. All updates are designed to be full overwrites of the entity. - -For an entity to be updated, it must meet preconditions which are distinct for each entity: -- Agencies: Must have either an agency row updated or an agency/location link updated or deleted. -- Data Sources: One of the following must be updated: - - The URL table - - The record type table - - The optional data sources metadata table - - The agency link table (either an addition or deletion) -- Meta URLs: Must be a URL that has been internally validated as a meta URL and linked to an agency. Either the URL table or the agency link table (addition or deletion) must be updated. - -## Delete - -Deletes the given entities from DS. - -These are denoted with the `/{entity}/delete` path in the DS API. - -This consists of submitting a set of DS IDs to the requisite endpoint, and removing the associated DS app link entry in the SM database. - -When an entity with a corresponding DS App Link is deleted from the Source Manager, the core data is removed but a deletion flag is appended to the DS App Link entry, indicating that the entry is not yet removed from the DS App. The deletion task uses this flag to identify entities to be deleted, submits the deletion request to the DS API, and removes both the flag and the DS App Link. \ No newline at end of file