feat(dbsync): Introduce DBSyncSnapshotService and snapshot freshness test #3222

ArturWieczorek · 2025-11-14T22:36:45Z

Introduces new components to cleanly implement the Cardano DB-Sync snapshot freshness check using the S3 REST API.

DBSyncSnapshotService: A new service class responsible for interacting with the IOHK S3 repository, encapsulating API calls and robust XML parsing logic (including handling S3 namespaces and date formats).
Clarity: Uses dataclass for snapshot metadata and standard Pytest best practices.

pytest cardano_node_tests/tests/test_dbsync.py::TestDBSyncSnapshot::test_latest_snapshot_freshness -v
 WARNING: Using `cardano-cli` from custom path!
 WARNING: Using `cardano-node` from custom path!
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.11.14, pytest-8.4.1, pluggy-1.5.0 -- /home/artur/Projects/cnt/cardano-node-tests/dev_workdir/.venv/bin/python
cachedir: .pytest_cache
metadata: {'Python': '3.11.14', 'Platform': 'Linux-5.15.0-161-generic-x86_64-with-glibc2.40', 'Packages': {'pytest': '8.4.1', 'pluggy': '1.5.0'}, 'Plugins': {'html': '4.1.1', 'xdist': '3.7.0', 'metadata': '3.1.1', 'select': '0.1.2', 'order': '1.3.0', 'subtests': '0.14.2', 'allure-pytest': '2.14.3', 'hypothesis': '6.135.17'}, 'cardano-node': '10.3.1', 'cardano-node rev': 'b3f237b75e64f4d8142af95b053e2828221d707f', 'cardano-node ghc': 'ghc-9.6', 'cardano-cli': '10.7.0.0', 'cardano-cli rev': 'b3f237b75e64f4d8142af95b053e2828221d707f', 'cardano-cli ghc': 'ghc-9.6', 'CLUSTER_ERA': 'conway', 'COMMAND_ERA': 'conway', 'TESTNET_VARIANT': 'conway_fast', 'ENABLE_LEGACY': 'False', 'MIXED_P2P': 'False', 'NUM_POOLS': '3', 'UTXO_BACKEND': '', 'MAX_TESTS_PER_CLUSTER': 8, 'CLUSTERS_COUNT': 1, 'HAS_CC': 'True', 'cardano-node-tests rev': 'dd09cb20c9a3ea819a0c8c795b861987d064808c', 'cardano-node-tests url': 'https://github.com/IntersectMBO/cardano-node-tests/tree/dd09cb20c9a3ea819a0c8c795b861987d064808c', 'CARDANO_NODE_SOCKET_PATH': '/home/artur/Projects/cnt/cardano-node-tests/dev_workdir/state-cluster0/bft1.socket', 'cardano-cli exe': '/home/artur/.local/bin/cardano-cli', 'cardano-node exe': '/home/artur/.local/bin/cardano-node', 'cardano-submit-api exe': '/nix/store/knjnvq0d6m3hn9sqfv552b443y7p62h5-cardano-submit-api-exe-cardano-submit-api-10.1.1/bin/cardano-submit-api', 'network magic': 42, 'HAS_DBSYNC': 'True', 'db-sync': '13.6.0.5', 'db-sync rev': '0000000000000000000000000000000000000000', 'db-sync ghc': 'ghc-9.6', 'db-sync exe': '/home/artur/.local/bin/cardano-db-sync', 'HAS_SMASH': 'False'}
hypothesis profile 'default'
rootdir: /home/artur/Projects/cnt/cardano-node-tests
configfile: pyproject.toml
plugins: html-4.1.1, xdist-3.7.0, metadata-3.1.1, select-0.1.2, order-1.3.0, subtests-0.14.2, allure-pytest-2.14.3, hypothesis-6.135.17
collected 1 item                                                                                                                                                                                                   

cardano_node_tests/tests/test_dbsync.py::TestDBSyncSnapshot::test_latest_snapshot_freshness 
-------------------------------------------------------------------------------------------------- live log setup --------------------------------------------------------------------------------------------------
INFO     cardano_node_tests.tests.conftest:conftest.py:237 Changed CWD to '/home/artur/Projects/cnt/cardano-node-tests/dev_workdir/tmp/pytest-of-artur/pytest-30'.
-------------------------------------------------------------------------------------------------- live log call ---------------------------------------------------------------------------------------------------
INFO     cardano_node_tests.tests.test_dbsync:test_dbsync.py:415 Latest db-sync version: 13.6
INFO     cardano_node_tests.tests.test_dbsync:test_dbsync.py:420 Latest snapshot: db-sync-snapshot-schema-13.6-block-12633962-x86_64.tgz
INFO     cardano_node_tests.tests.test_dbsync:test_dbsync.py:421 Snapshot date: 2025-11-10T23:41:00+00:00
INFO     cardano_node_tests.tests.test_dbsync:test_dbsync.py:422 Snapshot size: 69.60 GB
INFO     cardano_node_tests.tests.test_dbsync:test_dbsync.py:434 Success: The latest snapshot is recent (within 5-day limit).
PASSED                                                                                                                                                                                                       [100%]

================================================================================================ 1 passed in 0.43s =================================================================================================

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.

cardano_node_tests/utils/dbsync_snapshot_service.py

…test Introduces new components to cleanly implement the Cardano DB-Sync snapshot freshness check using the S3 REST API. * **DBSyncSnapshotService:** A new service class responsible for interacting with the IOHK S3 repository, encapsulating API calls and robust XML parsing logic (including handling S3 namespaces and date formats). * **Clarity:** Uses `dataclass` for snapshot metadata and standard Pytest best practices.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.

cardano_node_tests/utils/dbsync_snapshot_service.py

cardano_node_tests/tests/test_dbsync.py

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.

Copilot · 2025-12-02T13:02:56Z

cardano_node_tests/tests/test_dbsync.py

+
+        assert latest_snapshot.last_modified >= five_days_ago, (
+            f"The latest snapshot is too old. "
+            f"Age: {(now_utc - latest_snapshot.last_modified).days} days. "


The age calculation using .days may be incorrect for snapshots less than a full day old. The .days attribute returns only the integer number of complete days in a timedelta, so a snapshot that's 23 hours old would show as 0 days in the error message. Consider using total_seconds() / 86400 for a more accurate age representation, or clarify that this is intentional.

Suggested change

f"Age: {(now_utc - latest_snapshot.last_modified).days} days. "

f"Age: {(now_utc - latest_snapshot.last_modified).total_seconds() / 86400:.1f} days. "

cardano_node_tests/utils/dbsync_snapshot_service.py

- Refactor `get_test_id` to accept either ClusterLib or ClusterManager. - Update `TestDBSyncSnapshot` to use ClusterManager and instantiate DBSyncSnapshotService directly, removing the fixture. - Move imports to use module import for dbsync_snapshot_service. - Use `dataclasses` and `typing` consistently in dbsync_snapshot_service.py. - Minor docstring and formatting improvements.

ArturWieczorek force-pushed the artur/add-db-sync-snapshot-test branch 5 times, most recently from a8f523c to db17a54 Compare November 15, 2025 01:55

ArturWieczorek marked this pull request as ready for review November 18, 2025 12:38

ArturWieczorek requested review from mkoura and saratomaz as code owners November 18, 2025 12:38

mkoura force-pushed the artur/add-db-sync-snapshot-test branch 2 times, most recently from 01807d5 to 86dc869 Compare November 25, 2025 14:47

mkoura requested a review from Copilot November 25, 2025 14:59

Copilot AI reviewed Nov 25, 2025

View reviewed changes

Copilot started reviewing on behalf of mkoura November 25, 2025 15:15 View session

Copilot finished reviewing on behalf of mkoura November 25, 2025 15:17

mkoura requested a review from Copilot November 25, 2025 15:30

Copilot started reviewing on behalf of mkoura November 25, 2025 15:31 View session

Copilot finished reviewing on behalf of mkoura November 25, 2025 15:33

Copilot AI reviewed Nov 25, 2025

View reviewed changes

mkoura force-pushed the artur/add-db-sync-snapshot-test branch from 86dc869 to be0c004 Compare December 2, 2025 11:46

mkoura requested a review from Copilot December 2, 2025 11:47

Copilot started reviewing on behalf of mkoura December 2, 2025 11:49 View session

Copilot finished reviewing on behalf of mkoura December 2, 2025 11:52

Copilot AI reviewed Dec 2, 2025

View reviewed changes

cardano_node_tests/utils/dbsync_snapshot_service.py Outdated Show resolved Hide resolved

cardano_node_tests/utils/dbsync_snapshot_service.py Show resolved Hide resolved

cardano_node_tests/tests/test_dbsync.py Outdated Show resolved Hide resolved

mkoura force-pushed the artur/add-db-sync-snapshot-test branch 2 times, most recently from f64d3a8 to e3c64fa Compare December 2, 2025 12:59

mkoura requested a review from Copilot December 2, 2025 12:59

Copilot started reviewing on behalf of mkoura December 2, 2025 13:00 View session

Copilot finished reviewing on behalf of mkoura December 2, 2025 13:02

Copilot AI reviewed Dec 2, 2025

View reviewed changes

mkoura force-pushed the artur/add-db-sync-snapshot-test branch from e3c64fa to 96be26c Compare December 2, 2025 13:06

mkoura merged commit 8f50d7d into master Dec 2, 2025
3 checks passed

mkoura deleted the artur/add-db-sync-snapshot-test branch December 2, 2025 13:28

	f"Age: {(now_utc - latest_snapshot.last_modified).days} days. "
	f"Age: {(now_utc - latest_snapshot.last_modified).total_seconds() / 86400:.1f} days. "

feat(dbsync): Introduce DBSyncSnapshotService and snapshot freshness test #3222

feat(dbsync): Introduce DBSyncSnapshotService and snapshot freshness test #3222

Uh oh!

Conversation

ArturWieczorek commented Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ArturWieczorek commented Nov 14, 2025 •

edited

Loading