diff --git a/.readthedocs.yml b/.readthedocs.yml index 512a0db..fe7f0ca 100644 --- a/.readthedocs.yml +++ b/.readthedocs.yml @@ -8,9 +8,9 @@ version: 2 build: - os: "ubuntu-22.04" + os: "ubuntu-24.04" tools: - python: "3.11" + python: "3.14" python: install: diff --git a/doc/cdc/index.md b/doc/cdc/index.md index d5dc0f2..ce90898 100644 --- a/doc/cdc/index.md +++ b/doc/cdc/index.md @@ -3,27 +3,22 @@ `commons-codec` includes CDC -> SQL transformer components for AWS DMS, DynamoDB, and MongoDB. -## DynamoDB -- Blog: [Replicating CDC Events from DynamoDB to CrateDB] -- Documentation: [DynamoDB CDC Relay for CrateDB] +:DynamoDB: + - Blog: [Replicating CDC Events from DynamoDB to CrateDB] + - Documentation: [DynamoDB CDC Relay for CrateDB] -## MongoDB -- Introduction: [](project:#mongodb-cdc) -- Documentation: [MongoDB CDC Relay for CrateDB] +:MongoDB: + - Usage guide: [](project:#mongodb-cdc) + - Documentation: [MongoDB CDC Relay for CrateDB] ```{toctree} :hidden: -mongodb +MongoDB ``` -:::{note} -Please note relevant components are still in their infancy (beta), -and need further curation and improvements. -::: - ## Prior Art diff --git a/doc/cdc/mongodb.md b/doc/cdc/mongodb.md index 52c9898..07029e4 100644 --- a/doc/cdc/mongodb.md +++ b/doc/cdc/mongodb.md @@ -1,44 +1,53 @@ (mongodb-cdc)= # Relay MongoDB Change Stream into CrateDB -## About -[mongodb_cdc_cratedb.py] demonstrates a basic example program to relay event -records from [MongoDB Change Streams] into [CrateDB]. +:::{rubric} What's inside +::: + +Documentation and [example program][mongodb_cdc_cratedb.py] how to relay data +from MongoDB into [CrateDB], using [MongoDB Change Streams]. > Change streams allow applications to access real-time data changes without the prior > complexity and risk of manually tailing the oplog. Applications can use change streams > to subscribe to all data changes on a single collection, a database, or an entire > deployment, and immediately react to them. > -> - https://www.mongodb.com/docs/manual/changeStreams/ -> - https://www.mongodb.com/developer/languages/python/python-change-streams/ +> - [MongoDB Change Streams] +> - [Monitor Data with Change Streams] + +:::{note} +`commons-codec` includes `MongoDBFullLoadTranslator` and `MongoDBCDCTranslator`. +This document and example program is exclusively about the latter. +::: +## Prerequisites -## Services +Start services CrateDB and MongoDB. -### CrateDB -Start CrateDB. +:::{rubric} Start CrateDB +::: ```shell -docker run --rm -it --name=cratedb --publish=4200:4200 --env=CRATE_HEAP_SIZE=2g \ - crate:5.7 -Cdiscovery.type=single-node +docker run --rm --name=cratedb --publish=4200:4200 --env=CRATE_HEAP_SIZE=2g \ + docker.io/crate:latest '-Cdiscovery.type=single-node' ``` -### MongoDB -Start MongoDB. +:::{rubric} Start MongoDB +::: Please note that change streams are only available for replica sets and sharded clusters, so let's define a replica set by using the `--replSet rs-testdrive` option when starting the MongoDB server. ```shell -docker run -it --rm --name=mongodb --publish=27017:27017 \ - mongo:7 mongod --replSet rs-testdrive +docker run --rm --name=mongodb --publish=27017:27017 \ + docker.io/mongo:8 mongod --replSet rs-testdrive ``` Now, initialize the replica set, by using the `mongosh` command to invoke the `rs.initiate()` operation. ```shell export MONGODB_URL="mongodb://localhost/" -docker run -i --rm --network=host mongo:7 mongosh ${MONGODB_URL} < Example: [Kotori Sensor.Community integration] -- [Tasmota open source firmware for ESP devices]. +:Tasmota: + [Tasmota open source firmware for ESP devices]. +
Example: [Kotori Tasmota integration] -- Devices connected to [The Things Stack (TTS)] / [The Things Network (TTN)]. +:TTN: + Devices connected to [The Things Stack (TTS)] / [The Things Network (TTN)]. +
Example: [Kotori TTN/TTS integration] diff --git a/doc/transform.md b/doc/transform.md index 451062a..d172f25 100644 --- a/doc/transform.md +++ b/doc/transform.md @@ -1,8 +1,8 @@ -# Transform +# Transformations -Data transformations are now delegated to [tikray], an external library that -implements the same rule-based pipeline formerly shipped as **zyp**. -See the Tikray documentation for usage examples and migration tips. +Data transformations are delegated to Tikray, an external library that +implements a miniature yet elegant rule-based pipeline system. +See the [Tikray documentation] for usage examples and migration tips. -[tikray]: https://tikray.readthedocs.io/ +[Tikray documentation]: https://tikray.readthedocs.io/ diff --git a/pyproject.toml b/pyproject.toml index e80dded..bd78238 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -128,6 +128,7 @@ optional-dependencies.develop = [ optional-dependencies.doc = [ "furo==2025.9.25", # Check visited link style on newer versions! "myst-parser[linkify]>=0.18,<5", + "sphinx<8", "sphinx-autobuild==2021.3.14", # Newer versions stopped "watching" appropriately? "sphinx-copybutton", "sphinx-design-elements<1",