Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .readthedocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
version: 2

build:
os: "ubuntu-22.04"
os: "ubuntu-24.04"
tools:
python: "3.11"
python: "3.14"

python:
install:
Expand Down
19 changes: 7 additions & 12 deletions doc/cdc/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,27 +3,22 @@
`commons-codec` includes CDC -> SQL transformer components for AWS DMS,
DynamoDB, and MongoDB.

## DynamoDB
- Blog: [Replicating CDC Events from DynamoDB to CrateDB]
- Documentation: [DynamoDB CDC Relay for CrateDB]
:DynamoDB:
- Blog: [Replicating CDC Events from DynamoDB to CrateDB]
- Documentation: [DynamoDB CDC Relay for CrateDB]

## MongoDB
- Introduction: [](project:#mongodb-cdc)
- Documentation: [MongoDB CDC Relay for CrateDB]
:MongoDB:
- Usage guide: [](project:#mongodb-cdc)
- Documentation: [MongoDB CDC Relay for CrateDB]


```{toctree}
:hidden:

mongodb
MongoDB <mongodb>
```


:::{note}
Please note relevant components are still in their infancy (beta),
and need further curation and improvements.
:::


## Prior Art

Expand Down
40 changes: 25 additions & 15 deletions doc/cdc/mongodb.md
Original file line number Diff line number Diff line change
@@ -1,44 +1,53 @@
(mongodb-cdc)=
# Relay MongoDB Change Stream into CrateDB

## About
[mongodb_cdc_cratedb.py] demonstrates a basic example program to relay event
records from [MongoDB Change Streams] into [CrateDB].
:::{rubric} What's inside
:::

Documentation and [example program][mongodb_cdc_cratedb.py] how to relay data
from MongoDB into [CrateDB], using [MongoDB Change Streams].

> Change streams allow applications to access real-time data changes without the prior
> complexity and risk of manually tailing the oplog. Applications can use change streams
> to subscribe to all data changes on a single collection, a database, or an entire
> deployment, and immediately react to them.
>
> - https://www.mongodb.com/docs/manual/changeStreams/
> - https://www.mongodb.com/developer/languages/python/python-change-streams/
> - [MongoDB Change Streams]
> - [Monitor Data with Change Streams]

:::{note}
`commons-codec` includes `MongoDBFullLoadTranslator` and `MongoDBCDCTranslator`.
This document and example program is exclusively about the latter.
:::

## Prerequisites

## Services
Start services CrateDB and MongoDB.

### CrateDB
Start CrateDB.
:::{rubric} Start CrateDB
:::
```shell
docker run --rm -it --name=cratedb --publish=4200:4200 --env=CRATE_HEAP_SIZE=2g \
crate:5.7 -Cdiscovery.type=single-node
docker run --rm --name=cratedb --publish=4200:4200 --env=CRATE_HEAP_SIZE=2g \
docker.io/crate:latest '-Cdiscovery.type=single-node'
```

### MongoDB
Start MongoDB.
:::{rubric} Start MongoDB
:::
Please note that change streams are only available for replica sets and
sharded clusters, so let's define a replica set by using the
`--replSet rs-testdrive` option when starting the MongoDB server.
```shell
docker run -it --rm --name=mongodb --publish=27017:27017 \
mongo:7 mongod --replSet rs-testdrive
docker run --rm --name=mongodb --publish=27017:27017 \
docker.io/mongo:8 mongod --replSet rs-testdrive
```

Now, initialize the replica set, by using the `mongosh` command to invoke
the `rs.initiate()` operation.
```shell
export MONGODB_URL="mongodb://localhost/"
docker run -i --rm --network=host mongo:7 mongosh ${MONGODB_URL} <<EOF
docker run -i --rm --network=host docker.io/mongo:8 mongosh ${MONGODB_URL} <<EOF

disableTelemetry()
config = {
_id: "rs-testdrive",
members: [{ _id : 0, host : "localhost:27017"}]
Expand Down Expand Up @@ -100,3 +109,4 @@ time, but, again, it has not been correctly initialized.
[CrateDB]: https://github.com/crate/crate
[mongodb_cdc_cratedb.py]: https://github.com/daq-tools/commons-codec/raw/main/examples/mongodb_cdc_cratedb.py
[MongoDB Change Streams]: https://www.mongodb.com/docs/manual/changeStreams/
[Monitor Data with Change Streams]: https://www.mongodb.com/developer/languages/python/python-change-streams/
2 changes: 1 addition & 1 deletion doc/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
]

templates_path = ["_templates"]
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store", "readme.md"]


# -- Options for HTML output -------------------------------------------------
Expand Down
6 changes: 3 additions & 3 deletions doc/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,9 @@
| [Kotori]
| [LorryStream]

```{include} readme.md
:::{include} readme.md
:start-line: 11
```
:::


```{toctree}
Expand All @@ -33,7 +33,7 @@
:hidden:

cdc/index
decode
telemetry
transform
```

Expand Down
16 changes: 10 additions & 6 deletions doc/decode.md → doc/telemetry.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,24 @@
# Decoder Collection
# Telemetry decoders

`commons-codec` includes telemetry data decoders for individual popular sensor
appliances.

## Sensor.Community

- Air quality measurement appliances of [Sensor.Community].
:Sensor.Community:

Air quality measurement appliances of [Sensor.Community].
<br>
Example: [Kotori Sensor.Community integration]

- [Tasmota open source firmware for ESP devices].
:Tasmota:

[Tasmota open source firmware for ESP devices].
<br>
Example: [Kotori Tasmota integration]

- Devices connected to [The Things Stack (TTS)] / [The Things Network (TTN)].
:TTN:

Devices connected to [The Things Stack (TTS)] / [The Things Network (TTN)].
<br>
Example: [Kotori TTN/TTS integration]


Expand Down
10 changes: 5 additions & 5 deletions doc/transform.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Transform
# Transformations

Data transformations are now delegated to [tikray], an external library that
implements the same rule-based pipeline formerly shipped as **zyp**.
See the Tikray documentation for usage examples and migration tips.
Data transformations are delegated to Tikray, an external library that
implements a miniature yet elegant rule-based pipeline system.
See the [Tikray documentation] for usage examples and migration tips.


[tikray]: https://tikray.readthedocs.io/
[Tikray documentation]: https://tikray.readthedocs.io/
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ optional-dependencies.develop = [
optional-dependencies.doc = [
"furo==2025.9.25", # Check visited link style on newer versions!
"myst-parser[linkify]>=0.18,<5",
"sphinx<8",
"sphinx-autobuild==2021.3.14", # Newer versions stopped "watching" appropriately?
"sphinx-copybutton",
"sphinx-design-elements<1",
Expand Down