Rust POC #2689
Closed
Rust POC #2689
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Create Rust project with PyO3 and bson crate dependencies - Implement encode/decode for basic BSON types - Add benchmark functions for performance testing - Create comparison benchmarks showing 2.89x average speedup - Document findings in RUST_SPIKE_RESULTS.md - Add build and test scripts Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
- Create examples_rust.py demonstrating all functionality - Add RUST_DECISION_MATRIX.md for stakeholder review - Document ROI, risks, and migration strategy - All examples pass with cross-compatibility verified Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
- Extract test document creation into helper functions - Remove duplicate doc! macros in benchmark functions - Improve code maintainability per code review feedback Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
- Add IMPLEMENTATION_SUMMARY.md with complete findings - Document all deliverables and recommendations - Spike complete with 2.89x average performance improvement - Ready for stakeholder review Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
- Create QUICK_START.md with role-specific guidance - Include TL;DR sections for managers, developers, security, ops, PM, architects - Provide clear next steps and decision points - Ready for stakeholder presentation Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
- Create RUST_SPIKE_README.md as entry point - Include quick links to all documentation - Provide clear getting started instructions - Spike fully documented and ready for review Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
…-extensions Investigate Rust for C Extensions - Performance Spike
Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
…better docs Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
…ry-handling Add codec_options and Binary subtype support to Rust BSON extension
Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
…ify regex flag documentation Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
…on-types Implement DateTime, ObjectId, Regex, and Timestamp BSON types in Rust extension
Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
…tions Replace Python CI workflows with Rust CI testing
- Updated Cargo.toml to use PyO3 0.27 - Migrated code to PyO3 0.27 API changes: - Removed _bound suffix from methods (import_bound -> import, new_bound -> new, etc.) - Updated into_py_dict to handle Result return type - Changed to_object calls to use PyInt, PyFloat, PyString, PyBool types - Changed downcast to cast as recommended - Updated PyObject type to Py<PyAny> - All builds and tests pass successfully Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
Remove .clone() from PyInt, PyFloat, and PyString conversions as these types already return owned objects. Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
Fix macOS CI: Upgrade PyO3 0.22 → 0.27 for Python 3.14 support
Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
…on-compatibility Pin Python 3.13 in Rust CI to fix macOS ARM64 linking failures
Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
- Support non-dict mapping types (SON, OrderedDict) via items() method - Add UUID encoding/decoding support (subtypes 3 and 4) - Support compiled regex patterns (re.Pattern objects) - Add MaxKey/MinKey support - Allow bytes as dictionary keys (auto-convert to UTF-8 strings) - Support bytearray in addition to bytes for data parameter - Respect tz_aware codec option for naive vs timezone-aware datetimes - Return Int64 objects instead of plain Python ints for BSON Int64 - Add Code type support (with and without scope) - Add Decimal128 support - Ensure _id field is always first in encoded documents - Add Symbol and Undefined type handling in decoder Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
…time handling - Make regex Pattern detection more robust across Python versions by checking if name starts with 'Pattern' - Add hasattr checks for pattern and flags attributes for extra safety - Fix UUID representation logic to properly check enum values (decode as UUID when != UNSPECIFIED/0) - Replace deprecated utcfromtimestamp with fromtimestamp + timezone.utc for naive datetimes - Add detailed comments explaining PyMongo UuidRepresentation enum values Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
…SON extension - Added DBRef encoding/decoding with support for kwargs - Added Int64 type marker support for explicit Int64 encoding - Fixed _id field ordering (only at top level, not nested) - Improved decode_bson to accept buffer protocol objects - Added DBPointer to DBRef decoding - Test progress: 53/88 passing (was 49/88) Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
…rror handling Co-authored-by: aclark4life <72164+aclark4life@users.noreply.github.com>
Enable Rust BSON extension as C extension replacement
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| python-version: '3.13' | ||
|
|
||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@stable |
Check failure
Code scanning / zizmor
unpinned action reference Error
| python-version: '3.13' | ||
|
|
||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@stable |
Check failure
Code scanning / zizmor
unpinned action reference Error
| python-version: '3.13' | ||
|
|
||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@stable |
Check failure
Code scanning / zizmor
unpinned action reference Error
| python-version: '3.13' | ||
|
|
||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@stable |
Check failure
Code scanning / zizmor
unpinned action reference Error
Comment on lines
+19
to
+67
| test: | ||
| name: Test - ${{ matrix.os }} | ||
| runs-on: ${{ matrix.os }} | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| os: [ubuntu-latest, macos-latest, windows-latest] | ||
|
|
||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Set up Python 3.13 | ||
| uses: actions/setup-python@v6 | ||
| with: | ||
| python-version: '3.13' | ||
|
|
||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@stable | ||
|
|
||
| - name: Cache cargo registry | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: ~/.cargo/registry | ||
| key: ${{ runner.os }}-cargo-registry-${{ hashFiles('**/Cargo.lock') }} | ||
|
|
||
| - name: Cache cargo index | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: ~/.cargo/git | ||
| key: ${{ runner.os }}-cargo-index-${{ hashFiles('**/Cargo.lock') }} | ||
|
|
||
| - name: Cache cargo build | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: rust/target | ||
| key: ${{ runner.os }}-cargo-build-target-${{ hashFiles('**/Cargo.lock') }} | ||
|
|
||
| - name: Build | ||
| working-directory: ./rust | ||
| run: cargo build --verbose | ||
|
|
||
| - name: Run tests | ||
| working-directory: ./rust | ||
| run: cargo test --verbose | ||
|
|
||
| # Linting with clippy |
Check warning
Code scanning / zizmor
overly broad permissions Warning
Comment on lines
+68
to
+110
| clippy: | ||
| name: Clippy (linting) | ||
| runs-on: ubuntu-latest | ||
|
|
||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Set up Python 3.13 | ||
| uses: actions/setup-python@v6 | ||
| with: | ||
| python-version: '3.13' | ||
|
|
||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@stable | ||
| with: | ||
| components: clippy | ||
|
|
||
| - name: Cache cargo registry | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: ~/.cargo/registry | ||
| key: ${{ runner.os }}-cargo-registry-${{ hashFiles('**/Cargo.lock') }} | ||
|
|
||
| - name: Cache cargo index | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: ~/.cargo/git | ||
| key: ${{ runner.os }}-cargo-index-${{ hashFiles('**/Cargo.lock') }} | ||
|
|
||
| - name: Cache cargo build | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: rust/target | ||
| key: ${{ runner.os }}-cargo-build-target-${{ hashFiles('**/Cargo.lock') }} | ||
|
|
||
| - name: Run clippy | ||
| working-directory: ./rust | ||
| run: cargo clippy --all-targets --all-features -- -D warnings | ||
|
|
||
| # Formatting check |
Check warning
Code scanning / zizmor
overly broad permissions Warning
Comment on lines
+111
to
+135
| fmt: | ||
| name: Rustfmt (formatting) | ||
| runs-on: ubuntu-latest | ||
|
|
||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Set up Python 3.13 | ||
| uses: actions/setup-python@v6 | ||
| with: | ||
| python-version: '3.13' | ||
|
|
||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@stable | ||
| with: | ||
| components: rustfmt | ||
|
|
||
| - name: Check formatting | ||
| working-directory: ./rust | ||
| run: cargo fmt --all -- --check | ||
|
|
||
| # Build in release mode to ensure optimized builds work |
Check warning
Code scanning / zizmor
overly broad permissions Warning
Comment on lines
+20
to
+68
| name: Test - ${{ matrix.os }} | ||
| runs-on: ${{ matrix.os }} | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| os: [ubuntu-latest, macos-latest, windows-latest] | ||
|
|
||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Set up Python 3.13 | ||
| uses: actions/setup-python@v6 | ||
| with: | ||
| python-version: '3.13' | ||
|
|
||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@stable | ||
|
|
||
| - name: Cache cargo registry | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: ~/.cargo/registry | ||
| key: ${{ runner.os }}-cargo-registry-${{ hashFiles('**/Cargo.lock') }} | ||
|
|
||
| - name: Cache cargo index | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: ~/.cargo/git | ||
| key: ${{ runner.os }}-cargo-index-${{ hashFiles('**/Cargo.lock') }} | ||
|
|
||
| - name: Cache cargo build | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: rust/target | ||
| key: ${{ runner.os }}-cargo-build-target-${{ hashFiles('**/Cargo.lock') }} | ||
|
|
||
| - name: Build | ||
| working-directory: ./rust | ||
| run: cargo build --verbose | ||
|
|
||
| - name: Run tests | ||
| working-directory: ./rust | ||
| run: cargo test --verbose | ||
|
|
||
| # Linting with clippy | ||
| clippy: |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
| python-version: '3.13' | ||
|
|
||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@stable |
Check warning
Code scanning / CodeQL
Unpinned tag for a non-immutable Action in workflow Medium
Comment on lines
+69
to
+111
| name: Clippy (linting) | ||
| runs-on: ubuntu-latest | ||
|
|
||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Set up Python 3.13 | ||
| uses: actions/setup-python@v6 | ||
| with: | ||
| python-version: '3.13' | ||
|
|
||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@stable | ||
| with: | ||
| components: clippy | ||
|
|
||
| - name: Cache cargo registry | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: ~/.cargo/registry | ||
| key: ${{ runner.os }}-cargo-registry-${{ hashFiles('**/Cargo.lock') }} | ||
|
|
||
| - name: Cache cargo index | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: ~/.cargo/git | ||
| key: ${{ runner.os }}-cargo-index-${{ hashFiles('**/Cargo.lock') }} | ||
|
|
||
| - name: Cache cargo build | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: rust/target | ||
| key: ${{ runner.os }}-cargo-build-target-${{ hashFiles('**/Cargo.lock') }} | ||
|
|
||
| - name: Run clippy | ||
| working-directory: ./rust | ||
| run: cargo clippy --all-targets --all-features -- -D warnings | ||
|
|
||
| # Formatting check | ||
| fmt: |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
| python-version: '3.13' | ||
|
|
||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@stable |
Check warning
Code scanning / CodeQL
Unpinned tag for a non-immutable Action in workflow Medium
Comment on lines
+112
to
+136
| name: Rustfmt (formatting) | ||
| runs-on: ubuntu-latest | ||
|
|
||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Set up Python 3.13 | ||
| uses: actions/setup-python@v6 | ||
| with: | ||
| python-version: '3.13' | ||
|
|
||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@stable | ||
| with: | ||
| components: rustfmt | ||
|
|
||
| - name: Check formatting | ||
| working-directory: ./rust | ||
| run: cargo fmt --all -- --check | ||
|
|
||
| # Build in release mode to ensure optimized builds work | ||
| build-release: |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
| python-version: '3.13' | ||
|
|
||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@stable |
Check warning
Code scanning / CodeQL
Unpinned tag for a non-immutable Action in workflow Medium
Comment on lines
+137
to
+174
| name: Build (release mode) | ||
| runs-on: ubuntu-latest | ||
|
|
||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v6 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Set up Python 3.13 | ||
| uses: actions/setup-python@v6 | ||
| with: | ||
| python-version: '3.13' | ||
|
|
||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@stable | ||
|
|
||
| - name: Cache cargo registry | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: ~/.cargo/registry | ||
| key: ${{ runner.os }}-cargo-registry-${{ hashFiles('**/Cargo.lock') }} | ||
|
|
||
| - name: Cache cargo index | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: ~/.cargo/git | ||
| key: ${{ runner.os }}-cargo-index-${{ hashFiles('**/Cargo.lock') }} | ||
|
|
||
| - name: Cache cargo build | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: rust/target | ||
| key: ${{ runner.os }}-cargo-build-target-release-${{ hashFiles('**/Cargo.lock') }} | ||
|
|
||
| - name: Build release | ||
| working-directory: ./rust | ||
| run: cargo build --release --verbose |
Check warning
Code scanning / CodeQL
Workflow does not contain permissions Medium
| python-version: '3.13' | ||
|
|
||
| - name: Install Rust toolchain | ||
| uses: dtolnay/rust-toolchain@stable |
Check warning
Code scanning / CodeQL
Unpinned tag for a non-immutable Action in workflow Medium
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[JIRA TICKET]
Changes in this PR
Test Plan
Checklist
Checklist for Author
Checklist for Reviewer