From 41f25cefc349a9d2add7d99bc94945d123104ca1 Mon Sep 17 00:00:00 2001 From: Koichi Nishitani Date: Thu, 23 Apr 2026 22:22:27 +0900 Subject: [PATCH 01/33] Upgrade to version 2.0.0 (#13) * Perf: Sync from enterprise 4e91aee (via .ignore) on 2026-04-17 (#7) * Perf: Sync from enterprise 4e91aee (via .ignore) on 2026-04-17 BREAKING CHANGE: Bug fixes for Data Quality and addition of Data Product Hub modules Signed-off-by: Koichi Nishitani * fix relative path of Actions scripts Signed-off-by: Koichi Nishitani * fix script condition Signed-off-by: Koichi Nishitani * fix script condition again Signed-off-by: Koichi Nishitani * add missing Makefile and fix documentation Signed-off-by: Koichi Nishitani * add dq tests and upgrade python Signed-off-by: Koichi Nishitani * try to extend build timeout Signed-off-by: Koichi Nishitani * upgrade to python 3.10 as 3.9 is EOL Signed-off-by: Koichi Nishitani * modify constraint_model to use custom StrEnum in python 3.10 Signed-off-by: Koichi Nishitani * add pylint Signed-off-by: Koichi Nishitani * skip linting until ready Signed-off-by: Koichi Nishitani * make sure to build before checking Signed-off-by: Koichi Nishitani --------- Signed-off-by: Koichi Nishitani * chore: Sync with enterprise (manual) on 2026-04-21 (#8) update build script to version Signed-off-by: Koichi Nishitani * chore: fix semantic release build (#9) Signed-off-by: Koichi Nishitani * chore: use push for semantic-release and update action versions (#10) Signed-off-by: Koichi Nishitani * Update version 1.0.0 -> 2.0.0 * chore(release): 2.0.0-rc.1 release notes # [2.0.0-rc.1](https://github.com/IBM/data-intelligence-sdk/compare/v1.0.0...v2.0.0-rc.1) (2026-04-22) ### Perf * Sync from enterprise 4e91aee (via .ignore) on 2026-04-17 ([#7](https://github.com/IBM/data-intelligence-sdk/issues/7)) ([eeeffba](https://github.com/IBM/data-intelligence-sdk/commit/eeeffba17ce0e399446099d47352f7738259feae)) ### BREAKING CHANGES * Bug fixes for Data Quality and addition of Data Product Hub modules Signed-off-by: Koichi Nishitani * fix relative path of Actions scripts Signed-off-by: Koichi Nishitani * fix script condition Signed-off-by: Koichi Nishitani * fix script condition again Signed-off-by: Koichi Nishitani * add missing Makefile and fix documentation Signed-off-by: Koichi Nishitani * add dq tests and upgrade python Signed-off-by: Koichi Nishitani * try to extend build timeout Signed-off-by: Koichi Nishitani * upgrade to python 3.10 as 3.9 is EOL Signed-off-by: Koichi Nishitani * modify constraint_model to use custom StrEnum in python 3.10 Signed-off-by: Koichi Nishitani * add pylint Signed-off-by: Koichi Nishitani * skip linting until ready Signed-off-by: Koichi Nishitani * make sure to build before checking Signed-off-by: Koichi Nishitani * chore: enable rc versions (#12) Signed-off-by: Koichi Nishitani --------- Signed-off-by: Koichi Nishitani Co-authored-by: semantic-release-bot --- .bumpversion.toml | 21 +- .github/workflows/build.yaml | 92 +- .github/workflows/deploy-package.yaml | 104 + .github/workflows/docs.yml | 16 +- .github/workflows/verify.yaml | 67 + .pylintrc | 498 + .releaserc | 2 +- .secrets.baseline | 188 +- CHANGELOG.md | 58 + Makefile.build | 58 + README.md | 95 +- docs/README.md | 2 +- docs/chapters/01_welcome/installation.rst | 6 +- docs/chapters/02_overview/faq.rst | 2 +- docs/chapters/02_overview/known_issues.rst | 2 +- docs/conf.py | 4 +- examples/__init__.py | 19 + examples/assets_usage.py | 72 +- examples/auth_provider_usage.py | 90 +- examples/basic_usage.py | 54 +- examples/checks_usage.py | 97 +- examples/consolidation_usage.py | 112 +- examples/data_product_recommender_example.py | 136 + examples/dimensions_usage.py | 56 +- examples/dq_workflow_usage.py | 80 +- examples/end_to_end_example.py | 404 + examples/glossary_usage.py | 86 +- examples/issues_usage.py | 204 +- examples/odcs_generator_example.py | 217 + .../odcs_generator_informatica_example.py | 415 + examples/pandas_dataframe_usage.py | 56 +- examples/spark_dataframe_usage.py | 88 +- examples/test_dph_v1_examples.py | 1600 ++ requirements-dev.txt | 17 + requirements.txt | 12 +- setup.py | 23 +- src/wxdi/__init__.py | 9 +- src/wxdi/data_product_recommender/README.md | 218 + src/wxdi/data_product_recommender/__init__.py | 24 + src/wxdi/data_product_recommender/base.py | 37 + src/wxdi/data_product_recommender/cli.py | 152 + .../data_product_recommender/platforms.py | 167 + .../data_product_recommender/recommender.py | 1292 ++ src/wxdi/dph_services/README.md | 458 + src/wxdi/dph_services/__init__.py | 22 + src/wxdi/dph_services/common.py | 75 + src/wxdi/dph_services/common_constants.py | 48 + src/wxdi/dph_services/dph_v1.py | 12411 ++++++++++++++ src/wxdi/dph_services/version.py | 20 + src/wxdi/dq_validator/issue_reporting.py | 581 +- src/wxdi/dq_validator/provider/checks.py | 117 +- .../dq_validator/provider/constraint_model.py | 8 +- src/wxdi/dq_validator/provider/issues.py | 314 +- .../dq_validator/provider/response_model.py | 5 +- .../README-GENERATE-ODCS-SCRIPT.md | 1095 ++ src/wxdi/odcs_generator/__init__.py | 55 + .../generate_odcs_from_collibra.py | 765 + .../generate_odcs_from_informatica.py | 500 + src/wxdi/version.py | 2 +- tests/data/data_asset_response.json | 739 + tests/data/term_draft.json | 54 + tests/data/term_latest_version.json | 31 + tests/data/term_response.json | 117 + tests/src/__init__.py | 19 + tests/src/auth/__init__.py | 18 + tests/src/auth/test_auth_config.py | 275 + tests/src/auth/test_auth_provider.py | 309 + .../src/auth/test_gov_cloud_authenticator.py | 259 + .../src/auth/test_gov_cloud_token_manager.py | 403 + .../src/data_product_recommender/__init__.py | 19 + .../test_data_product_recommender.py | 613 + .../test_data_product_recommender_cli.py | 306 + .../test_data_product_recommender_conftest.py | 144 + .../test_data_product_recommender_parsers.py | 236 + tests/src/dph_services/__init__.py | 19 + tests/src/dph_services/test_common.py | 50 + tests/src/dph_services/test_dph_v1.py | 14205 ++++++++++++++++ tests/src/dq_validator/__init__.py | 20 + .../src/dq_validator/provider/test_assets.py | 445 + tests/src/dq_validator/provider/test_cams.py | 513 + .../src/dq_validator/provider/test_checks.py | 796 + .../src/dq_validator/provider/test_config.py | 284 + .../dq_validator/provider/test_dimensions.py | 227 + .../dq_validator/provider/test_dq_search.py | 279 + .../dq_validator/provider/test_glossary.py | 697 + .../src/dq_validator/provider/test_issues.py | 1129 ++ .../provider/test_thread_safety.py | 129 + tests/src/dq_validator/test_case_check.py | 258 + .../src/dq_validator/test_comparison_check.py | 491 + .../dq_validator/test_completeness_check.py | 131 + .../src/dq_validator/test_data_asset_model.py | 187 + tests/src/dq_validator/test_datatype_check.py | 424 + tests/src/dq_validator/test_format_check.py | 357 + tests/src/dq_validator/test_integration.py | 466 + .../src/dq_validator/test_issue_reporting.py | 1609 ++ tests/src/dq_validator/test_length_check.py | 307 + .../src/dq_validator/test_pandas_validator.py | 535 + tests/src/dq_validator/test_range_check.py | 380 + tests/src/dq_validator/test_regex_check.py | 280 + .../dq_validator/test_result_consolidator.py | 471 + tests/src/dq_validator/test_rule_loader.py | 559 + .../src/dq_validator/test_spark_validator.py | 805 + .../dq_validator/test_valid_values_check.py | 394 + tests/src/dq_validator/test_version.py | 35 + tests/src/integration/README.md | 221 + tests/src/integration/__init__.py | 19 + .../src/integration/initial_setup_service.py | 137 + ...st_data_product_recommender_integration.py | 373 + .../test_odcs_generator_collibra.py | 611 + .../test_odcs_generator_informatica.py | 389 + tests/src/odcs_generator/__init__.py | 19 + .../test_odcs_generator_collibra.py | 739 + .../test_odcs_generator_informatica.py | 1060 ++ 113 files changed, 56213 insertions(+), 807 deletions(-) create mode 100644 .github/workflows/deploy-package.yaml create mode 100644 .github/workflows/verify.yaml create mode 100644 .pylintrc create mode 100644 Makefile.build create mode 100644 examples/__init__.py create mode 100644 examples/data_product_recommender_example.py create mode 100644 examples/end_to_end_example.py create mode 100644 examples/odcs_generator_example.py create mode 100644 examples/odcs_generator_informatica_example.py create mode 100644 examples/test_dph_v1_examples.py create mode 100644 requirements-dev.txt create mode 100644 src/wxdi/data_product_recommender/README.md create mode 100644 src/wxdi/data_product_recommender/__init__.py create mode 100644 src/wxdi/data_product_recommender/base.py create mode 100644 src/wxdi/data_product_recommender/cli.py create mode 100644 src/wxdi/data_product_recommender/platforms.py create mode 100644 src/wxdi/data_product_recommender/recommender.py create mode 100644 src/wxdi/dph_services/README.md create mode 100644 src/wxdi/dph_services/__init__.py create mode 100644 src/wxdi/dph_services/common.py create mode 100644 src/wxdi/dph_services/common_constants.py create mode 100644 src/wxdi/dph_services/dph_v1.py create mode 100644 src/wxdi/dph_services/version.py create mode 100644 src/wxdi/odcs_generator/README-GENERATE-ODCS-SCRIPT.md create mode 100644 src/wxdi/odcs_generator/__init__.py create mode 100644 src/wxdi/odcs_generator/generate_odcs_from_collibra.py create mode 100644 src/wxdi/odcs_generator/generate_odcs_from_informatica.py create mode 100644 tests/data/data_asset_response.json create mode 100644 tests/data/term_draft.json create mode 100644 tests/data/term_latest_version.json create mode 100644 tests/data/term_response.json create mode 100644 tests/src/__init__.py create mode 100644 tests/src/auth/__init__.py create mode 100644 tests/src/auth/test_auth_config.py create mode 100644 tests/src/auth/test_auth_provider.py create mode 100644 tests/src/auth/test_gov_cloud_authenticator.py create mode 100644 tests/src/auth/test_gov_cloud_token_manager.py create mode 100644 tests/src/data_product_recommender/__init__.py create mode 100644 tests/src/data_product_recommender/test_data_product_recommender.py create mode 100644 tests/src/data_product_recommender/test_data_product_recommender_cli.py create mode 100644 tests/src/data_product_recommender/test_data_product_recommender_conftest.py create mode 100644 tests/src/data_product_recommender/test_data_product_recommender_parsers.py create mode 100644 tests/src/dph_services/__init__.py create mode 100644 tests/src/dph_services/test_common.py create mode 100644 tests/src/dph_services/test_dph_v1.py create mode 100644 tests/src/dq_validator/__init__.py create mode 100644 tests/src/dq_validator/provider/test_assets.py create mode 100644 tests/src/dq_validator/provider/test_cams.py create mode 100644 tests/src/dq_validator/provider/test_checks.py create mode 100644 tests/src/dq_validator/provider/test_config.py create mode 100644 tests/src/dq_validator/provider/test_dimensions.py create mode 100644 tests/src/dq_validator/provider/test_dq_search.py create mode 100644 tests/src/dq_validator/provider/test_glossary.py create mode 100644 tests/src/dq_validator/provider/test_issues.py create mode 100644 tests/src/dq_validator/provider/test_thread_safety.py create mode 100644 tests/src/dq_validator/test_case_check.py create mode 100644 tests/src/dq_validator/test_comparison_check.py create mode 100644 tests/src/dq_validator/test_completeness_check.py create mode 100644 tests/src/dq_validator/test_data_asset_model.py create mode 100644 tests/src/dq_validator/test_datatype_check.py create mode 100644 tests/src/dq_validator/test_format_check.py create mode 100644 tests/src/dq_validator/test_integration.py create mode 100644 tests/src/dq_validator/test_issue_reporting.py create mode 100644 tests/src/dq_validator/test_length_check.py create mode 100644 tests/src/dq_validator/test_pandas_validator.py create mode 100644 tests/src/dq_validator/test_range_check.py create mode 100644 tests/src/dq_validator/test_regex_check.py create mode 100644 tests/src/dq_validator/test_result_consolidator.py create mode 100644 tests/src/dq_validator/test_rule_loader.py create mode 100644 tests/src/dq_validator/test_spark_validator.py create mode 100644 tests/src/dq_validator/test_valid_values_check.py create mode 100644 tests/src/dq_validator/test_version.py create mode 100644 tests/src/integration/README.md create mode 100644 tests/src/integration/__init__.py create mode 100644 tests/src/integration/initial_setup_service.py create mode 100644 tests/src/integration/test_data_product_recommender_integration.py create mode 100644 tests/src/integration/test_odcs_generator_collibra.py create mode 100644 tests/src/integration/test_odcs_generator_informatica.py create mode 100644 tests/src/odcs_generator/__init__.py create mode 100644 tests/src/odcs_generator/test_odcs_generator_collibra.py create mode 100644 tests/src/odcs_generator/test_odcs_generator_informatica.py diff --git a/.bumpversion.toml b/.bumpversion.toml index 9c72852..86b135e 100644 --- a/.bumpversion.toml +++ b/.bumpversion.toml @@ -12,9 +12,28 @@ # limitations under the License. [tool.bumpversion] -current_version = "1.0.0" +current_version = "2.0.0-rc.1" commit = true message = "Update version {current_version} -> {new_version}" +parse = """(?x) + (?P0|[1-9]\\d*)\\. + (?P0|[1-9]\\d*)\\. + (?P0|[1-9]\\d*) + (?: + - # dash separator for pre-release section + (?Prc|stable) # pre-release label + \\.(?P0|[1-9]\\d*) # pre-release version number + )? # pre-release section is optional +""" + +serialize = [ + "{major}.{minor}.{patch}-{pre_label}.{pre_num}", + "{major}.{minor}.{patch}", +] + +[tool.bumpversion.parts.pre_label] +values = ["rc", "stable"] +optional_value = "stable" [[tool.bumpversion.files]] filename = "src/wxdi/version.py" diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 17c7e1c..ef4d3db 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -16,80 +16,37 @@ on: push: branches: - main - tags: ["*"] # Required to trigger PyPI job + - sandbox pull_request: - branches: ["**"] + branches: + - main + - sandbox workflow_dispatch: jobs: - detect-secrets: - if: "!contains(github.event.head_commit.message, '[skip ci]')" - name: detect-secrets - runs-on: ubuntu-latest - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Setup Python - uses: actions/setup-python@v5 - with: - python-version: "3.14" - - - name: Install detect-secrets - run: | - pip install --upgrade "git+https://github.com/ibm/detect-secrets.git@master#egg=detect-secrets" - - - name: Run detect-secrets - run: | - detect-secrets scan --update .secrets.baseline - detect-secrets -v audit --report --fail-on-unaudited --fail-on-live --fail-on-audited-real .secrets.baseline - - build: - if: "!contains(github.event.head_commit.message, '[skip ci]')" - name: build (python ${{ matrix.python-version }}) - needs: detect-secrets - runs-on: ubuntu-latest - strategy: - matrix: - python-version: ["3.9", "3.10", "3.11"] - - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v5 - with: - python-version: ${{ matrix.python-version }} - - - name: Install Python dependencies - run: | - python -m pip install --upgrade pip build twine - - - name: Build & Verify - run: | - python -m build - python -m twine check dist/* + build-and-verify: + uses: ./.github/workflows/verify.yaml publish-release: - if: "github.ref_name == 'main' && github.event_name != 'pull_request'" - needs: build + if: contains(fromJSON('["main", "sandbox"]'), github.ref_name) && (github.event_name != 'pull_request') + needs: build-and-verify name: semantic-release runs-on: ubuntu-latest steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@v6 with: persist-credentials: false fetch-depth: 0 + ref: ${{ github.ref }} - name: Setup Node.js - uses: actions/setup-node@v4 + uses: actions/setup-node@v6 with: node-version: 24 - name: Set up Python - uses: actions/setup-python@v5 + uses: actions/setup-python@v6 with: python-version: "3.14" @@ -102,28 +59,3 @@ jobs: env: GITHUB_TOKEN: ${{ secrets.GH_TOKEN }} run: npm run semantic-release - - deploy-pypi: - if: startsWith(github.ref, 'refs/tags/') - name: deploy-pypi - runs-on: ubuntu-latest - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Set up Python - uses: actions/setup-python@v5 - with: - python-version: "3.14" - - - name: Install Build Dependencies - run: | - python -m pip install --upgrade pip setuptools wheel build twine - - - name: Build and Publish to PyPI - env: - TWINE_USERNAME: __token__ - TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }} - run: | - python -m build - twine upload dist/* diff --git a/.github/workflows/deploy-package.yaml b/.github/workflows/deploy-package.yaml new file mode 100644 index 0000000..4a8dced --- /dev/null +++ b/.github/workflows/deploy-package.yaml @@ -0,0 +1,104 @@ +# Copyright 2026 IBM Corporation +# Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +name: Deploy package + +on: + push: + tags: ['**'] + workflow_dispatch: + +jobs: + build: + uses: ./.github/workflows/verify.yaml + + deploy-pypi: + if: startsWith(github.ref, 'refs/tags/') && !contains(github.ref, '-rc.') + name: deploy-pypi + runs-on: ubuntu-latest + needs: build + steps: + - name: Checkout repository + uses: actions/checkout@v6 + + - name: Set up Python + uses: actions/setup-python@v6 + with: + python-version: "3.14" + + - name: Install Build Dependencies + run: | + python -m pip install --upgrade pip setuptools wheel build twine + + - name: Build and Publish to PyPI + env: + TWINE_USERNAME: __token__ + TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }} + run: | + python -m build + twine upload dist/* + + deploy-testpypi: + if: startsWith(github.ref, 'refs/tags/') && contains(github.ref, '-rc.') + name: deploy-testpypi + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v6 + with: + ref: 'refs/heads/sandbox' + + - name: Set up Python + uses: actions/setup-python@v6 + with: + python-version: "3.14" + + - name: Install Build Dependencies + run: | + python -m pip install --upgrade pip setuptools wheel build twine + + - name: Build and Publish to PyPI + env: + TWINE_USERNAME: __token__ + TWINE_PASSWORD: ${{ secrets.TESTPYPI_TOKEN }} + run: | + python -m build + twine upload --repository testpypi dist/* + + build-documentation: + if: startsWith(github.ref, 'refs/tags/') && !contains(github.ref, '-rc.') + name: Build and deploy documentation + uses: ./.github/workflows/docs.yml + +# update-sandbox: +# if: startsWith(github.ref, 'refs/tags/') && !contains(github.ref, '-rc.') +# name: Update sandbox branch +# needs: build-documentation +# runs-on: ubuntu-latest +# steps: +# - name: Checkout main +# uses: actions/checkout@v6 +# with: +# persist-credentials: false +# fetch-depth: 0 +# ref: 'refs/heads/main' +# +# - name: Also get the sandbox branch +# run: | +# git remote set-branches --add origin sandbox +# git fetch +# - name: Checkout main +# run: git checkout sandbox +# - name: Merge main into sandbox +# run: git merge --signoff main +# - name: Push changes to sandbox +# run: git push -v origin diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 2032a4e..a22c199 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -13,14 +13,12 @@ name: Build and Deploy Documentation on: - push: - branches: - - main - tags: ['*'] pull_request: branches: - main + - sandbox workflow_dispatch: + workflow_call: permissions: contents: read @@ -38,10 +36,10 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@v6 - name: Set up Python - uses: actions/setup-python@v5 + uses: actions/setup-python@v6 with: python-version: '3.14' cache: 'pip' @@ -61,13 +59,13 @@ jobs: sphinx-build -b html -W --keep-going . _build/html - name: Upload artifact - uses: actions/upload-pages-artifact@v3 + uses: actions/upload-pages-artifact@v5 with: path: docs/_build/html deploy: # Only deploy on push to main/master, not on PRs - if: contains(fromJSON('["push", "workflow_dispatch"]'), github.event_name) && github.ref == 'refs/heads/main' + if: github.event_name == 'push' && github.ref == 'refs/heads/main' environment: name: github-pages @@ -79,6 +77,6 @@ jobs: steps: - name: Deploy to GitHub Pages id: deployment - uses: actions/deploy-pages@v4 + uses: actions/deploy-pages@v5 # Made with Bob diff --git a/.github/workflows/verify.yaml b/.github/workflows/verify.yaml new file mode 100644 index 0000000..3386a57 --- /dev/null +++ b/.github/workflows/verify.yaml @@ -0,0 +1,67 @@ +# Copyright 2026 IBM Corporation +# Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +name: verify + +on: + workflow_call: + +jobs: + detect-secrets: + if: "!contains(github.event.head_commit.message, '[skip ci]')" + name: detect-secrets + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v6 + + - name: Setup Python + uses: actions/setup-python@v6 + with: + python-version: "3.14" + + - name: Install detect-secrets + run: | + pip install --upgrade "git+https://github.com/ibm/detect-secrets.git@master#egg=detect-secrets" + + - name: Run detect-secrets + run: | + detect-secrets scan --update .secrets.baseline + detect-secrets -v audit --report --fail-on-unaudited --fail-on-live --fail-on-audited-real .secrets.baseline + + build: + if: "!contains(github.event.head_commit.message, '[skip ci]')" + name: build (python ${{ matrix.python-version }}) + needs: detect-secrets + runs-on: ubuntu-latest + strategy: + matrix: + python-version: ["3.10", "3.11", "3.12"] + + steps: + - name: Checkout repository + uses: actions/checkout@v6 + + - name: Set up Python ${{ matrix.python-version }} + uses: actions/setup-python@v6 + with: + python-version: ${{ matrix.python-version }} + + - name: Install Python dependencies + run: | + python -m pip install --upgrade pip build twine + + - name: Build & Verify + run: | + make -f Makefile.build ci + python -m build + python -m twine check dist/* diff --git a/.pylintrc b/.pylintrc new file mode 100644 index 0000000..3edec5a --- /dev/null +++ b/.pylintrc @@ -0,0 +1,498 @@ +[MASTER] + +# Use multiple processes to speed up Pylint. Specifying 0 will auto-detect the +# number of processors available to use. +jobs=1 + +# Control the amount of potential inferred values when inferring a single +# object. This can help the performance when dealing with large functions or +# complex, nested conditions. +limit-inference-results=100 + +# List of plugins (as comma separated values of python module names) to load, +# usually to register additional checkers. +#load-plugins= + +# Pickle collected data for later comparisons. +persistent=yes + +# Specify a configuration file. +#rcfile= + +# When enabled, pylint would attempt to guess common misconfiguration and emit +# user-friendly hints instead of false-positive error messages. +suggestion-mode=yes + +# Allow loading of arbitrary C extensions. Extensions are imported into the +# active Python interpreter and may run arbitrary code. +#unsafe-load-any-extension=no + + +[MESSAGES CONTROL] + +# Only show warnings with the listed confidence levels. Leave empty to show +# all. Valid levels: HIGH, INFERENCE, INFERENCE_FAILURE, UNDEFINED. +confidence=HIGH + +# Disable the message, report, category or checker with the given id(s). You +# can either give multiple identifiers separated by comma (,) or put this +# option multiple times (only on the command line, not in the configuration +# file where it should appear only once). You can also use "--disable=all" to +# disable everything first and then reenable specific checks. For example, if +# you want to run only the similarities checker, you can use "--disable=all +# --enable=similarities". If you want to run only the classes checker, but have +# no Warning level messages displayed, use "--disable=all --enable=classes +# --disable=W". +disable=too-many-arguments, + too-many-public-methods, + too-few-public-methods, + too-many-instance-attributes, + too-many-locals, + too-many-branches, + too-many-lines, + line-too-long, + similarities, + import-error, + raw-checker-failed, + bad-inline-option, + locally-disabled, + file-ignored, + suppressed-message, + useless-suppression, + deprecated-pragma, + use-symbolic-message-instead, + invalid-name, + global-statement, + use-implicit-booleaness-not-comparison + +# Enable the message, report, category or checker with the given id(s). You can +# either give multiple identifier separated by comma (,) or put this option +# multiple time (only on the command line, not in the configuration file where +# it should appear only once). See also the "--disable" option for examples. +#enable=c-extension-no-member + + +[REPORTS] + +# Python expression which should return a score less than or equal to 10. You +# have access to the variables 'error', 'warning', 'refactor', and 'convention' +# which contain the number of messages in each category, as well as 'statement' +# which is the total number of statements analyzed. This score is used by the +# global evaluation report (RP0004). +evaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10) + +# Template used to display messages. This is a python new-style format string +# used to format the message information. See doc for all details. +#msg-template= + +# Set the output format. Available formats are text, parseable, colorized, json +# and msvs (visual studio). You can also give a reporter class, e.g. +# mypackage.mymodule.MyReporterClass. +output-format=text + +# Tells whether to display a full report or only the messages. +reports=no + +# Activate the evaluation score. +score=yes + + +[REFACTORING] + +# Maximum number of nested blocks for function / method body +max-nested-blocks=5 + +# Complete name of functions that never returns. When checking for +# inconsistent-return-statements if a never returning function is called then +# it will be considered as an explicit return statement and no message will be +# printed. +never-returning-functions=sys.exit + + +[LOGGING] + +# Format style used to check logging format string. `old` means using % +# formatting, `new` is for `{}` formatting,and `fstr` is for f-strings. +logging-format-style=old + +# Logging modules to check that the string format arguments are in logging +# function parameter format. +logging-modules=logging + + +[FORMAT] + +# Expected format of line ending, e.g. empty (any line ending), LF or CRLF. +expected-line-ending-format= + +# Regexp for a line that is allowed to be longer than the limit. +ignore-long-lines=^\s*(# )??$ + +# Number of spaces of indent required inside a hanging or continued line. +indent-after-paren=4 + +# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1 +# tab). +indent-string=' ' + +# Maximum number of characters on a single line. +max-line-length=120 + +# Maximum number of lines in a module. +max-module-lines=3000 + +# List of optional constructs for which whitespace checking is disabled. `dict- +# separator` is used to allow tabulation in dicts, etc.: {1 : 1,\n222: 2}. +# `trailing-comma` allows a space between comma and closing bracket: (a, ). +# `empty-line` allows space-only lines. +no-space-check=trailing-comma, + dict-separator + +# Allow the body of a class to be on the same line as the declaration if body +# contains single statement. +single-line-class-stmt=no + +# Allow the body of an if to be on the same line as the test if there is no +# else. +single-line-if-stmt=no + + +[SPELLING] + +# Limits count of emitted suggestions for spelling mistakes. +max-spelling-suggestions=4 + +# Spelling dictionary name. Available dictionaries: none. To make it work, +# install the python-enchant package. +spelling-dict= + +# List of comma separated words that should not be checked. +spelling-ignore-words= + +# A path to a file that contains the private dictionary; one word per line. +spelling-private-dict-file= + +# Tells whether to store unknown words to the private dictionary (see the +# --spelling-private-dict-file option) instead of raising a message. +spelling-store-unknown-words=no + + +[BASIC] + +# Naming style matching correct argument names. +argument-naming-style=snake_case + +# Regular expression matching correct argument names. Overrides argument- +# naming-style. +#argument-rgx= + +# Naming style matching correct attribute names. +attr-naming-style=snake_case + +# Regular expression matching correct attribute names. Overrides attr-naming- +# style. +#attr-rgx= + +# Naming style matching correct class attribute names. +class-attribute-naming-style=any + +# Regular expression matching correct class attribute names. Overrides class- +# attribute-naming-style. +#class-attribute-rgx= + +# Naming style matching correct class names. +class-naming-style=PascalCase + +# Regular expression matching correct class names. Overrides class-naming- +# style. +#class-rgx= + +# Naming style matching correct constant names. +const-naming-style=UPPER_CASE + +# Regular expression matching correct constant names. Overrides const-naming- +# style. +#const-rgx= + +# Minimum line length for functions/classes that require docstrings, shorter +# ones are exempt. +docstring-min-length=-1 + +# Naming style matching correct function names. +function-naming-style=snake_case + +# Regular expression matching correct function names. Overrides function- +# naming-style. +#function-rgx= + +# Good variable names which should always be accepted, separated by a comma. +good-names=i, + j, + k, + ex, + Run, + _ + +# Include a hint for the correct naming format with invalid-name. +include-naming-hint=no + +# Naming style matching correct inline iteration names. +inlinevar-naming-style=any + +# Regular expression matching correct inline iteration names. Overrides +# inlinevar-naming-style. +#inlinevar-rgx= + +# Naming style matching correct method names. +method-naming-style=snake_case + +# Regular expression matching correct method names. Overrides method-naming- +# style. +#method-rgx= + +# Naming style matching correct module names. +module-naming-style=snake_case + +# Regular expression matching correct module names. Overrides module-naming- +# style. +#module-rgx= + +# Colon-delimited sets of names that determine each other's naming style when +# the name regexes allow several styles. +name-group= + +# Regular expression which should only match function or class names that do +# not require a docstring. +no-docstring-rgx=^_ + +# List of decorators that produce properties, such as abc.abstractproperty. Add +# to this list to register other decorators that produce valid properties. +# These decorators are taken in consideration only for invalid-name. +property-classes=abc.abstractproperty + +# Naming style matching correct variable names. +variable-naming-style=snake_case + +# Regular expression matching correct variable names. Overrides variable- +# naming-style. +#variable-rgx= + + +[STRING] + +# This flag controls whether the implicit-str-concat-in-sequence should +# generate a warning on implicit string concatenation in sequences defined over +# several lines. +check-str-concat-over-line-jumps=no + + +[TYPECHECK] + +# List of decorators that produce context managers, such as +# contextlib.contextmanager. Add to this list to register other decorators that +# produce valid context managers. +contextmanager-decorators=contextlib.contextmanager + +# List of members which are set dynamically and missed by pylint inference +# system, and so shouldn't trigger E1101 when accessed. Python regular +# expressions are accepted. +generated-members= + +# Tells whether missing members accessed in mixin class should be ignored. A +# mixin class is detected if its name ends with "mixin" (case insensitive). +ignore-mixin-members=yes + +# Tells whether to warn about missing members when the owner of the attribute +# is inferred to be None. +ignore-none=yes + +# This flag controls whether pylint should warn about no-member and similar +# checks whenever an opaque object is returned when inferring. The inference +# can return multiple potential results while evaluating a Python object, but +# some branches might not be evaluated, which results in partial inference. In +# that case, it might be useful to still emit no-member and other checks for +# the rest of the inferred objects. +ignore-on-opaque-inference=yes + +# List of class names for which member attributes should not be checked (useful +# for classes with dynamically set attributes). This supports the use of +# qualified names. +ignored-classes=optparse.Values,thread._local,_thread._local + +# List of module names for which member attributes should not be checked +# (useful for modules/projects where namespaces are manipulated during runtime +# and thus existing member attributes cannot be deduced by static analysis). It +# supports qualified module names, as well as Unix pattern matching. +ignored-modules= + +# Show a hint with possible names when a member name was not found. The aspect +# of finding the hint is based on edit distance. +missing-member-hint=yes + +# The minimum edit distance a name should have in order to be considered a +# similar match for a missing member name. +missing-member-hint-distance=1 + +# The total number of similar names that should be taken in consideration when +# showing a hint for a missing member. +missing-member-max-choices=1 + +# List of decorators that change the signature of a decorated function. +signature-mutators= + + +[SIMILARITIES] + +# Ignore comments when computing similarities. +ignore-comments=yes + +# Ignore docstrings when computing similarities. +ignore-docstrings=yes + +# Ignore imports when computing similarities. +ignore-imports=no + +# Minimum lines number of a similarity. +min-similarity-lines=4 + + +[MISCELLANEOUS] + +# List of note tags to take in consideration, separated by a comma. +notes=FIXME, + XXX, + TODO + + +[VARIABLES] + +# List of additional names supposed to be defined in builtins. Remember that +# you should avoid defining new builtins when possible. +additional-builtins= + +# Tells whether unused global variables should be treated as a violation. +allow-global-unused-variables=yes + +# List of strings which can identify a callback function by name. A callback +# name must start or end with one of those strings. +callbacks=cb_, + _cb + +# A regular expression matching the name of dummy variables (i.e. expected to +# not be used). +dummy-variables-rgx=_+$|(_[a-zA-Z0-9_]*[a-zA-Z0-9]+?$)|dummy|^ignored_|^unused_ + +# Argument names that match this expression will be ignored. Default to name +# with leading underscore. +ignored-argument-names=_.*|^ignored_|^unused_ + +# Tells whether we should check for unused import in __init__ files. +init-import=no + +# List of qualified module names which can have objects that can redefine +# builtins. +redefining-builtins-modules=six.moves,past.builtins,future.builtins,builtins,io + + +[DESIGN] + +# Maximum number of arguments for function / method. +max-args=5 + +# Maximum number of attributes for a class (see R0902). +max-attributes=7 + +# Maximum number of boolean expressions in an if statement (see R0916). +max-bool-expr=5 + +# Maximum number of branch for function / method body. +max-branches=12 + +# Maximum number of locals for function / method body. +max-locals=15 + +# Maximum number of parents for a class (see R0901). +max-parents=7 + +# Maximum number of public methods for a class (see R0904). +max-public-methods=20 + +# Maximum number of return / yield for function / method body. +max-returns=6 + +# Maximum number of statements in function / method body. +max-statements=50 + +# Minimum number of public methods for a class (see R0903). +min-public-methods=2 + + +[IMPORTS] + +# List of modules that can be imported at any level, not just the top level +# one. +allow-any-import-level= + +# Allow wildcard imports from modules that define __all__. +allow-wildcard-with-all=no + +# Analyse import fallback blocks. This can be used to support both Python 2 and +# 3 compatible code, which means that the block might have code that exists +# only in one or another interpreter, leading to false positives when analysed. +analyse-fallback-blocks=no + +# Deprecated modules which should not be used, separated by a comma. +deprecated-modules=optparse,tkinter.tix + +# Create a graph of external dependencies in the given file (report RP0402 must +# not be disabled). +ext-import-graph= + +# Create a graph of every (i.e. internal and external) dependencies in the +# given file (report RP0402 must not be disabled). +import-graph= + +# Create a graph of internal dependencies in the given file (report RP0402 must +# not be disabled). +int-import-graph= + +# Force import order to recognize a module as part of the standard +# compatibility libraries. +known-standard-library= + +# Force import order to recognize a module as part of a third party library. +known-third-party=enchant + +# Couples of modules and preferred modules, separated by a comma. +preferred-modules= + + +[CLASSES] + +# List of method names used to declare (i.e. assign) instance attributes. +defining-attr-methods=__init__, + __new__, + setUp, + __post_init__ + +# List of member names, which should be excluded from the protected access +# warning. +exclude-protected=_asdict, + _fields, + _replace, + _source, + _make + +# List of valid names for the first argument in a class method. +valid-classmethod-first-arg=cls + +# List of valid names for the first argument in a metaclass class method. +valid-metaclass-classmethod-first-arg=cls + + +[EXCEPTIONS] + +# Exceptions that will emit a warning when being caught. Defaults to +# "BaseException, Exception". +overgeneral-exceptions=builtins.BaseException, + builtins.Exception \ No newline at end of file diff --git a/.releaserc b/.releaserc index 8d0e9f8..ebc4e19 100644 --- a/.releaserc +++ b/.releaserc @@ -1,6 +1,6 @@ { "debug": true, - "branches": [ "main" ], + "branches": [ "main", {"name": "sandbox", "prerelease": "rc"} ], "plugins": [ "@semantic-release/commit-analyzer", "@semantic-release/release-notes-generator", diff --git a/.secrets.baseline b/.secrets.baseline index 68bc281..e1737e6 100644 --- a/.secrets.baseline +++ b/.secrets.baseline @@ -3,7 +3,7 @@ "files": "^.secrets.baseline$", "lines": null }, - "generated_at": "2026-03-06T04:33:02Z", + "generated_at": "2026-04-09T04:39:13Z", "plugins_used": [ { "name": "AWSKeyDetector" @@ -82,7 +82,7 @@ "hashed_secret": "11fa7c37d697f30e6aee828b4426a10f83ab2380", "is_secret": false, "is_verified": false, - "line_number": 350, + "line_number": 354, "type": "Secret Keyword", "verified_result": null } @@ -136,7 +136,7 @@ "hashed_secret": "aecdccc1cf64595b34e0cc152d238daabb32183a", "is_secret": false, "is_verified": false, - "line_number": 845, + "line_number": 915, "type": "Secret Keyword", "verified_result": null }, @@ -144,7 +144,7 @@ "hashed_secret": "f38bc57e171a3c9c5e4f405adb19a64667f875f5", "is_secret": false, "is_verified": false, - "line_number": 875, + "line_number": 945, "type": "Secret Keyword", "verified_result": null }, @@ -152,7 +152,7 @@ "hashed_secret": "e30ca465df4f3e851bad07da49e40817b45bae75", "is_secret": false, "is_verified": false, - "line_number": 888, + "line_number": 958, "type": "Secret Keyword", "verified_result": null }, @@ -160,7 +160,7 @@ "hashed_secret": "45d676e7c6ab44cf4b8fa366ef2d8fccd3e6d6e6", "is_secret": false, "is_verified": false, - "line_number": 902, + "line_number": 972, "type": "Secret Keyword", "verified_result": null }, @@ -168,7 +168,7 @@ "hashed_secret": "11fa7c37d697f30e6aee828b4426a10f83ab2380", "is_secret": false, "is_verified": false, - "line_number": 918, + "line_number": 988, "type": "Secret Keyword", "verified_result": null } @@ -178,7 +178,7 @@ "hashed_secret": "aecdccc1cf64595b34e0cc152d238daabb32183a", "is_secret": false, "is_verified": false, - "line_number": 783, + "line_number": 887, "type": "Secret Keyword", "verified_result": null }, @@ -186,7 +186,7 @@ "hashed_secret": "f38bc57e171a3c9c5e4f405adb19a64667f875f5", "is_secret": false, "is_verified": false, - "line_number": 813, + "line_number": 917, "type": "Secret Keyword", "verified_result": null }, @@ -194,7 +194,7 @@ "hashed_secret": "e30ca465df4f3e851bad07da49e40817b45bae75", "is_secret": false, "is_verified": false, - "line_number": 826, + "line_number": 930, "type": "Secret Keyword", "verified_result": null }, @@ -202,7 +202,7 @@ "hashed_secret": "45d676e7c6ab44cf4b8fa366ef2d8fccd3e6d6e6", "is_secret": false, "is_verified": false, - "line_number": 840, + "line_number": 944, "type": "Secret Keyword", "verified_result": null }, @@ -210,7 +210,7 @@ "hashed_secret": "11fa7c37d697f30e6aee828b4426a10f83ab2380", "is_secret": false, "is_verified": false, - "line_number": 856, + "line_number": 960, "type": "Secret Keyword", "verified_result": null } @@ -220,17 +220,7 @@ "hashed_secret": "11fa7c37d697f30e6aee828b4426a10f83ab2380", "is_secret": false, "is_verified": false, - "line_number": 83, - "type": "Secret Keyword", - "verified_result": null - } - ], - "docs/chapters/02_overview/release_notes.rst": [ - { - "hashed_secret": "a62f2225bf70bfaccbc7f1ef2a397836717377de", - "is_secret": false, - "is_verified": false, - "line_number": 141, + "line_number": 98, "type": "Secret Keyword", "verified_result": null } @@ -240,7 +230,7 @@ "hashed_secret": "1042188d51afe0c0b267d5c98e5ac2f2c741b28f", "is_secret": false, "is_verified": false, - "line_number": 74, + "line_number": 89, "type": "Secret Keyword", "verified_result": null }, @@ -248,7 +238,7 @@ "hashed_secret": "da6d6e9daf684cee4efd410e4d165ec5a2ee39a2", "is_secret": false, "is_verified": false, - "line_number": 113, + "line_number": 128, "type": "Secret Keyword", "verified_result": null }, @@ -256,7 +246,7 @@ "hashed_secret": "c58be5891091085cf51bf3e4d19317dce52767ae", "is_secret": false, "is_verified": false, - "line_number": 145, + "line_number": 160, "type": "Secret Keyword", "verified_result": null }, @@ -264,7 +254,7 @@ "hashed_secret": "ac52c3fa11d37a567f21c61397670e9dff7b8a52", "is_secret": false, "is_verified": false, - "line_number": 177, + "line_number": 192, "type": "Secret Keyword", "verified_result": null }, @@ -272,7 +262,7 @@ "hashed_secret": "45d676e7c6ab44cf4b8fa366ef2d8fccd3e6d6e6", "is_secret": false, "is_verified": false, - "line_number": 194, + "line_number": 209, "type": "Secret Keyword", "verified_result": null }, @@ -280,7 +270,7 @@ "hashed_secret": "a033a528b603fed46f861d4b3542c417b99d41c8", "is_secret": false, "is_verified": false, - "line_number": 228, + "line_number": 243, "type": "Secret Keyword", "verified_result": null }, @@ -288,7 +278,7 @@ "hashed_secret": "a62f2225bf70bfaccbc7f1ef2a397836717377de", "is_secret": false, "is_verified": false, - "line_number": 417, + "line_number": 432, "type": "Secret Keyword", "verified_result": null }, @@ -296,7 +286,7 @@ "hashed_secret": "11fa7c37d697f30e6aee828b4426a10f83ab2380", "is_secret": false, "is_verified": false, - "line_number": 454, + "line_number": 469, "type": "Secret Keyword", "verified_result": null } @@ -306,7 +296,7 @@ "hashed_secret": "11fa7c37d697f30e6aee828b4426a10f83ab2380", "is_secret": false, "is_verified": false, - "line_number": 33, + "line_number": 48, "type": "Secret Keyword", "verified_result": null } @@ -369,6 +359,16 @@ "verified_result": null } ], + "examples/end_to_end_example.py": [ + { + "hashed_secret": "df5cc5832dc34a455c18662ac84587ea19cf2435", + "is_secret": false, + "is_verified": false, + "line_number": 63, + "type": "Secret Keyword", + "verified_result": null + } + ], "examples/glossary_usage.py": [ { "hashed_secret": "aecdccc1cf64595b34e0cc152d238daabb32183a", @@ -379,6 +379,16 @@ "verified_result": null } ], + "examples/odcs_generator_informatica_example.py": [ + { + "hashed_secret": "0eb9a6a3306220b901c7b4920cd9896899f219be", + "is_secret": false, + "is_verified": false, + "line_number": 54, + "type": "Secret Keyword", + "verified_result": null + } + ], "src/wxdi/common/auth/__init__.py": [ { "hashed_secret": "11fa7c37d697f30e6aee828b4426a10f83ab2380", @@ -419,7 +429,33 @@ "verified_result": null } ], - "tests/auth/test_auth_config.py": [ + "src/wxdi/odcs_generator/README-GENERATE-ODCS-SCRIPT.md": [ + { + "hashed_secret": "ddc861617551c5e789c290865300f615e6f51cc7", + "is_secret": false, + "is_verified": false, + "line_number": 160, + "type": "Secret Keyword", + "verified_result": null + }, + { + "hashed_secret": "91dfd9ddb4198affc5c194cd8ce6d338fde470e2", + "is_secret": false, + "is_verified": false, + "line_number": 592, + "type": "Secret Keyword", + "verified_result": null + }, + { + "hashed_secret": "c359253705924b5bda99d65a476c651b43d5a9ab", + "is_secret": false, + "is_verified": false, + "line_number": 656, + "type": "Secret Keyword", + "verified_result": null + } + ], + "tests/src/auth/test_auth_config.py": [ { "hashed_secret": "2e7a7ee14caebf378fc32d6cf6f557f347c96773", "is_secret": false, @@ -437,7 +473,7 @@ "verified_result": null } ], - "tests/auth/test_auth_provider.py": [ + "tests/src/auth/test_auth_provider.py": [ { "hashed_secret": "3acfb2c2b433c0ea7ff107e33df91b18e52f960f", "is_secret": false, @@ -487,7 +523,7 @@ "verified_result": null } ], - "tests/auth/test_gov_cloud_authenticator.py": [ + "tests/src/auth/test_gov_cloud_authenticator.py": [ { "hashed_secret": "2e7a7ee14caebf378fc32d6cf6f557f347c96773", "is_secret": false, @@ -513,7 +549,7 @@ "verified_result": null } ], - "tests/auth/test_gov_cloud_token_manager.py": [ + "tests/src/auth/test_gov_cloud_token_manager.py": [ { "hashed_secret": "9a7ac473b40a179da10d64f3ed185894d54f0973", "is_secret": false, @@ -534,7 +570,7 @@ "hashed_secret": "2e7a7ee14caebf378fc32d6cf6f557f347c96773", "is_secret": false, "is_verified": false, - "line_number": 336, + "line_number": 318, "type": "Secret Keyword", "verified_result": null }, @@ -542,7 +578,7 @@ "hashed_secret": "3acfb2c2b433c0ea7ff107e33df91b18e52f960f", "is_secret": false, "is_verified": false, - "line_number": 387, + "line_number": 370, "type": "Secret Keyword", "verified_result": null }, @@ -550,12 +586,12 @@ "hashed_secret": "042857cf4ffc7022a50002083e0a34208b9333a8", "is_secret": false, "is_verified": false, - "line_number": 408, + "line_number": 391, "type": "Secret Keyword", "verified_result": null } ], - "tests/provider/test_config.py": [ + "tests/src/dq_validator/provider/test_config.py": [ { "hashed_secret": "74ba31d41223751c75cc0a453dd7df04889bdc72", "is_secret": false, @@ -572,6 +608,78 @@ "type": "Secret Keyword", "verified_result": null } + ], + "tests/src/integration/README.md": [ + { + "hashed_secret": "ba707e7fc663168cdca48990656bcce07d058474", + "is_secret": false, + "is_verified": false, + "line_number": 92, + "type": "Secret Keyword", + "verified_result": null + }, + { + "hashed_secret": "c5afd23baeb1464149e957bb12f02d39543b0c26", + "is_secret": false, + "is_verified": false, + "line_number": 109, + "type": "Secret Keyword", + "verified_result": null + } + ], + "tests/src/integration/test_odcs_generator_collibra.py": [ + { + "hashed_secret": "fca268ae2442d5cabc3e12d87b349adf8bf7d76c", + "is_secret": false, + "is_verified": false, + "line_number": 423, + "type": "Secret Keyword", + "verified_result": null + }, + { + "hashed_secret": "206c80413b9a96c1312cc346b7d2517b84463edd", + "is_secret": false, + "is_verified": false, + "line_number": 546, + "type": "Secret Keyword", + "verified_result": null + }, + { + "hashed_secret": "46e1833f63ae8d02127adb8316b25bea3e2051f4", + "is_secret": false, + "is_verified": false, + "line_number": 577, + "type": "Secret Keyword", + "verified_result": null + } + ], + "tests/src/integration/test_odcs_generator_informatica.py": [ + { + "hashed_secret": "206c80413b9a96c1312cc346b7d2517b84463edd", + "is_secret": false, + "is_verified": false, + "line_number": 106, + "type": "Secret Keyword", + "verified_result": null + }, + { + "hashed_secret": "7560d06f2f0b04f5b40643b505a1c8a4048a20da", + "is_secret": false, + "is_verified": false, + "line_number": 311, + "type": "Secret Keyword", + "verified_result": null + } + ], + "tests/src/odcs_generator/test_odcs_generator_collibra.py": [ + { + "hashed_secret": "206c80413b9a96c1312cc346b7d2517b84463edd", + "is_secret": false, + "is_verified": false, + "line_number": 312, + "type": "Secret Keyword", + "verified_result": null + } ] }, "version": "0.13.1+ibm.64.dss", diff --git a/CHANGELOG.md b/CHANGELOG.md index aa2844f..a42bb2b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,61 @@ +# [2.0.0-rc.1](https://github.com/IBM/data-intelligence-sdk/compare/v1.0.0...v2.0.0-rc.1) (2026-04-22) + + +### Perf + +* Sync from enterprise 4e91aee (via .ignore) on 2026-04-17 ([#7](https://github.com/IBM/data-intelligence-sdk/issues/7)) ([eeeffba](https://github.com/IBM/data-intelligence-sdk/commit/eeeffba17ce0e399446099d47352f7738259feae)) + + +### BREAKING CHANGES + +* Bug fixes for Data Quality and addition of Data Product Hub modules + +Signed-off-by: Koichi Nishitani + +* fix relative path of Actions scripts + +Signed-off-by: Koichi Nishitani + +* fix script condition + +Signed-off-by: Koichi Nishitani + +* fix script condition again + +Signed-off-by: Koichi Nishitani + +* add missing Makefile and fix documentation + +Signed-off-by: Koichi Nishitani + +* add dq tests and upgrade python + +Signed-off-by: Koichi Nishitani + +* try to extend build timeout + +Signed-off-by: Koichi Nishitani + +* upgrade to python 3.10 as 3.9 is EOL + +Signed-off-by: Koichi Nishitani + +* modify constraint_model to use custom StrEnum in python 3.10 + +Signed-off-by: Koichi Nishitani + +* add pylint + +Signed-off-by: Koichi Nishitani + +* skip linting until ready + +Signed-off-by: Koichi Nishitani + +* make sure to build before checking + +Signed-off-by: Koichi Nishitani + # [1.0.0](https://github.com/IBM/data-intelligence-sdk/compare/v0.5.3...v1.0.0) (2026-03-17) diff --git a/Makefile.build b/Makefile.build new file mode 100644 index 0000000..d85e603 --- /dev/null +++ b/Makefile.build @@ -0,0 +1,58 @@ +# Copyright 2026 IBM Corporation +# Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# See the LICENSE file in the project root for license information. + +PYTHON=python +LINT_DIRS=src/wxdi/dph_services src/wxdi/data_product_recommender src/wxdi/odcs_generator tests/src/data_product_recommender tests/src/dph_services tests/src/odcs_generator tests/src/integration examples + +setup: deps dev_deps install_project + +all: upgrade_pip setup test-unit lint + +ci: setup test-unit # lint # enable when ready + +upgrade_pip: + ${PYTHON} -m pip install --upgrade pip + +deps: + ${PYTHON} -m pip install -r requirements.txt + +dev_deps: + ${PYTHON} -m pip install -r requirements-dev.txt + +install_project: + ${PYTHON} -m pip install -e . + +test: test-unit test-int + +test-unit: + ${PYTHON} -m pytest --cov=src/wxdi/dph_services --cov=src/wxdi/data_product_recommender --cov=src/wxdi/odcs_generator \ + --cov-report=xml:coverage.xml --cov-report=html --cov-report=term tests/src/dph_services tests/src/data_product_recommender \ + tests/src/odcs_generator tests/src/dq_validator tests/src/auth + +test-int: + ${PYTHON} -m pytest tests/src/integration + +coverage: + ${PYTHON} -m pytest --cov=src/wxdi/dph_services --cov-report=xml:coverage.xml --cov-report=html --cov-report=term tests/src/dph_services tests/src/integration + +test-examples: + ${PYTHON} -m pytest examples + +lint: + ${PYTHON} -m pylint ${LINT_DIRS} + black --check ${LINT_DIRS} + +lint-fix: + black ${LINT_DIRS} \ No newline at end of file diff --git a/README.md b/README.md index 5d9a43c..9040d1a 100644 --- a/README.md +++ b/README.md @@ -13,9 +13,13 @@ See the License for the specific language governing permissions and limitations under the License. --> -# IBM watsonx.data intelligence SDK Version 1.0.0 +# IBM watsonx.data intelligence SDK Version 2.0.0 -A comprehensive Python SDK for performing data quality validations on streaming data records (arrays), Pandas DataFrames, and PySpark DataFrames with complete REST API integration for IBM Cloud Pak for Data. +A comprehensive Python SDK for data intelligence operations including: +- **Data Quality Validation**: Validate streaming data records, Pandas DataFrames, and PySpark DataFrames +- **Data Product Hub Services**: Complete Python client for IBM Data Product Hub API +- **ODCS Generation**: Generate Open Data Contract Standard (ODCS) files from Collibra and Informatica +- **Data Product Recommendations**: Analyze query logs to identify high-value data products ## Features @@ -48,7 +52,7 @@ A comprehensive Python SDK for performing data quality validations on streaming ### REST API Integration - **GlossaryProvider**: Fetch glossary terms and data quality constraints from IBM Cloud Pak for Data - **CamsProvider**: Fetch data assets from CAMS (Catalog Asset Management System) -- **IssuesProvider**: Manage data quality issues (occurrences, tested records, ignored status) +- **IssuesProvider**: Manage data quality issues (create single/bulk occurrences, tested records, ignored status, update metrics, ignored status) - **DQSearchProvider**: Search for DQ checks and assets by native ID - **Thread-Safe**: Concurrent access support with thread-local sessions @@ -57,6 +61,14 @@ A comprehensive Python SDK for performing data quality validations on streaming - **Automatic Protocol Handling**: Environment-specific authentication methods - **Type-Safe Configuration**: Full type hints and validation - **SSL Control**: Configurable SSL verification for on-premises +### Additional Modules + +For detailed documentation on additional modules, see their respective READMEs: + +- **[Data Product Hub Services](src/wxdi/dph_services/README.md)**: Complete Python client for IBM Data Product Hub API +- **[ODCS Generator](src/wxdi/odcs_generator/README-GENERATE-ODCS-SCRIPT.md)**: Generate ODCS v3.1.0 compliant YAML from Collibra and Informatica +- **[Data Product Recommender](src/wxdi/data_product_recommender/README.md)**: Analyze query logs to identify high-value data products + ## Installation @@ -263,23 +275,37 @@ df_expanded.select('name', 'dq_is_valid', 'dq_score', 'dq_pass_rate').show() # Write validation report spark_validator.write_validation_report(df, output_path='validation_report', format='parquet') -``` -## Core Concepts +### Data Product Hub Services -### AssetMetadata +```python +from wxdi.dph_services import DphV1 +from ibm_cloud_sdk_core.authenticators import IAMAuthenticator -Defines the structure of your data asset (table) with column information: +# Initialize the service +authenticator = IAMAuthenticator('your-api-key') +dph_service = DphV1(authenticator=authenticator) +dph_service.set_service_url('https://your-dph-instance.com') -```python -metadata = AssetMetadata( - table_name='my_table', - columns=[ - ColumnMetadata('id', DataType.INTEGER), - ColumnMetadata('name', DataType.STRING, length=100), - ColumnMetadata('amount', DataType.DECIMAL, precision=10, scale=2), - ] +# Initialize a container +container_response = dph_service.initialize( + include=['delivery_methods', 'data_product_samples', 'domains_multi_industry'] +) + +# Create a data product +data_product = dph_service.create_data_product( + drafts=[{ + 'version': '2.0.0', + 'name': 'My Data Product', + 'description': 'A sample data product', + 'asset': { + 'id': 'asset-123', + 'container': {'id': 'container-456'} + } + }] ) + +print(f"Created data product: {data_product.result['id']}") ``` ### ValidationRule @@ -737,21 +763,33 @@ for column in asset.column_info: ### IssuesProvider -Manage data quality issues (occurrences, tested records, ignored status). +Manage data quality issues (create single/bulk create single/bulk occurrences, tested records, ignored status, update metrics, ignored status). ```python from wxdi.dq_validator.provider import IssuesProvider issues = IssuesProvider(config) -# Update issue occurrences -issues.update_issue_occurrences(issue_id="issue-123", occurrences=767) - -# Update tested records -issues.update_tested_records(issue_id="issue-123", tested_records=1000) +# Create single issue +issue_id = issues.create_issue( + dq_check_id="check-123", + reported_for_id="asset-456", + number_of_occurrences=10, + number_of_tested_records=100, + project_id="project-123" +) -# Set ignored status -issues.set_issue_ignored(issue_id="issue-123", ignored=True) +# Create multiple issues in bulk +bulk_payload = { + "issues": [...], + "assets": [...], + "existing_checks": [...] +} +response = issues.create_issues_bulk( + payload=bulk_payload, + project_id="project-123", + incremental_reporting=False +) # Update issue metrics issues.update_issue_values( @@ -819,18 +857,19 @@ from wxdi.dq_validator.provider import ChecksProvider checks = ChecksProvider(config) -# Create a new check +# Create a new check (returns check ID as string) check_id = checks.create_check( name="Format Check", native_id="asset-id/column-name", check_type="format", dimension_id="dimension-id", - project_id="project-id" + project_id="project-id", + parent_id=None # Optional: for parent-child hierarchy ) # Get existing checks checks_list = checks.get_checks( - asset_id="asset-id", + dq_asset_id="asset-id", check_type="format", project_id="project-id" ) @@ -1147,5 +1186,5 @@ For issues, questions, or contributions, please open an issue on GitHub. - pytest-cov >= 4.0.0 - pytest-mock >= 3.7.0 - black >= 26.3.1 -- mypy >= 1.0.0 -- flake8 >= 6.0.0 +- mypy >= 2.0.0 + diff --git a/docs/README.md b/docs/README.md index 377f7de..3fe648e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -273,4 +273,4 @@ When contributing documentation: For documentation issues or questions: - Open an issue on GitHub -- Contact: data-intelligence-sdk@ibm.com \ No newline at end of file +- Contact: Data_Intelligence_SDK@wwpdl.vnet.ibm.com \ No newline at end of file diff --git a/docs/chapters/01_welcome/installation.rst b/docs/chapters/01_welcome/installation.rst index 2e55d91..0ca04d3 100644 --- a/docs/chapters/01_welcome/installation.rst +++ b/docs/chapters/01_welcome/installation.rst @@ -100,7 +100,7 @@ To verify that the SDK is installed correctly: >>> import wxdi.dq_validator >>> from wxdi.common.auth import AuthProvider >>> print(dq_validator.__version__) - 1.0.0 + 2.0.0 Versioning ---------- @@ -116,7 +116,7 @@ Version numbers follow the format ``MAJOR.MINOR.PATCH``: Current Version ~~~~~~~~~~~~~~~ -The current version of the SDK is **1.0.0**. +The current version of the SDK is **2.0.0**. Checking Your Version ~~~~~~~~~~~~~~~~~~~~~ @@ -133,7 +133,7 @@ Or programmatically: >>> import wxdi.dq_validator >>> print(dq_validator.__version__) - 1.0.0 + 2.0.0 Upgrading --------- diff --git a/docs/chapters/02_overview/faq.rst b/docs/chapters/02_overview/faq.rst index 23f51b5..ae8df01 100644 --- a/docs/chapters/02_overview/faq.rst +++ b/docs/chapters/02_overview/faq.rst @@ -329,6 +329,6 @@ If your question isn't answered here: * Check the :ref:`API Reference` * Review the code examples * Open an issue on GitHub -* Contact: data-intelligence-sdk@ibm.com +* Contact: Data_Intelligence_SDK@wwpdl.vnet.ibm.com .. Made with Bob diff --git a/docs/chapters/02_overview/known_issues.rst b/docs/chapters/02_overview/known_issues.rst index 0238282..afcbc97 100644 --- a/docs/chapters/02_overview/known_issues.rst +++ b/docs/chapters/02_overview/known_issues.rst @@ -275,7 +275,7 @@ If you're experiencing issues: * Check the :ref:`API Reference` for detailed documentation * Search GitHub issues for similar problems * Open a new issue with detailed information -* Contact: data-intelligence-sdk@ibm.com +* Contact: Data_Intelligence_SDK@wwpdl.vnet.ibm.com We appreciate your patience as we continue to improve the SDK! diff --git a/docs/conf.py b/docs/conf.py index 363a656..917b9ea 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -38,9 +38,9 @@ # the built documents. # # The short X.Y version. -version = "1.0.0" +version = "2.0.0" # The full version, including alpha/beta/rc tags. -release = "1.0.0" +release = "2.0.0" # -- General configuration --------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration diff --git a/examples/__init__.py b/examples/__init__.py new file mode 100644 index 0000000..a5165bf --- /dev/null +++ b/examples/__init__.py @@ -0,0 +1,19 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Examples""" + +# This file is only here to get pylint to check the files in this directory diff --git a/examples/assets_usage.py b/examples/assets_usage.py index 510a2b2..a1c04aa 100644 --- a/examples/assets_usage.py +++ b/examples/assets_usage.py @@ -33,58 +33,58 @@ def main(): """Main function demonstrating AssetProvider usage.""" - + print("=" * 70) print("AssetProvider - Usage Examples") print("=" * 70) - + # Step 1: Configure the provider with your instance URL and authentication token config = ProviderConfig( url="https://your-instance.cloud.ibm.com", auth_token="Bearer your-auth-token-here" ) - + print("\nConfiguration:") print(f" URL: {config.url}") print(f" Auth Token: {config.auth_token[:50]}...") - + # Step 2: Create a DQAssetsProvider instance asset_provider = DQAssetsProvider(config) print("\n✓ DQAssetsProvider initialized") - + # Define IDs that will be used throughout the examples project_id = "your-project-id-here" # Replace with your actual project ID catalog_id = "your-catalog-id-here" # Alternative to project_id - + # ========================================================================= # Example 1: Get data assets with children (using project_id) # ========================================================================= print("\n" + "=" * 70) print("Example 1: Get Data Assets with Children (using project_id)") print("=" * 70) - + try: print(f"\nParameters:") print(f" project_id: {project_id}") print(f" include_children: True") print(f" asset_type: data_asset") - + assets_response = asset_provider.get_assets( project_id=project_id, include_children=True, asset_type="data_asset", limit=5 ) - + print(f"\n✓ Successfully retrieved data assets") - + assets = assets_response.get("assets", []) total_count = assets_response.get("total_count", 0) - + print(f"\nSummary:") print(f" Total Count: {total_count}") print(f" Assets Returned: {len(assets)}") - + if assets: asset = assets[0] print(f"\nFirst asset:") @@ -93,72 +93,72 @@ def main(): print(f" Type: {asset.get('type', 'N/A')}") children = asset.get('children', []) print(f" Children Count: {len(children)}") - + except ValueError as e: print(f"\n✗ Error: {e}") - + # ========================================================================= # Example 2: Get column assets without children (using project_id) # ========================================================================= print("\n" + "=" * 70) print("Example 2: Get Column Assets without Children (using project_id)") print("=" * 70) - + try: print(f"\nParameters:") print(f" project_id: {project_id}") print(f" include_children: False") print(f" asset_type: column") - + assets_response = asset_provider.get_assets( project_id=project_id, include_children=False, asset_type="column", limit=5 ) - + print(f"\n✓ Successfully retrieved column assets") - + assets = assets_response.get("assets", []) print(f"\nAssets Returned: {len(assets)}") - + if assets: print(f"\nFirst 2 columns:") for idx, asset in enumerate(assets[:2], 1): print(f" {idx}. {asset.get('name', 'N/A')} (ID: {asset.get('id')})") - + except ValueError as e: print(f"\n✗ Error: {e}") - + # ========================================================================= # Example 3: Get data assets using catalog_id # ========================================================================= print("\n" + "=" * 70) print("Example 3: Get Data Assets using catalog_id") print("=" * 70) - + try: print(f"\nParameters:") print(f" catalog_id: {catalog_id}") print(f" include_children: True") print(f" asset_type: data_asset") - + assets_response = asset_provider.get_assets( catalog_id=catalog_id, include_children=True, asset_type="data_asset", limit=5 ) - + print(f"\n✓ Successfully retrieved data assets using catalog_id") - + assets = assets_response.get("assets", []) total_count = assets_response.get("total_count", 0) - + print(f"\nSummary:") print(f" Total Count: {total_count}") print(f" Assets Returned: {len(assets)}") - + if assets: asset = assets[0] print(f"\nFirst asset:") @@ -166,41 +166,41 @@ def main(): print(f" Name: {asset.get('name', 'N/A')}") children = asset.get('children', []) print(f" Children Count: {len(children)}") - + except ValueError as e: print(f"\n✗ Error: {e}") - + # ========================================================================= # Example 4: Get column assets using catalog_id # ========================================================================= print("\n" + "=" * 70) print("Example 4: Get Column Assets using catalog_id") print("=" * 70) - + try: print(f"\nParameters:") print(f" catalog_id: {catalog_id}") print(f" asset_type: column") - + assets_response = asset_provider.get_assets( catalog_id=catalog_id, asset_type="column", limit=5 ) - + print(f"\n✓ Successfully retrieved column assets using catalog_id") - + assets = assets_response.get("assets", []) print(f"\nAssets Returned: {len(assets)}") - + if assets: print(f"\nFirst 2 columns:") for idx, asset in enumerate(assets[:2], 1): print(f" {idx}. {asset.get('name', 'N/A')} (ID: {asset.get('id')})") - + except ValueError as e: print(f"\n✗ Error: {e}") - + print("\n" + "=" * 70) print("Examples Complete!") print("=" * 70) diff --git a/examples/auth_provider_usage.py b/examples/auth_provider_usage.py index 5700088..3508984 100644 --- a/examples/auth_provider_usage.py +++ b/examples/auth_provider_usage.py @@ -65,7 +65,7 @@ def example_ibm_cloud(): """ Example: IBM Cloud authentication using IAMAuthenticator - + Requirements: - api_key: Your IBM Cloud API key - url: Not required for production. Provide only for non-production environments. @@ -73,7 +73,7 @@ def example_ibm_cloud(): print("=" * 60) print("IBM CLOUD AUTHENTICATION") print("=" * 60) - + # Create environment configuration # Required: environment_type, api_key # url: Not required for production @@ -85,28 +85,28 @@ def example_ibm_cloud(): # url='https://host_name', # This is not required for production env. For non-default custom add this property. # disable_ssl_verification=True # Optional, default is True ) - + print(f"Environment Type: {config.environment_type.value}") print(f"URL: {config.url}") - + # Create auth provider auth_provider = AuthProvider(config) print(f"Authenticator: {type(auth_provider.authenticator).__name__}") - + # Get token (uncomment when you have valid credentials) # try: # token = auth_provider.get_token() # print(f"Token obtained successfully: {token[:20]}...") # except Exception as e: # print(f"Error getting token: {e}") - + print() def example_aws_cloud(): """ Example: AWS Cloud authentication using MCSPV2Authenticator - + Requirements: - api_key: Your AWS API key - account_id: Your AWS account ID @@ -115,7 +115,7 @@ def example_aws_cloud(): print("=" * 60) print("AWS CLOUD AUTHENTICATION") print("=" * 60) - + # Create environment configuration # Required: environment_type, api_key, account_id # url: Not required for production @@ -128,29 +128,29 @@ def example_aws_cloud(): # url='https://host_name', # This is not required for production env. For non-default custom add this property. # disable_ssl_verification=True # Optional, default is True ) - + print(f"Environment Type: {config.environment_type.value}") print(f"URL: {config.url}") print(f"Account ID: {config.account_id}") - + # Create auth provider auth_provider = AuthProvider(config) print(f"Authenticator: {type(auth_provider.authenticator).__name__}") - + # Get token (uncomment when you have valid credentials) # try: # token = auth_provider.get_token() # print(f"Token obtained successfully: {token[:20]}...") # except Exception as e: # print(f"Error getting token: {e}") - + print() def example_gov_cloud(): """ Example: Government Cloud authentication using GovCloudAuthenticator - + Requirements: - api_key: Your Government Cloud API key - url: Not required for production. Provide only for non-production environments. @@ -158,7 +158,7 @@ def example_gov_cloud(): print("=" * 60) print("GOVERNMENT CLOUD AUTHENTICATION") print("=" * 60) - + # Create environment configuration # Required: environment_type, api_key # url: Not required for production environments @@ -170,39 +170,39 @@ def example_gov_cloud(): # url='https://host_name', # This is not required for production env. For non-default custom add this property. # disable_ssl_verification=True # Optional, default is True ) - + print(f"Environment Type: {config.environment_type.value}") print(f"URL: {config.url}") - + # Create auth provider auth_provider = AuthProvider(config) print(f"Authenticator: {type(auth_provider.authenticator).__name__}") - + # Get token (uncomment when you have valid credentials) # try: # token = auth_provider.get_token() # print(f"Token obtained successfully: {token[:20]}...") # except Exception as e: # print(f"Error getting token: {e}") - + print() def example_on_prem_with_apikey(): """ Example: On-Premises authentication using CloudPakForDataAuthenticator with API key - + Requirements: - url: Your on-premises CP4D URL (required) - username: Your username - api_key: Your API key - + Note: When using api_key, username is required but password should NOT be provided. """ print("=" * 60) print("ON-PREMISES AUTHENTICATION (with API Key)") print("=" * 60) - + # Create environment configuration # Required: environment_type, url, username, api_key (or password) # Optional: disable_ssl_verification (default: True) @@ -214,41 +214,41 @@ def example_on_prem_with_apikey(): api_key='your-cp4d-api-key-here' # disable_ssl_verification=True # Optional, default is True ) - + print(f"Environment Type: {config.environment_type.value}") print(f"URL: {config.url}") print(f"Username: {config.username}") print(f"Authentication Method: API Key") - + # Create auth provider auth_provider = AuthProvider(config) print(f"Authenticator: {type(auth_provider.authenticator).__name__}") - + # Get token (uncomment when you have valid credentials) # try: # token = auth_provider.get_token() # print(f"Token obtained successfully: {token[:20]}...") # except Exception as e: # print(f"Error getting token: {e}") - + print() def example_on_prem_with_password(): """ Example: On-Premises authentication using CloudPakForDataAuthenticator with password - + Requirements: - url: Your on-premises CP4D URL (required) - username: Your username - password: Your password - + Note: When using password, api_key should NOT be provided. """ print("=" * 60) print("ON-PREMISES AUTHENTICATION (with Username/Password)") print("=" * 60) - + # Create environment configuration # Required: environment_type, url, username, password (or api_key) # Optional: disable_ssl_verification (default: True) @@ -259,37 +259,37 @@ def example_on_prem_with_password(): password='your-password', # disable_ssl_verification=True # Optional, default is True ) - + print(f"Environment Type: {config.environment_type.value}") print(f"URL: {config.url}") print(f"Username: {config.username}") print(f"Authentication Method: Username/Password") - + # Create auth provider auth_provider = AuthProvider(config) print(f"Authenticator: {type(auth_provider.authenticator).__name__}") - + # Get token (uncomment when you have valid credentials) # try: # token = auth_provider.get_token() # print(f"Token obtained successfully: {token[:20]}...") # except Exception as e: # print(f"Error getting token: {e}") - + print() def example_custom_url(): """ Example: Using custom URL for non-production environments - + Provide url only for non-production environments (staging, preprod, dev). For production, omit url and the default production URL will be used automatically. """ print("=" * 60) print("NON-production CUSTOM URL EXAMPLE") print("=" * 60) - + # Provide url only for non-production environments # For production, omit url - the default production URL is used automatically # Trailing slashes are automatically stripped @@ -299,14 +299,14 @@ def example_custom_url(): api_key='your-api-key-here', # disable_ssl_verification=True # Optional, default is True ) - + print(f"Environment Type: {config.environment_type.value}") print(f"Custom URL: {config.url}") - + # Create auth provider auth_provider = AuthProvider(config) print(f"Authenticator: {type(auth_provider.authenticator).__name__}") - + # Get token (uncomment when you have valid credentials) # try: # token = auth_provider.get_token() @@ -320,13 +320,13 @@ def example_custom_url(): def example_error_handling(): """ Example: Error handling and validation - + The AuthConfig validates required fields based on environment type. """ print("=" * 60) print("ERROR HANDLING EXAMPLES") print("=" * 60) - + # Example 1: Missing API key for IBM_CLOUD print("\n1. Missing API key for IBM_CLOUD:") try: @@ -336,7 +336,7 @@ def example_error_handling(): ) except ValueError as e: print(f" ✓ Caught error: {e}") - + # Example 2: Missing account_id for AWS_CLOUD print("\n2. Missing account_id for AWS_CLOUD:") try: @@ -347,7 +347,7 @@ def example_error_handling(): ) except ValueError as e: print(f" ✓ Caught error: {e}") - + # Example 3: Missing username for ON_PREM print("\n3. Missing username for ON_PREM:") try: @@ -359,7 +359,7 @@ def example_error_handling(): ) except ValueError as e: print(f" ✓ Caught error: {e}") - + # Example 4: Missing URL for ON_PREM print("\n4. Missing URL for ON_PREM:") try: @@ -371,7 +371,7 @@ def example_error_handling(): ) except ValueError as e: print(f" ✓ Caught error: {e}") - + print() @@ -379,7 +379,7 @@ def example_error_handling(): print("\n" + "=" * 60) print("AUTHPROVIDER USAGE EXAMPLES") print("=" * 60 + "\n") - + # Run all examples example_ibm_cloud() example_aws_cloud() @@ -388,7 +388,7 @@ def example_error_handling(): example_on_prem_with_password() example_custom_url() example_error_handling() - + print("=" * 60) print("All examples completed!") print("=" * 60) \ No newline at end of file diff --git a/examples/basic_usage.py b/examples/basic_usage.py index ad7f4c6..8b06bff 100644 --- a/examples/basic_usage.py +++ b/examples/basic_usage.py @@ -34,7 +34,7 @@ def main(): print("=" * 60) print("IBM watsonx.data intelligence SDK - Basic Usage Example") print("=" * 60) - + metadata = AssetMetadata( table_name='employee_data', columns=[ @@ -52,25 +52,25 @@ def main(): ColumnMetadata('employee_code', DataType.STRING, length=10) ] ) - + print(f"\nAsset: {metadata.table_name}") print(f"Columns: {len(metadata.columns)}") - + # Step 2: Create validator with rules validator = Validator(metadata) - + # Add length check for name (works with any type) validator.add_rule( ValidationRule('name') .add_check(LengthCheck(min_length=2, max_length=100)) ) - + # Add length check for emp_id (integer converted to string) validator.add_rule( ValidationRule('emp_id') .add_check(LengthCheck(min_length=4, max_length=6)) ) - + # Add valid values check for department (case-insensitive) validator.add_rule( ValidationRule('department') @@ -79,7 +79,7 @@ def main(): case_sensitive=False )) ) - + # Add comparison checks using enum validator.add_rule( ValidationRule('age') @@ -92,7 +92,7 @@ def main(): target_value=65 )) ) - + validator.add_rule( ValidationRule('salary') .add_check(ComparisonCheck( @@ -100,43 +100,43 @@ def main(): target_column='min_salary' )) ) - + # Add CaseCheck for email (must be lowercase) validator.add_rule( ValidationRule('email') .add_check(CaseCheck(case_type=ColumnCaseEnum.LOWER_CASE)) ) - + # Add CompletenessCheck for name (required field) validator.add_rule( ValidationRule('name') .add_check(CompletenessCheck(missing_values_allowed=False)) ) - + # Add RangeCheck for bonus (0 to 50000) validator.add_rule( ValidationRule('bonus') .add_check(RangeCheck(min_value=0, max_value=50000)) ) - + # Add RegexCheck for phone (format: XXX-XXX-XXXX) validator.add_rule( ValidationRule('phone') .add_check(RegexCheck(pattern=r'^\d{3}-\d{3}-\d{4}$')) ) - + # Add CaseCheck for status (must be NameCase) validator.add_rule( ValidationRule('status') .add_check(CaseCheck(case_type=ColumnCaseEnum.NAME_CASE)) ) - + # Add DataTypeCheck for age (must be INTEGER) validator.add_rule( ValidationRule('age') .add_check(DataTypeCheck(expected_type=DType(dtype=DataTypeEnum.INT32))) ) - + # Add FormatCheck for hire_date (must be valid date format) # NEW: Using DateTimeFormats constants for readable format names validator.add_rule( @@ -150,7 +150,7 @@ def main(): } )) ) - + print(f"\nValidator configured with {len(validator.rules)} rules") print("Checks included:") print(" - LengthCheck (name, emp_id)") @@ -162,12 +162,12 @@ def main(): print(" - RegexCheck (phone)") print(" - DataTypeCheck (age)") print(" - FormatCheck (hire_date)") - + # Step 3: Validate records print("\n" + "=" * 60) print("Validating Records") print("=" * 60) - + records = [ # [emp_id, name, email, age, department, salary, min_salary, phone, status, bonus, hire_date, employee_code] [1001, 'John Doe', 'john@company.com', 30, 'engineering', 75000.00, 60000.00, '555-123-4567', 'Active', 5000.00, '2020-01-15', 'EMP001'], @@ -177,39 +177,39 @@ def main(): [1005, 'Charlie Brown', 'charlie@company.com', 40, 'Finance', 55000.00, 60000.00, '555-567-8901', 'Active', 4000.00, '2023-06-10', 'EMP005'], [1006, None, 'test@company.com', 28, 'Engineering', 70000.00, 65000.00, '555-678-9012', 'Active', 3500.00, '07/20/2024', 'EMP006'], # Name is None (completeness check) ] - + results = validator.validate_batch(records) - + # Step 4: Display results for idx, result in enumerate(results): status_symbol = '[PASS]' if result.is_valid else '[FAIL]' - + print(f"\nRecord {idx + 1}: {status_symbol} (Score: {result.score}, Pass Rate: {result.pass_rate:.1f}%)") - + if not result.is_valid: for error in result.errors: print(f" - {error.column_name}: {error.message}") - + # Step 5: Summary statistics print("\n" + "=" * 60) print("Summary") print("=" * 60) - + total_records = len(results) valid_records = sum(1 for r in results if r.is_valid) invalid_records = total_records - valid_records overall_pass_rate = (valid_records / total_records) * 100 - + print(f"Total Records: {total_records}") print(f"Valid Records: {valid_records}") print(f"Invalid Records: {invalid_records}") print(f"Overall Pass Rate: {overall_pass_rate:.1f}%") - + # Step 6: Detailed validation result for first failed record print("\n" + "=" * 60) print("Detailed Result Example (First Failed Record)") print("=" * 60) - + failed_result = next((r for r in results if not r.is_valid), None) if failed_result: print(f"\nRecord Index: {failed_result.record_index}") diff --git a/examples/checks_usage.py b/examples/checks_usage.py index 3048cc8..63f3e75 100644 --- a/examples/checks_usage.py +++ b/examples/checks_usage.py @@ -24,7 +24,7 @@ - Returns: The check ID (string) get_checks(): Retrieve checks for a specific asset filtered by check type - - Requires: asset_id, check_type, and either project_id OR catalog_id + - Requires: dq_asset_id, check_type, and either project_id OR catalog_id - Optional: include_children (defaults to True) - Returns: List of check objects matching the criteria @@ -36,36 +36,36 @@ def main(): """Main function demonstrating CheckProvider usage.""" - + print("=" * 70) print("CheckProvider - Usage Examples") print("=" * 70) - + # Step 1: Configure the provider with your instance URL and authentication token config = ProviderConfig( url="https://your-instance.com", auth_token="Bearer your-auth-token-here" ) - + print("\nConfiguration:") print(f" URL: {config.url}") print(f" Auth Token: {config.auth_token[:50]}...") - + # Step 2: Create a ChecksProvider instance check_provider = ChecksProvider(config) print("\n✓ ChecksProvider initialized") - + # Define IDs that will be used throughout the examples project_id = "your-project-id-here" catalog_id = "your-catalog-id-here" # Alternative to project_id - + # ========================================================================= # Example 1: Create a new check without specifying check_type (using project_id) # ========================================================================= print("\n" + "=" * 70) print("Example 1: Create Check without check_type (defaults to name)") print("=" * 70) - + try: # Note: Update these values with your actual data # When check_type is not provided, it defaults to the check name @@ -78,38 +78,43 @@ def main(): ) print(f"\n✓ Successfully created check") print(f" New Check ID: {check_id}") - + except ValueError as e: print(f"\n✗ Error creating check: {e}") - + # ========================================================================= - # Example 2: Create a comparison check (using project_id) + # Example 2: Create a comparison check with parent_check_id (using project_id) # ========================================================================= print("\n" + "=" * 70) - print("Example 2: Create a Comparison Check (using project_id)") + print("Example 2: Create a Comparison Check with parent_check_id (using project_id)") print("=" * 70) - + try: + # First, let's assume we have a parent check ID from Example 1 + parent_check_id = "848aaddc-7401-4a43-ad2b-96a0946d4674" # Replace with actual parent ID + check_id = check_provider.create_check( name="Example Comparison Check", dimension_id="ec453723-669c-48bb-82c1-11b69b3b8c93", # Validity dimension native_id="your-asset-id/your-check-id-2", check_type="comparison", - project_id=project_id + project_id=project_id, + parent_check_id=parent_check_id # Optional: Link to parent check ) - print(f"\n✓ Successfully created comparison check") + print(f"\n✓ Successfully created comparison check with parent") print(f" New Check ID: {check_id}") - + print(f" Parent Check ID: {parent_check_id}") + except ValueError as e: print(f"\n✗ Error creating check: {e}") - + # ========================================================================= # Example 3: Create a check using catalog_id instead of project_id # ========================================================================= print("\n" + "=" * 70) print("Example 3: Create Check using catalog_id") print("=" * 70) - + try: check_id = check_provider.create_check( name="Catalog Check Example", @@ -120,17 +125,17 @@ def main(): ) print(f"\n✓ Successfully created check using catalog_id") print(f" New Check ID: {check_id}") - + except ValueError as e: print(f"\n✗ Error creating check: {e}") - + # ========================================================================= # Example 4: Create multiple checks with different types # ========================================================================= print("\n" + "=" * 70) print("Example 4: Create Multiple Checks with Different Types") print("=" * 70) - + check_configs = [ { "name": "Uniqueness Check", @@ -142,25 +147,31 @@ def main(): "name": "Completeness Check", "dimension_id": "ec453723-669c-48bb-82c1-11b69b3b8c93", "native_id": "asset-1/completeness-check", - "check_type": "data_rule" + "check_type": "data_rule", + "parent_check_id": "parent-check-id-123" # Optional: Add parent for hierarchical checks }, { "name": "Format Validation Check", "dimension_id": "371114cd-5516-4691-8b2e-1e66edf66486", "native_id": "asset-1/format-check", - "check_type": "comparison" + "check_type": "comparison", + "parent_check_id": "parent-check-id-456" # Optional: Add parent for hierarchical checks } ] - + created_checks = [] for config in check_configs: try: + # Get parent_check_id if it exists in config, otherwise None + parent_check_id = config.get("parent_check_id") + check_id = check_provider.create_check( name=config["name"], dimension_id=config["dimension_id"], native_id=config["native_id"], check_type=config["check_type"], - project_id=project_id + project_id=project_id, + parent_check_id=parent_check_id ) created_checks.append({ "id": check_id, @@ -172,16 +183,16 @@ def main(): print(f" Type: {config['check_type']}") except ValueError as e: print(f"\n✗ Error creating {config['name']}: {e}") - + print(f"\n\nSummary: Successfully created {len(created_checks)} checks") - + # ========================================================================= # Example 5: Error handling - Missing required parameters # ========================================================================= print("\n" + "=" * 70) print("Example 5: Error Handling - Missing project_id and catalog_id") print("=" * 70) - + try: # This should fail because neither project_id nor catalog_id is provided check_provider.create_check( @@ -192,14 +203,14 @@ def main(): ) except ValueError as e: print(f"\n✓ Expected error caught: {e}") - + # ========================================================================= # Example 6: Error handling - Both project_id and catalog_id provided # ========================================================================= print("\n" + "=" * 70) print("Example 6: Error Handling - Both project_id and catalog_id provided") print("=" * 70) - + try: # This should fail because both project_id and catalog_id are provided check_provider.create_check( @@ -211,30 +222,30 @@ def main(): ) except ValueError as e: print(f"\n✓ Expected error caught: {e}") - + # ========================================================================= # Example 7: Get checks for a specific asset filtered by check type # ========================================================================= print("\n" + "=" * 70) print("Example 7: Get Checks for an Asset (filtered by check_type)") print("=" * 70) - + try: # Retrieve all checks for a specific column asset filtered by check type column_asset_id = "your-column-asset-id-here" check_type = "case" # e.g., "case", "completeness", "comparison", etc. - + checks = check_provider.get_checks( - asset_id=column_asset_id, + dq_asset_id=column_asset_id, check_type=check_type, project_id=project_id ) - + print(f"\n✓ Successfully retrieved checks") print(f" Asset ID: {column_asset_id}") print(f" Check Type Filter: {check_type}") print(f" Number of checks found: {len(checks)}") - + # Display details of each check for i, check in enumerate(checks, 1): print(f"\n Check {i}:") @@ -242,31 +253,31 @@ def main(): print(f" Name: {check.get('name')}") print(f" Type: {check.get('type')}") print(f" Native ID: {check.get('native_id')}") - + except ValueError as e: print(f"\n✗ Error retrieving checks: {e}") - + # ========================================================================= # Example 8: Get checks with include_children parameter # ========================================================================= print("\n" + "=" * 70) print("Example 8: Get Checks with include_children=False") print("=" * 70) - + try: checks = check_provider.get_checks( - asset_id="your-asset-id", + dq_asset_id="your-asset-id", check_type="completeness", project_id=project_id, include_children=False # Don't include child checks ) - + print(f"\n✓ Successfully retrieved checks (without children)") print(f" Number of checks found: {len(checks)}") - + except ValueError as e: print(f"\n✗ Error retrieving checks: {e}") - + print("\n" + "=" * 70) print("Examples Complete!") print("=" * 70) diff --git a/examples/consolidation_usage.py b/examples/consolidation_usage.py index 20745eb..0926d1b 100644 --- a/examples/consolidation_usage.py +++ b/examples/consolidation_usage.py @@ -36,7 +36,7 @@ def main(): print("=" * 70) print("IBM watsonx.data intelligence SDK - Consolidation Usage Example") print("=" * 70) - + metadata = AssetMetadata( table_name='employee_data', columns=[ @@ -54,24 +54,24 @@ def main(): ColumnMetadata('employee_code', DataType.STRING, length=10) ] ) - + print(f"\nAsset: {metadata.table_name}") print(f"Columns: {len(metadata.columns)}") - + # Step 2: Create validator with rules (same as basic_usage.py) validator = Validator(metadata) - + validator.add_rule( ValidationRule('name') .add_check(LengthCheck(min_length=2, max_length=100)) .add_check(CompletenessCheck(missing_values_allowed=False)) ) - + validator.add_rule( ValidationRule('emp_id') .add_check(LengthCheck(min_length=4, max_length=6)) ) - + validator.add_rule( ValidationRule('department') .add_check(ValidValuesCheck( @@ -79,7 +79,7 @@ def main(): case_sensitive=False )) ) - + validator.add_rule( ValidationRule('age') .add_check(ComparisonCheck( @@ -92,7 +92,7 @@ def main(): )) .add_check(DataTypeCheck(expected_type=DType(dtype=DataTypeEnum.INT32))) ) - + validator.add_rule( ValidationRule('salary') .add_check(ComparisonCheck( @@ -100,27 +100,27 @@ def main(): target_column='min_salary' )) ) - + validator.add_rule( ValidationRule('email') .add_check(CaseCheck(case_type=ColumnCaseEnum.LOWER_CASE)) ) - + validator.add_rule( ValidationRule('bonus') .add_check(RangeCheck(min_value=0, max_value=50000)) ) - + validator.add_rule( ValidationRule('phone') .add_check(RegexCheck(pattern=r'^\d{3}-\d{3}-\d{4}$')) ) - + validator.add_rule( ValidationRule('status') .add_check(CaseCheck(case_type=ColumnCaseEnum.NAME_CASE)) ) - + validator.add_rule( ValidationRule('hire_date') .add_check(FormatCheck( @@ -132,14 +132,14 @@ def main(): } )) ) - + print(f"\nValidator configured with {len(validator.rules)} rules") - + # Step 3: Validate records print("\n" + "=" * 70) print("Validating Records") print("=" * 70) - + records = [ [1001, 'John Doe', 'john@company.com', 30, 'engineering', 75000.00, 60000.00, '555-123-4567', 'Active', 5000.00, '2020-01-15', 'EMP001'], [12, 'Jane Smith', 'jane@company.com', 25, 'SALES', 65000.00, 55000.00, '555-234-5678', 'Active', 3000.00, '01/15/2021', 'EMP002'], @@ -148,23 +148,23 @@ def main(): [1005, 'Charlie Brown', 'charlie@company.com', 40, 'Finance', 55000.00, 60000.00, '555-567-8901', 'Active', 4000.00, '2023-06-10', 'EMP005'], [1006, None, 'test@company.com', 28, 'Engineering', 70000.00, 65000.00, '555-678-9012', 'Active', 3500.00, '07/20/2024', 'EMP006'], ] - + results = validator.validate_batch(records) - + # Display basic results for idx, result in enumerate(results): status_symbol = '[PASS]' if result.is_valid else '[FAIL]' print(f"Record {idx + 1}: {status_symbol} (Score: {result.score})") - + # Step 4: NEW - Use ValidationResultConsolidated for detailed statistics print("\n" + "=" * 70) print("Consolidated Statistics (NEW FEATURE)") print("=" * 70) - + # Create consolidator with error storage enabled consolidator = ValidationResultConsolidated(validator=validator, store_errors=True) consolidator.add_results(results) - + # Overall statistics print("\n[Overall Statistics]") overall = consolidator.get_overall_statistics() @@ -173,43 +173,43 @@ def main(): print(f" Invalid Records: {overall['invalid_records']}") print(f" Pass Rate: {overall['pass_rate']:.1f}%") print(f" Total Errors: {overall['total_errors']}") - + # Statistics by column print("\n[Statistics by Column]") print(f" {'Column':<20} {'Passed':<10} {'Failed':<10} {'Total':<10}") print(f" {'-'*20} {'-'*10} {'-'*10} {'-'*10}") - + for column in sorted(consolidator.get_columns()): stats = consolidator.get_column_statistics(column) print(f" {column:<20} {stats['passed']:<10} {stats['failed']:<10} {stats['total']:<10}") - + # Statistics by check type print("\n[Statistics by Check Type]") print(f" {'Check Type':<30} {'Passed':<10} {'Failed':<10} {'Total':<10}") print(f" {'-'*30} {'-'*10} {'-'*10} {'-'*10}") - + for check in sorted(consolidator.get_checks()): stats = consolidator.get_check_statistics(check) print(f" {check:<30} {stats['passed']:<10} {stats['failed']:<10} {stats['total']:<10}") - + # Combined statistics (column + check) print("\n[Combined Statistics (Column + Check Type)]") combined = consolidator.get_combined_statistics() - + for column in sorted(combined.keys()): print(f"\n {column}:") for check, stats in sorted(combined[column].items()): print(f" {check:<28} Failed: {stats['failed']} Passed: {stats['passed']}") - + # Error details for specific columns print("\n" + "=" * 70) print("Error Details by Column") print("=" * 70) - + # Show errors for columns with failures - columns_with_errors = [col for col in consolidator.get_columns() + columns_with_errors = [col for col in consolidator.get_columns() if consolidator.get_column_statistics(col)['failed'] > 0] - + for column in sorted(columns_with_errors)[:3]: # Show first 3 columns with errors errors = consolidator.get_errors_by_column(column) print(f"\n[X] {column} ({len(errors)} error(s)):") @@ -218,27 +218,27 @@ def main(): print(f" Value: {error['value']}") if error['expected']: print(f" Expected: {error['expected']}") - + # Error details by check type print("\n" + "=" * 70) print("Error Details by Check Type") print("=" * 70) - + # Show errors for specific check types check_types_with_errors = [check for check in consolidator.get_checks() if consolidator.get_check_statistics(check)['failed'] > 0] - + for check in sorted(check_types_with_errors)[:3]: # Show first 3 check types errors = consolidator.get_errors_by_check(check) print(f"\n[Check: {check}] ({len(errors)} error(s)):") for error in errors[:2]: # Show first 2 errors per check print(f" Column '{error['column']}' at record {error['record_index']}: {error['message']}") - + # Specific column + check combination print("\n" + "=" * 70) print("Specific Column + Check Combination") print("=" * 70) - + # Example: Get all case_check errors for email column if 'email' in consolidator.get_columns() and 'case_check' in consolidator.get_checks(): email_case_errors = consolidator.get_errors_by_column_and_check('email', 'case_check') @@ -246,42 +246,42 @@ def main(): for error in email_case_errors: print(f" Record {error['record_index']}: {error['message']}") print(f" Value: '{error['value']}'") - + # Issues by Data Quality Dimension print("\n" + "=" * 70) print("Issues by Data Quality Dimension") print("=" * 70) - + # 1. Get issues for all dimensions print("\n[All Dimensions]") print(f" {'Dimension':<20} {'Issues':<10}") print(f" {'-'*20} {'-'*10}") - + all_dimension_issues = consolidator.get_all_dimension_issues() for dimension_name, issue_count in sorted(all_dimension_issues.items()): print(f" {dimension_name:<20} {issue_count:<10}") - + # 2. Get issues for only the VALIDITY dimension print("\n[VALIDITY Dimension Only]") validity_issues = consolidator.get_issues_by_dimension(DataQualityDimension.VALIDITY) print(f" VALIDITY: {validity_issues} issue(s)") - + # Step 5: Report Issues to CPD (Optional) print("\n" + "=" * 70) print("Reporting Issues to CPD (Optional)") print("=" * 70) - + # Uncomment and configure the following section to report issues to CAMS from wxdi.dq_validator.provider import ProviderConfig from wxdi.dq_validator.issue_reporting import IssueReporter - + # Configure provider config = ProviderConfig( url="https://your-cpd-instance.com", auth_token="Bearer your-token-here", project_id="your-project-id-here" ) - + # Initialize IssueReporter reporter = IssueReporter(config) print("\n[IssueReporter Initialized]") @@ -289,14 +289,14 @@ def main(): print(" ✓ IssuesProvider ready") print(" ✓ DimensionProvider ready") print(" ✓ AssetProvider ready") - + # Report issues cams_asset_id = "your-asset-id-here" - + print(f"\n[Configuration]") print(f" Asset ID: {cams_asset_id}") print(f" Project ID: {config.project_id}") - + try: reporter.report_issues( stats=combined, @@ -304,50 +304,50 @@ def main(): validator=validator ) print("\n[SUCCESS] Issues reported to CPD successfully!") - + except Exception as e: print(f"\n[ERROR] Failed to report issues: {str(e)}") - + print("\n[Note] Issue reporting to CPD is optional and requires:") print(" 1. Valid CPD instance URL") print(" 2. Authentication token") print(" 3. CAMS asset ID") print(" 4. Project ID or Catalog ID") print(" Uncomment the code above and configure to enable.") - + # Export to dictionary print("\n" + "=" * 70) print("Export Consolidated Data") print("=" * 70) - + consolidated_dict = consolidator.to_dict() print(f"\n[Consolidated data structure]") print(f" - Overall statistics: {len(consolidated_dict['overall'])} metrics") print(f" - Columns tracked: {len(consolidated_dict['columns'])}") print(f" - Check types tracked: {len(consolidated_dict['checks'])}") print(f" - Total errors stored: {consolidated_dict['error_count']}") - + # Memory-efficient mode demonstration print("\n" + "=" * 70) print("Memory-Efficient Mode (No Error Storage)") print("=" * 70) - + # Create consolidator without error storage consolidator_lite = ValidationResultConsolidated(validator=validator, store_errors=False) consolidator_lite.add_results(results) - + print("\n[Memory-efficient consolidator]") print(f" Total Records: {consolidator_lite.total_records}") print(f" Valid Records: {consolidator_lite.valid_records}") print(f" Statistics available: YES") print(f" Error details available: NO (memory efficient)") - + # Try to access error details (will raise error) try: consolidator_lite.get_errors_by_column('email') except RuntimeError as e: print(f"\n Expected error when accessing error details: {str(e)[:50]}...") - + print("\n" + "=" * 70) print("[SUCCESS] Consolidation example completed!") print("=" * 70) diff --git a/examples/data_product_recommender_example.py b/examples/data_product_recommender_example.py new file mode 100644 index 0000000..13a1523 --- /dev/null +++ b/examples/data_product_recommender_example.py @@ -0,0 +1,136 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Example usage of Data Product Recommender + +This example demonstrates how to use the Data Product Recommender to analyze +query logs and generate recommendations for data products. +""" + +from wxdi.data_product_recommender.platforms import SnowflakeQueryParser +from wxdi.data_product_recommender.recommender import DataProductRecommender + + +def main(): + """Example: Analyze Snowflake query logs from a CSV file""" + + # Initialize platform-specific parser + print("Initializing Snowflake query parser...") + parser = SnowflakeQueryParser() + + # Initialize recommender + recommender = DataProductRecommender(parser) + + # Load query logs from CSV file + # Replace with your actual query log file path + csv_file_path = 'path/to/your/query_logs.csv' + + print(f"\nLoading query logs from: {csv_file_path}") + try: + recommender.load_query_logs_from_csv_file(csv_file_path) + except FileNotFoundError: + print(f"Error: File not found: {csv_file_path}") + print("Please update the csv_file_path variable with your actual query log file.") + return + + # Calculate metrics + print("\nCalculating table metrics...") + recommender.calculate_metrics() + + # Generate recommendations + print("\nGenerating recommendations...") + recommendations = recommender.recommend_data_products( + num_recommendations=20, + min_score=50.0 # Only include tables with score >= 50 + ) + + # Export recommendations as Markdown (human-readable) + output_file_md = 'output/recommendations_example.md' + print(f"\nExporting recommendations to: {output_file_md}") + recommender.export_recommendations_markdown(recommendations, output_file_md) + + # Export recommendations as JSON (agent-consumable) + output_file_json = 'output/recommendations_example.json' + print(f"Exporting recommendations to: {output_file_json}") + recommender.export_recommendations_json(recommendations, output_file_json) + + # Print summary + print("\n✓ Analysis complete!") + print(f" - Queries analyzed: {len(recommender.query_logs):,}") + print(f" - Tables identified: {len(recommender.table_metrics)}") + print(f" - Recommendations: {len(recommendations['individual_tables'])}") + if 'table_groups' in recommendations: + print(f" - Table groups: {len(recommendations['table_groups'])}") + print(f" - Markdown output: {output_file_md}") + print(f" - JSON output: {output_file_json}") + + +def example_with_json_input(): + """Example: Analyze query logs from a JSON file""" + + from wxdi.data_product_recommender.platforms import DatabricksQueryParser + + # Initialize for Databricks + parser = DatabricksQueryParser() + recommender = DataProductRecommender(parser) + + # Load from JSON file + json_file_path = 'path/to/your/query_logs.json' + recommender.load_query_logs_from_json_file(json_file_path) + + # Calculate and generate recommendations + recommender.calculate_metrics() + recommendations = recommender.recommend_data_products(num_recommendations=15) + + # Export + recommender.export_recommendations_markdown(recommendations, 'output/databricks_recommendations.md') + + +def example_with_custom_scoring(): + """Example: Use custom scoring weights""" + + from wxdi.data_product_recommender.platforms import BigQueryQueryParser + + parser = BigQueryQueryParser() + recommender = DataProductRecommender(parser) + + # Load data + recommender.load_query_logs_from_csv_file('path/to/query_logs.csv') + recommender.calculate_metrics() + + # Score tables with custom weights + # Emphasize user diversity over query frequency + scored_tables = recommender.score_tables( + query_weight=0.25, # 25% weight on query frequency + user_weight=0.50, # 50% weight on user diversity + recency_weight=0.15, # 15% weight on recency + consistency_weight=0.10 # 10% weight on consistency + ) + + print("\nTop 10 tables by custom scoring:") + print(scored_tables[['table', 'recommendation_score', 'query_count', 'unique_users']].head(10)) + + +if __name__ == '__main__': + # Run the main example + main() + + # Uncomment to run other examples: + # example_with_json_input() + # example_with_custom_scoring() + +# Made with Bob diff --git a/examples/dimensions_usage.py b/examples/dimensions_usage.py index c4d9e4c..4b5a008 100644 --- a/examples/dimensions_usage.py +++ b/examples/dimensions_usage.py @@ -29,103 +29,103 @@ def main(): """Main function demonstrating DimensionsProvider usage.""" - + print("=" * 70) print("DimensionsProvider - Usage Examples") print("=" * 70) - + # Step 1: Configure the provider with your instance URL and authentication token config = ProviderConfig( url="https://your-instance.cloud.ibm.com", auth_token="Bearer your-auth-token-here" ) - + print("\nConfiguration:") print(f" URL: {config.url}") print(f" Auth Token: {config.auth_token[:50]}...") - + # Step 2: Create a DimensionsProvider instance dimension_provider = DimensionsProvider(config) print("\n✓ DimensionsProvider initialized") - + # ========================================================================= # Example 1: Get Completeness dimension # ========================================================================= print("\n" + "=" * 70) print("Example 1: Get Completeness Dimension") print("=" * 70) - + try: dimension_name = "Completeness" dimension_id = dimension_provider.search_dimension(dimension_name) - + print(f"\n✓ Successfully retrieved dimension: {dimension_name}") print(f" Dimension ID: {dimension_id}") - + except ValueError as e: print(f"\n✗ Error getting dimension: {e}") - + # ========================================================================= # Example 2: Get Accuracy dimension # ========================================================================= print("\n" + "=" * 70) print("Example 2: Get Accuracy Dimension") print("=" * 70) - + try: dimension_name = "Accuracy" dimension_id = dimension_provider.search_dimension(dimension_name) - + print(f"\n✓ Successfully retrieved dimension: {dimension_name}") print(f" Dimension ID: {dimension_id}") - + except ValueError as e: print(f"\n✗ Error getting dimension: {e}") - + # ========================================================================= # Example 3: Get Consistency dimension # ========================================================================= print("\n" + "=" * 70) print("Example 3: Get Consistency Dimension") print("=" * 70) - + try: dimension_name = "Consistency" dimension_id = dimension_provider.search_dimension(dimension_name) - + print(f"\n✓ Successfully retrieved dimension: {dimension_name}") print(f" Dimension ID: {dimension_id}") - + except ValueError as e: print(f"\n✗ Error getting dimension: {e}") - + # ========================================================================= # Example 4: Test case-insensitive matching # ========================================================================= print("\n" + "=" * 70) print("Example 4: Case-Insensitive Matching (lowercase 'completeness')") print("=" * 70) - + try: dimension_name = "completeness" # lowercase dimension_id = dimension_provider.search_dimension(dimension_name) - + print(f"\n✓ Successfully retrieved dimension with lowercase name") print(f" Dimension Name: {dimension_name}") print(f" Dimension ID: {dimension_id}") - + except ValueError as e: print(f"\n✗ Error getting dimension: {e}") - + # ========================================================================= # Example 5: Get multiple dimensions in a loop # ========================================================================= print("\n" + "=" * 70) print("Example 5: Get Multiple Dimensions") print("=" * 70) - + dimension_names = ["Completeness", "Accuracy", "Consistency"] - + print("\nRetrieving multiple dimensions:") for idx, dim_name in enumerate(dimension_names, 1): try: @@ -133,25 +133,25 @@ def main(): print(f"{idx}. ✓ {dim_name}: {dim_id}") except ValueError as e: print(f"{idx}. ✗ {dim_name}: Failed - {e}") - + # ========================================================================= # Example 6: Try to get a non-existent dimension (error handling) # ========================================================================= print("\n" + "=" * 70) print("Example 6: Error Handling - Non-existent Dimension") print("=" * 70) - + try: dimension_name = "NonExistentDimension" dimension_id = dimension_provider.search_dimension(dimension_name) - + print(f"\n✓ Retrieved dimension: {dimension_name}") print(f" Dimension ID: {dimension_id}") - + except ValueError as e: print(f"\n✗ Expected error for non-existent dimension:") print(f" Error: {e}") - + print("\n" + "=" * 70) print("Examples Complete!") print("=" * 70) diff --git a/examples/dq_workflow_usage.py b/examples/dq_workflow_usage.py index 85cd414..443ba30 100644 --- a/examples/dq_workflow_usage.py +++ b/examples/dq_workflow_usage.py @@ -39,51 +39,51 @@ def main(): catalog_id = None # Alternative: use catalog_id instead of project_id column_name = "RTN" # Column to check check_type = "format" # Type of check to find - + config = ProviderConfig( url="https://cpd-ikc.apps.dqdev.ibm.com", auth_token="Bearer your-token-here", project_id=project_id ) - + # Initialize providers cams_provider = CamsProvider(config) dq_search = DQSearchProvider(config) issues_provider = IssuesProvider(config) - + # Step 1: Fetch CAMS asset by data_asset_id and project_id print("=" * 70) print("STEP 1: Fetch CAMS Asset") print("=" * 70) - + try: cams_asset = cams_provider.get_asset_by_id( asset_id=data_asset_id, options={"hide_deprecated_response_fields": "false"} ) - + print(f"✓ Fetched CAMS Asset:") print(f" Asset ID: {cams_asset.metadata.asset_id}") print(f" Name: {cams_asset.metadata.name}") print(f" Asset Type: {cams_asset.metadata.asset_type}") - + except ValueError as e: print(f"✗ Error fetching CAMS asset: {e}") return - + # Step 2: Read the CAMS object to get check_id from column_info # Iterate through column_checks and find the first check with matching type print("\n" + "=" * 70) print("STEP 2: Extract Check ID from CAMS Asset") print("=" * 70) - + check_id = None - + try: # Navigate to column_info for the specified column if hasattr(cams_asset.entity, 'column_info') and column_name in cams_asset.entity.column_info: column_info = cams_asset.entity.column_info[column_name] - + # Iterate through column_checks to find the check with matching type if hasattr(column_info, 'column_checks') and column_info.column_checks: for check in column_info.column_checks: @@ -93,7 +93,7 @@ def main(): print(f" Check ID: {check_id}") print(f" Check Type: {check.metadata.type}") break - + if not check_id: print(f"✗ No check with type '{check_type}' found for column '{column_name}'") return @@ -103,19 +103,19 @@ def main(): else: print(f"✗ Column '{column_name}' not found in asset") return - + except Exception as e: print(f"✗ Error extracting check ID: {e}") return - + # Step 3: Search for DQ check by native_id # The native_id format is: / print("\n" + "=" * 70) print("STEP 3: Search for DQ Check") print("=" * 70) - + check_native_id = f"{data_asset_id}/{check_id}" - + try: check_result = dq_search.search_dq_check( native_id=check_native_id, @@ -123,28 +123,28 @@ def main(): project_id=project_id, include_children=True ) - + print(f"✓ Found DQ Check:") print(f" ID: {check_result['id']}") print(f" Name: {check_result['name']}") print(f" Type: {check_result['type']}") print(f" Native ID: {check_result['native_id']}") - + dq_check_id = check_result['id'] - + except ValueError as e: print(f"✗ Error searching for check: {e}") return - + # Step 4: Search for DQ asset by native_id # The native_id format is: / print("\n" + "=" * 70) print("STEP 4: Search for DQ Asset") print("=" * 70) - + asset_native_id = f"{data_asset_id}/{column_name}" asset_type = "column" - + try: asset_result = dq_search.search_dq_asset( native_id=asset_native_id, @@ -153,24 +153,24 @@ def main(): include_children=True, get_actual_asset=False ) - + print(f"✓ Found DQ Asset:") print(f" ID: {asset_result['id']}") print(f" Name: {asset_result['name']}") print(f" Type: {asset_result['type']}") print(f" Native ID: {asset_result['native_id']}") - + dq_asset_id = asset_result['id'] - + except ValueError as e: print(f"✗ Error searching for asset: {e}") return - + # Step 5: Search issue for a specific asset and check print("\n" + "=" * 70) print("STEP 5: Search Issue for Asset and Check") print("=" * 70) - + try: # Use either project_id or catalog_id issue_id = issues_provider.get_issue_id( @@ -178,19 +178,19 @@ def main(): check_id=dq_check_id, project_id=project_id # Or use catalog_id=catalog_id ) - + print(f"✓ Found Issue:") print(f" Issue ID: {issue_id}") - + except ValueError as e: print(f"✗ Error searching for issue: {e}") return - + # Step 6: Update (patch) the issue by issue_id print("\n" + "=" * 70) print("STEP 6: Update Issue (Patch)") print("=" * 70) - + try: # Update both occurrences and tested records in a single call new_occurrences = 100 @@ -205,16 +205,16 @@ def main(): ) print(f"✓ Updated occurrences: {update_result.get('number_of_occurrences', 'N/A')}") print(f"✓ Updated tested records: {update_result.get('number_of_tested_records', 'N/A')}") - + # The percent_occurrences is typically calculated automatically by the system # based on number_of_occurrences and number_of_tested_records if 'percent_occurrences' in update_result: print(f"✓ Calculated percent occurrences: {update_result['percent_occurrences']}%") - + except ValueError as e: print(f"✗ Error updating issue: {e}") return - + print("\n" + "=" * 70) print("WORKFLOW COMPLETED SUCCESSFULLY") print("=" * 70) @@ -228,16 +228,16 @@ def example_with_add_operations(): url="https://cpd-ikc.apps.dqdev.ibm.com", auth_token="Bearer your-token-here" ) - + issues_provider = IssuesProvider(config) issue_id = "b8f4252b-cd35-4668-9b35-4635bfc6e2e0" - + print("=" * 70) print("EXAMPLE: Add Operations") print("=" * 70) - + project_id = "your-project-id-here" - + try: # Add 10 more occurrences and 50 more tested records to the existing counts print("\nAdding 10 occurrences and 50 tested records to existing counts...") @@ -250,7 +250,7 @@ def example_with_add_operations(): ) print(f"✓ New occurrences count: {result.get('number_of_occurrences', 'N/A')}") print(f"✓ New tested records count: {result.get('number_of_tested_records', 'N/A')}") - + except ValueError as e: print(f"✗ Error: {e}") @@ -269,10 +269,10 @@ def example_with_add_operations(): print("6. Update (patch) the issue by issue_id") print("\nNote: All methods support either project_id OR catalog_id (but not both)") print("\n" + "=" * 70 + "\n") - + # Run the main workflow main() - + # Uncomment to run additional examples: # print("\n\n") # example_with_add_operations() diff --git a/examples/end_to_end_example.py b/examples/end_to_end_example.py new file mode 100644 index 0000000..574a2c7 --- /dev/null +++ b/examples/end_to_end_example.py @@ -0,0 +1,404 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +End-to-End Data Quality Validation Example + +This comprehensive example demonstrates the complete workflow: +1. Generate authentication token for IBM Cloud environment +2. Load asset metadata from CAMS API using DQAssetsProvider +3. Load data from a CSV file using Pandas DataFrame +4. Add additional validation rules (e.g., range check) +5. Run validation on the DataFrame +6. Print and analyze the results + +Prerequisites: +- IBM Cloud API key +- Project ID or Catalog ID +- Asset ID from CAMS +- CSV file matching the asset metadata +- pandas library installed +""" + +import pandas as pd +from wxdi.common.auth import AuthConfig, EnvironmentType, AuthProvider +from wxdi.dq_validator.provider import ProviderConfig +from wxdi.dq_validator.provider.cams import CamsProvider +from wxdi.dq_validator.rule_loader import RuleLoader +from wxdi.dq_validator import ValidationRule, RangeCheck, ComparisonCheck, ComparisonOperator +from wxdi.dq_validator.integrations import PandasValidator + + +def main(): + """ + Main function demonstrating end-to-end data quality validation workflow. + """ + print("=" * 80) + print("END-TO-END DATA QUALITY VALIDATION EXAMPLE") + print("=" * 80) + + # ========================================================================= + # Step 1: Generate Authentication Token for IBM Cloud + # ========================================================================= + print("\n" + "=" * 80) + print("STEP 1: Generate Authentication Token") + print("=" * 80) + + # Configure authentication for IBM Cloud environment + # Replace with your actual IBM Cloud API key + auth_config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='your-ibm-cloud-api-key-here', + url='https://iam.test.cloud.ibm.com', # Optional, default URL is used if not provided + + # url is optional for production IBM Cloud (default URL is used) + # disable_ssl_verification=True # Optional, default is True + ) + + print(f"\nEnvironment Type: {auth_config.environment_type.value}") + print(f"Authentication URL: {auth_config.url}") + + # Create auth provider and get token + auth_provider = AuthProvider(auth_config) + print(f"Authenticator Type: {type(auth_provider.authenticator).__name__}") + + # Get the authentication token + # Uncomment the following lines when you have valid credentials + # try: + # auth_token = auth_provider.get_token() + # print(f"✓ Token generated successfully: {auth_token[:20]}...") + # except Exception as e: + # print(f"✗ Error generating token: {e}") + # return + + # For demonstration purposes, using a placeholder token + auth_token = "Bearer your-auth-token-here" + print(f"✓ Using token: {auth_token[:30]}...") + + # ========================================================================= + # Step 2: Load Asset Metadata from CAMS API + # ========================================================================= + print("\n" + "=" * 80) + print("STEP 2: Load Asset Metadata from CAMS") + print("=" * 80) + + # Configure the provider with your instance URL and authentication + base_url = "https://your-instance.cloud.ibm.com" + project_id = "your-project-id-here" # Or use catalog_id instead + asset_id = "your-asset-id-here" + + print(f"\nConfiguration:") + print(f" Base URL: {base_url}") + print(f" Project ID: {project_id}") + print(f" Asset ID: {asset_id}") + + # Create provider configuration + provider_config = ProviderConfig( + url=base_url, + auth_token=auth_token, + project_id=project_id + ) + + # Create CAMS provider + cams_provider = CamsProvider(provider_config) + print(f"\n✓ CAMS Provider initialized") + + # Load asset metadata and validation rules from CAMS + # Uncomment the following lines when you have valid credentials + # try: + # data_asset = cams_provider.get_asset_by_id(asset_id) + # print(f"✓ Asset loaded: {data_asset.metadata.name}") + # print(f" Asset Type: {data_asset.metadata.asset_type}") + # print(f" Columns: {len(data_asset.entity.data_asset.columns)}") + # + # # Use RuleLoader to extract metadata and validation rules + # rule_loader = RuleLoader(base_url, auth_token) + # validator = rule_loader.load_from_data_asset(data_asset) + # print(f"✓ Validator created with {len(validator.rules)} rules from CAMS") + # except Exception as e: + # print(f"✗ Error loading asset: {e}") + # return + + # For demonstration, create sample metadata manually + from wxdi.dq_validator import AssetMetadata, ColumnMetadata, DataType, Validator + + metadata = AssetMetadata( + table_name='customer_data', + columns=[ + ColumnMetadata('customer_id', DataType.INTEGER), + ColumnMetadata('name', DataType.STRING, length=100), + ColumnMetadata('email', DataType.STRING, length=255), + ColumnMetadata('age', DataType.INTEGER), + ColumnMetadata('account_balance', DataType.DECIMAL, precision=10, scale=2), + ColumnMetadata('registration_date', DataType.DATE), + ColumnMetadata('status', DataType.STRING, length=20), + ] + ) + + print(f"\n✓ Using sample metadata:") + print(f" Table: {metadata.table_name}") + print(f" Columns: {len(metadata.columns)}") + for col in metadata.columns: + print(f" - {col.name} ({col.data_type.value})") + + # Create validator with metadata + validator = Validator(metadata) + print(f"\n✓ Validator initialized") + + # ========================================================================= + # Step 3: Load Data from CSV File using Pandas + # ========================================================================= + print("\n" + "=" * 80) + print("STEP 3: Load Data from CSV File") + print("=" * 80) + + # For demonstration, create sample data that matches the metadata + # In production, you would load from an actual CSV file: + # df = pd.read_csv('customer_data.csv') + + sample_data = { + 'customer_id': [1001, 1002, 1003, 1004, 1005, 1006], + 'name': ['John Doe', 'Jane Smith', 'Bob Johnson', 'Alice Williams', 'Charlie Brown', 'Diana Prince'], + 'email': ['john@example.com', 'jane@example.com', 'bob@invalid', 'alice@example.com', 'charlie@example.com', 'diana@example.com'], + 'age': [25, 30, 17, 45, 35, 28], # Note: 17 is below typical minimum age + 'account_balance': [1500.50, 2500.75, 500.00, 5000.00, 3500.25, 1200.00], + 'registration_date': ['2023-01-15', '2023-02-20', '2023-03-10', '2023-04-05', '2023-05-12', '2023-06-18'], + 'status': ['active', 'active', 'pending', 'active', 'inactive', 'active'] + } + + df = pd.DataFrame(sample_data) + + print(f"\n✓ Data loaded from CSV (sample data)") + print(f" Rows: {len(df)}") + print(f" Columns: {list(df.columns)}") + print(f"\nFirst 3 rows:") + print(df.head(3).to_string(index=False)) + + # ========================================================================= + # Step 4: Add Additional Validation Rules + # ========================================================================= + print("\n" + "=" * 80) + print("STEP 4: Add Additional Validation Rules") + print("=" * 80) + + # Add range check for age (must be between 18 and 100) + age_rule = ValidationRule('age') + age_rule.add_check(RangeCheck(min_value=18, max_value=100)) + validator.add_rule(age_rule) + print(f"\n✓ Added range check for 'age' column (18-100)") + + # Add range check for account_balance (must be >= 1000) + balance_rule = ValidationRule('account_balance') + balance_rule.add_check(ComparisonCheck( + operator=ComparisonOperator.GREATER_THAN_OR_EQUAL, + target_value=1000.00 + )) + validator.add_rule(balance_rule) + print(f"✓ Added comparison check for 'account_balance' column (>= 1000)") + + # Add valid values check for status + from wxdi.dq_validator import ValidValuesCheck + status_rule = ValidationRule('status') + status_rule.add_check(ValidValuesCheck( + valid_values=['active', 'inactive', 'suspended'], + case_sensitive=False + )) + validator.add_rule(status_rule) + print(f"✓ Added valid values check for 'status' column") + + print(f"\n✓ Total validation rules: {len(validator.rules)}") + for rule in validator.rules: + print(f" - {rule.column_name}: {len(rule.checks)} check(s)") + + # ========================================================================= + # Step 5: Run Validation on DataFrame + # ========================================================================= + print("\n" + "=" * 80) + print("STEP 5: Run Validation") + print("=" * 80) + + # Create Pandas validator + pandas_validator = PandasValidator(validator, chunk_size=1000) + print(f"\n✓ Pandas Validator created (chunk_size=1000)") + + # Get summary statistics + print(f"\nRunning validation...") + summary = pandas_validator.get_summary_statistics(df) + + print(f"\n✓ Validation completed!") + + # ========================================================================= + # Step 6: Print and Analyze Results + # ========================================================================= + print("\n" + "=" * 80) + print("STEP 6: Validation Results") + print("=" * 80) + + # Print summary statistics + print(f"\n{'SUMMARY STATISTICS':^80}") + print("-" * 80) + print(f"Total Rows: {summary['total_rows']:>10}") + print(f"Valid Rows: {summary['valid_rows']:>10}") + print(f"Invalid Rows: {summary['invalid_rows']:>10}") + print(f"Pass Rate: {summary['pass_rate']:>9.2f}%") + print("-" * 80) + print(f"Total Checks: {summary['total_checks']:>10}") + print(f"Passed Checks: {summary['passed_checks']:>10}") + print(f"Failed Checks: {summary['failed_checks']:>10}") + print("-" * 80) + + # Add validation column to DataFrame + df_validated = pandas_validator.add_validation_column(df) + + # Print detailed results for each row + print(f"\n{'DETAILED VALIDATION RESULTS':^80}") + print("-" * 80) + + for idx, row in df_validated.iterrows(): + validation = row['dq_validation_result'] + is_valid = bool(validation['is_valid']) + status_icon = "✓" if is_valid else "✗" + status_text = "PASS" if is_valid else "FAIL" + row_num = idx + 1 if isinstance(idx, int) else idx # type: ignore[operator] + + print(f"\nRow {row_num}: {status_icon} {status_text}") + print(f" Customer: {row['name']} (ID: {row['customer_id']})") + print(f" Age: {row['age']}, Balance: ${row['account_balance']:.2f}, Status: {row['status']}") + print(f" Validation Score: {validation['score']} ({validation['pass_rate']:.1f}%)") + + if not is_valid: + print(f" Errors: {validation['error_count']}") + errors = validation['errors'] + if errors is not None and len(errors) > 0: + for error in validation['errors'][:3]: # Show first 3 errors + print(f" - {error['column']}: {error['message']}") + + # Get and display invalid rows + print(f"\n{'INVALID ROWS DETAILS':^80}") + print("-" * 80) + + invalid_df = pandas_validator.get_invalid_rows(df) + + if len(invalid_df) > 0: + print(f"\nFound {len(invalid_df)} invalid row(s):\n") + + for idx, row in invalid_df.iterrows(): + validation = row['dq_validation_result'] + row_num = idx + 1 if isinstance(idx, int) else idx # type: ignore[operator] + print(f"Row {row_num}:") + print(f" Customer ID: {row['customer_id']}") + print(f" Name: {row['name']}") + print(f" Age: {row['age']}") + print(f" Balance: ${row['account_balance']:.2f}") + print(f" Status: {row['status']}") + print(f" Validation Score: {validation['score']} ({validation['pass_rate']:.1f}%)") + print(f" Failed Checks: {validation['error_count']}") + + errors = validation['errors'] + if errors is not None and len(errors) > 0: + print(f" Errors:") + for error in validation['errors']: + print(f" - Column '{error['column']}': {error['message']}") + if error.get('value') is not None: + print(f" Value: {error['value']}") + if error.get('expected') is not None: + print(f" Expected: {error['expected']}") + print() + else: + print("\n✓ All rows passed validation!") + + # Expand validation columns for analysis + print(f"\n{'EXPANDED VALIDATION COLUMNS':^80}") + print("-" * 80) + + df_expanded = pandas_validator.expand_validation_column(df_validated) + + # Show key validation columns + validation_cols = ['customer_id', 'name', 'dq_is_valid', 'dq_score', 'dq_pass_rate', 'dq_error_count'] + print(f"\nValidation Summary by Row:") + print(df_expanded[validation_cols].to_string(index=False)) + + # Save results to files + print(f"\n{'SAVING RESULTS':^80}") + print("-" * 80) + + # Save invalid rows + if len(invalid_df) > 0: + invalid_df.to_csv('invalid_customers.csv', index=False) + print(f"\n✓ Saved {len(invalid_df)} invalid rows to: invalid_customers.csv") + + # Save full validation results + df_expanded.to_csv('validation_results.csv', index=False) + print(f"✓ Saved full validation results to: validation_results.csv") + + # Save summary report + with open('validation_summary.txt', 'w') as f: + f.write("=" * 80 + "\n") + f.write("DATA QUALITY VALIDATION SUMMARY REPORT\n") + f.write("=" * 80 + "\n\n") + f.write(f"Asset: {metadata.table_name}\n") + f.write(f"Total Rows: {summary['total_rows']}\n") + f.write(f"Valid Rows: {summary['valid_rows']}\n") + f.write(f"Invalid Rows: {summary['invalid_rows']}\n") + f.write(f"Pass Rate: {summary['pass_rate']:.2f}%\n\n") + f.write(f"Total Checks: {summary['total_checks']}\n") + f.write(f"Passed Checks: {summary['passed_checks']}\n") + f.write(f"Failed Checks: {summary['failed_checks']}\n\n") + f.write("Validation Rules Applied:\n") + for rule in validator.rules: + f.write(f" - {rule.column_name}: {len(rule.checks)} check(s)\n") + + print(f"✓ Saved summary report to: validation_summary.txt") + + # ========================================================================= + # Completion + # ========================================================================= + print("\n" + "=" * 80) + print("END-TO-END VALIDATION COMPLETE!") + print("=" * 80) + + print(f"\nKey Takeaways:") + print(f" • Authenticated with IBM Cloud using API key") + print(f" • Loaded asset metadata from CAMS (or created sample)") + print(f" • Loaded {len(df)} rows from CSV file") + print(f" • Applied {len(validator.rules)} validation rules") + print(f" • Validated data with {summary['pass_rate']:.1f}% pass rate") + print(f" • Identified {summary['invalid_rows']} invalid row(s)") + print(f" • Saved results to CSV files for further analysis") + + print(f"\nNext Steps:") + print(f" 1. Review invalid_customers.csv for data quality issues") + print(f" 2. Analyze validation_results.csv for detailed insights") + print(f" 3. Update source data or adjust validation rules as needed") + print(f" 4. Re-run validation to verify improvements") + + print("\n" + "=" * 80) + + +if __name__ == '__main__': + try: + main() + except ImportError as e: + print(f"Error: {e}") + print("\nTo run this example, install required dependencies:") + print(" pip install pandas") + print("Or install with all dependencies:") + print(" pip install wxdi[pandas]") + except Exception as e: + print(f"\nUnexpected error: {e}") + import traceback + traceback.print_exc() + +# Made with Bob diff --git a/examples/glossary_usage.py b/examples/glossary_usage.py index 492f115..e51a8e3 100644 --- a/examples/glossary_usage.py +++ b/examples/glossary_usage.py @@ -42,73 +42,73 @@ def main(): print("=" * 70) print("GlossaryProvider - Usage Examples") print("=" * 70) - + # ========================================================================= # Configuration Option 1: Using static auth token # ========================================================================= print("\n" + "-" * 70) print("Configuration Option 1: Using Static Auth Token") print("-" * 70) - + config_with_token = ProviderConfig( url="https://your-instance.cloud.ibm.com", auth_token="Bearer your-auth-token-here" ) - + print("\nConfiguration:") print(f" URL: {config_with_token.url}") print(f" Auth Token: {config_with_token.auth_token[:50]}...") - + # ========================================================================= # Configuration Option 2: Using AuthConfig (recommended) # ========================================================================= print("\n" + "-" * 70) print("Configuration Option 2: Using AuthConfig (Recommended)") print("-" * 70) - + # Example with IBM Cloud auth_config = AuthConfig( environment_type=EnvironmentType.IBM_CLOUD, api_key="your-api-key-here" ) - + config_with_auth = ProviderConfig( url="https://your-instance.cloud.ibm.com", auth_config=auth_config ) - + print("\nConfiguration:") print(f" URL: {config_with_auth.url}") print(f" Environment: {auth_config.environment_type.value}") print( " Auth Provider: Configured") - + # For the examples below, we'll use config_with_token # In production, use config_with_auth for automatic token management config = config_with_token - + # Step 2: Create a GlossaryProvider instance glossary_provider = GlossaryProvider(config) print("\n✓ GlossaryProvider initialized") - + # Define IDs that will be used throughout the examples artifact_id = "your-artifact-id-here" # Replace with your actual artifact ID version_id = "your-version-id-here" # Replace with your actual version ID - + # ========================================================================= # Example 1: Get published glossary term by artifact ID # ========================================================================= print("\n" + "=" * 70) print("Example 1: Get Published Glossary Term by Artifact ID") print("=" * 70) - + try: print(PARAMETERS_HEADER) print(f" artifact_id: {artifact_id}") - + term = glossary_provider.get_published_artifact_by_id(artifact_id) - + print( "\n✓ Successfully retrieved glossary term") - + print(TERM_DETAILS_HEADER) print(f" Name: {term.metadata.name}") print(f" Artifact Type: {term.metadata.artifact_type}") @@ -117,73 +117,73 @@ def main(): print(f" Version ID: {term.metadata.version_id}") print(f" Created At: {term.metadata.created_at}") print(f" Modified At: {term.metadata.modified_at}") - + if term.metadata.tags: print(f" Tags: {', '.join(term.metadata.tags)}") - + # Display entity information if available if hasattr(term.entity, 'extended_attribute_groups') and term.entity.extended_attribute_groups: dq_constraints = term.entity.extended_attribute_groups.dq_constraints print(f" Data Quality Constraints: {len(dq_constraints)}") - + except ValueError as e: print(f"\n✗ Error: {e}") except Exception as e: print(f"\n✗ Unexpected error: {e}") - + # ========================================================================= # Example 2: Get published term with additional options # ========================================================================= print("\n" + "=" * 70) print("Example 2: Get Published Term with Additional Options") print("=" * 70) - + try: print(PARAMETERS_HEADER) print(f" artifact_id: {artifact_id}") - + # Options can include query parameters like 'include', 'limit', etc. options = { - "include": "relationships" + "include_relationship": "all" } print(f" options: {options}") - + term = glossary_provider.get_published_artifact_by_id( artifact_id, options=options ) - + print( "\n✓ Successfully retrieved glossary term with options") - + print(TERM_DETAILS_HEADER) print(f" Name: {term.metadata.name}") print(f" State: {term.metadata.state}") print(f" Revision: {term.metadata.revision}") - + except ValueError as e: print(f"\n✗ Error: {e}") except Exception as e: print(f"\n✗ Unexpected error: {e}") - + # ========================================================================= # Example 3: Get specific version of a glossary term # ========================================================================= print("\n" + "=" * 70) print("Example 3: Get Specific Version of a Glossary Term") print("=" * 70) - + try: print(PARAMETERS_HEADER) print(f" artifact_id: {artifact_id}") print(f" version_id: {version_id}") - + term = glossary_provider.get_term_by_version_id( artifact_id, version_id ) - + print( "\n✓ Successfully retrieved glossary term version") - + print(TERM_DETAILS_HEADER) print(f" Name: {term.metadata.name}") print(f" Version ID: {term.metadata.version_id}") @@ -191,58 +191,58 @@ def main(): print(f" Effective Start Date: {term.metadata.effective_start_date}") print(f" Created By: {term.metadata.created_by}") print(f" Modified By: {term.metadata.modified_by}") - + if term.metadata.draft_ancestor_id: print(f" Draft Ancestor ID: {term.metadata.draft_ancestor_id}") - + except ValueError as e: print(f"\n✗ Error: {e}") except Exception as e: print(f"\n✗ Unexpected error: {e}") - + # ========================================================================= # Example 4: Get term version with options # ========================================================================= print("\n" + "=" * 70) print("Example 4: Get Term Version with Options") print("=" * 70) - + try: print(PARAMETERS_HEADER) print(f" artifact_id: {artifact_id}") print(f" version_id: {version_id}") - + options = { - "include": "all" + "all_parents": "true" } print(f" options: {options}") - + term = glossary_provider.get_term_by_version_id( artifact_id, version_id, options=options ) - + print( "\n✓ Successfully retrieved glossary term version with options") - + print(TERM_DETAILS_HEADER) print(f" Name: {term.metadata.name}") print(f" Version ID: {term.metadata.version_id}") print(f" State: {term.metadata.state}") print(f" Global ID: {term.metadata.global_id}") print(f" Source Repository ID: {term.metadata.source_repository_id}") - + # Display steward information if available if term.metadata.steward_ids: print(f" Steward IDs: {', '.join(term.metadata.steward_ids)}") if term.metadata.steward_group_ids: print(f" Steward Group IDs: {', '.join(term.metadata.steward_group_ids)}") - + except ValueError as e: print(f"\n✗ Error: {e}") except Exception as e: print(f"\n✗ Unexpected error: {e}") - + print("\n" + "=" * 70) print("Examples Complete!") print("=" * 70) diff --git a/examples/issues_usage.py b/examples/issues_usage.py index e86e587..9d4d321 100644 --- a/examples/issues_usage.py +++ b/examples/issues_usage.py @@ -16,7 +16,7 @@ """ Example usage of IssuesProvider for managing data quality issues. -This example demonstrates four main operations: +This example demonstrates five main operations: 1. get_issues(): Retrieve a specific issue for a DQ asset, check type, and check_id - Requires: dq_asset_id, check_type, check_id, and either project_id OR catalog_id @@ -28,17 +28,22 @@ - Optional: operation ("add" or "replace", default: "add") 3. update_issue_metrics(): Update issue metrics using CAMS asset and check IDs OR check_native_id - - Option A: Provide cams_asset_id and cams_check_id - - Option B: Provide check_native_id (format: "/") + - Option A: Provide asset_id and check_id + - Option B: Provide check_native_id (format: "/") - Also requires: occurrences, tested_records, column_name, check_type, and either project_id OR catalog_id - Optional: asset_type (default: "column"), operation (default: "add") - This method automatically searches for the DQ asset, check, and issue before updating 4. create_issue(): Create a new data quality issue for a check - - Requires: check_id, reported_for_id, number_of_occurrences, number_of_tested_records, and either project_id OR catalog_id + - Requires: dq_check_id, reported_for_id, number_of_occurrences, number_of_tested_records, and either project_id OR catalog_id - Optional: status (default: "actual"), ignored (default: False) - Returns: The created issue_id +5. create_issues_bulk(): Create multiple issues, assets, and checks in a single API call + - Requires: payload (dict with issues, assets, existing_checks), and either project_id OR catalog_id + - Optional: incremental_reporting (default: False), refresh_assets (default: False) + - Returns: The full API response with created issues + Both update methods require occurrences and tested_records as mandatory parameters. The operation parameter (default: "add") applies to both metrics. Either project_id or catalog_id must be provided (but not both). @@ -49,27 +54,27 @@ def main(): """Main function demonstrating IssuesProvider usage.""" - + # Step 1: Configure the provider with your instance URL and authentication token config = ProviderConfig( url="https://your-instance.cloud.ibm.com", auth_token="Bearer your-auth-token-here" ) - + # Step 2: Create an IssuesProvider instance issues_provider = IssuesProvider(config) - + # Define IDs that will be used throughout the examples issue_id = "your-issue-id-here" project_id = "your-project-id-here" catalog_id = "your-catalog-id-here" # Alternative to project_id - + # Step 3: Get issues for a specific DQ asset, check type, and check_id print("\n--- Getting issues for a DQ asset with check_id filter ---") dq_asset_id = "08b139ca-35a6-4b61-b87b-aa832870d89c" check_type = "format" check_id = "065c2b72-4600-4d15-8c48-298a2abf66cd" # The check ID to filter by - + try: # Get issues using catalog_id and check_id issue_result = issues_provider.get_issues( @@ -93,7 +98,7 @@ def main(): print(f"No issue found matching check_id {check_id}") except ValueError as e: print(f"Error getting issues: {e}") - + # Example: Get issues using project_id instead try: issue_result = issues_provider.get_issues( @@ -110,7 +115,7 @@ def main(): print(f"No matching issue found") except ValueError as e: print(f"Error getting issues with project_id: {e}") - + # Step 4: Update both occurrences and tested records using issue ID directly # Note: Both occurrences and tested_records are mandatory # Either project_id OR catalog_id must be provided (but not both) @@ -127,7 +132,7 @@ def main(): print(f"Response: {result}") except ValueError as e: print(f"Error updating issue: {e}") - + # Step 4: Use catalog_id instead of project_id try: result = issues_provider.update_issue_values( @@ -140,7 +145,7 @@ def main(): print(f"Response: {result}") except ValueError as e: print(f"Error updating issue with catalog_id: {e}") - + # Step 5: Use replace operation instead of add try: result = issues_provider.update_issue_values( @@ -154,7 +159,7 @@ def main(): print(f"Response: {result}") except ValueError as e: print(f"Error replacing metrics: {e}") - + # Example: Using replace operation with different values try: # Replace both metrics with specific values @@ -169,14 +174,14 @@ def main(): print(f"Response: {result}") except ValueError as e: print(f"Error resetting metrics: {e}") - + # Example: Update multiple issues with both metrics issues_to_update = [ ("issue-123", 10, 100), ("issue-456", 25, 250), ("issue-789", 50, 500), ] - + print("\n--- Updating multiple issues with both metrics ---") for issue_id, occurrences, records in issues_to_update: try: @@ -189,14 +194,14 @@ def main(): print(f"✓ Updated {issue_id}: +{occurrences} occurrences, +{records} tested records") except ValueError as e: print(f"✗ Failed to update {issue_id}: {e}") - + # Step 6: Update issue metrics using CAMS IDs print("\n--- Updating issue metrics using CAMS IDs ---") try: # Update issue metrics by providing CAMS asset and check IDs with project_id result = issues_provider.update_issue_metrics( - cams_asset_id="b2debda2-6ab9-4a39-8c23-17954e004dcf", - cams_check_id="7377e2cd-ac0e-4833-8760-fd0e8cb682aa", + asset_id="b2debda2-6ab9-4a39-8c23-17954e004dcf", + check_id="7377e2cd-ac0e-4833-8760-fd0e8cb682aa", occurrences=10, tested_records=100, column_name="RTN", # Required for column type assets @@ -208,12 +213,12 @@ def main(): print(f"Response: {result}") except ValueError as e: print(f"Error updating issue with CAMS IDs: {e}") - + # Example: Using catalog_id instead of project_id try: result = issues_provider.update_issue_metrics( - cams_asset_id="b2debda2-6ab9-4a39-8c23-17954e004dcf", - cams_check_id="7377e2cd-ac0e-4833-8760-fd0e8cb682aa", + asset_id="b2debda2-6ab9-4a39-8c23-17954e004dcf", + check_id="7377e2cd-ac0e-4833-8760-fd0e8cb682aa", occurrences=10, tested_records=100, column_name="RTN", @@ -225,7 +230,7 @@ def main(): print(f"Response: {result}") except ValueError as e: print(f"Error updating issue with CAMS IDs and catalog_id: {e}") - + # Example: Using check_native_id instead of cams_asset_id and cams_check_id print("\n--- Updating issue metrics using check_native_id ---") try: @@ -244,7 +249,7 @@ def main(): print(f"Response: {result}") except ValueError as e: print(f"Error updating issue with check_native_id: {e}") - + # Example: Using check_native_id for comparison check with target column try: # For comparison checks with target columns, the native_id includes the operator and target column @@ -262,12 +267,12 @@ def main(): print(f"Response: {result}") except ValueError as e: print(f"Error updating comparison check with check_native_id: {e}") - + # Example: Update multiple issues using CAMS IDs cams_updates = [ { - "cams_asset_id": "asset-1", - "cams_check_id": "check-1", + "asset_id": "asset-1", + "check_id": "check-1", "occurrences": 5, "tested_records": 50, "column_name": "test_column", @@ -276,8 +281,8 @@ def main(): "asset_type": "column" }, { - "cams_asset_id": "asset-2", - "cams_check_id": "check-2", + "asset_id": "asset-2", + "check_id": "check-2", "occurrences": 15, "tested_records": 150, "column_name": "another_column", @@ -286,15 +291,15 @@ def main(): "asset_type": "table" }, ] - + print("\n--- Updating multiple issues using CAMS IDs ---") for update_params in cams_updates: try: result = issues_provider.update_issue_metrics(**update_params) - print(f"✓ Updated issue for asset {update_params['cams_asset_id']}") + print(f"✓ Updated issue for asset {update_params['asset_id']}") except ValueError as e: - print(f"✗ Failed to update issue for asset {update_params['cams_asset_id']}: {e}") - + print(f"✗ Failed to update issue for asset {update_params['asset_id']}: {e}") + # Step 7: Create a new issue for a check (using project_id) print("\n--- Creating a new issue for a check (using project_id) ---") try: @@ -303,7 +308,7 @@ def main(): # reported_for_id: The ID of the DQ asset being reported on issue_check_id = "c3c97a92-8a45-4456-be6e-ab0d13b65a33" reported_for_id = "3f73a9d8-1664-482b-829f-9c879a4dd5d6" - + print(f"\nCreating issue with:") print(f" Check ID: {issue_check_id}") print(f" Reported For ID: {reported_for_id}") @@ -312,9 +317,9 @@ def main(): print(f" Status: actual") print(f" Ignored: False") print(f" Project ID: {project_id}") - + new_issue_id = issues_provider.create_issue( - check_id=issue_check_id, + dq_check_id=issue_check_id, reported_for_id=reported_for_id, number_of_occurrences=25, number_of_tested_records=1000, @@ -322,18 +327,18 @@ def main(): ignored=False, project_id=project_id ) - + print(f"\n✓ Successfully created issue!") print(f" New Issue ID: {new_issue_id}") - + except ValueError as e: print(f"\n✗ Error creating issue: {e}") - + # Step 8: Create issue with different parameters (using project_id) print("\n--- Creating issue with ignored=True (using project_id) ---") try: new_issue_id = issues_provider.create_issue( - check_id="c3c97a92-8a45-4456-be6e-ab0d13b65a33", + dq_check_id="c3c97a92-8a45-4456-be6e-ab0d13b65a33", reported_for_id="3f73a9d8-1664-482b-829f-9c879a4dd5d6", number_of_occurrences=100, number_of_tested_records=5000, @@ -341,21 +346,21 @@ def main(): ignored=True, # This issue is ignored project_id=project_id ) - + print(f"\n✓ Successfully created issue with ignored=True") print(f" New Issue ID: {new_issue_id}") print(f" Occurrences: 100") print(f" Tested Records: 5000") print(f" Ignored: True") - + except ValueError as e: print(f"\n✗ Error creating issue: {e}") - + # Step 9: Create issue using catalog_id print("\n--- Creating issue using catalog_id ---") try: new_issue_id = issues_provider.create_issue( - check_id="c3c97a92-8a45-4456-be6e-ab0d13b65a33", + dq_check_id="c3c97a92-8a45-4456-be6e-ab0d13b65a33", reported_for_id="3f73a9d8-1664-482b-829f-9c879a4dd5d6", number_of_occurrences=50, number_of_tested_records=2000, @@ -363,10 +368,117 @@ def main(): ignored=False, catalog_id=catalog_id ) - + print(f"\n✓ Successfully created issue using catalog_id") print(f" New Issue ID: {new_issue_id}") - + + except ValueError as e: + print(f"\n✗ Error: {e}") + + # Step 10: Create multiple issues in bulk (using project_id) + print("\n" + "=" * 70) + print("Creating Multiple Issues in Bulk") + print("=" * 70) + + # Construct the bulk payload with issues, assets, and existing_checks + bulk_payload = { + "issues": [ + { + "check": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/Validity", + "type": "format" + }, + "reported_for": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f", + "type": "data_asset" + }, + "number_of_occurrences": 200, + "number_of_tested_records": 1000, + "status": "aggregation", + "ignored": False + }, + { + "check": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/sample3", + "type": "format" + }, + "reported_for": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/NAME", + "type": "column" + }, + "number_of_occurrences": 200, + "number_of_tested_records": 1000, + "status": "actual", + "ignored": False + } + ], + "assets": [ + { + "name": "ACCOUNT_HOLDERS.csv", + "type": "data_asset", + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f", + "weight": 1 + }, + { + "name": "NAME", + "type": "column", + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/NAME", + "parent": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f", + "type": "data_asset" + }, + "weight": 1 + } + ], + "existing_checks": [ + { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/Validity", + "type": "format" + }, + { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/sample3", + "type": "format" + } + ] + } + + try: + print(f"\nCreating bulk issues with:") + print(f" Number of issues: {len(bulk_payload['issues'])}") + print(f" Number of assets: {len(bulk_payload['assets'])}") + print(f" Number of checks: {len(bulk_payload['existing_checks'])}") + print(f" Project ID: {project_id}") + print(f" Incremental Reporting: False") + print(f" Refresh Assets: False") + + response = issues_provider.create_issues_bulk( + payload=bulk_payload, + project_id=project_id, + incremental_reporting=False, + refresh_assets=False + ) + + print(f"\n✓ Successfully created issues in bulk!") + print(f" Response keys: {list(response.keys())}") + + except ValueError as e: + print(f"\n✗ Error creating bulk issues: {e}") + + # Step 11: Create bulk issues with incremental reporting (using catalog_id) + print("\n--- Creating bulk issues with incremental reporting (using catalog_id) ---") + try: + # Same payload structure but with incremental_reporting=True + response = issues_provider.create_issues_bulk( + payload=bulk_payload, + catalog_id=catalog_id, + incremental_reporting=True, # Adds archived counts to new issues + refresh_assets=False + ) + + print(f"\n✓ Successfully created bulk issues with incremental reporting!") + print(f" Using catalog_id: {catalog_id}") + print(f" Incremental reporting enabled") + except ValueError as e: print(f"\n✗ Error: {e}") diff --git a/examples/odcs_generator_example.py b/examples/odcs_generator_example.py new file mode 100644 index 0000000..1165ed2 --- /dev/null +++ b/examples/odcs_generator_example.py @@ -0,0 +1,217 @@ +#!/usr/bin/env python3 +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Example usage of ODCS Generator to create ODCS YAML files from Collibra assets. + +This example demonstrates: +1. Connecting to Collibra using credentials +2. Generating ODCS from a Collibra asset +3. Saving the ODCS to a YAML file +4. Using the generator programmatically +""" + +import os +import sys +import yaml +from wxdi.odcs_generator import CollibraClient, ODCSGenerator + + +def example_basic_usage(): + """Basic example: Generate ODCS from a Collibra asset""" + + # Get credentials from environment variables + collibra_url = os.getenv('COLLIBRA_URL') + username = os.getenv('COLLIBRA_USERNAME') + password = os.getenv('COLLIBRA_PASSWORD') + + if not all([collibra_url, username, password]): + print("Error: Please set COLLIBRA_URL, COLLIBRA_USERNAME, and COLLIBRA_PASSWORD environment variables") + sys.exit(1) + + # Replace with your actual asset ID + asset_id = "019a57f9-62d2-7aa0-9f22-4fa2cea1180b" + + print(f"Connecting to Collibra at {collibra_url}...") + + # Initialize Collibra client + client = CollibraClient( + base_url=collibra_url, + username=username, + password=password + ) + + # Create ODCS generator + generator = ODCSGenerator(client) + + print(f"Generating ODCS for asset: {asset_id}") + + # Generate ODCS from asset + odcs_data = generator.generate_odcs(asset_id) + + # Save to YAML file + output_file = f"{odcs_data.get('name', 'asset').lower().replace(' ', '-')}-odcs.yaml" + + print(f"Writing ODCS to {output_file}...") + with open(output_file, 'w') as f: + yaml.dump(odcs_data, f, default_flow_style=False, sort_keys=False, allow_unicode=True) + + print(f"✓ Successfully generated ODCS file: {output_file}") + + return odcs_data + + +def example_custom_processing(): + """Advanced example: Generate ODCS and perform custom processing""" + + # Get credentials from environment variables + collibra_url = os.getenv('COLLIBRA_URL') + username = os.getenv('COLLIBRA_USERNAME') + password = os.getenv('COLLIBRA_PASSWORD') + + if not all([collibra_url, username, password]): + print("Error: Please set COLLIBRA_URL, COLLIBRA_USERNAME, and COLLIBRA_PASSWORD environment variables") + sys.exit(1) + + # Replace with your actual asset ID + asset_id = "019a57f9-62d2-7aa0-9f22-4fa2cea1180b" + + # Initialize client and generator + client = CollibraClient(base_url=collibra_url, username=username, password=password) + generator = ODCSGenerator(client) + + # Generate ODCS + odcs_data = generator.generate_odcs(asset_id) + + # Custom processing: Update contract metadata + odcs_data['dataProduct'] = 'My Custom Data Product' + odcs_data['version'] = '2.0.0' + odcs_data['name'] = 'Custom Contract Name' + + # Custom processing: Add quality rules + if 'schema' in odcs_data and len(odcs_data['schema']) > 0: + schema = odcs_data['schema'][0] + schema['quality'] = [ + { + 'type': 'completeness', + 'dimension': 'completeness', + 'mustBe': 100, + 'mustNotBe': None + } + ] + + # Custom processing: Update server configuration + if 'servers' in odcs_data and len(odcs_data['servers']) > 0: + server = odcs_data['servers'][0] + server['server'] = 'prod.snowflake.mycompany.com' + server['type'] = 'snowflake' + server['account'] = 'my_account' + server['database'] = 'my_database' + server['schema'] = 'my_schema' + + # Save customized ODCS + output_file = 'custom-odcs.yaml' + with open(output_file, 'w') as f: + yaml.dump(odcs_data, f, default_flow_style=False, sort_keys=False, allow_unicode=True) + + print(f"✓ Successfully generated customized ODCS file: {output_file}") + + return odcs_data + + +def example_batch_processing(): + """Example: Generate ODCS for multiple assets""" + + # Get credentials from environment variables + collibra_url = os.getenv('COLLIBRA_URL') + username = os.getenv('COLLIBRA_USERNAME') + password = os.getenv('COLLIBRA_PASSWORD') + + if not all([collibra_url, username, password]): + print("Error: Please set COLLIBRA_URL, COLLIBRA_USERNAME, and COLLIBRA_PASSWORD environment variables") + sys.exit(1) + + # List of asset IDs to process + asset_ids = [ + "019a57f9-62d2-7aa0-9f22-4fa2cea1180b", + # Add more asset IDs here + ] + + # Initialize client and generator once + client = CollibraClient(base_url=collibra_url, username=username, password=password) + generator = ODCSGenerator(client) + + results = [] + + for asset_id in asset_ids: + try: + print(f"\nProcessing asset: {asset_id}") + odcs_data = generator.generate_odcs(asset_id) + + # Save to file + output_file = f"{odcs_data.get('name', asset_id).lower().replace(' ', '-')}-odcs.yaml" + with open(output_file, 'w') as f: + yaml.dump(odcs_data, f, default_flow_style=False, sort_keys=False, allow_unicode=True) + + results.append({'asset_id': asset_id, 'status': 'success', 'file': output_file}) + print(f"✓ Generated: {output_file}") + + except Exception as e: + results.append({'asset_id': asset_id, 'status': 'failed', 'error': str(e)}) + print(f"✗ Failed: {str(e)}") + + # Print summary + print("\n=== Batch Processing Summary ===") + success_count = sum(1 for r in results if r['status'] == 'success') + failed_count = sum(1 for r in results if r['status'] == 'failed') + print(f"Total: {len(results)}, Success: {success_count}, Failed: {failed_count}") + + return results + + +if __name__ == '__main__': + print("=== ODCS Generator Examples ===\n") + + # Run basic example + print("1. Basic Usage Example") + print("-" * 50) + try: + example_basic_usage() + except Exception as e: + print(f"Error: {str(e)}") + + print("\n" + "=" * 50 + "\n") + + # Uncomment to run other examples: + + # print("2. Custom Processing Example") + # print("-" * 50) + # try: + # example_custom_processing() + # except Exception as e: + # print(f"Error: {str(e)}") + + # print("\n" + "=" * 50 + "\n") + + # print("3. Batch Processing Example") + # print("-" * 50) + # try: + # example_batch_processing() + # except Exception as e: + # print(f"Error: {str(e)}") + +# Made with Bob diff --git a/examples/odcs_generator_informatica_example.py b/examples/odcs_generator_informatica_example.py new file mode 100644 index 0000000..73b2905 --- /dev/null +++ b/examples/odcs_generator_informatica_example.py @@ -0,0 +1,415 @@ +#!/usr/bin/env python3 +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Example usage of ODCS Generator to create ODCS YAML files from Informatica CDGC assets. + +This example demonstrates: +1. Connecting to Informatica CDGC using credentials +2. Generating ODCS from an Informatica asset +3. Saving the ODCS to a YAML file +4. Using the generator programmatically +5. Batch processing multiple assets +""" + +import os +import sys +import yaml +from concurrent.futures import ThreadPoolExecutor, as_completed +from wxdi.odcs_generator.generate_odcs_from_informatica import ( + InformaticaClient, + generate_odcs_yaml, + write_yaml_file, + extract_column_position +) + + +def example_basic_usage(): + """Basic example: Generate ODCS from an Informatica asset""" + + # Get credentials from environment variables + cdgc_url = os.getenv('INFORMATICA_CDGC_URL') + username = os.getenv('INFORMATICA_USERNAME') + password = os.getenv('INFORMATICA_PASSWORD') + + if not all([cdgc_url, username, password]): + print("Error: Please set INFORMATICA_CDGC_URL, INFORMATICA_USERNAME, and INFORMATICA_PASSWORD environment variables") + print("\nExample:") + print(" export INFORMATICA_CDGC_URL='https://cdgc.dm-us.informaticacloud.com'") + print(" export INFORMATICA_USERNAME='your_username'") + print(" export INFORMATICA_PASSWORD='your_pwd'") + sys.exit(1) + + # Replace with your actual asset ID + asset_id = "1b5fc805-252d-4ba2-bd90-e943103e411b" + + print(f"Connecting to Informatica CDGC at {cdgc_url}...") + + # Initialize Informatica client + client = InformaticaClient(cdgc_url, username, password) + + print(f"Fetching asset details for {asset_id}...") + + # Fetch asset data + asset_data = client.get_asset_details(asset_id) + + # Get column IDs from hierarchy + column_ids = [col['core.identity'] for col in asset_data.get('hierarchy', [])] + + print(f"Found {len(column_ids)} columns. Fetching column details...") + + # Fetch column details concurrently + column_details = [] + with ThreadPoolExecutor(max_workers=10) as executor: + future_to_col_id = { + executor.submit(client.get_column_details, col_id): col_id + for col_id in column_ids + } + + for future in as_completed(future_to_col_id): + try: + col_data = future.result() + column_details.append(col_data) + except Exception as e: + col_id = future_to_col_id[future] + print(f"Warning: Failed to fetch column {col_id}: {e}") + + # Sort columns by position + column_details.sort(key=extract_column_position) + + print("Generating ODCS YAML...") + + # Generate ODCS + odcs_data = generate_odcs_yaml(asset_data, column_details, client.base_url) + + # Determine output filename + asset_name = odcs_data.get('name', 'asset').lower().replace(' ', '-') + output_file = f"{asset_name}-odcs.yaml" + + # Write to file + write_yaml_file(output_file, odcs_data) + + print(f"✓ Successfully generated ODCS file: {output_file}") + + return odcs_data + + +def example_custom_processing(): + """Example: Generate ODCS with custom processing""" + + cdgc_url = os.getenv('INFORMATICA_CDGC_URL') + username = os.getenv('INFORMATICA_USERNAME') + password = os.getenv('INFORMATICA_PASSWORD') + + if not all([cdgc_url, username, password]): + print("Error: Environment variables not set") + sys.exit(1) + + asset_id = "1b5fc805-252d-4ba2-bd90-e943103e411b" + + print("Example: Custom ODCS Processing") + print("=" * 50) + + # Initialize client + client = InformaticaClient(cdgc_url, username, password) + + # Fetch data + asset_data = client.get_asset_details(asset_id) + column_ids = [col['core.identity'] for col in asset_data.get('hierarchy', [])] + + # Fetch columns + column_details = [] + with ThreadPoolExecutor(max_workers=10) as executor: + futures = [executor.submit(client.get_column_details, col_id) for col_id in column_ids] + for future in as_completed(futures): + try: + column_details.append(future.result()) + except Exception as e: + print(f"Warning: {e}") + + column_details.sort(key=extract_column_position) + + # Generate base ODCS + odcs_data = generate_odcs_yaml(asset_data, column_details, client.base_url) + + # Customize the ODCS + print("\nCustomizing ODCS metadata...") + + # Update contract metadata + odcs_data['domain'] = 'Finance' + odcs_data['dataProduct'] = 'Customer Analytics' + odcs_data['version'] = '2.0.0' + odcs_data['name'] = 'Customer Transaction Contract' + + # Add quality rules + odcs_data['quality'] = [ + { + 'type': 'completeness', + 'description': 'All required fields must be populated', + 'dimension': 'completeness', + 'weight': 'high' + }, + { + 'type': 'uniqueness', + 'description': 'Primary key must be unique', + 'dimension': 'uniqueness', + 'weight': 'critical' + } + ] + + # Update server configuration with actual values + if odcs_data.get('servers'): + odcs_data['servers'][0]['server'] = 'prod.snowflake.acme.com' + odcs_data['servers'][0]['database'] = 'ANALYTICS_DB' + odcs_data['servers'][0]['account'] = 'acme_prod' + + # Add stakeholders + odcs_data['stakeholders'] = [ + { + 'username': 'data.owner@acme.com', + 'role': 'Data Owner', + 'dateIn': '2024-01-01', + 'dateOut': None, + 'replaced': False + }, + { + 'username': 'data.steward@acme.com', + 'role': 'Data Steward', + 'dateIn': '2024-01-01', + 'dateOut': None, + 'replaced': False + } + ] + + # Save customized ODCS + output_file = "customized-customer-contract-odcs.yaml" + write_yaml_file(output_file, odcs_data) + + print(f"✓ Customized ODCS saved to: {output_file}") + + return odcs_data + + +def example_batch_processing(): + """Example: Process multiple assets in batch""" + + cdgc_url = os.getenv('INFORMATICA_CDGC_URL') + username = os.getenv('INFORMATICA_USERNAME') + password = os.getenv('INFORMATICA_PASSWORD') + + if not all([cdgc_url, username, password]): + print("Error: Environment variables not set") + sys.exit(1) + + # List of asset IDs to process + asset_ids = [ + "1b5fc805-252d-4ba2-bd90-e943103e411b", + "2c6gd916-363e-5cb1-af01-5gb3dfb2291c", + "3d7he027-474f-6dc2-bg12-6hc4egc3302d" + ] + + print("Example: Batch Processing Multiple Assets") + print("=" * 50) + + # Initialize client + client = InformaticaClient(cdgc_url, username, password) + + results = { + 'success': [], + 'failed': [] + } + + for i, asset_id in enumerate(asset_ids, 1): + print(f"\nProcessing asset {i}/{len(asset_ids)}: {asset_id}") + + try: + # Fetch asset data + asset_data = client.get_asset_details(asset_id) + column_ids = [col['core.identity'] for col in asset_data.get('hierarchy', [])] + + # Fetch columns + column_details = [] + with ThreadPoolExecutor(max_workers=10) as executor: + futures = [executor.submit(client.get_column_details, col_id) for col_id in column_ids] + for future in as_completed(futures): + try: + column_details.append(future.result()) + except Exception: + pass + + column_details.sort(key=extract_column_position) + + # Generate ODCS + odcs_data = generate_odcs_yaml(asset_data, column_details, client.base_url) + + # Save to file + asset_name = odcs_data.get('name', f'asset-{i}').lower().replace(' ', '-') + output_file = f"batch-{asset_name}-odcs.yaml" + write_yaml_file(output_file, odcs_data) + + results['success'].append({ + 'asset_id': asset_id, + 'output_file': output_file + }) + + print(f" ✓ Success: {output_file}") + + except Exception as e: + results['failed'].append({ + 'asset_id': asset_id, + 'error': str(e) + }) + print(f" ✗ Failed: {e}") + + # Print summary + print("\n" + "=" * 50) + print("Batch Processing Summary") + print("=" * 50) + print(f"Total assets: {len(asset_ids)}") + print(f"Successful: {len(results['success'])}") + print(f"Failed: {len(results['failed'])}") + + if results['failed']: + print("\nFailed assets:") + for item in results['failed']: + print(f" - {item['asset_id']}: {item['error']}") + + return results + + +def example_programmatic_usage(): + """Example: Using the module programmatically in your code""" + + print("Example: Programmatic Usage") + print("=" * 50) + + # Configuration + config = { + 'cdgc_url': os.getenv('INFORMATICA_CDGC_URL'), + 'username': os.getenv('INFORMATICA_USERNAME'), + 'password': os.getenv('INFORMATICA_PASSWORD'), + 'asset_id': '1b5fc805-252d-4ba2-bd90-e943103e411b' + } + + if not all([config['cdgc_url'], config['username'], config['password']]): + print("Error: Environment variables not set") + return None + + try: + # Step 1: Initialize client + print("Step 1: Initializing Informatica client...") + client = InformaticaClient( + config['cdgc_url'], + config['username'], + config['password'] + ) + + # Step 2: Fetch asset metadata + print("Step 2: Fetching asset metadata...") + asset_data = client.get_asset_details(config['asset_id']) + table_name = asset_data['summary']['core.name'] + print(f" Found table: {table_name}") + + # Step 3: Fetch column metadata + print("Step 3: Fetching column metadata...") + column_ids = [col['core.identity'] for col in asset_data.get('hierarchy', [])] + print(f" Found {len(column_ids)} columns") + + column_details = [] + with ThreadPoolExecutor(max_workers=10) as executor: + futures = [executor.submit(client.get_column_details, col_id) for col_id in column_ids] + for future in as_completed(futures): + try: + column_details.append(future.result()) + except Exception as e: + print(f" Warning: {e}") + + column_details.sort(key=extract_column_position) + + # Step 4: Generate ODCS + print("Step 4: Generating ODCS...") + odcs_data = generate_odcs_yaml(asset_data, column_details, client.base_url) + + # Step 5: Process or save ODCS + print("Step 5: Saving ODCS...") + output_file = f"{table_name.lower()}-programmatic-odcs.yaml" + write_yaml_file(output_file, odcs_data) + + print(f"\n✓ ODCS generated successfully: {output_file}") + + # You can now use odcs_data in your application + print(f"\nODCS Summary:") + print(f" - Contract ID: {odcs_data['id']}") + print(f" - Table: {odcs_data['schema'][0]['name']}") + print(f" - Columns: {len(odcs_data['schema'][0]['properties'])}") + print(f" - Server Type: {odcs_data['servers'][0]['type']}") + + return odcs_data + + except Exception as e: + print(f"\n✗ Error: {e}") + return None + + +def main(): + """Run all examples""" + + print("\n" + "=" * 70) + print("INFORMATICA ODCS GENERATOR - EXAMPLES") + print("=" * 70) + + examples = [ + ("Basic Usage", example_basic_usage), + ("Custom Processing", example_custom_processing), + ("Batch Processing", example_batch_processing), + ("Programmatic Usage", example_programmatic_usage) + ] + + print("\nAvailable examples:") + for i, (name, _) in enumerate(examples, 1): + print(f" {i}. {name}") + + print("\nTo run a specific example, uncomment the corresponding line below:") + print("Or run all examples by uncommenting the loop.\n") + + # Uncomment to run specific examples: + # example_basic_usage() + # example_custom_processing() + # example_batch_processing() + # example_programmatic_usage() + + # Uncomment to run all examples: + # for name, example_func in examples: + # print(f"\n{'=' * 70}") + # print(f"Running: {name}") + # print('=' * 70) + # try: + # example_func() + # except Exception as e: + # print(f"Error in {name}: {e}") + # print() + + print("\nNote: Make sure to set the following environment variables:") + print(" - INFORMATICA_CDGC_URL") + print(" - INFORMATICA_USERNAME") + print(" - INFORMATICA_PASSWORD") + print("\nAnd update the asset IDs in the examples with your actual asset IDs.") + + +if __name__ == '__main__': + main() + diff --git a/examples/pandas_dataframe_usage.py b/examples/pandas_dataframe_usage.py index 84f90a0..fcd994f 100644 --- a/examples/pandas_dataframe_usage.py +++ b/examples/pandas_dataframe_usage.py @@ -33,7 +33,7 @@ def main(): print("=" * 70) print("Pandas DataFrame Validation Example") print("=" * 70) - + # Step 1: Define metadata metadata = AssetMetadata( table_name='employees', @@ -46,24 +46,24 @@ def main(): ColumnMetadata('min_salary', DataType.DECIMAL, precision=10, scale=2), ] ) - + print(f"\nAsset: {metadata.table_name}") print(f"Columns: {len(metadata.columns)}") - + # Step 2: Create validator with rules validator = Validator(metadata) - + # Add validation rules validator.add_rule( ValidationRule('emp_id') .add_check(LengthCheck(min_length=4, max_length=6)) ) - + validator.add_rule( ValidationRule('name') .add_check(LengthCheck(min_length=2, max_length=100)) ) - + validator.add_rule( ValidationRule('age') .add_check(ComparisonCheck( @@ -75,7 +75,7 @@ def main(): target_value=65 )) ) - + validator.add_rule( ValidationRule('department') .add_check(ValidValuesCheck( @@ -83,7 +83,7 @@ def main(): case_sensitive=False )) ) - + validator.add_rule( ValidationRule('salary') .add_check(ComparisonCheck( @@ -91,9 +91,9 @@ def main(): target_column='min_salary' )) ) - + print(f"\nValidator configured with {len(validator.rules)} rules") - + # Step 3: Create sample DataFrame df = pd.DataFrame({ 'emp_id': [1001, 12, 1003, 1004, 1005], @@ -103,20 +103,20 @@ def main(): 'salary': [75000.00, 65000.00, 50000.00, 80000.00, 55000.00], 'min_salary': [60000.00, 55000.00, 45000.00, 70000.00, 60000.00] }) - + print(f"\nOriginal DataFrame ({len(df)} rows):") print(df.to_string(index=False)) - + # Step 4: Create Pandas validator pandas_validator = PandasValidator(validator, chunk_size=1000) - + print(f"\nPandas Validator: {pandas_validator}") - + # Step 5: Get summary statistics (memory efficient) print("\n" + "=" * 70) print("Validation Summary Statistics") print("=" * 70) - + summary = pandas_validator.get_summary_statistics(df) print(f"Total Rows: {summary['total_rows']}") print(f"Valid Rows: {summary['valid_rows']}") @@ -125,14 +125,14 @@ def main(): print(f"Total Checks: {summary['total_checks']}") print(f"Passed Checks: {summary['passed_checks']}") print(f"Failed Checks: {summary['failed_checks']}") - + # Step 6: Add validation column print("\n" + "=" * 70) print("DataFrame with Validation Column") print("=" * 70) - + df_validated = pandas_validator.add_validation_column(df) - + print(f"\nColumns: {list(df_validated.columns)}") print(f"\nValidation results (struct column):") for idx, result in enumerate(df_validated['dq_validation_result']): @@ -140,15 +140,15 @@ def main(): print(f"Row {idx}: {status} - Score: {result['score']}, Pass Rate: {result['pass_rate']:.1f}%") if not result['is_valid']: print(f" Errors: {result['error_count']}") - + # Step 7: Get invalid rows print("\n" + "=" * 70) print("Invalid Rows") print("=" * 70) - + invalid_df = pandas_validator.get_invalid_rows(df) print(f"\nFound {len(invalid_df)} invalid rows:") - + for idx, row in invalid_df.iterrows(): print(f"\nRow {idx}:") print(f" emp_id: {row['emp_id']}") @@ -158,31 +158,31 @@ def main(): validation = row['dq_validation_result'] print(f" Validation: {validation['score']} ({validation['pass_rate']:.1f}%)") print(f" Errors: {validation['error_count']}") - + # Step 8: Expand validation column print("\n" + "=" * 70) print("Expanded Validation Columns") print("=" * 70) - + df_expanded = pandas_validator.expand_validation_column(df_validated) - + print(f"\nExpanded columns: {[c for c in df_expanded.columns if c.startswith('dq_')]}") print(f"\nSample of expanded data:") print(df_expanded[['name', 'dq_is_valid', 'dq_score', 'dq_pass_rate', 'dq_error_count']].to_string(index=False)) - + # Step 9: Save results print("\n" + "=" * 70) print("Saving Results") print("=" * 70) - + # Save invalid rows invalid_df.to_csv('invalid_employees.csv', index=False) print("✓ Saved invalid rows to: invalid_employees.csv") - + # Save expanded results df_expanded.to_csv('validation_results.csv', index=False) print("✓ Saved validation results to: validation_results.csv") - + print("\n" + "=" * 70) print("Example Complete!") print("=" * 70) diff --git a/examples/spark_dataframe_usage.py b/examples/spark_dataframe_usage.py index 443d26c..37b3851 100644 --- a/examples/spark_dataframe_usage.py +++ b/examples/spark_dataframe_usage.py @@ -35,16 +35,16 @@ def main(): print("=" * 70) print("PySpark DataFrame Validation Example") print("=" * 70) - + # Step 1: Initialize Spark Session spark = SparkSession.builder \ .appName("DataQualityValidation") \ .master("local[*]") \ .getOrCreate() - + print(f"\nSpark Version: {spark.version}") print(f"Spark Master: {spark.sparkContext.master}") - + # Step 2: Define metadata metadata = AssetMetadata( table_name='employees', @@ -57,24 +57,24 @@ def main(): ColumnMetadata('min_salary', DataType.DECIMAL, precision=10, scale=2), ] ) - + print(f"\nAsset: {metadata.table_name}") print(f"Columns: {len(metadata.columns)}") - + # Step 3: Create validator with rules validator = Validator(metadata) - + # Add validation rules validator.add_rule( ValidationRule('emp_id') .add_check(LengthCheck(min_length=4, max_length=6)) ) - + validator.add_rule( ValidationRule('name') .add_check(LengthCheck(min_length=2, max_length=100)) ) - + validator.add_rule( ValidationRule('age') .add_check(ComparisonCheck( @@ -86,7 +86,7 @@ def main(): target_value=65 )) ) - + validator.add_rule( ValidationRule('department') .add_check(ValidValuesCheck( @@ -94,7 +94,7 @@ def main(): case_sensitive=False )) ) - + validator.add_rule( ValidationRule('salary') .add_check(ComparisonCheck( @@ -102,9 +102,9 @@ def main(): target_column='min_salary' )) ) - + print(f"\nValidator configured with {len(validator.rules)} rules") - + # Step 4: Create sample DataFrame schema = StructType([ StructField('emp_id', IntegerType(), True), @@ -114,7 +114,7 @@ def main(): StructField('salary', DecimalType(10, 2), True), StructField('min_salary', DecimalType(10, 2), True), ]) - + data = [ (1001, 'John Doe', 30, 'Engineering', 75000.00, 60000.00), (12, 'Jane Smith', 25, 'SALES', 65000.00, 55000.00), @@ -122,22 +122,22 @@ def main(): (1004, 'Alice Johnson', 35, 'Marketing', 80000.00, 70000.00), (1005, 'Charlie Brown', 40, 'Finance', 55000.00, 60000.00), ] - + df = spark.createDataFrame(data, schema) - + print(f"\nOriginal DataFrame ({df.count()} rows):") df.show(truncate=False) - + # Step 5: Create Spark validator spark_validator = SparkValidator(validator) - + print(f"\nSpark Validator: {spark_validator}") - + # Step 6: Get summary statistics (distributed aggregation) print("\n" + "=" * 70) print("Validation Summary Statistics") print("=" * 70) - + summary = spark_validator.get_summary_statistics(df) print(f"Total Rows: {summary['total_rows']}") print(f"Valid Rows: {summary['valid_rows']}") @@ -146,47 +146,47 @@ def main(): print(f"Total Checks: {summary['total_checks']}") print(f"Passed Checks: {summary['passed_checks']}") print(f"Failed Checks: {summary['failed_checks']}") - + # Step 7: Add validation column print("\n" + "=" * 70) print("DataFrame with Validation Column") print("=" * 70) - + df_validated = spark_validator.add_validation_column(df) - + print(f"\nColumns: {df_validated.columns}") print(f"\nSchema of validation column:") df_validated.select('dq_validation_result').printSchema() - + print(f"\nValidation results (struct column):") df_validated.select('name', 'dq_validation_result').show(truncate=False) - + # Step 8: Get invalid rows print("\n" + "=" * 70) print("Invalid Rows") print("=" * 70) - + invalid_df = spark_validator.get_invalid_rows(df) print(f"\nFound {invalid_df.count()} invalid rows:") invalid_df.show(truncate=False) - + # Step 9: Expand validation column print("\n" + "=" * 70) print("Expanded Validation Columns") print("=" * 70) - + df_expanded = spark_validator.expand_validation_column(df_validated) - + expanded_cols = [c for c in df_expanded.columns if c.startswith('dq_')] print(f"\nExpanded columns: {expanded_cols}") print(f"\nSample of expanded data:") df_expanded.select('name', 'dq_is_valid', 'dq_score', 'dq_pass_rate', 'dq_error_count').show(truncate=False) - + # Step 10: Get error samples print("\n" + "=" * 70) print("Error Samples") print("=" * 70) - + error_samples = spark_validator.get_error_sample(df, limit=10) print(f"\nFound {len(error_samples)} error samples:") for i, errors in enumerate(error_samples, 1): @@ -194,12 +194,12 @@ def main(): print(f" Errors:") for error in errors: print(f" - {error}") - + # Step 11: Write validation report print("\n" + "=" * 70) print("Writing Validation Report") print("=" * 70) - + output_path = "validation_report" spark_validator.write_validation_report( df, @@ -207,51 +207,51 @@ def main(): format='parquet' ) print(f"✓ Validation report written to: {output_path}") - + # Step 12: Save results print("\n" + "=" * 70) print("Saving Results") print("=" * 70) - + # Save invalid rows invalid_df.write.mode('overwrite').parquet('invalid_employees_spark') print("✓ Saved invalid rows to: invalid_employees_spark/") - + # Save expanded results df_expanded.write.mode('overwrite').parquet('validation_results_spark') print("✓ Saved validation results to: validation_results_spark/") - + # Step 13: Performance demonstration with larger dataset print("\n" + "=" * 70) print("Performance Test with Larger Dataset") print("=" * 70) - + # Create a larger dataset by repeating the data large_df = df for _ in range(10): # 5 * 2^10 = 5,120 rows large_df = large_df.union(df) - + print(f"\nLarge DataFrame: {large_df.count()} rows") - + # Validate large dataset (distributed processing) import time start_time = time.time() - + large_summary = spark_validator.get_summary_statistics(large_df) - + elapsed_time = time.time() - start_time - + print(f"\nValidation completed in {elapsed_time:.2f} seconds") print(f"Total Rows: {large_summary['total_rows']}") print(f"Valid Rows: {large_summary['valid_rows']}") print(f"Invalid Rows: {large_summary['invalid_rows']}") print(f"Pass Rate: {large_summary['pass_rate']:.2f}%") print(f"Throughput: {large_summary['total_rows'] / elapsed_time:.0f} rows/second") - + print("\n" + "=" * 70) print("Example Complete!") print("=" * 70) - + # Stop Spark session spark.stop() diff --git a/examples/test_dph_v1_examples.py b/examples/test_dph_v1_examples.py new file mode 100644 index 0000000..f89041b --- /dev/null +++ b/examples/test_dph_v1_examples.py @@ -0,0 +1,1600 @@ +# -*- coding: utf-8 -*- +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Examples for DphV1 +""" + +from ibm_cloud_sdk_core import ApiException, read_external_sources +from ibm_cloud_sdk_core.utils import datetime_to_string, string_to_datetime +import io +import os +import pytest +from wxdi.dph_services.dph_v1 import * + +# +# This file provides an example of how to use the DPH service. +# +# The following configuration properties are assumed to be defined: +# DPH_URL= +# DPH_AUTH_TYPE=iam +# DPH_APIKEY= +# DPH_AUTH_URL= +# +# These configuration properties can be exported as environment variables, or stored +# in a configuration file and then: +# export IBM_CREDENTIALS_FILE= +# +config_file = 'dph_v1.env' + +dph_service = None + +config = None + +# Variables to hold link values +complete_a_draft_by_contract_terms_id_link = None +complete_a_draft_by_draft_id_link = None +complete_contract_terms_document_by_document_id_link = None +complete_draft_contract_terms_by_data_product_id_link = None +create_a_contract_terms_doc_by_contract_terms_id_link = None +create_a_contract_terms_doc_by_draft_id_link = None +create_data_product_by_catalog_id_link = None +create_draft_by_container_id_link = None +create_new_draft_by_data_product_id_link = None +delete_a_contract_document_by_draft_id_link = None +delete_a_draft_by_contract_terms_id_link = None +delete_a_draft_by_draft_id_link = None +delete_contract_document_by_data_product_id_link = None +delete_contract_terms_document_by_document_id_link = None +delete_draft_of_data_product_by_data_product_id_link = None +get_a_draft_by_contract_terms_id_link = None +get_a_draft_contract_document_by_draft_id_link = None +get_a_draft_of_data_product_by_data_product_id_link = None +get_a_release_by_release_id_link = None +get_a_release_contract_terms_by_contract_terms_id_link = None +get_a_release_contract_terms_by_release_id_link = None +get_a_release_of_data_product_by_data_product_id_link = None +get_contract_document_by_data_product_id_link = None +get_contract_terms_document_by_id_document_id_link = None +get_data_product_by_data_product_id_link = None +get_draft_by_draft_id_link = None +get_list_of_data_product_drafts_by_data_product_id_link = None +get_list_of_releases_of_data_product_by_data_product_id_link = None +get_release_contract_document_by_data_product_id_link = None +get_release_contract_document_by_document_id_link = None +get_status_by_catalog_id_link = None +publish_a_draft_by_draft_id_link = None +publish_a_draft_of_data_product_by_data_product_id_link = None +retire_a_release_contract_terms_by_release_id_link = None +retire_a_releases_of_data_product_by_data_product_id_link = None +update_a_draft_by_contract_terms_id_link = None +update_a_draft_by_draft_id_link = None +update_a_release_by_release_id_link = None +update_contract_document_by_data_product_id_link = None +update_contract_document_by_draft_id_link = None +update_contract_terms_document_by_document_id_link = None +update_draft_of_data_product_by_data_product_id_link = None +update_release_of_data_product_by_data_product_id_link = None +upload_contract_terms_doc_by_data_product_id_link = None + + +############################################################################## +# Start of Examples for Service: DphV1 +############################################################################## +# region +class TestDphV1Examples: + """ + Example Test Class for DphV1 + """ + + @classmethod + def setup_class(cls): + global dph_service + if os.path.exists(config_file): + os.environ['IBM_CREDENTIALS_FILE'] = config_file + + # begin-common + + dph_service = DphV1.new_instance( + ) + + # end-common + assert dph_service is not None + + # Load the configuration + global config + config = read_external_sources(DphV1.DEFAULT_SERVICE_NAME) + + print('Setup complete.') + + needscredentials = pytest.mark.skipif( + not os.path.exists(config_file), reason="External configuration not available, skipping..." + ) + + @needscredentials + def test_initialize_example(self): + """ + initialize request example + """ + try: + global create_draft_by_container_id_link + global create_data_product_by_catalog_id_link + global get_status_by_catalog_id_link + + print('\ninitialize() result:') + + # begin-initialize + + response = dph_service.initialize( + include=['delivery_methods', 'domains_multi_industry', 'data_product_samples', 'workflows', 'project', 'catalog_configurations'], + ) + initialize_resource = response.get_result() + + print(json.dumps(initialize_resource, indent=2)) + + # end-initialize + + create_draft_by_container_id_link = initialize_resource['container']['id'] + create_data_product_by_catalog_id_link = initialize_resource['container']['id'] + get_status_by_catalog_id_link = initialize_resource['container']['id'] + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_create_data_product_example(self): + """ + create_data_product request example + """ + try: + global create_new_draft_by_data_product_id_link + global get_contract_document_by_data_product_id_link + global retire_a_releases_of_data_product_by_data_product_id_link + global get_data_product_by_data_product_id_link + global update_draft_of_data_product_by_data_product_id_link + global update_contract_document_by_data_product_id_link + global delete_draft_of_data_product_by_data_product_id_link + global get_a_release_of_data_product_by_data_product_id_link + global complete_draft_contract_terms_by_data_product_id_link + global delete_contract_document_by_data_product_id_link + global get_list_of_data_product_drafts_by_data_product_id_link + global get_a_draft_of_data_product_by_data_product_id_link + global get_release_contract_document_by_data_product_id_link + global publish_a_draft_of_data_product_by_data_product_id_link + global get_list_of_releases_of_data_product_by_data_product_id_link + global update_release_of_data_product_by_data_product_id_link + global upload_contract_terms_doc_by_data_product_id_link + + print('\ncreate_data_product() result:') + + # begin-create_data_product + + container_identity_model = { + 'id': 'd29c42eb-7100-4b7a-8257-c196dbcca1cd', + } + + asset_prototype_model = { + 'container': container_identity_model, + } + + data_product_draft_prototype_model = { + 'name': 'My New Data Product', + 'asset': asset_prototype_model, + } + + response = dph_service.create_data_product( + drafts=[data_product_draft_prototype_model], + ) + data_product = response.get_result() + + print(json.dumps(data_product, indent=2)) + + # end-create_data_product + + create_new_draft_by_data_product_id_link = data_product['id'] + get_contract_document_by_data_product_id_link = data_product['id'] + retire_a_releases_of_data_product_by_data_product_id_link = data_product['id'] + get_data_product_by_data_product_id_link = data_product['id'] + update_draft_of_data_product_by_data_product_id_link = data_product['id'] + update_contract_document_by_data_product_id_link = data_product['id'] + delete_draft_of_data_product_by_data_product_id_link = data_product['id'] + get_a_release_of_data_product_by_data_product_id_link = data_product['id'] + complete_draft_contract_terms_by_data_product_id_link = data_product['id'] + delete_contract_document_by_data_product_id_link = data_product['id'] + get_list_of_data_product_drafts_by_data_product_id_link = data_product['id'] + get_a_draft_of_data_product_by_data_product_id_link = data_product['id'] + get_release_contract_document_by_data_product_id_link = data_product['id'] + publish_a_draft_of_data_product_by_data_product_id_link = data_product['id'] + get_list_of_releases_of_data_product_by_data_product_id_link = data_product['id'] + update_release_of_data_product_by_data_product_id_link = data_product['id'] + upload_contract_terms_doc_by_data_product_id_link = data_product['id'] + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_create_data_product_draft_example(self): + """ + create_data_product_draft request example + """ + try: + global get_a_draft_contract_document_by_draft_id_link + global update_a_draft_by_contract_terms_id_link + global create_a_contract_terms_doc_by_contract_terms_id_link + global update_contract_document_by_draft_id_link + global get_a_release_contract_terms_by_contract_terms_id_link + global complete_a_draft_by_contract_terms_id_link + global get_draft_by_draft_id_link + global publish_a_draft_by_draft_id_link + global update_a_draft_by_draft_id_link + global create_a_contract_terms_doc_by_draft_id_link + global delete_a_contract_document_by_draft_id_link + global delete_a_draft_by_contract_terms_id_link + global delete_a_draft_by_draft_id_link + global complete_a_draft_by_draft_id_link + global get_a_draft_by_contract_terms_id_link + + print('\ncreate_data_product_draft() result:') + + # begin-create_data_product_draft + + container_identity_model = { + 'id': 'd29c42eb-7100-4b7a-8257-c196dbcca1cd', + } + + asset_prototype_model = { + 'container': container_identity_model, + } + + data_product_draft_version_release_model = { + 'id': '8bf83660-11fe-4427-a72a-8d8359af24e3', + } + + data_product_identity_model = { + 'id': 'b38df608-d34b-4d58-8136-ed25e6c6684e', + 'release': data_product_draft_version_release_model, + } + + response = dph_service.create_data_product_draft( + data_product_id=create_new_draft_by_data_product_id_link, + asset=asset_prototype_model, + version='1.2.0', + data_product=data_product_identity_model, + ) + data_product_draft = response.get_result() + + print(json.dumps(data_product_draft, indent=2)) + + # end-create_data_product_draft + + get_a_draft_contract_document_by_draft_id_link = data_product_draft['id'] + update_a_draft_by_contract_terms_id_link = data_product_draft['contract_terms'][0]['id'] + create_a_contract_terms_doc_by_contract_terms_id_link = data_product_draft['contract_terms'][0]['id'] + update_contract_document_by_draft_id_link = data_product_draft['id'] + get_a_release_contract_terms_by_contract_terms_id_link = data_product_draft['contract_terms'][0]['id'] + complete_a_draft_by_contract_terms_id_link = data_product_draft['contract_terms'][0]['id'] + get_draft_by_draft_id_link = data_product_draft['id'] + publish_a_draft_by_draft_id_link = data_product_draft['id'] + update_a_draft_by_draft_id_link = data_product_draft['id'] + create_a_contract_terms_doc_by_draft_id_link = data_product_draft['id'] + delete_a_contract_document_by_draft_id_link = data_product_draft['id'] + delete_a_draft_by_contract_terms_id_link = data_product_draft['contract_terms'][0]['id'] + delete_a_draft_by_draft_id_link = data_product_draft['id'] + complete_a_draft_by_draft_id_link = data_product_draft['id'] + get_a_draft_by_contract_terms_id_link = data_product_draft['contract_terms'][0]['id'] + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_create_draft_contract_terms_document_example(self): + """ + create_draft_contract_terms_document request example + """ + try: + global get_release_contract_document_by_document_id_link + global delete_contract_terms_document_by_document_id_link + global get_contract_terms_document_by_id_document_id_link + global update_contract_terms_document_by_document_id_link + global complete_contract_terms_document_by_document_id_link + + print('\ncreate_draft_contract_terms_document() result:') + + # begin-create_draft_contract_terms_document + + response = dph_service.create_draft_contract_terms_document( + data_product_id=upload_contract_terms_doc_by_data_product_id_link, + draft_id=create_a_contract_terms_doc_by_draft_id_link, + contract_terms_id=create_a_contract_terms_doc_by_contract_terms_id_link, + type='terms_and_conditions', + name='Terms and conditions document', + ) + contract_terms_document = response.get_result() + + print(json.dumps(contract_terms_document, indent=2)) + + # end-create_draft_contract_terms_document + + get_release_contract_document_by_document_id_link = contract_terms_document['id'] + delete_contract_terms_document_by_document_id_link = contract_terms_document['id'] + get_contract_terms_document_by_id_document_id_link = contract_terms_document['id'] + update_contract_terms_document_by_document_id_link = contract_terms_document['id'] + complete_contract_terms_document_by_document_id_link = contract_terms_document['id'] + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_publish_data_product_draft_example(self): + """ + publish_data_product_draft request example + """ + try: + global update_a_release_by_release_id_link + global get_a_release_contract_terms_by_release_id_link + global retire_a_release_contract_terms_by_release_id_link + global get_a_release_by_release_id_link + + print('\npublish_data_product_draft() result:') + + # begin-publish_data_product_draft + + response = dph_service.publish_data_product_draft( + data_product_id=publish_a_draft_of_data_product_by_data_product_id_link, + draft_id=publish_a_draft_by_draft_id_link, + ) + data_product_release = response.get_result() + + print(json.dumps(data_product_release, indent=2)) + + # end-publish_data_product_draft + + update_a_release_by_release_id_link = data_product_release['id'] + get_a_release_contract_terms_by_release_id_link = data_product_release['id'] + retire_a_release_contract_terms_by_release_id_link = data_product_release['id'] + get_a_release_by_release_id_link = data_product_release['id'] + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_initialize_status_example(self): + """ + get_initialize_status request example + """ + try: + print('\nget_initialize_status() result:') + + # begin-get_initialize_status + + response = dph_service.get_initialize_status() + initialize_resource = response.get_result() + + print(json.dumps(initialize_resource, indent=2)) + + # end-get_initialize_status + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_service_id_credentials_example(self): + """ + get_service_id_credentials request example + """ + try: + print('\nget_service_id_credentials() result:') + + # begin-get_service_id_credentials + + response = dph_service.get_service_id_credentials() + service_id_credentials = response.get_result() + + print(json.dumps(service_id_credentials, indent=2)) + + # end-get_service_id_credentials + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_manage_api_keys_example(self): + """ + manage_api_keys request example + """ + try: + # begin-manage_api_keys + + response = dph_service.manage_api_keys() + + # end-manage_api_keys + print('\nmanage_api_keys() response status code: ', response.get_status_code()) + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_create_data_asset_visualization_example(self): + """ + create_data_asset_visualization request example + """ + try: + print('\ncreate_data_asset_visualization() result:') + + # begin-create_data_asset_visualization + + container_reference_model = { + 'id': '2be8f727-c5d2-4cb0-9216-f9888f428048', + 'type': 'catalog', + } + + asset_reference_model = { + 'id': 'caeee3f3-756e-47d5-846d-da4600809e22', + 'container': container_reference_model, + } + + data_asset_relationship_model = { + 'asset': asset_reference_model, + 'related_asset': asset_reference_model, + } + + response = dph_service.create_data_asset_visualization( + assets=[data_asset_relationship_model], + ) + data_asset_visualization_res = response.get_result() + + print(json.dumps(data_asset_visualization_res, indent=2)) + + # end-create_data_asset_visualization + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_reinitiate_data_asset_visualization_example(self): + """ + reinitiate_data_asset_visualization request example + """ + try: + print('\nreinitiate_data_asset_visualization() result:') + + # begin-reinitiate_data_asset_visualization + + container_reference_model = { + 'id': '2be8f727-c5d2-4cb0-9216-f9888f428048', + 'type': 'catalog', + } + + asset_reference_model = { + 'id': 'caeee3f3-756e-47d5-846d-da4600809e22', + 'container': container_reference_model, + } + + data_asset_relationship_model = { + 'asset': asset_reference_model, + 'related_asset': asset_reference_model, + } + + response = dph_service.reinitiate_data_asset_visualization( + assets=[data_asset_relationship_model], + ) + data_asset_visualization_res = response.get_result() + + print(json.dumps(data_asset_visualization_res, indent=2)) + + # end-reinitiate_data_asset_visualization + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_list_data_products_example(self): + """ + list_data_products request example + """ + try: + print('\nlist_data_products() result:') + + # begin-list_data_products + + all_results = [] + pager = DataProductsPager( + client=dph_service, + limit=10, + ) + while pager.has_next(): + next_page = pager.get_next() + assert next_page is not None + all_results.extend(next_page) + + print(json.dumps(all_results, indent=2)) + + # end-list_data_products + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_data_product_example(self): + """ + get_data_product request example + """ + try: + print('\nget_data_product() result:') + + # begin-get_data_product + + response = dph_service.get_data_product( + data_product_id=get_data_product_by_data_product_id_link, + ) + data_product = response.get_result() + + print(json.dumps(data_product, indent=2)) + + # end-get_data_product + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_complete_draft_contract_terms_document_example(self): + """ + complete_draft_contract_terms_document request example + """ + try: + print('\ncomplete_draft_contract_terms_document() result:') + + # begin-complete_draft_contract_terms_document + + response = dph_service.complete_draft_contract_terms_document( + data_product_id=complete_draft_contract_terms_by_data_product_id_link, + draft_id=complete_a_draft_by_draft_id_link, + contract_terms_id=complete_a_draft_by_contract_terms_id_link, + document_id=complete_contract_terms_document_by_document_id_link, + ) + contract_terms_document = response.get_result() + + print(json.dumps(contract_terms_document, indent=2)) + + # end-complete_draft_contract_terms_document + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_list_data_product_drafts_example(self): + """ + list_data_product_drafts request example + """ + try: + print('\nlist_data_product_drafts() result:') + + # begin-list_data_product_drafts + + all_results = [] + pager = DataProductDraftsPager( + client=dph_service, + data_product_id=get_list_of_data_product_drafts_by_data_product_id_link, + asset_container_id='testString', + version='testString', + limit=10, + ) + while pager.has_next(): + next_page = pager.get_next() + assert next_page is not None + all_results.extend(next_page) + + print(json.dumps(all_results, indent=2)) + + # end-list_data_product_drafts + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_data_product_draft_example(self): + """ + get_data_product_draft request example + """ + try: + print('\nget_data_product_draft() result:') + + # begin-get_data_product_draft + + response = dph_service.get_data_product_draft( + data_product_id=get_a_draft_of_data_product_by_data_product_id_link, + draft_id=get_draft_by_draft_id_link, + ) + data_product_draft = response.get_result() + + print(json.dumps(data_product_draft, indent=2)) + + # end-get_data_product_draft + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_update_data_product_draft_example(self): + """ + update_data_product_draft request example + """ + try: + print('\nupdate_data_product_draft() result:') + + # begin-update_data_product_draft + + json_patch_operation_model = { + 'op': 'add', + 'path': 'testString', + } + + response = dph_service.update_data_product_draft( + data_product_id=update_draft_of_data_product_by_data_product_id_link, + draft_id=update_a_draft_by_draft_id_link, + json_patch_instructions=[json_patch_operation_model], + ) + data_product_draft = response.get_result() + + print(json.dumps(data_product_draft, indent=2)) + + # end-update_data_product_draft + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_draft_contract_terms_document_example(self): + """ + get_draft_contract_terms_document request example + """ + try: + print('\nget_draft_contract_terms_document() result:') + + # begin-get_draft_contract_terms_document + + response = dph_service.get_draft_contract_terms_document( + data_product_id=get_contract_document_by_data_product_id_link, + draft_id=get_a_draft_contract_document_by_draft_id_link, + contract_terms_id=get_a_draft_by_contract_terms_id_link, + document_id=get_contract_terms_document_by_id_document_id_link, + ) + contract_terms_document = response.get_result() + + print(json.dumps(contract_terms_document, indent=2)) + + # end-get_draft_contract_terms_document + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_update_draft_contract_terms_document_example(self): + """ + update_draft_contract_terms_document request example + """ + try: + print('\nupdate_draft_contract_terms_document() result:') + + # begin-update_draft_contract_terms_document + + json_patch_operation_model = { + 'op': 'add', + 'path': 'testString', + } + + response = dph_service.update_draft_contract_terms_document( + data_product_id=update_contract_document_by_data_product_id_link, + draft_id=update_contract_document_by_draft_id_link, + contract_terms_id=update_a_draft_by_contract_terms_id_link, + document_id=update_contract_terms_document_by_document_id_link, + json_patch_instructions=[json_patch_operation_model], + ) + contract_terms_document = response.get_result() + + print(json.dumps(contract_terms_document, indent=2)) + + # end-update_draft_contract_terms_document + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_data_product_draft_contract_terms_example(self): + """ + get_data_product_draft_contract_terms request example + """ + try: + print('\nget_data_product_draft_contract_terms() result:') + + # begin-get_data_product_draft_contract_terms + + response = dph_service.get_data_product_draft_contract_terms( + data_product_id='testString', + draft_id='testString', + contract_terms_id='testString', + ) + contract_terms = response.get_result() + + print(json.dumps(contract_terms, indent=2)) + + # end-get_data_product_draft_contract_terms + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_replace_data_product_draft_contract_terms_example(self): + """ + replace_data_product_draft_contract_terms request example + """ + try: + print('\nreplace_data_product_draft_contract_terms() result:') + + # begin-replace_data_product_draft_contract_terms + + contract_terms_document_model = { + 'url': 'https://ibm.com/document', + 'type': 'terms_and_conditions', + 'name': 'Terms and Conditions', + 'id': 'b38df608-d34b-4d58-8136-ed25e6c6684e', + } + + domain_model = { + 'id': 'b38df608-d34b-4d58-8136-ed25e6c6684e', + 'name': 'domain_name', + } + + overview_model = { + 'api_version': 'v3.0.1', + 'kind': 'DataContract', + 'name': 'Sample Data Contract', + 'version': 'v0.0', + 'domain': domain_model, + 'more_info': 'List of links to sources that provide more details on the data contract.', + } + + contract_terms_more_info_model = { + 'type': 'privacy-statement', + 'url': 'https://www.moreinfo.example.coms', + } + + description_model = { + 'purpose': 'Intended purpose for the provided data.', + 'limitations': 'Technical, compliance, and legal limitations for data use.', + 'usage': 'Recommended usage of the data.', + 'more_info': [contract_terms_more_info_model], + 'custom_properties': 'Custom properties that are not part of the standard.', + } + + contract_template_organization_model = { + 'user_id': 'IBMid-691000IN4G', + 'role': 'owner', + } + + roles_model = { + 'role': 'IAM Role', + } + + pricing_model = { + 'amount': 'Amount', + 'currency': 'Currency', + 'unit': 'Unit', + } + + contract_template_sla_property_model = { + 'property': 'slaproperty', + 'value': 'slavalue', + } + + contract_template_sla_model = { + 'default_element': 'sladefaultelement', + 'properties': [contract_template_sla_property_model], + } + + contract_template_support_and_communication_model = { + 'channel': 'channel', + 'url': 'https://www.example.coms', + } + + contract_template_custom_property_model = { + 'key': 'The name of the key.', + 'value': 'The value of the key.', + } + + contract_asset_model = { + 'id': '684d6aa0-9f93-4564-8a20-e354bc469857', + 'name': 'PAYMENT_TRANSACTIONS1', + } + + contract_server_model = { + 'server': 'snowflake-server-01', + 'asset': contract_asset_model, + 'connection_id': '8d7701be-709a-49c0-ae4e-a7daeaae6def', + 'type': 'snowflake', + 'description': 'Snowflake analytics server', + 'environment': 'dev', + 'account': 'acc-456', + 'catalog': 'analytics_cat', + 'database': 'analytics_db', + 'dataset': 'customer_data', + 'delimiter': ',', + 'endpoint_url': 'https://xy12345.snowflakecomputing.com', + 'format': 'parquet', + 'host': 'xy12345.snowflakecomputing.com', + 'location': 'Mumbai', + 'path': '/analytics/data', + 'port': '443', + 'project': 'projectY', + 'region': 'ap-south-1', + 'region_name': 'Asia South 1', + 'schema': 'PAYMENT_TRANSACTIONS1', + 'service_name': 'snowflake', + 'staging_dir': '/snowflake/staging', + 'stream': 'stream_analytics', + 'warehouse': 'wh_xlarge', + 'custom_properties': [contract_template_custom_property_model], + } + + contract_schema_property_type_model = { + 'type': 'varchar', + 'length': '1024', + 'scale': '0', + 'nullable': 'true', + 'signed': 'false', + } + + contract_schema_property_model = { + 'name': 'product_brand_code', + 'type': contract_schema_property_type_model, + } + + contract_schema_model = { + 'asset_id': '09ca6b40-7c89-412a-8951-ad820da709d1', + 'connection_id': '6cc57d4d-2229-438f-91a0-2c455556422b', + 'name': '000000_0-2025-06-20-20-28-52.csv', + 'connection_path': '/dpx-test-bucket/000000_0-2025-06-20-20-28-52.csv', + 'physical_type': 'text/csv', + 'properties': [contract_schema_property_model], + } + + response = dph_service.replace_data_product_draft_contract_terms( + data_product_id='testString', + draft_id='testString', + contract_terms_id='testString', + documents=[contract_terms_document_model], + overview=overview_model, + description=description_model, + organization=[contract_template_organization_model], + roles=[roles_model], + price=pricing_model, + sla=[contract_template_sla_model], + support_and_communication=[contract_template_support_and_communication_model], + custom_properties=[contract_template_custom_property_model], + servers=[contract_server_model], + schema=[contract_schema_model], + ) + contract_terms = response.get_result() + + print(json.dumps(contract_terms, indent=2)) + + # end-replace_data_product_draft_contract_terms + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_update_data_product_draft_contract_terms_example(self): + """ + update_data_product_draft_contract_terms request example + """ + try: + print('\nupdate_data_product_draft_contract_terms() result:') + + # begin-update_data_product_draft_contract_terms + + json_patch_operation_model = { + 'op': 'add', + 'path': 'testString', + } + + response = dph_service.update_data_product_draft_contract_terms( + data_product_id='testString', + draft_id='testString', + contract_terms_id='testString', + json_patch_instructions=[json_patch_operation_model], + ) + contract_terms = response.get_result() + + print(json.dumps(contract_terms, indent=2)) + + # end-update_data_product_draft_contract_terms + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_contract_terms_in_specified_format_example(self): + """ + get_contract_terms_in_specified_format request example + """ + try: + print('\nget_contract_terms_in_specified_format() result:') + + # begin-get_contract_terms_in_specified_format + + response = dph_service.get_contract_terms_in_specified_format( + data_product_id='testString', + draft_id='testString', + contract_terms_id='testString', + format='testString', + format_version='testString', + ) + result = response.get_result() + + with open('/tmp/result.out', 'wb') as fp: + fp.write(result) + + # end-get_contract_terms_in_specified_format + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_data_product_release_example(self): + """ + get_data_product_release request example + """ + try: + print('\nget_data_product_release() result:') + + # begin-get_data_product_release + + response = dph_service.get_data_product_release( + data_product_id=get_a_release_of_data_product_by_data_product_id_link, + release_id=get_a_release_by_release_id_link, + ) + data_product_release = response.get_result() + + print(json.dumps(data_product_release, indent=2)) + + # end-get_data_product_release + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_update_data_product_release_example(self): + """ + update_data_product_release request example + """ + try: + print('\nupdate_data_product_release() result:') + + # begin-update_data_product_release + + json_patch_operation_model = { + 'op': 'add', + 'path': 'testString', + } + + response = dph_service.update_data_product_release( + data_product_id=update_release_of_data_product_by_data_product_id_link, + release_id='testString', + json_patch_instructions=[json_patch_operation_model], + ) + data_product_release = response.get_result() + + print(json.dumps(data_product_release, indent=2)) + + # end-update_data_product_release + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_release_contract_terms_document_example(self): + """ + get_release_contract_terms_document request example + """ + try: + print('\nget_release_contract_terms_document() result:') + + # begin-get_release_contract_terms_document + + response = dph_service.get_release_contract_terms_document( + data_product_id=get_release_contract_document_by_data_product_id_link, + release_id=get_a_release_contract_terms_by_release_id_link, + contract_terms_id=get_a_release_contract_terms_by_contract_terms_id_link, + document_id=get_release_contract_document_by_document_id_link, + ) + contract_terms_document = response.get_result() + + print(json.dumps(contract_terms_document, indent=2)) + + # end-get_release_contract_terms_document + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_published_data_product_draft_contract_terms_example(self): + """ + get_published_data_product_draft_contract_terms request example + """ + try: + print('\nget_published_data_product_draft_contract_terms() result:') + + # begin-get_published_data_product_draft_contract_terms + + response = dph_service.get_published_data_product_draft_contract_terms( + data_product_id='testString', + release_id='testString', + contract_terms_id='testString', + ) + result = response.get_result() + + with open('/tmp/result.out', 'wb') as fp: + fp.write(result) + + # end-get_published_data_product_draft_contract_terms + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_list_data_product_releases_example(self): + """ + list_data_product_releases request example + """ + try: + print('\nlist_data_product_releases() result:') + + # begin-list_data_product_releases + + all_results = [] + pager = DataProductReleasesPager( + client=dph_service, + data_product_id=get_list_of_releases_of_data_product_by_data_product_id_link, + asset_container_id='testString', + state=['available'], + version='testString', + limit=10, + ) + while pager.has_next(): + next_page = pager.get_next() + assert next_page is not None + all_results.extend(next_page) + + print(json.dumps(all_results, indent=2)) + + # end-list_data_product_releases + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_retire_data_product_release_example(self): + """ + retire_data_product_release request example + """ + try: + print('\nretire_data_product_release() result:') + + # begin-retire_data_product_release + + response = dph_service.retire_data_product_release( + data_product_id=retire_a_releases_of_data_product_by_data_product_id_link, + release_id=retire_a_release_contract_terms_by_release_id_link, + ) + data_product_release = response.get_result() + + print(json.dumps(data_product_release, indent=2)) + + # end-retire_data_product_release + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_create_revoke_access_process_example(self): + """ + create_revoke_access_process request example + """ + try: + print('\ncreate_revoke_access_process() result:') + + # begin-create_revoke_access_process + + response = dph_service.create_revoke_access_process( + data_product_id='testString', + release_id='testString', + body=io.BytesIO(b'This is a mock file.').getvalue(), + ) + revoke_access_response = response.get_result() + + print(json.dumps(revoke_access_response, indent=2)) + + # end-create_revoke_access_process + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_list_data_product_contract_template_example(self): + """ + list_data_product_contract_template request example + """ + try: + print('\nlist_data_product_contract_template() result:') + + # begin-list_data_product_contract_template + + response = dph_service.list_data_product_contract_template() + data_product_contract_template_collection = response.get_result() + + print(json.dumps(data_product_contract_template_collection, indent=2)) + + # end-list_data_product_contract_template + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_create_contract_template_example(self): + """ + create_contract_template request example + """ + try: + print('\ncreate_contract_template() result:') + + # begin-create_contract_template + + container_reference_model = { + 'id': '531f74a-01c8-4e91-8e29-b018db683c86', + 'type': 'catalog', + } + + domain_model = { + 'id': '0094ebe9-abc3-473b-80ea-c777ede095ea', + 'name': 'Test Domain New', + } + + overview_model = { + 'name': 'Sample Data Contract', + 'version': '0.0.0', + 'domain': domain_model, + 'more_info': 'List of links to sources that provide more details on the data contract.', + } + + contract_terms_more_info_model = { + 'type': 'privacy-statement', + 'url': 'https://www.moreinfo.example.coms', + } + + description_model = { + 'purpose': 'Intended purpose for the provided data.', + 'limitations': 'Technical, compliance, and legal limitations for data use.', + 'usage': 'Recommended usage of the data.', + 'more_info': [contract_terms_more_info_model], + 'custom_properties': 'Custom properties that are not part of the standard.', + } + + contract_template_organization_model = { + 'user_id': 'IBMid-691000IN4G', + 'role': 'owner', + } + + roles_model = { + 'role': 'IAM Role', + } + + pricing_model = { + 'amount': '100.00', + 'currency': 'USD', + 'unit': 'megabyte', + } + + contract_template_sla_property_model = { + 'property': 'slaproperty', + 'value': 'slavalue', + } + + contract_template_sla_model = { + 'default_element': 'sladefaultelement', + 'properties': [contract_template_sla_property_model], + } + + contract_template_support_and_communication_model = { + 'channel': 'channel', + 'url': 'https://www.example.coms', + } + + contract_template_custom_property_model = { + 'key': 'propertykey', + 'value': 'propertyvalue', + } + + contract_terms_model = { + 'overview': overview_model, + 'description': description_model, + 'organization': [contract_template_organization_model], + 'roles': [roles_model], + 'price': pricing_model, + 'sla': [contract_template_sla_model], + 'support_and_communication': [contract_template_support_and_communication_model], + 'custom_properties': [contract_template_custom_property_model], + } + + response = dph_service.create_contract_template( + container=container_reference_model, + name='Sample Data Contract Template', + contract_terms=contract_terms_model, + ) + data_product_contract_template = response.get_result() + + print(json.dumps(data_product_contract_template, indent=2)) + + # end-create_contract_template + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_contract_template_example(self): + """ + get_contract_template request example + """ + try: + print('\nget_contract_template() result:') + + # begin-get_contract_template + + response = dph_service.get_contract_template( + contract_template_id='testString', + container_id='testString', + ) + data_product_contract_template = response.get_result() + + print(json.dumps(data_product_contract_template, indent=2)) + + # end-get_contract_template + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_update_data_product_contract_template_example(self): + """ + update_data_product_contract_template request example + """ + try: + print('\nupdate_data_product_contract_template() result:') + + # begin-update_data_product_contract_template + + json_patch_operation_model = { + 'op': 'add', + 'path': 'testString', + } + + response = dph_service.update_data_product_contract_template( + contract_template_id='testString', + container_id='testString', + json_patch_instructions=[json_patch_operation_model], + ) + data_product_contract_template = response.get_result() + + print(json.dumps(data_product_contract_template, indent=2)) + + # end-update_data_product_contract_template + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_list_data_product_domains_example(self): + """ + list_data_product_domains request example + """ + try: + print('\nlist_data_product_domains() result:') + + # begin-list_data_product_domains + + response = dph_service.list_data_product_domains() + data_product_domain_collection = response.get_result() + + print(json.dumps(data_product_domain_collection, indent=2)) + + # end-list_data_product_domains + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_create_data_product_domain_example(self): + """ + create_data_product_domain request example + """ + try: + print('\ncreate_data_product_domain() result:') + + # begin-create_data_product_domain + + container_reference_model = { + 'id': 'ed580171-a6e4-4b93-973f-ae2f2f62991b', + 'type': 'catalog', + } + + initialize_sub_domain_model = { + 'name': 'Sub domain 1', + 'description': 'New sub domain 1', + } + + response = dph_service.create_data_product_domain( + container=container_reference_model, + name='Test domain', + description='The sample description for new domain', + sub_domains=[initialize_sub_domain_model], + ) + data_product_domain = response.get_result() + + print(json.dumps(data_product_domain, indent=2)) + + # end-create_data_product_domain + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_create_data_product_subdomain_example(self): + """ + create_data_product_subdomain request example + """ + try: + print('\ncreate_data_product_subdomain() result:') + + # begin-create_data_product_subdomain + + response = dph_service.create_data_product_subdomain( + domain_id='testString', + container_id='testString', + name='Sub domain 1', + description='New sub domain 1', + ) + initialize_sub_domain = response.get_result() + + print(json.dumps(initialize_sub_domain, indent=2)) + + # end-create_data_product_subdomain + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_domain_example(self): + """ + get_domain request example + """ + try: + print('\nget_domain() result:') + + # begin-get_domain + + response = dph_service.get_domain( + domain_id='testString', + ) + data_product_domain = response.get_result() + + print(json.dumps(data_product_domain, indent=2)) + + # end-get_domain + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_update_data_product_domain_example(self): + """ + update_data_product_domain request example + """ + try: + print('\nupdate_data_product_domain() result:') + + # begin-update_data_product_domain + + json_patch_operation_model = { + 'op': 'add', + 'path': 'testString', + } + + response = dph_service.update_data_product_domain( + domain_id='testString', + container_id='testString', + json_patch_instructions=[json_patch_operation_model], + ) + data_product_domain = response.get_result() + + print(json.dumps(data_product_domain, indent=2)) + + # end-update_data_product_domain + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_data_product_by_domain_example(self): + """ + get_data_product_by_domain request example + """ + try: + print('\nget_data_product_by_domain() result:') + + # begin-get_data_product_by_domain + + response = dph_service.get_data_product_by_domain( + domain_id='testString', + container_id='testString', + ) + data_product_version_collection = response.get_result() + + print(json.dumps(data_product_version_collection, indent=2)) + + # end-get_data_product_by_domain + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_create_s3_bucket_example(self): + """ + create_s3_bucket request example + """ + try: + print('\ncreate_s3_bucket() result:') + + # begin-create_s3_bucket + + response = dph_service.create_s3_bucket( + is_shared=True, + ) + bucket_response = response.get_result() + + print(json.dumps(bucket_response, indent=2)) + + # end-create_s3_bucket + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_s3_bucket_validation_example(self): + """ + get_s3_bucket_validation request example + """ + try: + print('\nget_s3_bucket_validation() result:') + + # begin-get_s3_bucket_validation + + response = dph_service.get_s3_bucket_validation( + bucket_name='testString', + ) + bucket_validation_response = response.get_result() + + print(json.dumps(bucket_validation_response, indent=2)) + + # end-get_s3_bucket_validation + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_revoke_access_process_state_example(self): + """ + get_revoke_access_process_state request example + """ + try: + print('\nget_revoke_access_process_state() result:') + + # begin-get_revoke_access_process_state + + response = dph_service.get_revoke_access_process_state( + release_id='testString', + ) + revoke_access_state_response = response.get_result() + + print(json.dumps(revoke_access_state_response, indent=2)) + + # end-get_revoke_access_process_state + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_delete_draft_contract_terms_document_example(self): + """ + delete_draft_contract_terms_document request example + """ + try: + # begin-delete_draft_contract_terms_document + + response = dph_service.delete_draft_contract_terms_document( + data_product_id=delete_contract_document_by_data_product_id_link, + draft_id=delete_a_contract_document_by_draft_id_link, + contract_terms_id=delete_a_draft_by_contract_terms_id_link, + document_id=delete_contract_terms_document_by_document_id_link, + ) + + # end-delete_draft_contract_terms_document + print('\ndelete_draft_contract_terms_document() response status code: ', response.get_status_code()) + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_delete_data_product_draft_example(self): + """ + delete_data_product_draft request example + """ + try: + # begin-delete_data_product_draft + + response = dph_service.delete_data_product_draft( + data_product_id=delete_draft_of_data_product_by_data_product_id_link, + draft_id=delete_a_draft_by_draft_id_link, + ) + + # end-delete_data_product_draft + print('\ndelete_data_product_draft() response status code: ', response.get_status_code()) + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_delete_data_product_contract_template_example(self): + """ + delete_data_product_contract_template request example + """ + try: + # begin-delete_data_product_contract_template + + response = dph_service.delete_data_product_contract_template( + contract_template_id='testString', + container_id='testString', + ) + + # end-delete_data_product_contract_template + print('\ndelete_data_product_contract_template() response status code: ', response.get_status_code()) + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_delete_domain_example(self): + """ + delete_domain request example + """ + try: + # begin-delete_domain + + response = dph_service.delete_domain( + domain_id='testString', + ) + + # end-delete_domain + print('\ndelete_domain() response status code: ', response.get_status_code()) + + except ApiException as e: + pytest.fail(str(e)) + + +# endregion +############################################################################## +# End of Examples for Service: DphV1 +############################################################################## diff --git a/requirements-dev.txt b/requirements-dev.txt new file mode 100644 index 0000000..0da4263 --- /dev/null +++ b/requirements-dev.txt @@ -0,0 +1,17 @@ +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +ibm-cloud-sdk-core==3.24.4 +black>=26.3.1 +pylint>=3.0.0 diff --git a/requirements.txt b/requirements.txt index 1b9de16..0951bc4 100644 --- a/requirements.txt +++ b/requirements.txt @@ -14,18 +14,24 @@ # Core dependencies (defined in setup.py, included here for development convenience) pydantic>=2.12.0 -requests>=2.28.0 +requests>=2.32.4 regex>=2023.0.0 +urllib3>=2.6.3 +python-dateutil>=2.5.3,<3.0.0 +PyJWT>=2.12.1 +pyyaml>=5.4.0,<7.0.0 +numpy>=1.24.0 # Note: ibm-cloud-sdk-core is defined in setup.py with exact version pin # Development dependencies pytest>=7.0.0 pytest-cov>=4.0.0 pytest-mock>=3.7.0 +responses>=0.20.0 # black is defined in setup.py extras_require['dev'] to avoid BOM conflicts mypy>=1.0.0 flake8>=6.0.0 # Optional dependencies for development/testing -pandas>=1.3.0 -pyspark>=3.0.0 \ No newline at end of file +pandas>=2.0.0 +pyspark>=3.0.0 diff --git a/setup.py b/setup.py index 03e0d0c..5c928a1 100644 --- a/setup.py +++ b/setup.py @@ -27,10 +27,11 @@ setup( name="data-intelligence-sdk", - version='0.5.3', + version='2.0.0', author="IBM", author_email="Data_Intelligence_SDK@wwpdl.vnet.ibm.com", - description="A Python SDK for performing data quality validations on streaming data records and DataFrames", + description="A Python SDK for IBM watsonx.data intelligence that provides data quality validation for streaming records and DataFrames, " \ + "wrapper methods to access Data Product Hub REST API services, ODCS file generation from data catalogs, and data product recommendations from query log analysis.", long_description=long_description, long_description_content_type="text/markdown", url="https://github.com/IBM/data-intelligence-sdk", @@ -49,11 +50,15 @@ "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", ], - python_requires=">=3.8", + python_requires=">=3.10", install_requires=[ "pydantic>=2.12.0", - "requests>=2.28.0", + "requests>=2.33.1", "regex>=2023.0.0", + "urllib3>=2.6.3", + "python-dateutil>=2.5.3,<3.0.0", + "pyyaml>=5.4.0,<7.0.0", + "numpy>=1.24.0", # Pinned to exact version to avoid CRA bom-generate pip resolver conflict. # CRA sees ibm-cloud-sdk-core from both setup.py and requirements.txt and # fails with ResolutionImpossible when constraints differ (bare vs >=). @@ -64,29 +69,33 @@ "pytest>=7.0.0", "pytest-cov>=4.0.0", "pytest-mock>=3.7.0", + "responses>=0.20.0", "black>=26.3.1", "mypy>=1.0.0", "flake8>=6.0.0", + "pylint>=3.0.0", ], "pandas": [ - "pandas>=1.3.0", + "pandas>=2.0.0", ], "spark": [ "pyspark>=3.0.0", ], "dataframes": [ - "pandas>=1.3.0", + "pandas>=2.0.0", "pyspark>=3.0.0", ], "all": [ - "pandas>=1.3.0", + "pandas>=2.0.0", "pyspark>=3.0.0", "pytest>=7.0.0", "pytest-cov>=4.0.0", "pytest-mock>=3.7.0", + "responses>=0.20.0", "black>=26.3.1", "mypy>=1.0.0", "flake8>=6.0.0", + "pylint>=3.0.0", ], }, entry_points={ diff --git a/src/wxdi/__init__.py b/src/wxdi/__init__.py index 497cd9a..63261f8 100644 --- a/src/wxdi/__init__.py +++ b/src/wxdi/__init__.py @@ -16,7 +16,8 @@ """ WXDI - IBM watsonx.data intelligence SDK -A comprehensive Python SDK for data quality validation and data intelligence operations. +A comprehensive Python SDK for data quality validation, data intelligence operations, +data product hub services, ODCS generation, and data product recommendations. """ # Re-export commonly used modules for convenience @@ -98,4 +99,10 @@ "DataQualityDimension", ] +# Note: dph_services, odcs_generator, and data_product_recommender are available as submodules +# Import them explicitly: +# from wxdi.dph_services import DphV1 +# from wxdi.odcs_generator import CollibraClient, ODCSGenerator, InformaticaClient +# from wxdi.data_product_recommender import DataProductRecommender + # Made with Bob diff --git a/src/wxdi/data_product_recommender/README.md b/src/wxdi/data_product_recommender/README.md new file mode 100644 index 0000000..5c71fbc --- /dev/null +++ b/src/wxdi/data_product_recommender/README.md @@ -0,0 +1,218 @@ + + +# Data Product Recommender + +A tool that analyzes database query logs from files to identify high-value tables and logical groupings that should be prioritized as data products in a data marketplace. + +## Purpose + +**Accelerate data product onboarding** by leveraging existing usage patterns rather than starting from scratch. + +Organizations often have valuable data assets already in active use across various teams and use cases. Instead of building new data pipelines or guessing which tables to promote, this tool analyzes actual query patterns to identify: + +- **High-value tables**: Frequently queried by diverse user groups +- **Logical groupings**: Tables that are commonly used together +- **Proven assets**: Tables with demonstrated business value through real usage + +## Features + +- **Multi-platform support**: Snowflake, Databricks, BigQuery, watsonx.data query log formats +- **File-based input**: Supports CSV and JSON query log files (no direct database connection required) +- **Intelligent scoring**: Combines query frequency, user diversity, recency, and consistency +- **Table grouping**: Identifies tables frequently used together +- **Query pattern analysis**: Shows common query patterns for each recommendation +- **Multiple output formats**: Markdown (human-readable) and JSON (agent-consumable) +- **CLI and Python API**: Use from command line or integrate into your applications + +## Installation + +From the data-intelligence-sdk repository: + +```bash +# Install the package with dependencies +pip install -e . +``` + +## Quick Start + +### Command Line Interface + +```bash +# Analyze Snowflake query logs from CSV +python -m wxdi.data_product_recommender.cli \ + --platform snowflake \ + --input-file samples/query_logs.csv \ + --output output \ + --num-recommendations 20 + +# Analyze with minimum score threshold +python -m wxdi.data_product_recommender.cli \ + --platform databricks \ + --input-file samples/query_logs.json \ + --min-score 60.0 \ + --output-format json +``` + +### Python API + +```python +from data_product_recommender.platforms import SnowflakeQueryParser +from data_product_recommender.recommender import DataProductRecommender + +# Initialize with platform-specific parser +parser = SnowflakeQueryParser() +recommender = DataProductRecommender(parser) + +# Load and analyze query logs from file +recommender.load_query_logs_from_csv_file('query_logs.csv') +recommender.calculate_metrics() +recommendations = recommender.recommend_data_products(num_recommendations=20) + +# Export results +recommender.export_recommendations_markdown(recommendations, 'output/recommendations.md') +recommender.export_recommendations_json(recommendations, 'output/recommendations.json') +``` + +## Methodology + +### Individual Table Scoring (0-100) + +- **37.5%** Query Count - Volume of usage +- **37.5%** User Diversity - Breadth of usage across teams +- **15%** Recency - Recent activity (prioritizes currently active tables) +- **10%** Consistency - Regular usage patterns (identifies stable, ongoing value) + +### Table Group Scoring (0-100) + +- **30%** Cohesion - How tightly tables are connected +- **20%** Usage - Relative usage compared to other groups +- **15%** User Reach - Percentage of users querying the group +- **20%** Recency - Recent activity across tables +- **10%** Consistency - Regular usage patterns +- **5%** Size - Number of tables in the group + +### Star Rating Scale + +- ⭐⭐⭐⭐⭐ **Excellent (80-100)**: Implement immediately +- ⭐⭐⭐⭐ **Good (60-79)**: Medium priority +- ⭐⭐⭐ **Fair (40-59)**: Consider splitting or implement later +- ⭐⭐ **Weak (20-39)**: Reconsider grouping +- ⭐ **Poor (0-19)**: Do not implement + +## CLI Options + +``` +--platform Database platform (snowflake, databricks, bigquery, watsonxdata) +--input-file Path to CSV or JSON query log file +--output Output directory (default: output) +--output-format Output format: markdown or json (default: markdown) +--num-recommendations Number of recommendations (default: 20) +--min-score Minimum score threshold 0-100 (filters low-scoring tables) +``` + +## Output Formats + +### Markdown (Human-Readable) +- Rich formatting with tables and collapsible sections +- Star ratings and detailed explanations +- Query pattern examples with syntax highlighting +- Comprehensive coverage statistics + +### JSON (Agent-Consumable) +- Clean, parseable structure +- All metrics as numeric values +- Rating labels (excellent, good, fair, weak, poor) +- Suitable for AI agents and automated workflows + +## Platform Support + +This tool supports query log files from the following platforms: + +- ✅ **Snowflake** - Export from `SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY` +- ✅ **Databricks** - Export from `system.query.history` (Unity Catalog) +- ✅ **BigQuery** - Export from `INFORMATION_SCHEMA.JOBS_BY_PROJECT` +- ✅ **watsonx.data** - Export from `system.runtime.queries` (Presto) + +**Note**: This tool requires pre-exported query log files in CSV or JSON format. It does not connect directly to databases. + +## Examples + +See the `examples/` directory for detailed usage examples: + +- `data_product_recommender_example.py` - Basic usage with different platforms +- Custom scoring weights +- JSON and CSV input formats + +## Testing + +```bash +# Run unit tests +pytest tests/src/data_product_recommender/ -v + +# Run with coverage +pytest tests/src/data_product_recommender/ --cov=wxdi.data_product_recommender --cov-report=html +``` + +## Architecture + +The tool uses an extensible design with abstract base classes: + +``` +QueryLogParser (Abstract) +├── SnowflakeQueryParser +├── DatabricksQueryParser +├── BigQueryQueryParser +└── WatsonxDataQueryParser + +DataProductRecommender +└── Uses parser for platform-specific query log formats +``` + +## Important Notes + +### Query Log Export + +To use this tool, you need to export query logs from your data platform: + +**Snowflake Example:** +```sql +SELECT query_text, user_name, start_time, end_time, execution_status +FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY +WHERE start_time >= DATEADD(day, -30, CURRENT_TIMESTAMP()) +``` + +**Databricks Example:** +```sql +SELECT statement_text, user_name, start_time, end_time, status +FROM system.query.history +WHERE start_time >= current_timestamp() - INTERVAL 30 DAYS +``` + +### Required Columns + +Query log files must contain these columns (column names will be normalized by the parser): +- `query_text` - The SQL query text +- `user` - User who executed the query +- `start_time` - Query execution timestamp + +### Data Privacy + +Query logs may contain sensitive information including user identities, table names, and query patterns. Ensure proper data handling and security measures are in place. + +## License + +Apache License 2.0 - See LICENSE file in the root directory. \ No newline at end of file diff --git a/src/wxdi/data_product_recommender/__init__.py b/src/wxdi/data_product_recommender/__init__.py new file mode 100644 index 0000000..d226660 --- /dev/null +++ b/src/wxdi/data_product_recommender/__init__.py @@ -0,0 +1,24 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Data Product Recommender - Query Log Analysis Tool + +Analyzes query logs from various data platforms to recommend data products +based on usage patterns, user diversity, and table relationships. +""" + +__version__ = "0.1.0" \ No newline at end of file diff --git a/src/wxdi/data_product_recommender/base.py b/src/wxdi/data_product_recommender/base.py new file mode 100644 index 0000000..89c8916 --- /dev/null +++ b/src/wxdi/data_product_recommender/base.py @@ -0,0 +1,37 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Abstract base class for query log parsers. +""" + +from abc import ABC, abstractmethod +from typing import List +import pandas as pd + + +class QueryLogParser(ABC): + """Abstract base class for parsing query logs""" + + @abstractmethod + def normalize_columns(self, df: pd.DataFrame) -> pd.DataFrame: + """Normalize column names to standard format""" + pass + + @abstractmethod + def extract_tables(self, query_text: str) -> List[str]: + """Extract table names from SQL query""" + pass \ No newline at end of file diff --git a/src/wxdi/data_product_recommender/cli.py b/src/wxdi/data_product_recommender/cli.py new file mode 100644 index 0000000..d884a90 --- /dev/null +++ b/src/wxdi/data_product_recommender/cli.py @@ -0,0 +1,152 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Command-line interface for the Data Product Recommender +""" + +import argparse +import json +from datetime import datetime, timedelta +from pathlib import Path + +from .platforms import ( + SnowflakeQueryParser, + DatabricksQueryParser, + BigQueryQueryParser, + WatsonxDataQueryParser +) +from .recommender import DataProductRecommender + + +def main(): + parser = argparse.ArgumentParser( + description='Analyze query logs to recommend data products' + ) + + # Platform selection + parser.add_argument( + '--platform', + choices=['snowflake', 'databricks', 'bigquery', 'watsonxdata'], + required=True, + help='Data platform to analyze' + ) + + # Input source (file-based only) + parser.add_argument( + '--input-file', + type=str, + required=True, + help='Path to CSV or JSON file containing query logs' + ) + + # Output options + parser.add_argument( + '--output', + type=str, + default='output', + help='Output directory for recommendations (default: output)' + ) + + parser.add_argument( + '--output-format', + choices=['markdown', 'json'], + default='markdown', + help='Output format: markdown (human-readable) or json (agent-consumable) (default: markdown)' + ) + + parser.add_argument( + '--num-recommendations', + type=int, + default=20, + help='Number of recommendations to generate (default: 20)' + ) + + parser.add_argument( + '--min-score', + type=float, + default=None, + help='Minimum recommendation score threshold (0-100). Tables below this score will be excluded.' + ) + + args = parser.parse_args() + + # Initialize platform-specific parser + print(f"Initializing {args.platform} query parser...") + + if args.platform == 'snowflake': + parser = SnowflakeQueryParser() + elif args.platform == 'databricks': + parser = DatabricksQueryParser() + elif args.platform == 'bigquery': + parser = BigQueryQueryParser() + elif args.platform == 'watsonxdata': + parser = WatsonxDataQueryParser() + + # Initialize recommender + recommender = DataProductRecommender(parser) + + # Load query logs from file + print(f"Loading query logs from file: {args.input_file}") + + if args.input_file.lower().endswith('.json'): + recommender.load_query_logs_from_json_file(args.input_file) + elif args.input_file.lower().endswith('.csv'): + recommender.load_query_logs_from_csv_file(args.input_file) + else: + raise ValueError("Invalid --input-file type. Supported file types are JSON and CSV.") + + # Calculate metrics + print("\nCalculating metrics...") + recommender.calculate_metrics() + + # Generate recommendations + print("\nGenerating recommendations...") + recommendations = recommender.recommend_data_products( + num_recommendations=args.num_recommendations, + min_score=args.min_score + ) + + # Create output directory + output_dir = Path(args.output) + output_dir.mkdir(parents=True, exist_ok=True) + + # Generate output filename based on format + timestamp = datetime.now().strftime('%Y%m%d_%H%M%S') + input_name = Path(args.input_file).stem if args.input_file else 'database' + file_extension = 'json' if args.output_format == 'json' else 'md' + output_file = output_dir / f"recommendations_{input_name}_{timestamp}.{file_extension}" + + # Export recommendations in selected format + print(f"\nExporting recommendations to {output_file}...") + if args.output_format == 'json': + recommender.export_recommendations_json(recommendations, str(output_file)) + else: + recommender.export_recommendations_markdown(recommendations, str(output_file)) + + print("\n✓ Analysis complete!") + print(f" - Queries analyzed: {len(recommender.query_logs):,}") + print(f" - Tables identified: {len(recommender.table_metrics)}") + print(f" - Recommendations: {len(recommendations['individual_tables'])}") + if 'table_groups' in recommendations: + print(f" - Table groups: {len(recommendations['table_groups'])}") + print(f" - Output file: {output_file}") + + +if __name__ == '__main__': + main() + +# Made with Bob diff --git a/src/wxdi/data_product_recommender/platforms.py b/src/wxdi/data_product_recommender/platforms.py new file mode 100644 index 0000000..b0c6d7b --- /dev/null +++ b/src/wxdi/data_product_recommender/platforms.py @@ -0,0 +1,167 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Platform-specific query log parsers for Snowflake, Databricks, BigQuery, and watsonx.data + +Note: This module only provides query parsing functionality. +Database connections are not supported - use file-based input instead. +""" + +import re +from typing import List +import pandas as pd + +from .base import QueryLogParser + +# Regex pattern for extracting schema.table from FROM/JOIN clauses +TABLE_PATTERN = r'(?:FROM|JOIN)\s+([\w]+\.[\w]+)(?:\s+[a-zA-Z])?' + + +# ============================================================================ +# SNOWFLAKE QUERY PARSER +# ============================================================================ + +class SnowflakeQueryParser(QueryLogParser): + """Snowflake-specific query parser""" + + def normalize_columns(self, df: pd.DataFrame) -> pd.DataFrame: + """Normalize Snowflake column names""" + return df.rename(columns={ + 'query_text': 'query_text', + 'user_name': 'user', + 'start_time': 'start_time' + }) + + def extract_tables(self, query_text: str) -> List[str]: + """Extract table names from SQL query using regex patterns""" + if not query_text or not isinstance(query_text, str): + return [] + + query_upper = query_text.upper() + tables = set() + + # Pattern: Match FROM and JOIN clauses with schema.table format + matches = re.findall(TABLE_PATTERN, query_upper) + tables.update(matches) + + return list(tables) + + +# ============================================================================ +# DATABRICKS QUERY PARSER +# ============================================================================ + +class DatabricksQueryParser(QueryLogParser): + """Databricks-specific query parser""" + + def normalize_columns(self, df: pd.DataFrame) -> pd.DataFrame: + """Normalize Databricks column names""" + return df.rename(columns={ + 'statement_text': 'query_text', + 'executed_by': 'user', + 'start_time': 'start_time' + }) + + def extract_tables(self, query_text: str) -> List[str]: + """Extract table names from SQL query using regex patterns""" + if not query_text or not isinstance(query_text, str): + return [] + + query_upper = query_text.upper() + tables = set() + + # Pattern: Match FROM and JOIN clauses with schema.table format + matches = re.findall(TABLE_PATTERN, query_upper) + tables.update(matches) + + return list(tables) + + +# ============================================================================ +# BIGQUERY QUERY PARSER +# ============================================================================ + +class BigQueryQueryParser(QueryLogParser): + """BigQuery-specific query parser""" + + def normalize_columns(self, df: pd.DataFrame) -> pd.DataFrame: + """Normalize BigQuery column names""" + return df.rename(columns={ + 'query': 'query_text', + 'user_email': 'user', + 'start_time': 'start_time' + }) + + def extract_tables(self, query_text: str) -> List[str]: + """Extract table names from BigQuery SQL query using regex patterns""" + if not query_text or not isinstance(query_text, str): + return [] + + query_upper = query_text.upper() + tables = set() + + # BigQuery pattern: Match backtick-quoted tables with project.dataset.table format + # Example: `retailco-project.PRODUCT.CATALOG` + pattern1 = r'`([\w-]+\.[\w]+\.[\w]+)`' + matches1 = re.findall(pattern1, query_text) # Use original case for project names + + # Simplify to DATASET.TABLE format (remove project prefix) + for match in matches1: + parts = match.split('.') + if len(parts) == 3: + # Keep only dataset.table + simplified = f"{parts[1]}.{parts[2]}" + tables.add(simplified.upper()) + + # Also try standard pattern for dataset.table without backticks + matches2 = re.findall(TABLE_PATTERN, query_upper) + tables.update(matches2) + + return list(tables) + + +# ============================================================================ +# WATSONX.DATA (PRESTO) QUERY PARSER +# ============================================================================ + +class WatsonxDataQueryParser(QueryLogParser): + """watsonx.data-specific query parser""" + + def normalize_columns(self, df: pd.DataFrame) -> pd.DataFrame: + """Normalize watsonx.data column names""" + return df.rename(columns={ + 'query': 'query_text', + 'user': 'user', + 'created': 'start_time' + }) + + def extract_tables(self, query_text: str) -> List[str]: + """Extract table names from SQL query using regex patterns""" + if not query_text or not isinstance(query_text, str): + return [] + + query_upper = query_text.upper() + tables = set() + + # Pattern: Match FROM and JOIN clauses with schema.table format + matches = re.findall(TABLE_PATTERN, query_upper) + tables.update(matches) + + return list(tables) + + +# Made with Bob \ No newline at end of file diff --git a/src/wxdi/data_product_recommender/recommender.py b/src/wxdi/data_product_recommender/recommender.py new file mode 100644 index 0000000..65f9732 --- /dev/null +++ b/src/wxdi/data_product_recommender/recommender.py @@ -0,0 +1,1292 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Data Product Recommender - Core recommendation engine +""" + +import json +import re +import pandas as pd +from datetime import datetime +from collections import Counter, defaultdict +from typing import List, Dict, Tuple, Optional + +from .base import QueryLogParser + +# Star rating constants +FIVE_STARS = "⭐⭐⭐⭐⭐" +FOUR_STARS = "⭐⭐⭐⭐" +THREE_STARS = "⭐⭐⭐" +TWO_STARS = "⭐⭐" +ONE_STAR = "⭐" + +EXCELLENT_CANDIDATE = "Excellent Candidate" +GOOD_CANDIDATE = "Good Candidate" +FAIR_CANDIDATE = "Fair Candidate" +WEAK_CANDIDATE = "Weak Candidate" +POOR_CANDIDATE = "Poor Candidate" + +# Report section headers +QUERY_LOG_METRICS_HEADER = "**Query Log Metrics:**\n" +TABLES_IN_GROUP_HEADER = "**Tables in Group:**\n\n" +TABLES_IN_GROUP_SIMPLE = "**Tables in Group:**\n" +TABLE_HEADER_ROW = "| Table | Individual Score | Queries | Users | Recency | Consistency |\n" +TABLE_SEPARATOR_ROW = "|-------|-----------------|---------|-------|---------|-------------|\n" + +# SQL code block markers +SQL_CODE_BLOCK_START = " ```sql\n" +SQL_CODE_BLOCK_END = " ```\n" +SQL_EXAMPLE_LABEL = " - Example:\n" +SQL_EXAMPLE_CODE_START = " ```sql\n" +SQL_EXAMPLE_CODE_END = " ```\n" + +# HTML collapsible section markers +DETAILS_START = "
\n" +DETAILS_SUMMARY = "Frequent Query Patterns (click to expand)\n\n" +DETAILS_END = "
\n\n" + + +def normalize_query_pattern(query_text: str) -> str: + """ + Normalize a SQL query into a pattern by replacing literals with placeholders. + + This helps group similar queries together by removing variable parts like: + - Numbers (123 -> ?) + - Quoted strings ('value' -> ?) + - Date literals ('2024-01-01' -> ?) + + Args: + query_text: The SQL query text to normalize + + Returns: + Normalized query pattern with literals replaced by ? + """ + if not query_text or not isinstance(query_text, str): + return "" + + # Convert to uppercase for consistency + pattern = query_text.upper() + + # Replace quoted strings (both single and double quotes) + # Handles escaped quotes within strings + pattern = re.sub(r"'(?:[^'\\]|\\.)*'", "?", pattern) + pattern = re.sub(r'"(?:[^"\\]|\\.)*"', "?", pattern) + + # Replace numbers (integers and decimals) + pattern = re.sub(r'\b\d+\.?\d*\b', '?', pattern) + + # Replace date/timestamp patterns that might remain + # Format: YYYY-MM-DD or YYYY-MM-DD HH:MM:SS + pattern = re.sub(r'\b\?-\?-\?\b', '?', pattern) + pattern = re.sub(r'\b\? \?:\?:\?\b', '?', pattern) + + # Normalize whitespace (multiple spaces/tabs/newlines to single space) + pattern = re.sub(r'\s+', ' ', pattern) + + # Trim leading/trailing whitespace + pattern = pattern.strip() + + return pattern + + +class DataProductRecommender: + """Analyzes query logs and recommends data products""" + + # Error message constants + ERROR_QUERY_LOGS_NOT_LOADED = "Query logs not loaded" + + def __init__(self, parser: QueryLogParser): + self.parser = parser + self.query_logs = None + self.table_metrics = None + self.query_patterns = None # Store query pattern information + + def load_query_logs_from_json_file(self, file_path: str): + """Load and normalize query logs from JSON file""" + print(f"Reading query logs from {file_path}...") + with open(file_path, 'r') as f: + data = json.load(f) + + df = pd.DataFrame(data) + df = self.parser.normalize_columns(df) + + print("Extracting table names from queries...") + df['tables'] = df['query_text'].apply(self.parser.extract_tables) + df = df[df['tables'].apply(len) > 0] + + print("Normalizing query patterns...") + df['query_pattern'] = df['query_text'].apply(normalize_query_pattern) + + self.query_logs = df + print(f"Processed {len(df):,} queries with table references") + return df + + def load_query_logs_from_csv_file(self, file_path: str): + """Load and normalize query logs from CSV or JSON file""" + print(f"Reading query logs from {file_path}...") + + # Read file based on extension + if file_path.lower().endswith('.json'): + df = pd.read_json(file_path) + else: + df = pd.read_csv(file_path) + + # Normalize column names based on platform + df = self.parser.normalize_columns(df) + + # Verify required columns exist + required_cols = ['query_text', 'user', 'start_time'] + missing_cols = [col for col in required_cols if col not in df.columns] + if missing_cols: + raise ValueError(f"CSV file missing required columns after normalization: {missing_cols}") + + print("Extracting table names from queries...") + df['tables'] = df['query_text'].apply(self.parser.extract_tables) + + # Filter out queries with no table references + df = df[df['tables'].apply(len) > 0] + + print("Normalizing query patterns...") + df['query_pattern'] = df['query_text'].apply(normalize_query_pattern) + + self.query_logs = df + print(f"Processed {len(df):,} queries with table references") + return df + + def _calculate_table_query_counts(self) -> Counter: + """Calculate query count per table""" + table_query_count = Counter() + for tables in self.query_logs['tables']: + table_query_count.update(tables) + return table_query_count + + def _calculate_table_users(self) -> dict: + """Calculate user diversity per table""" + table_users = defaultdict(set) + for _, row in self.query_logs.iterrows(): + for table in row['tables']: + table_users[table].add(row['user']) + return table_users + + def _calculate_table_cooccurrence(self) -> dict: + """Calculate table co-occurrence""" + table_cooccurrence = defaultdict(Counter) + for tables in self.query_logs['tables']: + for table1 in tables: + for table2 in tables: + if table1 != table2: + table_cooccurrence[table1][table2] += 1 + return table_cooccurrence + + def _calculate_table_timestamps(self) -> dict: + """Calculate temporal metrics per table""" + table_timestamps = defaultdict(list) + for _, row in self.query_logs.iterrows(): + for table in row['tables']: + table_timestamps[table].append(row['start_time']) + return table_timestamps + + def _calculate_recency_score(self, timestamps: list, reference_time) -> tuple: + """Calculate recency score and days since last query""" + days_since_last_query = max(0, (reference_time - max(timestamps)).days) + recency_score = 1.0 / (1.0 + days_since_last_query) + return recency_score, days_since_last_query + + def _calculate_consistency_score(self, timestamps: list) -> float: + """Calculate consistency score based on query intervals""" + if len(timestamps) <= 1: + return 0.0 + + sorted_times = sorted(timestamps) + intervals = [(sorted_times[i+1] - sorted_times[i]).days + for i in range(len(sorted_times)-1)] + total_interval = sum(intervals) + + if len(intervals) == 0 or total_interval == 0: + return 0.5 + + mean_interval = total_interval / len(intervals) + std_interval = (sum((x - mean_interval)**2 for x in intervals) / len(intervals))**0.5 + cv = std_interval / mean_interval if mean_interval > 0 else 0 + return 1.0 / (1.0 + cv) + + def calculate_metrics(self) -> pd.DataFrame: + """Calculate metrics for each table""" + if self.query_logs is None: + raise ValueError(self.ERROR_QUERY_LOGS_NOT_LOADED) + + print("Calculating table metrics...") + + # Convert start_time to datetime if it's not already + self.query_logs['start_time'] = pd.to_datetime(self.query_logs['start_time']) + + # Calculate all base metrics + table_query_count = self._calculate_table_query_counts() + table_users = self._calculate_table_users() + table_cooccurrence = self._calculate_table_cooccurrence() + table_timestamps = self._calculate_table_timestamps() + + # Use the most recent query in the dataset as reference for recency calculations + reference_time = self.query_logs['start_time'].max() + + # Build metrics dataframe + metrics = [] + for table, query_count in table_query_count.items(): + user_count = len(table_users[table]) + related_tables = [t for t, c in table_cooccurrence[table].most_common(10)] + timestamps = table_timestamps[table] + + recency_score, days_since_last_query = self._calculate_recency_score(timestamps, reference_time) + consistency_score = self._calculate_consistency_score(timestamps) + + metrics.append({ + 'table': table, + 'query_count': query_count, + 'unique_users': user_count, + 'related_tables': related_tables, + 'related_table_count': len(related_tables), + 'recency_score': recency_score, + 'consistency_score': consistency_score, + 'days_since_last_query': days_since_last_query, + 'first_query_date': min(timestamps), + 'last_query_date': max(timestamps) + }) + + self.table_metrics = pd.DataFrame(metrics) + print(f"Calculated metrics for {len(self.table_metrics):,} tables") + return self.table_metrics + + def score_tables(self, + query_weight=0.375, + user_weight=0.375, + recency_weight=0.15, + consistency_weight=0.10): + """ + Score tables based on weighted metrics + + Args: + query_weight: Weight for query frequency (default 0.375) + user_weight: Weight for user diversity (default 0.375) + recency_weight: Weight for recent activity (default 0.15) + consistency_weight: Weight for consistent usage over time (default 0.10) + + Returns: + DataFrame with scored tables sorted by recommendation_score + + Note: + Relationship metrics are not included as standalone tables are packaged + in isolation without their related tables. + """ + if self.table_metrics is None: + raise ValueError("Metrics not calculated") + + # Validate weights sum to 1.0 + total_weight = query_weight + user_weight + recency_weight + consistency_weight + if abs(total_weight - 1.0) > 0.001: + raise ValueError(f"Weights must sum to 1.0, got {total_weight}") + + df = self.table_metrics.copy() + + # Normalize metrics to 0-1 scale, handling edge cases + max_queries = df['query_count'].max() + max_users = df['unique_users'].max() + + df['query_score'] = df['query_count'] / max_queries if max_queries > 0 else 0 + df['user_score'] = df['unique_users'] / max_users if max_users > 0 else 0 + + # Temporal scores are already 0-1, but ensure no NaN values + df['recency_score'] = df['recency_score'].fillna(0) + df['consistency_score'] = df['consistency_score'].fillna(0) + + # Calculate weighted score + df['recommendation_score'] = ( + df['query_score'] * query_weight + + df['user_score'] * user_weight + + df['recency_score'] * recency_weight + + df['consistency_score'] * consistency_weight + ) * 100 + + return df.sort_values('recommendation_score', ascending=False) + def get_top_query_patterns(self, tables: List[str], top_n: int = 5) -> List[Dict]: + """ + Get the most frequent query patterns for a set of tables. + + Args: + tables: List of table names to analyze + top_n: Number of top patterns to return (default 5) + + Returns: + List of dictionaries with pattern info: + - pattern: The normalized query pattern + - count: Number of times this pattern appears + - example: An actual query text example + - tables_used: Tables from the group that appear in this pattern + """ + if self.query_logs is None: + raise ValueError(self.ERROR_QUERY_LOGS_NOT_LOADED) + + if 'query_pattern' not in self.query_logs.columns: + raise ValueError("Query patterns not computed. Reload query logs.") + + # Filter queries that reference any of the specified tables + relevant_queries = self.query_logs[ + self.query_logs['tables'].apply( + lambda query_tables: any(t in tables for t in query_tables) + ) + ].copy() + + if len(relevant_queries) == 0: + return [] + + # Count pattern frequencies + pattern_counts = Counter(relevant_queries['query_pattern']) + + # Get top N patterns + top_patterns = [] + for pattern, count in pattern_counts.most_common(top_n): + # Find an example query for this pattern + example_row = relevant_queries[relevant_queries['query_pattern'] == pattern].iloc[0] + example_query = example_row['query_text'] + + # Determine which tables from the group are used in this pattern + tables_in_pattern = [t for t in tables if t in example_row['tables']] + + top_patterns.append({ + 'pattern': pattern, + 'count': count, + 'example': example_query, + 'tables_used': tables_in_pattern + }) + + return top_patterns + + + def _count_group_queries(self, table_group: tuple) -> int: + """Count queries involving any table in the group""" + if self.query_logs is None: + return 1 # Avoid division by zero + + count = sum(1 for tables_in_query in self.query_logs['tables'] + if any(t in tables_in_query for t in table_group)) + return max(count, 1) # Ensure at least 1 to avoid division by zero + + def _get_pairwise_frequencies(self, table_group: tuple, table_pair_counts: dict, + group_query_count: int) -> tuple: + """Get pairwise join frequencies and percentages for a table group""" + frequencies = [] + percentages = [] + + for i, table1 in enumerate(table_group): + for table2 in table_group[i+1:]: + pair = tuple(sorted([table1, table2])) + freq = table_pair_counts.get(pair, 0) + frequencies.append(freq) + percentages.append(freq / group_query_count) + + return frequencies, percentages + + def _calculate_group_cohesion(self, table_group: tuple, table_pair_counts: dict) -> dict: + """ + Calculate cohesion metrics for a table group + + Args: + table_group: Tuple of table names in the group + table_pair_counts: Dictionary mapping (table1, table2) -> count + + Returns: + Dictionary with cohesion metrics including: + - avg_join_frequency: Average absolute co-occurrence count + - avg_join_percentage: Average co-occurrence as percentage of group queries (0-1) + - min_join_frequency: Minimum co-occurrence count + - max_join_frequency: Maximum co-occurrence count + - total_pairs: Number of table pairs in the group + """ + if len(table_group) < 2: + return { + 'avg_join_frequency': 0, + 'avg_join_percentage': 0, + 'min_join_frequency': 0, + 'max_join_frequency': 0, + 'total_pairs': 0 + } + + group_query_count = self._count_group_queries(table_group) + frequencies, percentages = self._get_pairwise_frequencies( + table_group, table_pair_counts, group_query_count + ) + + return { + 'avg_join_frequency': sum(frequencies) / len(frequencies) if frequencies else 0, + 'avg_join_percentage': sum(percentages) / len(percentages) if percentages else 0, + 'min_join_frequency': min(frequencies) if frequencies else 0, + 'max_join_frequency': max(frequencies) if frequencies else 0, + 'total_pairs': len(frequencies) + } + + def _count_table_occurrences(self) -> Counter: + """Count how many times each table appears in queries""" + table_query_counts = Counter() + for tables in self.query_logs['tables']: + table_query_counts.update(tables) + return table_query_counts + + def _count_table_pairs(self) -> Counter: + """Count pairwise co-occurrences of tables""" + table_pair_counts = Counter() + for tables in self.query_logs['tables']: + if len(tables) >= 2: + for i, table1 in enumerate(tables): + for table2 in tables[i+1:]: + pair = tuple(sorted([table1, table2])) + table_pair_counts[pair] += 1 + return table_pair_counts + + def _build_strong_connections(self, table_pair_counts: Counter, table_query_counts: Counter, + min_frequency_threshold: float) -> dict: + """Build adjacency list of strong table connections""" + strong_connections = defaultdict(set) + + for (table1, table2), count in table_pair_counts.items(): + freq1 = count / table_query_counts[table1] if table_query_counts[table1] > 0 else 0 + freq2 = count / table_query_counts[table2] if table_query_counts[table2] > 0 else 0 + min_freq = min(freq1, freq2) + + if min_freq >= min_frequency_threshold: + strong_connections[table1].add(table2) + strong_connections[table2].add(table1) + + return strong_connections + + def _find_best_candidate(self, candidates: set, cluster: set, strong_connections: dict) -> tuple: + """Find the best candidate to add to a cluster""" + best_candidate = None + best_connection_count = 0 + + for candidate in candidates: + connection_count = len(cluster & strong_connections[candidate]) + if connection_count > best_connection_count: + best_candidate = candidate + best_connection_count = connection_count + + return best_candidate, best_connection_count + + def _grow_cluster(self, core_table: str, strong_connections: dict, visited: set, + max_group_size: int) -> set: + """Grow a cluster starting from a core table""" + cluster = {core_table} + candidates = strong_connections[core_table] - visited + + while candidates and len(cluster) < max_group_size: + best_candidate, best_connection_count = self._find_best_candidate( + candidates, cluster, strong_connections + ) + + if best_candidate and best_connection_count > 0: + cluster.add(best_candidate) + candidates.remove(best_candidate) + candidates.update(strong_connections[best_candidate] - visited - cluster) + else: + break + + return cluster + + def _create_cluster_dict(self, cluster_tuple: tuple, table_pair_counts: Counter) -> dict: + """Create a cluster dictionary with cohesion metrics""" + cohesion = self._calculate_group_cohesion(cluster_tuple, table_pair_counts) + + return { + 'tables': list(cluster_tuple), + 'size': len(cluster_tuple), + 'avg_join_frequency': cohesion['avg_join_frequency'], + 'avg_join_percentage': cohesion['avg_join_percentage'], + 'min_join_frequency': cohesion['min_join_frequency'], + 'max_join_frequency': cohesion['max_join_frequency'] + } + + def _build_frequency_clusters(self, min_frequency_threshold=0.10, min_group_size=2, max_group_size=10): + """ + Build table clusters using frequency-based threshold approach + + Args: + min_frequency_threshold: Minimum join frequency as percentage (0.0-1.0). + Tables must join together in at least this % of queries + where either table appears. + min_group_size: Minimum number of tables in a group + max_group_size: Maximum number of tables in a group + + Returns: + List of dictionaries with cluster information + """ + if self.query_logs is None: + raise ValueError(self.ERROR_QUERY_LOGS_NOT_LOADED) + + print(f"Building frequency-based clusters (threshold: {min_frequency_threshold*100:.1f}%)...") + + table_query_counts = self._count_table_occurrences() + table_pair_counts = self._count_table_pairs() + strong_connections = self._build_strong_connections( + table_pair_counts, table_query_counts, min_frequency_threshold + ) + + visited = set() + clusters = [] + + sorted_tables = sorted(strong_connections.keys(), + key=lambda t: len(strong_connections[t]), + reverse=True) + + for core_table in sorted_tables: + if core_table in visited: + continue + + cluster = self._grow_cluster(core_table, strong_connections, visited, max_group_size) + + if len(cluster) >= min_group_size: + visited.update(cluster) + cluster_tuple = tuple(sorted(cluster)) + clusters.append(self._create_cluster_dict(cluster_tuple, table_pair_counts)) + + clusters.sort(key=lambda x: x['avg_join_frequency'], reverse=True) + + print(f"Built {len(clusters)} frequency-based clusters") + return clusters + + def _calculate_cluster_query_counts(self, clusters: list) -> list: + """Calculate query counts for all clusters""" + cluster_query_counts = [] + for cluster in clusters: + tables = cluster['tables'] + queries_with_group = sum( + 1 for tables_in_query in self.query_logs['tables'] + if any(t in tables_in_query for t in tables) + ) + cluster_query_counts.append(queries_with_group) + return cluster_query_counts + + def _get_cluster_users(self, tables: list) -> set: + """Get unique users querying a cluster""" + users_querying_group = set() + for _, row in self.query_logs.iterrows(): + if any(t in row['tables'] for t in tables): + users_querying_group.add(row['user']) + return users_querying_group + + def _get_temporal_scores(self, tables: list) -> tuple: + """Extract temporal metrics for tables in a cluster""" + recency_scores = [] + consistency_scores = [] + + for table in tables: + table_data = self.table_metrics[self.table_metrics['table'] == table] + if not table_data.empty: + recency_scores.append(table_data['recency_score'].values[0]) + consistency_scores.append(table_data['consistency_score'].values[0]) + + avg_recency = sum(recency_scores) / len(recency_scores) if recency_scores else 0.0 + avg_consistency = sum(consistency_scores) / len(consistency_scores) if consistency_scores else 0.0 + + return avg_recency, avg_consistency + + def _calculate_normalized_scores(self, cluster: dict, queries_with_group: int, + users_count: int, max_queries: int, max_group_size: int) -> dict: + """Calculate normalized scores for a cluster""" + cohesion_score = cluster['avg_join_percentage'] + usage_score = queries_with_group / max_queries if max_queries > 0 else 0 + total_users = self.query_logs['user'].nunique() + user_score = users_count / total_users if total_users > 0 else 0 + size_score = cluster['size'] / max_group_size if max_group_size > 0 else 0 + + return { + 'cohesion_score': cohesion_score, + 'usage_score': usage_score, + 'user_score': user_score, + 'size_score': size_score + } + + def _calculate_group_score(self, scores: dict, avg_recency: float, avg_consistency: float) -> float: + """Calculate weighted group score (0-100 scale)""" + return ( + scores['cohesion_score'] * 0.30 + + scores['usage_score'] * 0.20 + + scores['user_score'] * 0.15 + + avg_recency * 0.20 + + avg_consistency * 0.10 + + scores['size_score'] * 0.05 + ) * 100 + + def _score_table_groups(self, clusters: list) -> list: + """ + Calculate group-specific scores for table clusters + + Args: + clusters: List of cluster dictionaries from _build_frequency_clusters + + Returns: + List of clusters with added 'group_score' field + """ + if self.query_logs is None or self.table_metrics is None: + raise ValueError("Query logs and table metrics must be loaded") + + max_group_size = max((c['size'] for c in clusters), default=1) + cluster_query_counts = self._calculate_cluster_query_counts(clusters) + max_queries = max(cluster_query_counts) if cluster_query_counts else 1 + + scored_clusters = [] + for i, cluster in enumerate(clusters): + tables = cluster['tables'] + queries_with_group = cluster_query_counts[i] + + users_querying_group = self._get_cluster_users(tables) + avg_recency, avg_consistency = self._get_temporal_scores(tables) + + scores = self._calculate_normalized_scores( + cluster, queries_with_group, len(users_querying_group), + max_queries, max_group_size + ) + + group_score = self._calculate_group_score(scores, avg_recency, avg_consistency) + + scored_clusters.append({ + **cluster, + 'group_score': group_score, + 'total_queries': queries_with_group, + 'unique_users': len(users_querying_group), + 'avg_recency_score': avg_recency, + 'avg_consistency_score': avg_consistency, + 'cohesion_score': scores['cohesion_score'] * 100, + 'usage_score': scores['usage_score'] * 100, + 'user_score': scores['user_score'] * 100, + 'size_score': scores['size_score'] * 100 + }) + + scored_clusters.sort(key=lambda x: x['group_score'], reverse=True) + return scored_clusters + + def identify_table_groups(self, min_cooccurrence=None): + """ + Identify groups of frequently co-occurring tables + + Args: + min_cooccurrence: Minimum number of times tables must appear together. + If None, automatically calculated as 0.01% of total queries + (minimum 2, maximum 100) + + Returns: + List of tuples: (table_group, count) sorted by count descending + """ + if self.query_logs is None: + raise ValueError(self.ERROR_QUERY_LOGS_NOT_LOADED) + + print("Identifying table groups...") + + # Calculate dynamic threshold if not provided + if min_cooccurrence is None: + total_queries = len(self.query_logs) + # Use 0.01% of queries as threshold (1 in 10,000) + min_cooccurrence = max(2, min(100, int(total_queries * 0.0001))) + print(f"Auto-calculated co-occurrence threshold: {min_cooccurrence} (based on {total_queries:,} queries)") + + table_groups = Counter() + for tables in self.query_logs['tables']: + if len(tables) >= 2: + # Sort tables to ensure consistent grouping + table_groups[tuple(sorted(tables))] += 1 + + # Filter by minimum co-occurrence + filtered_groups = [(group, count) for group, count in table_groups.items() + if count >= min_cooccurrence] + + # Sort by count descending + filtered_groups.sort(key=lambda x: x[1], reverse=True) + + print(f"Found {len(filtered_groups)} table groups with {min_cooccurrence}+ co-occurrences") + return filtered_groups + + def recommend_data_products(self, + num_recommendations=10, + min_score=None, + min_frequency_threshold=0.10, + min_group_size=2, + max_group_size=10): + """ + Generate final data product recommendations using frequency-based clustering + + Args: + num_recommendations: Maximum number of top recommendations to return + min_score: Minimum recommendation score threshold (0-100). + Tables below this score will be excluded from all recommendations. + Also used to filter standalone (unclustered) tables. + If None, no score filtering is applied. + min_frequency_threshold: Minimum join frequency for clustering (0.0-1.0) + min_group_size: Minimum tables in a cluster + max_group_size: Maximum tables in a cluster + + Returns: + Dictionary with 'individual_tables', 'table_groups', and optionally 'standalone_tables' + """ + scored_tables = self.score_tables() + + # Store metadata before filtering + total_tables = len(scored_tables) + highest_score = scored_tables['recommendation_score'].max() if len(scored_tables) > 0 else 0 + + # Apply score threshold if specified + if min_score is not None: + scored_tables = scored_tables[scored_tables['recommendation_score'] >= min_score] + print(f"Applied score threshold: {min_score} (kept {len(scored_tables)} tables)") + + # Limit to top N + top_tables = scored_tables.head(num_recommendations) + + recommendations = { + 'individual_tables': top_tables.to_dict('records'), + 'metadata': { + 'total_tables': total_tables, + 'recommended_tables': len(top_tables), + 'highest_score': highest_score, + 'min_score_threshold': min_score, + 'clustering_enabled': True # Always enabled now + } + } + + # Build frequency-based clusters + clusters = self._build_frequency_clusters( + min_frequency_threshold=min_frequency_threshold, + min_group_size=min_group_size, + max_group_size=max_group_size + ) + scored_clusters = self._score_table_groups(clusters) + + # Track which tables are in clusters + tables_in_clusters = set() + for cluster in scored_clusters: + tables_in_clusters.update(cluster['tables']) + + # Add individual table details to each group + for cluster in scored_clusters[:num_recommendations]: + cluster['table_details'] = [] + for table_name in cluster['tables']: + table_info = scored_tables[scored_tables['table'] == table_name] + if not table_info.empty: + cluster['table_details'].append(table_info.iloc[0].to_dict()) + + recommendations['table_groups'] = scored_clusters[:num_recommendations] + + # Identify high-value standalone tables (not in any cluster) + # Use the same min_score threshold if provided + standalone_threshold = min_score if min_score is not None else 0 + standalone_tables = scored_tables[ + (~scored_tables['table'].isin(tables_in_clusters)) & + (scored_tables['recommendation_score'] >= standalone_threshold) + ] + + if len(standalone_tables) > 0: + recommendations['standalone_tables'] = standalone_tables.head(num_recommendations).to_dict('records') + recommendations['metadata']['standalone_tables_count'] = len(standalone_tables) + + return recommendations + + def _write_markdown_header(self, f): + """Write markdown file header""" + f.write("# Data Product Recommendations\n\n") + f.write(f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n\n") + + def _write_summary_stats(self, f): + """Write summary statistics section""" + f.write("## Summary Statistics\n\n") + if self.query_logs is not None: + f.write(f"- Total queries analyzed: {len(self.query_logs):,}\n") + f.write(f"- Unique users: {self.query_logs['user'].nunique()}\n") + if self.table_metrics is not None: + f.write(f"- Unique tables: {len(self.table_metrics)}\n") + + def _calculate_clustering_stats(self, recommendations: dict, metadata: dict) -> dict: + """Calculate statistics for clustering mode""" + num_groups = len(recommendations['table_groups']) + total_tables = metadata['total_tables'] + + tables_in_groups = set() + for group in recommendations['table_groups']: + tables_in_groups.update(group['tables']) + + tables_standalone = set() + num_standalone = 0 + if 'standalone_tables' in recommendations: + num_standalone = len(recommendations['standalone_tables']) + tables_standalone = {t['table'] for t in recommendations['standalone_tables']} + + total_recommended = len(tables_in_groups) + len(tables_standalone) + + return { + 'num_groups': num_groups, + 'num_standalone': num_standalone, + 'tables_in_groups_count': len(tables_in_groups), + 'tables_standalone_count': len(tables_standalone), + 'total_recommended': total_recommended, + 'total_tables': total_tables + } + + def _write_clustering_stats(self, f, stats: dict, metadata: dict): + """Write clustering statistics""" + total_coverage_pct = (stats['total_recommended'] / stats['total_tables'] * 100) if stats['total_tables'] > 0 else 0 + groups_coverage_pct = (stats['tables_in_groups_count'] / stats['total_tables'] * 100) if stats['total_tables'] > 0 else 0 + standalone_coverage_pct = (stats['tables_standalone_count'] / stats['total_tables'] * 100) if stats['total_tables'] > 0 else 0 + + f.write(f"- Total data products recommended: {stats['num_groups'] + stats['num_standalone']}\n") + f.write(f" - Multi-table products: {stats['num_groups']}\n") + f.write(f" - Single-table products: {stats['num_standalone']}\n") + f.write(f"- Recommendation coverage: {stats['total_recommended']} of {stats['total_tables']} ({total_coverage_pct:.1f}%)\n") + f.write(f" - In groups: {stats['tables_in_groups_count']} ({groups_coverage_pct:.1f}%)\n") + f.write(f" - Standalone: {stats['tables_standalone_count']} ({standalone_coverage_pct:.1f}%)\n") + + if metadata.get('min_score_threshold') is not None: + f.write(f"- Minimum score threshold: {metadata['min_score_threshold']:.1f}\n") + + def _write_non_clustering_stats(self, f, metadata: dict): + """Write non-clustering statistics""" + total = metadata['total_tables'] + recommended = metadata['recommended_tables'] + percentage = (recommended / total * 100) if total > 0 else 0 + f.write(f"- Total recommended tables: {recommended} ({percentage:.1f}% of all tables)\n") + f.write(f"- Highest table score: {metadata['highest_score']:.1f}\n") + if metadata.get('min_score_threshold') is not None: + f.write(f"- Minimum score threshold: {metadata['min_score_threshold']:.1f}\n") + + def _write_recommendation_stats(self, f, recommendations: dict): + """Write recommendation statistics section""" + if 'metadata' not in recommendations: + return + + metadata = recommendations['metadata'] + + if metadata.get('clustering_enabled', False) and 'table_groups' in recommendations: + stats = self._calculate_clustering_stats(recommendations, metadata) + self._write_clustering_stats(f, stats, metadata) + else: + self._write_non_clustering_stats(f, metadata) + + f.write("\n") + + def _write_individual_tables_header(self, f): + """Write individual tables section header""" + f.write("## Top Recommended Tables\n\n") + f.write("| Rank | Table | Score | Queries | Users | Recency | Consistency | Last Query | Related Tables |\n") + f.write("|------|-------|-------|---------|-------|---------|-------------|------------|----------------|\n") + + def _format_table_row(self, i: int, table: dict) -> str: + """Format a single table row for markdown""" + related = ', '.join(table['related_tables'][:3]) if table['related_tables'] else 'None' + recency_pct = f"{table.get('recency_score', 0)*100:.0f}%" + consistency_pct = f"{table.get('consistency_score', 0)*100:.0f}%" + days_ago = table.get('days_since_last_query', 'N/A') + days_ago_str = f"{days_ago}d ago" if isinstance(days_ago, (int, float)) else days_ago + + return (f"| {i} | {table['table']} | {table['recommendation_score']:.1f} | " + f"{table['query_count']} | {table['unique_users']} | " + f"{recency_pct} | {consistency_pct} | {days_ago_str} | {related} |\n") + + def _write_table_metric_definitions(self, f): + """Write table metric definitions""" + f.write("\n### Metric Definitions\n\n") + f.write("- **Score**: Overall recommendation score (0-100)\n") + f.write("- **Queries**: Total number of queries referencing this table\n") + f.write("- **Users**: Number of unique users querying this table\n") + f.write("- **Recency**: How recently the table was queried (100% = queried today)\n") + f.write("- **Consistency**: How consistently the table is queried over time (100% = perfectly regular)\n") + f.write("- **Last Query**: Days since the most recent query\n") + f.write("- **Related Tables**: Tables frequently queried together with this one\n") + f.write("\n") + + def _write_individual_tables_section(self, f, recommendations: dict): + """Write individual tables section""" + self._write_individual_tables_header(f) + + for i, table in enumerate(recommendations['individual_tables'], 1): + f.write(self._format_table_row(i, table)) + + self._write_table_metric_definitions(f) + + def _write_data_product_metric_definitions(self, f): + """Write data product metric definitions""" + f.write("\n### Data Product Metric Definitions\n\n") + f.write("- **Score**: Overall recommendation score for the data product (0-100)\n") + f.write(" - Star Rating Scale:\n") + f.write(f" - {FIVE_STARS} {EXCELLENT_CANDIDATE} (80-100): Strong data product candidate, implement immediately\n") + f.write(f" - {FOUR_STARS} {GOOD_CANDIDATE} (60-79): Solid candidate, implement as medium priority\n") + f.write(f" - {THREE_STARS} {FAIR_CANDIDATE} (40-59): Consider splitting or implement later\n") + f.write(f" - {TWO_STARS} {WEAK_CANDIDATE} (20-39): Reconsider grouping\n") + f.write(f" - {ONE_STAR} {POOR_CANDIDATE} (0-19): Do not implement as data product\n") + f.write("- **General Metrics:**\n") + f.write(" - **Total Queries**: Total number of queries that touched the table(s)\n") + f.write(" - **Unique Users**: Number of distinct users who queried the table(s)\n") + f.write("- **Table Metrics:**\n") + f.write(" - **Recency Score**: How recently the table was queried (relative to dataset's most recent query)\n") + f.write(" - **Consistency Score**: What % of days (in the log timespan) the table was queried\n") + f.write("- **Table Group Metrics:**\n") + f.write(" - **Group Cohesion Score**: What % of queries join multiple tables from the group together\n") + f.write(" - **Avg Join Frequency**: Average number of times per day tables in the group are joined together\n") + + def _get_star_rating(self, score: float) -> tuple: + """Get star rating and label for a score""" + if score >= 80: + return FIVE_STARS, EXCELLENT_CANDIDATE + elif score >= 60: + return FOUR_STARS, GOOD_CANDIDATE + elif score >= 40: + return THREE_STARS, FAIR_CANDIDATE + elif score >= 20: + return TWO_STARS, WEAK_CANDIDATE + else: + return ONE_STAR, POOR_CANDIDATE + + def _merge_and_sort_products(self, recommendations: dict) -> list: + """Merge groups and standalone tables, sorted by score""" + all_products = [] + + if 'table_groups' in recommendations: + for group in recommendations['table_groups']: + all_products.append({ + 'type': 'group', + 'score': group['group_score'], + 'data': group + }) + + if 'standalone_tables' in recommendations: + for table in recommendations['standalone_tables']: + all_products.append({ + 'type': 'standalone', + 'score': table['recommendation_score'], + 'data': table + }) + + all_products.sort(key=lambda x: x['score'], reverse=True) + return all_products + + def _write_group_metrics(self, f, group: dict): + """Write group-level metrics""" + f.write(QUERY_LOG_METRICS_HEADER) + f.write(f"- Total Queries: {group['total_queries']:,}\n") + f.write(f"- Unique Users: {group['unique_users']}\n") + f.write(f"- Tables in Group: {group['size']}\n") + f.write("- Group Metrics:\n") + f.write(f" - Cohesion Score: {group['cohesion_score']:.1f}%\n") + f.write(f" - Average Join Frequency: {group['avg_join_frequency']:.1f}\n") + f.write("\n") + + def _write_group_tables(self, f, group: dict): + """Write tables in group section""" + if 'table_details' in group and group['table_details']: + f.write(TABLES_IN_GROUP_HEADER) + f.write(TABLE_HEADER_ROW) + f.write(TABLE_SEPARATOR_ROW) + + for table in group['table_details']: + recency_pct = f"{table.get('recency_score', 0)*100:.0f}%" + consistency_pct = f"{table.get('consistency_score', 0)*100:.0f}%" + f.write(f"| {table['table']} | {table['recommendation_score']:.1f} | " + f"{table['query_count']} | {table['unique_users']} | " + f"{recency_pct} | {consistency_pct} |\n") + else: + f.write(TABLES_IN_GROUP_SIMPLE) + for table in group['tables']: + f.write(f"- {table}\n") + f.write("\n") + + def _truncate_text(self, text: str, max_length: int = 200) -> str: + """Truncate text if too long""" + if len(text) > max_length: + return text[:max_length] + "..." + return text + + def _write_query_pattern(self, f, idx: int, pattern_info: dict): + """Write a single query pattern""" + f.write(f"{idx}. **Pattern** (used {pattern_info['count']} times):\n") + f.write(SQL_CODE_BLOCK_START) + f.write(f" {self._truncate_text(pattern_info['pattern'])}\n") + f.write(SQL_CODE_BLOCK_END) + + if 'tables_used' in pattern_info: + f.write(f" - Tables used: {', '.join(pattern_info['tables_used'])}\n") + + f.write(SQL_EXAMPLE_LABEL) + f.write(SQL_EXAMPLE_CODE_START) + f.write(f" {self._truncate_text(pattern_info['example'])}\n") + f.write(SQL_EXAMPLE_CODE_END) + f.write("\n") + + def _write_query_patterns(self, f, tables: list, product_id: int): + """Write query patterns section""" + try: + top_patterns = self.get_top_query_patterns(tables, top_n=5) + if top_patterns: + f.write(DETAILS_START) + f.write(DETAILS_SUMMARY) + for idx, pattern_info in enumerate(top_patterns, 1): + self._write_query_pattern(f, idx, pattern_info) + f.write(DETAILS_END) + except Exception as e: + print(f"Warning: Could not generate query patterns for product {product_id}: {e}") + f.write("\n") + + def _write_group_product(self, f, i: int, group: dict): + """Write a group data product""" + score = group['group_score'] + stars, rating = self._get_star_rating(score) + + f.write(f"### Data Product {i} - Score: {score:.1f} {stars} ({rating})\n\n") + self._write_group_metrics(f, group) + self._write_group_tables(f, group) + self._write_query_patterns(f, group['tables'], i) + + def _write_standalone_product(self, f, i: int, table: dict): + """Write a standalone table data product""" + score = table['recommendation_score'] + stars, rating = self._get_star_rating(score) + + f.write(f"### Data Product {i} - Score: {score:.1f} {stars} ({rating})\n\n") + + f.write(QUERY_LOG_METRICS_HEADER) + f.write(f"- Total Queries: {table['query_count']:,}\n") + f.write(f"- Unique Users: {table['unique_users']}\n") + f.write("- Tables in Group: 1\n") + f.write("\n") + + recency_pct = f"{table.get('recency_score', 0)*100:.0f}%" + consistency_pct = f"{table.get('consistency_score', 0)*100:.0f}%" + + f.write(TABLES_IN_GROUP_HEADER) + f.write(TABLE_HEADER_ROW) + f.write(TABLE_SEPARATOR_ROW) + f.write(f"| {table['table']} | {table['recommendation_score']:.1f} | " + f"{table['query_count']} | {table['unique_users']} | " + f"{recency_pct} | {consistency_pct} |\n") + f.write("\n") + + self._write_query_patterns(f, [table['table']], i) + + def _write_clustered_products(self, f, recommendations: dict): + """Write clustered data products section""" + f.write("\n## Recommended Data Products\n\n") + f.write("*Sorted by recommendation score (descending). Groups identified using frequency-based clustering.*\n\n") + f.write("*Note: Group scores and individual table scores use different weighting formulas and are not directly comparable, but both indicate relative value within their category.*\n\n") + + all_products = self._merge_and_sort_products(recommendations) + + for i, product in enumerate(all_products, 1): + if product['type'] == 'group': + self._write_group_product(f, i, product['data']) + else: + self._write_standalone_product(f, i, product['data']) + + def _write_original_groups(self, f, recommendations: dict): + """Write original co-occurrence format groups""" + f.write("\n## Recommended Table Groups\n\n") + f.write("Tables that are frequently queried together:\n\n") + + for i, group in enumerate(recommendations['table_groups'], 1): + f.write(f"### Group {i} (Co-occurrence: {group['co_occurrence_count']})\n\n") + for table in group['tables']: + f.write(f"- {table}\n") + f.write("\n") + + def export_recommendations_markdown(self, recommendations: dict, output_file: str): + """Export recommendations to markdown file""" + with open(output_file, 'w', encoding='utf-8') as f: + self._write_markdown_header(f) + self._write_summary_stats(f) + self._write_recommendation_stats(f, recommendations) + + clustering_enabled = recommendations.get('metadata', {}).get('clustering_enabled', False) + + if not clustering_enabled: + self._write_individual_tables_section(f, recommendations) + + if 'table_groups' in recommendations: + self._write_data_product_metric_definitions(f) + + if clustering_enabled: + self._write_clustered_products(f, recommendations) + else: + self._write_original_groups(f, recommendations) + + + def _build_json_metadata(self, recommendations: dict) -> dict: + """Build metadata section for JSON export""" + from datetime import datetime + return { + "generated_at": datetime.now().isoformat(), + "total_queries_analyzed": len(self.query_logs) if self.query_logs is not None else 0, + "unique_users": int(self.query_logs['user'].nunique()) if self.query_logs is not None else 0, + "unique_tables": len(self.table_metrics) if self.table_metrics is not None else 0, + "platform": self.parser.__class__.__name__.replace('QueryParser', '').lower(), + "clustering_enabled": recommendations.get('metadata', {}).get('clustering_enabled', False), + "min_score_threshold": recommendations.get('metadata', {}).get('min_score_threshold') + } + + def _add_table_details_to_product(self, product: dict, group: dict): + """Add table details to a group product""" + if 'table_details' in group and group['table_details']: + for table_info in group['table_details']: + product["table_details"].append({ + "table": table_info['table'], + "score": round(table_info['recommendation_score'], 1), + "queries": table_info['query_count'], + "users": table_info['unique_users'], + "recency_score": round(table_info.get('recency_score', 0), 2), + "consistency_score": round(table_info.get('consistency_score', 0), 2) + }) + + def _add_query_patterns_to_product(self, product: dict, tables: list, include_tables_used: bool = True): + """Add query patterns to a product""" + try: + patterns = self.get_top_query_patterns(tables, top_n=5) + for pattern_info in patterns: + pattern_dict = { + "pattern": pattern_info['pattern'], + "count": pattern_info['count'], + "example": pattern_info['example'] + } + if include_tables_used and 'tables_used' in pattern_info: + pattern_dict["tables_used"] = pattern_info['tables_used'] + product["query_patterns"].append(pattern_dict) + except Exception: + pass # Skip patterns if error + + def _create_group_product_json(self, idx: int, group: dict) -> dict: + """Create JSON product for a group""" + product = { + "id": f"dp_{idx:03d}", + "type": "group", + "score": round(group['group_score'], 1), + "rating": self._get_rating_label(group['group_score']), + "tables": group['tables'], + "metrics": { + "total_queries": group['total_queries'], + "unique_users": group['unique_users'], + "table_count": group['size'], + "cohesion_score": round(group['cohesion_score'], 1), + "avg_join_frequency": round(group['avg_join_frequency'], 1), + "recency_score": round(group.get('avg_recency_score', 0), 2), + "consistency_score": round(group.get('avg_consistency_score', 0), 2) + }, + "table_details": [], + "query_patterns": [] + } + + self._add_table_details_to_product(product, group) + self._add_query_patterns_to_product(product, group['tables'], include_tables_used=True) + + return product + + def _create_standalone_product_json(self, idx: int, table: dict) -> dict: + """Create JSON product for a standalone table""" + product = { + "id": f"dp_{idx:03d}", + "type": "standalone", + "score": round(table['recommendation_score'], 1), + "rating": self._get_rating_label(table['recommendation_score']), + "tables": [table['table']], + "metrics": { + "total_queries": table['query_count'], + "unique_users": table['unique_users'], + "table_count": 1, + "recency_score": round(table.get('recency_score', 0), 2), + "consistency_score": round(table.get('consistency_score', 0), 2) + }, + "query_patterns": [] + } + + self._add_query_patterns_to_product(product, [table['table']], include_tables_used=False) + + return product + + def _process_clustered_recommendations(self, recommendations: dict) -> list: + """Process recommendations in clustering mode""" + all_products = [] + + if 'table_groups' in recommendations: + for idx, group in enumerate(recommendations['table_groups'], 1): + product = self._create_group_product_json(idx, group) + all_products.append(product) + + if 'standalone_tables' in recommendations: + start_idx = len(all_products) + 1 + for idx, table in enumerate(recommendations['standalone_tables'], start_idx): + product = self._create_standalone_product_json(idx, table) + all_products.append(product) + + all_products.sort(key=lambda x: x['score'], reverse=True) + return all_products + + def _process_non_clustered_recommendations(self, recommendations: dict) -> list: + """Process recommendations in non-clustering mode""" + products = [] + + if 'individual_tables' in recommendations: + for idx, table in enumerate(recommendations['individual_tables'], 1): + product = self._create_standalone_product_json(idx, table) + # Add tables_used for non-clustered mode + try: + patterns = self.get_top_query_patterns([table['table']], top_n=5) + product["query_patterns"] = [] + for pattern_info in patterns: + product["query_patterns"].append({ + "pattern": pattern_info['pattern'], + "count": pattern_info['count'], + "tables_used": pattern_info.get('tables_used', []), + "example": pattern_info['example'] + }) + except Exception: + pass + + products.append(product) + + return products + + def export_recommendations_json(self, recommendations: dict, output_file: str): + """Export recommendations to JSON file for agent consumption""" + import json + + output = { + "recommendations": [], + "metadata": self._build_json_metadata(recommendations) + } + + clustering_enabled = recommendations.get('metadata', {}).get('clustering_enabled', False) + + if clustering_enabled: + output["recommendations"] = self._process_clustered_recommendations(recommendations) + else: + output["recommendations"] = self._process_non_clustered_recommendations(recommendations) + + with open(output_file, 'w', encoding='utf-8') as f: + json.dump(output, f, indent=2, ensure_ascii=False) + + def _get_rating_label(self, score: float) -> str: + """Convert numeric score to rating label""" + if score >= 80: + return "excellent" + elif score >= 60: + return "good" + elif score >= 40: + return "fair" + elif score >= 20: + return "weak" + else: + return "poor" + +# Made with Bob diff --git a/src/wxdi/dph_services/README.md b/src/wxdi/dph_services/README.md new file mode 100644 index 0000000..3610050 --- /dev/null +++ b/src/wxdi/dph_services/README.md @@ -0,0 +1,458 @@ + + +# Data Product Hub Services (DPH Services) + +Python client library for IBM Data Product Hub API, providing programmatic access to data product management, container operations, contract terms, and asset visualization. + +## Overview + +The `dph_services` module provides a complete Python SDK for interacting with IBM Data Product Hub services. It enables developers to: + +- Initialize and manage data product containers +- Create, update, and publish data products +- Manage data product drafts and releases +- Handle contract terms and documents +- Create and manage data asset visualizations +- Manage domains and subdomains +- Work with contract templates + +## Installation + +The module is included in the data-intelligence-sdk package: + +```bash +pip install -e . +``` + +## Quick Start + +### Basic Setup + +```python +from wxdi.dph_services import DphV1 +from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + +# Initialize authenticator +authenticator = IAMAuthenticator('your-api-key') + +# Create service instance +dph_service = DphV1(authenticator=authenticator) +dph_service.set_service_url('https://your-dph-instance.com') +``` + +### Initialize a Container + +```python +# Initialize container with default settings +response = dph_service.initialize( + include=['delivery_methods', 'data_product_samples', 'domains_multi_industry'] +) + +print(f"Container initialized: {response.result}") +``` + +### Create a Data Product + +```python +# Create a new data product with a draft +data_product = dph_service.create_data_product( + drafts=[{ + 'version': '1.0.0', + 'name': 'Customer Analytics Data Product', + 'description': 'Comprehensive customer analytics dataset', + 'asset': { + 'id': 'asset-123', + 'container': {'id': 'container-456'} + }, + 'domain': { + 'id': 'domain-789', + 'name': 'Customer Analytics' + } + }] +) + +print(f"Data product created: {data_product.result['id']}") +``` + +### List Data Products + +```python +# List all data products +response = dph_service.list_data_products(limit=50) + +for product in response.result['data_products']: + print(f"- {product['name']} (v{product['version']})") +``` + +### Get a Specific Data Product + +```python +# Get data product by ID +data_product_id = 'your-data-product-id' +response = dph_service.get_data_product(data_product_id=data_product_id) + +print(f"Data Product: {response.result['name']}") +print(f"Description: {response.result['description']}") +``` + +## Core Features + +### 1. Container Management + +Initialize and manage data product containers: + +```python +# Initialize container +response = dph_service.initialize( + include=['delivery_methods', 'data_product_samples'] +) + +# Get initialization status +status = dph_service.get_initialize_status() +print(f"Status: {status.result['status']}") +``` + +### 2. Data Product Operations + +Complete lifecycle management: + +```python +# Create data product +product = dph_service.create_data_product(drafts=[...]) + +# Update data product +updated = dph_service.update_data_product( + data_product_id=product_id, + json_patch_instructions=[ + {'op': 'replace', 'path': '/description', 'value': 'Updated description'} + ] +) + +# Delete data product (if needed) +dph_service.delete_data_product(data_product_id=product_id) +``` + +### 3. Draft Management + +Work with data product drafts: + +```python +# Create a draft +draft = dph_service.create_data_product_draft( + data_product_id=product_id, + asset={'id': 'asset-123', 'container': {'id': 'container-456'}}, + version='1.1.0', + name='Updated Version' +) + +# List drafts +drafts = dph_service.list_data_product_drafts(data_product_id=product_id) + +# Get specific draft +draft_detail = dph_service.get_data_product_draft( + data_product_id=product_id, + draft_id=draft_id +) + +# Update draft +updated_draft = dph_service.update_data_product_draft( + data_product_id=product_id, + draft_id=draft_id, + json_patch_instructions=[...] +) + +# Publish draft +published = dph_service.publish_data_product_draft( + data_product_id=product_id, + draft_id=draft_id +) +``` + +### 4. Contract Terms Management + +Manage contract terms and documents: + +```python +# Create contract terms document +doc = dph_service.create_draft_contract_terms_document( + data_product_id=product_id, + draft_id=draft_id, + contract_terms_id=terms_id, + type='terms_and_conditions', + name='Terms and Conditions', + url='https://example.com/terms.pdf' +) + +# Get contract terms +terms = dph_service.get_data_product_draft_contract_terms( + data_product_id=product_id, + draft_id=draft_id +) + +# Update contract terms document +updated_doc = dph_service.update_draft_contract_terms_document( + data_product_id=product_id, + draft_id=draft_id, + contract_terms_id=terms_id, + document_id=doc_id, + json_patch_instructions=[...] +) + +# Delete contract terms document +dph_service.delete_draft_contract_terms_document( + data_product_id=product_id, + draft_id=draft_id, + contract_terms_id=terms_id, + document_id=doc_id +) +``` + +### 5. Release Management + +Manage data product releases: + +```python +# List releases +releases = dph_service.list_data_product_releases( + data_product_id=product_id +) + +# Get specific release +release = dph_service.get_data_product_release( + data_product_id=product_id, + release_id=release_id +) + +# Update release +updated_release = dph_service.update_data_product_release( + data_product_id=product_id, + release_id=release_id, + json_patch_instructions=[...] +) + +# Retire release +retired = dph_service.retire_data_product_release( + data_product_id=product_id, + release_id=release_id +) +``` + +### 6. Asset Visualization + +Create and manage data asset visualizations: + +```python +# Create visualization +visualization = dph_service.create_data_asset_visualization( + container={'id': 'container-123'}, + assets=[ + {'id': 'asset-1', 'container': {'id': 'container-123'}}, + {'id': 'asset-2', 'container': {'id': 'container-123'}} + ] +) + +# Reinitiate visualization +reinitiated = dph_service.reinitiate_data_asset_visualization( + container={'id': 'container-123'}, + assets=[...] +) +``` + +### 7. Domain Management + +Organize data products by domains: + +```python +# List domains +domains = dph_service.list_data_product_domains(limit=50) + +# Create domain +domain = dph_service.create_data_product_domain( + name='Customer Analytics', + description='Customer-related data products', + container={'id': 'container-123'} +) + +# Create subdomain +subdomain = dph_service.create_data_product_subdomain( + domain_id=domain_id, + name='Customer Segmentation', + description='Customer segmentation datasets' +) + +# Get domain +domain_detail = dph_service.get_domain(domain_id=domain_id) + +# Update domain +updated_domain = dph_service.update_data_product_domain( + domain_id=domain_id, + json_patch_instructions=[...] +) + +# Delete domain +dph_service.delete_domain(domain_id=domain_id) +``` + +### 8. Contract Templates + +Manage reusable contract templates: + +```python +# Create contract template +template = dph_service.create_contract_template( + name='Standard Terms Template', + description='Standard contract terms for data products', + contract_terms_documents=[...] +) + +# List templates +templates = dph_service.list_data_product_contract_template(limit=50) + +# Get template +template_detail = dph_service.get_contract_template( + contract_template_id=template_id +) + +# Update template +updated_template = dph_service.update_data_product_contract_template( + contract_template_id=template_id, + json_patch_instructions=[...] +) + +# Delete template +dph_service.delete_data_product_contract_template( + contract_template_id=template_id +) +``` + +## Advanced Usage + +### Pagination + +Handle large result sets with pagination: + +```python +# Using pager for data products +all_products = [] +pager = dph_service.list_data_products_with_pager(limit=50) + +for page in pager: + all_products.extend(page['data_products']) + +print(f"Total products: {len(all_products)}") +``` + +### Error Handling + +```python +from ibm_cloud_sdk_core import ApiException + +try: + response = dph_service.get_data_product(data_product_id='invalid-id') +except ApiException as e: + print(f"Error: {e.code} - {e.message}") +``` + +### Custom Headers + +```python +# Add custom headers to requests +response = dph_service.get_data_product( + data_product_id=product_id, + headers={'Custom-Header': 'value'} +) +``` + +## API Reference + +### Main Classes + +- **`DphV1`**: Main service class for Data Product Hub operations +- **`DataProduct`**: Data product model +- **`DataProductDraft`**: Draft model +- **`ContractTerms`**: Contract terms model +- **`Domain`**: Domain model + +### Key Methods + +#### Container Operations +- `initialize()` - Initialize container +- `get_initialize_status()` - Get initialization status +- `get_service_id_credentials()` - Get service credentials +- `manage_api_keys()` - Manage API keys + +#### Data Product Operations +- `create_data_product()` - Create new data product +- `list_data_products()` - List all data products +- `get_data_product()` - Get specific data product +- `update_data_product()` - Update data product +- `delete_data_product()` - Delete data product + +#### Draft Operations +- `create_data_product_draft()` - Create draft +- `list_data_product_drafts()` - List drafts +- `get_data_product_draft()` - Get draft details +- `update_data_product_draft()` - Update draft +- `delete_data_product_draft()` - Delete draft +- `publish_data_product_draft()` - Publish draft + +#### Release Operations +- `list_data_product_releases()` - List releases +- `get_data_product_release()` - Get release details +- `update_data_product_release()` - Update release +- `retire_data_product_release()` - Retire release + +## Examples + +See the `examples/test_dph_v1_examples.py` file for comprehensive usage examples. + +## Testing + +Run the unit tests: + +```bash +pytest tests/src/dph_services/ -v +``` + +Run integration tests (requires service credentials): + +```bash +pytest tests/src/integration/test_dph_v1.py -v +``` + +## Requirements + +- Python 3.8+ +- ibm-cloud-sdk-core >= 3.16.7 +- requests >= 2.32.4 +- python-dateutil >= 2.5.3 + +## License + +Apache License 2.0 + +## Support + +For issues and questions: +- GitHub Issues: https://github.com/IBM/data-intelligence-sdk/issues +- Documentation: See main README.md + +## Related Modules + +- **dq_validator**: Data quality validation +- **odcs_generator**: ODCS file generation +- **data_product_recommender**: Query log analysis \ No newline at end of file diff --git a/src/wxdi/dph_services/__init__.py b/src/wxdi/dph_services/__init__.py new file mode 100644 index 0000000..ad33907 --- /dev/null +++ b/src/wxdi/dph_services/__init__.py @@ -0,0 +1,22 @@ +# coding: utf-8 +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Python client library for the DPH Services""" + +from ibm_cloud_sdk_core import IAMTokenManager, DetailedResponse, BaseService, ApiException + +from .common import get_sdk_headers +from .version import __version__ +from .dph_v1 import DphV1 diff --git a/src/wxdi/dph_services/common.py b/src/wxdi/dph_services/common.py new file mode 100644 index 0000000..cb538ca --- /dev/null +++ b/src/wxdi/dph_services/common.py @@ -0,0 +1,75 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +This module provides common methods for use across all service modules. +""" + +import platform +from wxdi.dph_services.version import __version__ + +HEADER_NAME_USER_AGENT = 'User-Agent' +SDK_NAME = 'data-product-python-sdk' + + +def get_system_info(): + """ + Get information about the system to be inserted into the User-Agent header. + """ + return 'lang={0}; arch={1}; os={2}; python.version={3}'.format( + 'python', platform.machine(), platform.system(), platform.python_version() # Architecture # OS + ) # Python version + + +def get_user_agent(): + """ + Get the value to be sent in the User-Agent header. + """ + return USER_AGENT + + +USER_AGENT = '{0}/{1} ({2})'.format(SDK_NAME, __version__, get_system_info()) + + +def get_sdk_headers(service_name=None, service_version=None, operation_id=None): + """ + Get the request headers to be sent in requests by the SDK. + + If you plan to gather metrics for your SDK, the User-Agent header value must + be a string similar to the following: + my-python-sdk/0.0.1 (lang=python; arch=x86_64; os=Linux; python.version=3.7.4) + + In the example above, the analytics tool will parse the user-agent header and + use the following properties: + "my-python-sdk" - the name of your sdk + "0.0.1"- the version of your sdk + "lang=python" - the language of the current sdk + "arch=x86_64; os=Linux; python.version=3.7.4" - system information + + Note: It is very important that the sdk name ends with the string `-sdk`, + as the analytics data collector uses this to gather usage data. + + :param service_name: The name of the service (optional, for future use) + :param service_version: The version of the service (optional, for future use) + :param operation_id: The operation ID (optional, for future use) + """ + # Parameters are accepted for API compatibility + # They may be utilized in future versions for enhanced telemetry + _ = (service_name, service_version, operation_id) + + headers = {} + headers[HEADER_NAME_USER_AGENT] = get_user_agent() + return headers diff --git a/src/wxdi/dph_services/common_constants.py b/src/wxdi/dph_services/common_constants.py new file mode 100644 index 0000000..585c79c --- /dev/null +++ b/src/wxdi/dph_services/common_constants.py @@ -0,0 +1,48 @@ +# coding: utf-8 + +# Copyright 2019, 2020 IBM All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +This module provides common constants for use across all service modules. +""" + +# Dph Api Paths +URL_GET_INITIALIZE_STATUS = '/data_product_exchange/v1/configuration/initialize/status' +URL_GET_SERVICEID_CREDENTIALS = '/data_product_exchange/v1/configuration/credentials' +URL_INITIALIZE = '/data_product_exchange/v1/configuration/initialize' +URL_MANAGE_APIKEYS = '/data_product_exchange/v1/configuration/rotate_credentials' +URL_LIST_DATA_PRODUCTS = '/data_product_exchange/v1/data_products' +URL_CREATE_DATA_PRODUCT = '/data_product_exchange/v1/data_products' +URL_GET_DATA_PRODUCT = '/data_product_exchange/v1/data_products/{data_product_id}' +URL_COMPLETE_DRAFT_CONTRACT_TERMS_DOCUMENT = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}/documents/{document_id}/complete' +URL_LIST_DATA_PRODUCT_DRAFTS = '/data_product_exchange/v1/data_products/{data_product_id}/drafts' +URL_CREATE_DATA_PRODUCT_DRAFT = '/data_product_exchange/v1/data_products/{data_product_id}/drafts' +URL_CREATE_DRAFT_CONTRACT_TERMS_DOCUMENT = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}/documents' +URL_GET_DATA_PRODUCT_DRAFT = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}' +URL_GET_DRAFT_CONTRACT_TERMS_DOCUMENT = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}/documents/{document_id}' +URL_PUBLISH_DATA_PRODUCT_DRAFT = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/publish' +URL_GET_DATA_PRODUCT_RELEASE = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}' +URL_UPDATE_DATA_PRODUCT_RELEASE = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}' +URL_GET_RELEASE_CONTRACT_TERMS_DOCUMENT = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}/contract_terms/{contract_terms_id}/documents/{document_id}' +URL_LIST_DATA_PRODUCT_RELEASES = '/data_product_exchange/v1/data_products/{data_product_id}/releases' +URL_RETIRE_DATA_PRODUCT_RELEASE = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}/retire' + +# Dph Api Headers +CONTENT_TYPE_JSON = 'application/json' +CONTENT_TYPE_PATCH_JSON = 'application/json-patch+json' + +SERVICE_NAME = 'data_product_hub_api_service' +SERVICE_VERSION = 'V1' + diff --git a/src/wxdi/dph_services/dph_v1.py b/src/wxdi/dph_services/dph_v1.py new file mode 100644 index 0000000..a05cec9 --- /dev/null +++ b/src/wxdi/dph_services/dph_v1.py @@ -0,0 +1,12411 @@ +# coding: utf-8 + +# (C) Copyright IBM Corp. 2025. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# IBM OpenAPI SDK Code Generator Version: 3.96.0-d6dec9d7-20241008-212902 + +""" +Data Product Hub API Service + +API Version: 1 +""" + +from datetime import datetime +from enum import Enum +from typing import BinaryIO, Dict, List, Optional +import json + +from ibm_cloud_sdk_core import BaseService, DetailedResponse +from ibm_cloud_sdk_core.authenticators.authenticator import Authenticator +from ibm_cloud_sdk_core.get_authenticator import get_authenticator_from_environment +from ibm_cloud_sdk_core.utils import convert_list, convert_model, datetime_to_string, string_to_datetime + +from .common import get_sdk_headers +from .common_constants import * + +############################################################################## +# Service +############################################################################## + + +class DphV1(BaseService): + """The DPH V1 service.""" + + DEFAULT_SERVICE_URL = 'https://api.dataplatform.dev.cloud.ibm.com/' + DEFAULT_SERVICE_NAME = SERVICE_NAME + + @classmethod + def new_instance( + cls, + service_name: str = DEFAULT_SERVICE_NAME, + ) -> 'DphV1': + """ + Return a new client for the DPH service using the specified parameters and + external configuration. + """ + authenticator = get_authenticator_from_environment(service_name) + service = cls( + authenticator + ) + service.configure_service(service_name) + return service + + def __init__( + self, + authenticator: Authenticator = None, + ) -> None: + """ + Construct a new client for the DPH service. + + :param Authenticator authenticator: The authenticator specifies the authentication mechanism. + Get up to date information from https://github.com/IBM/python-sdk-core/blob/main/README.md + about initializing the authenticator of your choice. + """ + BaseService.__init__(self, service_url=self.DEFAULT_SERVICE_URL, authenticator=authenticator) + + ######################### + # Helper Methods + ######################### + + def _prepare_headers(self, operation_id: str, **kwargs) -> Dict: + """ + Prepare request headers with SDK headers and custom headers from kwargs. + + :param operation_id: The operation ID for SDK headers + :param kwargs: Additional keyword arguments that may contain 'headers' + :return: Dictionary of prepared headers + """ + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id=operation_id, + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + + return headers + + def _set_accept_header(self, headers: Dict, accept_type: Optional[str] = CONTENT_TYPE_JSON) -> None: + """ + Set the Accept header if not already present. + + :param headers: Headers dictionary to update + :param accept_type: The accept type to set (default: CONTENT_TYPE_JSON) + """ + if accept_type and 'Accept' not in headers: + headers['Accept'] = accept_type + + def _prepare_json_data(self, data: Dict) -> str: + """ + Prepare JSON data by removing None values and converting to JSON string. + + :param data: Dictionary of data to prepare + :return: JSON string + """ + data = {k: v for (k, v) in data.items() if v is not None} + return json.dumps(data) + + ######################### + # Configuration + ######################### + + def get_initialize_status( + self, + *, + container_id: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Get resource initialization status. + + Use this API to get the status of resource initialization in Data Product + Hub.

If the data product catalog exists but has never been initialized, + the status will be "not_started".

If the data product catalog exists and + has been or is being initialized, the response will contain the status of the last + or current initialization. If the initialization failed, the "errors" and "trace" + fields will contain the error(s) encountered during the initialization, including + the ID to trace the error(s).

If the data product catalog doesn't exist, + an HTTP 404 response is returned. + + :param str container_id: (optional) Container ID of the data product + catalog. If not supplied, the data product catalog is looked up by using + the uid of the default data product catalog. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `InitializeResource` object + """ + + headers = self._prepare_headers('get_initialize_status', **kwargs) + self._set_accept_header(headers) + + params = { + 'container.id': container_id, + } + + url = '/data_product_exchange/v1/configuration/initialize/status' + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def get_service_id_credentials( + self, + **kwargs, + ) -> DetailedResponse: + """ + Get service id credentials. + + Use this API to get the information of service id credentials in Data Product Hub. + + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `ServiceIdCredentials` object + """ + + headers = self._prepare_headers('get_service_id_credentials', **kwargs) + self._set_accept_header(headers) + + url = '/data_product_exchange/v1/configuration/credentials' + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + def initialize( + self, + *, + container: Optional['ContainerReference'] = None, + include: Optional[List[str]] = None, + **kwargs, + ) -> DetailedResponse: + """ + Initialize resources. + + Use this API to initialize default assets for data product hub.

You can + initialize:
  • `delivery_methods` - Methods through which data product + parts can be delivered to consumers of the data product + hub
  • `domains_multi_industry` - Taxonomy of domains and use cases + applicable to multiple industries
  • `data_product_samples` - Sample data + products used to illustrate capabilities of the data product + hub
  • `workflows` - Workflows to enable restricted data + products
  • `project` - A default project for exporting data assets to + files
  • `catalog_configurations` - Catalog configurations for the default + data product catalog


If a resource depends on resources that + are not specified in the request, these dependent resources will be automatically + initialized. E.g., initializing `data_product_samples` will also initialize + `domains_multi_industry` and `delivery_methods` even if they are not specified in + the request because it depends on them.

If initializing the data product + hub for the first time, do not specify a container. The default data product + catalog will be created.
For first time initialization, it is recommended that + at least `delivery_methods` and `domains_multi_industry` is included in the + initialize operation.

If the data product hub has already been + initialized, you may call this API again to initialize new resources, such as new + delivery methods. In this case, specify the default data product catalog container + information. + + :param ContainerReference container: (optional) Container reference. + :param List[str] include: (optional) List of configuration options to + (re-)initialize. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `InitializeResource` object + """ + + if container is not None: + container = convert_model(container) + + headers = self._prepare_headers('initialize', **kwargs) + self._set_accept_header(headers) + + data = self._prepare_json_data({ + 'container': container, + 'include': include, + }) + headers['content-type'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/configuration/initialize' + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def manage_api_keys( + self, + **kwargs, + ) -> DetailedResponse: + """ + Rotate credentials for a Data Product Hub instance. + + Use this API to rotate credentials for a Data Product Hub instance. + + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse + """ + + headers = self._prepare_headers('manage_api_keys', **kwargs) + + url = '/data_product_exchange/v1/configuration/rotate_credentials' + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + ######################### + # Data Asset Visualization + ######################### + + def create_data_asset_visualization( + self, + *, + assets: Optional[List['DataAssetRelationship']] = None, + **kwargs, + ) -> DetailedResponse: + """ + Create visualization asset and initialize profiling for the provided data assets. + + Use this API to create visualization asset and initialize profiling for the + provided data assets

Provide the below required fields

Required + fields:

- catalog_id
- Collection of assetId with it's related asset + id

. + + :param List[DataAssetRelationship] assets: (optional) Data product hub + asset and it's related part asset. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataAssetVisualizationRes` object + """ + + if assets is not None: + assets = [convert_model(x) for x in assets] + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='create_data_asset_visualization', + ) + headers.update(sdk_headers) + + data = { + 'assets': assets, + } + data = {k: v for (k, v) in data.items() if v is not None} + data = json.dumps(data) + headers['content-type'] = CONTENT_TYPE_JSON + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/data_asset/visualization' + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def reinitiate_data_asset_visualization( + self, + *, + assets: Optional[List['DataAssetRelationship']] = None, + **kwargs, + ) -> DetailedResponse: + """ + Reinitiate visualization for an asset. + + Use this API to Reinitiate visualization for an asset which is in below + scenarios

- Previous bucket got deleted and new bucket is created.
- + Data visualization attachment is missing in asset details.
- Visualization + asset reference is missing in related asset details.

. + + :param List[DataAssetRelationship] assets: (optional) Data product hub + asset and it's related part asset. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataAssetVisualizationRes` object + """ + + if assets is not None: + assets = [convert_model(x) for x in assets] + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='reinitiate_data_asset_visualization', + ) + headers.update(sdk_headers) + + data = { + 'assets': assets, + } + data = {k: v for (k, v) in data.items() if v is not None} + data = json.dumps(data) + headers['content-type'] = CONTENT_TYPE_JSON + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/data_asset/visualization/reinitiate' + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + ######################### + # Data Products + ######################### + + def list_data_products( + self, + *, + limit: Optional[int] = None, + start: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a list of data products. + + Retrieve a list of data products. + + :param int limit: (optional) Limit the number of data products in the + results. The maximum limit is 200. + :param str start: (optional) Start token for pagination. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductCollection` object + """ + + headers = self._prepare_headers('list_data_products', **kwargs) + self._set_accept_header(headers) + + params = { + 'limit': limit, + 'start': start, + } + + url = '/data_product_exchange/v1/data_products' + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def create_data_product( + self, + drafts: List['DataProductDraftPrototype'], + *, + limit: Optional[int] = None, + start: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Create a new data product. + + Use this API to create a new data product.

Provide the initial draft of + the data product.

Required fields:

- name
- + container

If `version` is not specified, the default version **1.0.0** + will be used.

The `domain` is optional. + + :param List[DataProductDraftPrototype] drafts: Collection of data products + drafts to add to data product. + :param int limit: (optional) Limit the number of data products in the + results. The maximum limit is 200. + :param str start: (optional) Start token for pagination. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProduct` object + """ + + if drafts is None: + raise ValueError('drafts must be provided') + drafts = [convert_model(x) for x in drafts] + + headers = self._prepare_headers('create_data_product', **kwargs) + self._set_accept_header(headers) + + params = { + 'limit': limit, + 'start': start, + } + + data = self._prepare_json_data({'drafts': drafts}) + headers['content-type'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/data_products' + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + params=params, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def get_data_product( + self, + data_product_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a data product identified by id. + + Retrieve a data product identified by id. + + :param str data_product_id: Data product ID. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProduct` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + + headers = self._prepare_headers('get_data_product', **kwargs) + self._set_accept_header(headers) + + path_param_keys = ['data_product_id'] + path_param_values = self.encode_path_vars(data_product_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + ######################### + # Data Product Drafts + ######################### + + def complete_draft_contract_terms_document( + self, + data_product_id: str, + draft_id: str, + contract_terms_id: str, + document_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Complete a contract document upload operation. + + After uploading a file to the provided signed URL, call this endpoint to mark the + upload as complete. After the upload operation is marked as complete, the file is + available to download. Use '-' for the `data_product_id` to skip specifying the + data product ID explicitly. + - After the upload is marked as complete, the returned URL is displayed in the + "url" field. The signed URL is used to download the document. + - Calling complete on referential documents results in an error. + - Calling complete on attachment documents for which the file has not been + uploaded will result in an error. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param str contract_terms_id: Contract terms id. + :param str document_id: Document id. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `ContractTermsDocument` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + if not document_id: + raise ValueError('document_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='complete_draft_contract_terms_document', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'draft_id', 'contract_terms_id', 'document_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id, contract_terms_id, document_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}/documents/{document_id}/complete'.format(**path_param_dict) + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + def list_data_product_drafts( + self, + data_product_id: str, + *, + asset_container_id: Optional[str] = None, + version: Optional[str] = None, + limit: Optional[int] = None, + start: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a list of data product drafts. + + Retrieve a list of data product drafts. Use '-' for the `data_product_id` to skip + specifying the data product ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str asset_container_id: (optional) Filter the list of data product + drafts by container id. + :param str version: (optional) Filter the list of data product drafts by + version number. + :param int limit: (optional) Limit the number of data product drafts in the + results. The maximum limit is 200. + :param str start: (optional) Start token for pagination. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductDraftCollection` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='list_data_product_drafts', + ) + headers.update(sdk_headers) + + params = { + 'asset.container.id': asset_container_id, + 'version': version, + 'limit': limit, + 'start': start, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id'] + path_param_values = self.encode_path_vars(data_product_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def create_data_product_draft( + self, + data_product_id: str, + asset: 'AssetPrototype', + *, + version: Optional[str] = None, + state: Optional[str] = None, + data_product: Optional['DataProductIdentity'] = None, + name: Optional[str] = None, + description: Optional[str] = None, + tags: Optional[List[str]] = None, + use_cases: Optional[List['UseCase']] = None, + types: Optional[List[str]] = None, + contract_terms: Optional[List['ContractTerms']] = None, + domain: Optional['Domain'] = None, + parts_out: Optional[List['DataProductPart']] = None, + workflows: Optional['DataProductWorkflows'] = None, + dataview_enabled: Optional[bool] = None, + comments: Optional[str] = None, + access_control: Optional['AssetListAccessControl'] = None, + last_updated_at: Optional[datetime] = None, + sub_container: Optional['ContainerIdentity'] = None, + is_restricted: Optional[bool] = None, + **kwargs, + ) -> DetailedResponse: + """ + Create a new draft of an existing data product. + + Create a new draft of an existing data product. + + :param str data_product_id: Data product ID. + :param AssetPrototype asset: New asset input properties. + :param str version: (optional) The data product version number. + :param str state: (optional) The state of the data product version. If not + specified, the data product version will be created in `draft` state. + :param DataProductIdentity data_product: (optional) Data product + identifier. + :param str name: (optional) The name that refers to the new data product + version. If this is a new data product, this value must be specified. If + this is a new version of an existing data product, the name will default to + the name of the previous data product version. A name can contain letters, + numbers, understores, dashes, spaces or periods. A name must contain at + least one non-space character. + :param str description: (optional) Description of the data product version. + If this is a new version of an existing data product, the description will + default to the description of the previous version of the data product. + :param List[str] tags: (optional) Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated + with the data product version. + :param List[str] types: (optional) Types of parts on the data product. + :param List[ContractTerms] contract_terms: (optional) Contract terms + binding various aspects of the data product. + :param Domain domain: (optional) Domain that the data product version + belongs to. If this is the first version of a data product, this field is + required. If this is a new version of an existing data product, the domain + will default to the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: (optional) The outgoing parts of + this data product version to be delivered to consumers. If this is the + first version of a data product, this field defaults to an empty list. If + this is a new version of an existing data product, the data product parts + will default to the parts list from the previous version of the data + product. + :param DataProductWorkflows workflows: (optional) The workflows associated + with the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided + either at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control + object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for + a IBM knowledge catalog container (catalog/project/space). + :param bool is_restricted: (optional) Indicates whether the data product is + restricted or not. A restricted data product indicates that orders of the + data product requires explicit approval before data is delivered. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductDraft` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if asset is None: + raise ValueError('asset must be provided') + asset = convert_model(asset) + if data_product is not None: + data_product = convert_model(data_product) + if use_cases is not None: + use_cases = [convert_model(x) for x in use_cases] + if contract_terms is not None: + contract_terms = [convert_model(x) for x in contract_terms] + if domain is not None: + domain = convert_model(domain) + if parts_out is not None: + parts_out = [convert_model(x) for x in parts_out] + if workflows is not None: + workflows = convert_model(workflows) + if access_control is not None: + access_control = convert_model(access_control) + if last_updated_at is not None: + last_updated_at = datetime_to_string(last_updated_at) + if sub_container is not None: + sub_container = convert_model(sub_container) + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='create_data_product_draft', + ) + headers.update(sdk_headers) + + data = { + 'asset': asset, + 'version': version, + 'state': state, + 'data_product': data_product, + 'name': name, + 'description': description, + 'tags': tags, + 'use_cases': use_cases, + 'types': types, + 'contract_terms': contract_terms, + 'domain': domain, + 'parts_out': parts_out, + 'workflows': workflows, + 'dataview_enabled': dataview_enabled, + 'comments': comments, + 'access_control': access_control, + 'last_updated_at': last_updated_at, + 'sub_container': sub_container, + 'is_restricted': is_restricted, + } + data = {k: v for (k, v) in data.items() if v is not None} + data = json.dumps(data) + headers['content-type'] = CONTENT_TYPE_JSON + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id'] + path_param_values = self.encode_path_vars(data_product_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts'.format(**path_param_dict) + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def create_draft_contract_terms_document( + self, + data_product_id: str, + draft_id: str, + contract_terms_id: str, + type: str, + name: str, + *, + url: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Upload a contract document to the data product draft contract terms. + + Upload a contract document to the data product draft identified by draft_id. Use + '-' for the `data_product_id` to skip specifying the data product ID explicitly. + - If the request object contains a "url" parameter, a referential document is + created to store the provided url. + - If the request object does not contain a "url" parameter, an attachment document + is created, and a signed url will be returned in an "upload_url" parameter. The + data product producer can upload the document using the provided "upload_url". + After the upload is completed, call "complete_contract_terms_document" for the + given document needs to be called to mark the upload as completed. After + completion of the upload, "get_contract_terms_document" for the given document + returns a signed "url" parameter that can be used to download the attachment + document. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param str contract_terms_id: Contract terms id. + :param str type: Type of the contract document. + :param str name: Name of the contract document. + :param str url: (optional) URL that can be used to retrieve the contract + document. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `ContractTermsDocument` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + if type is None: + raise ValueError('type must be provided') + if name is None: + raise ValueError('name must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='create_draft_contract_terms_document', + ) + headers.update(sdk_headers) + + data = { + 'type': type, + 'name': name, + 'url': url, + } + data = {k: v for (k, v) in data.items() if v is not None} + data = json.dumps(data) + headers['content-type'] = CONTENT_TYPE_JSON + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'draft_id', 'contract_terms_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id, contract_terms_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}/documents'.format(**path_param_dict) + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def get_data_product_draft( + self, + data_product_id: str, + draft_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Get a draft of an existing data product. + + Get a draft of an existing data product. Use '-' for the `data_product_id` to skip + specifying the data product ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductDraft` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_data_product_draft', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'draft_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + def delete_data_product_draft( + self, + data_product_id: str, + draft_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Delete a data product draft identified by ID. + + Delete a data product draft identified by a valid ID. Use '-' for the + `data_product_id` to skip specifying the data product ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='delete_data_product_draft', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + + path_param_keys = ['data_product_id', 'draft_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}'.format(**path_param_dict) + request = self.prepare_request( + method='DELETE', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + def update_data_product_draft( + self, + data_product_id: str, + draft_id: str, + json_patch_instructions: List['JsonPatchOperation'], + **kwargs, + ) -> DetailedResponse: + """ + Update the data product draft identified by ID. + + Use this API to update the properties of a data product draft identified by a + valid ID. Use '-' for the `data_product_id` to skip specifying the data product ID + explicitly.

Specify patch operations using http://jsonpatch.com/ + syntax.

Supported patch operations include:

- Update the + properties of a data product

- Add/Remove parts from a data product (up + to 20 parts)

- Add/Remove use cases from a data product

- Update + the data product state

. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param List[JsonPatchOperation] json_patch_instructions: A set of patch + operations as defined in RFC 6902. See http://jsonpatch.com/ for more + information. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductDraft` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if json_patch_instructions is None: + raise ValueError('json_patch_instructions must be provided') + json_patch_instructions = [convert_model(x) for x in json_patch_instructions] + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='update_data_product_draft', + ) + headers.update(sdk_headers) + + data = json.dumps(json_patch_instructions) + headers['content-type'] = 'application/json-patch+json' + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'draft_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}'.format(**path_param_dict) + request = self.prepare_request( + method='PATCH', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def get_draft_contract_terms_document( + self, + data_product_id: str, + draft_id: str, + contract_terms_id: str, + document_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Get a contract document. + + If a document has a completed attachment, the response contains the `url` which + can be used to download the attachment. If a document does not have a completed + attachment, the response contains the `url` which was submitted at document + creation. If a document has an attachment that is incomplete, an error is returned + to prompt the user to upload the document file and complete it. Use '-' for the + `data_product_id` to skip specifying the data product ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param str contract_terms_id: Contract terms id. + :param str document_id: Document id. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `ContractTermsDocument` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + if not document_id: + raise ValueError('document_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_draft_contract_terms_document', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'draft_id', 'contract_terms_id', 'document_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id, contract_terms_id, document_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}/documents/{document_id}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + def delete_draft_contract_terms_document( + self, + data_product_id: str, + draft_id: str, + contract_terms_id: str, + document_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Delete a contract document. + + Delete an existing contract document. + Contract documents can only be deleted for data product versions that are in DRAFT + state. Use '-' for the `data_product_id` to skip specifying the data product ID + explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param str contract_terms_id: Contract terms id. + :param str document_id: Document id. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + if not document_id: + raise ValueError('document_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='delete_draft_contract_terms_document', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + + path_param_keys = ['data_product_id', 'draft_id', 'contract_terms_id', 'document_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id, contract_terms_id, document_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}/documents/{document_id}'.format(**path_param_dict) + request = self.prepare_request( + method='DELETE', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + def update_draft_contract_terms_document( + self, + data_product_id: str, + draft_id: str, + contract_terms_id: str, + document_id: str, + json_patch_instructions: List['JsonPatchOperation'], + **kwargs, + ) -> DetailedResponse: + """ + Update a contract document. + + Use this API to update the properties of a contract document that is identified by + a valid ID. + Specify patch operations using http://jsonpatch.com/ syntax. + Supported patch operations include: + - Update the url of document if it does not have an attachment. + - Update the type of the document. +

Contract terms documents can only be updated if the associated data + product version is in DRAFT state. Use '-' for the `data_product_id` to skip + specifying the data product ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param str contract_terms_id: Contract terms id. + :param str document_id: Document id. + :param List[JsonPatchOperation] json_patch_instructions: A set of patch + operations as defined in RFC 6902. See http://jsonpatch.com/ for more + information. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `ContractTermsDocument` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + if not document_id: + raise ValueError('document_id must be provided') + if json_patch_instructions is None: + raise ValueError('json_patch_instructions must be provided') + json_patch_instructions = [convert_model(x) for x in json_patch_instructions] + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='update_draft_contract_terms_document', + ) + headers.update(sdk_headers) + + data = json.dumps(json_patch_instructions) + headers['content-type'] = 'application/json-patch+json' + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'draft_id', 'contract_terms_id', 'document_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id, contract_terms_id, document_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}/documents/{document_id}'.format(**path_param_dict) + request = self.prepare_request( + method='PATCH', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def get_data_product_draft_contract_terms( + self, + data_product_id: str, + draft_id: str, + contract_terms_id: str, + *, + accept: Optional[str] = None, + include_contract_documents: Optional[bool] = None, + autopopulate_server_information: Optional[bool] = None, + server_asset_id: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a data product contract terms identified by id. + + Retrieve a data product contract terms identified by id. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param str contract_terms_id: Contract terms id. + :param str accept: (optional) The type of the response: application/json or + application/odcs+yaml. + :param bool include_contract_documents: (optional) Set to false to exclude + external contract documents (e.g., Terms and Conditions URLs) from the + response. By default, these are included. + :param bool autopopulate_server_information: (optional) Set to true to + autopopulate server information from connection details. Default is false. + :param str server_asset_id: (optional) Asset ID of the server used for + autopopulating connection details. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `ContractTerms` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + headers = { + 'Accept': accept, + } + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_data_product_draft_contract_terms', + ) + headers.update(sdk_headers) + + params = { + 'include_contract_documents': include_contract_documents, + 'autopopulate_server_information': autopopulate_server_information, + 'server_asset_id': server_asset_id, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + + path_param_keys = ['data_product_id', 'draft_id', 'contract_terms_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id, contract_terms_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def replace_data_product_draft_contract_terms( + self, + data_product_id: str, + draft_id: str, + contract_terms_id: str, + *, + asset: Optional['AssetReference'] = None, + id: Optional[str] = None, + documents: Optional[List['ContractTermsDocument']] = None, + error_msg: Optional[str] = None, + overview: Optional['Overview'] = None, + description: Optional['Description'] = None, + organization: Optional[List['ContractTemplateOrganization']] = None, + roles: Optional[List['Roles']] = None, + price: Optional['Pricing'] = None, + sla: Optional[List['ContractTemplateSLA']] = None, + support_and_communication: Optional[List['ContractTemplateSupportAndCommunication']] = None, + custom_properties: Optional[List['ContractTemplateCustomProperty']] = None, + contract_test: Optional['ContractTest'] = None, + servers: Optional[List['ContractServer']] = None, + schema: Optional[List['ContractSchema']] = None, + **kwargs, + ) -> DetailedResponse: + """ + Update a data product contract terms identified by id. + + Update a data product contract terms identified by id. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param str contract_terms_id: Contract terms id. + :param AssetReference asset: (optional) The reference schema for a asset in + a container. + :param str id: (optional) ID of the contract terms. + :param List[ContractTermsDocument] documents: (optional) Collection of + contract terms documents. + :param str error_msg: (optional) An error message, if existing, relating to + the contract terms. + :param Overview overview: (optional) Overview details of a data contract. + :param Description description: (optional) Description details of a data + contract. + :param List[ContractTemplateOrganization] organization: (optional) List of + sub domains to be added within a domain. + :param List[Roles] roles: (optional) List of roles associated with the + contract. + :param Pricing price: (optional) Represents the pricing details of the + contract. + :param List[ContractTemplateSLA] sla: (optional) Service Level Agreement + details. + :param List[ContractTemplateSupportAndCommunication] + support_and_communication: (optional) Support and communication details for + the contract. + :param List[ContractTemplateCustomProperty] custom_properties: (optional) + Custom properties that are not part of the standard contract. + :param ContractTest contract_test: (optional) Contains the contract test + status and related metadata. + :param List[ContractServer] servers: (optional) List of server definitions. + :param List[ContractSchema] schema: (optional) Schema details of the data + asset. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `ContractTerms` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + if asset is not None: + asset = convert_model(asset) + if documents is not None: + documents = [convert_model(x) for x in documents] + if overview is not None: + overview = convert_model(overview) + if description is not None: + description = convert_model(description) + if organization is not None: + organization = [convert_model(x) for x in organization] + if roles is not None: + roles = [convert_model(x) for x in roles] + if price is not None: + price = convert_model(price) + if sla is not None: + sla = [convert_model(x) for x in sla] + if support_and_communication is not None: + support_and_communication = [convert_model(x) for x in support_and_communication] + if custom_properties is not None: + custom_properties = [convert_model(x) for x in custom_properties] + if contract_test is not None: + contract_test = convert_model(contract_test) + if servers is not None: + servers = [convert_model(x) for x in servers] + if schema is not None: + schema = [convert_model(x) for x in schema] + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='replace_data_product_draft_contract_terms', + ) + headers.update(sdk_headers) + + data = { + 'asset': asset, + 'id': id, + 'documents': documents, + 'error_msg': error_msg, + 'overview': overview, + 'description': description, + 'organization': organization, + 'roles': roles, + 'price': price, + 'sla': sla, + 'support_and_communication': support_and_communication, + 'custom_properties': custom_properties, + 'contract_test': contract_test, + 'servers': servers, + 'schema': schema, + } + data = {k: v for (k, v) in data.items() if v is not None} + data = json.dumps(data) + headers['content-type'] = CONTENT_TYPE_JSON + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'draft_id', 'contract_terms_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id, contract_terms_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}'.format(**path_param_dict) + request = self.prepare_request( + method='PUT', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def update_data_product_draft_contract_terms( + self, + data_product_id: str, + draft_id: str, + contract_terms_id: str, + json_patch_instructions: List['JsonPatchOperation'], + **kwargs, + ) -> DetailedResponse: + """ + Update a contract terms property. + + Use this API to update the properties of a contract terms that is identified by a + valid ID. + Specify patch operations using http://jsonpatch.com/ syntax. + Supported patch operations include: + - Update the contract terms properties. +

Contract terms can only be updated if the associated data product + version is in DRAFT state. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param str contract_terms_id: Contract terms id. + :param List[JsonPatchOperation] json_patch_instructions: A set of patch + operations as defined in RFC 6902. See http://jsonpatch.com/ for more + information. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `ContractTerms` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + if json_patch_instructions is None: + raise ValueError('json_patch_instructions must be provided') + json_patch_instructions = [convert_model(x) for x in json_patch_instructions] + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='update_data_product_draft_contract_terms', + ) + headers.update(sdk_headers) + + data = json.dumps(json_patch_instructions) + headers['content-type'] = 'application/json-patch+json' + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'draft_id', 'contract_terms_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id, contract_terms_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}'.format(**path_param_dict) + request = self.prepare_request( + method='PATCH', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def get_contract_terms_in_specified_format( + self, + data_product_id: str, + draft_id: str, + contract_terms_id: str, + format: str, + format_version: str, + *, + accept: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a data product contract terms identified by id in specified format. + + Retrieve a data product contract terms identified by id in specified format. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param str contract_terms_id: Contract terms id. + :param str format: The format for returning contract terms. For example: + odcs. + :param str format_version: The version of the format for returning contract + terms. For example: 3. + :param str accept: (optional) The type of the response: + application/odcs+yaml or application/json. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `BinaryIO` result + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + if not format: + raise ValueError('format must be provided') + if not format_version: + raise ValueError('format_version must be provided') + headers = { + 'Accept': accept, + } + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_contract_terms_in_specified_format', + ) + headers.update(sdk_headers) + + params = { + 'format': format, + 'format_version': format_version, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + + path_param_keys = ['data_product_id', 'draft_id', 'contract_terms_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id, contract_terms_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}/format'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def publish_data_product_draft( + self, + data_product_id: str, + draft_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Publish a draft of an existing data product. + + Publish a draft of an existing data product. Use '-' for the `data_product_id` to + skip specifying the data product ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductRelease` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='publish_data_product_draft', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'draft_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/publish'.format(**path_param_dict) + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + ######################### + # Data Product Releases + ######################### + + def get_data_product_release( + self, + data_product_id: str, + release_id: str, + *, + check_caller_approval: Optional[bool] = None, + **kwargs, + ) -> DetailedResponse: + """ + Get a release of an existing data product. + + Get a release of an existing data product. Use '-' for the `data_product_id` to + skip specifying the data product ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str release_id: Data product release id. + :param bool check_caller_approval: (optional) If the value is true, then it + will be verfied whether the caller is present in the data access request + pre-approved user list. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductRelease` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not release_id: + raise ValueError('release_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_data_product_release', + ) + headers.update(sdk_headers) + + params = { + 'check_caller_approval': check_caller_approval, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'release_id'] + path_param_values = self.encode_path_vars(data_product_id, release_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def update_data_product_release( + self, + data_product_id: str, + release_id: str, + json_patch_instructions: List['JsonPatchOperation'], + **kwargs, + ) -> DetailedResponse: + """ + Update the data product release identified by ID. + + Use this API to update the properties of a data product release identified by a + valid ID. Use '-' for the `data_product_id` to skip specifying the data product ID + explicitly.

Specify patch operations using http://jsonpatch.com/ + syntax.

Supported patch operations include:

- Update the + properties of a data product

- Add/remove parts from a data product (up + to 20 parts)

- Add/remove use cases from a data product

. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str release_id: Data product release id. + :param List[JsonPatchOperation] json_patch_instructions: A set of patch + operations as defined in RFC 6902. See http://jsonpatch.com/ for more + information. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductRelease` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not release_id: + raise ValueError('release_id must be provided') + if json_patch_instructions is None: + raise ValueError('json_patch_instructions must be provided') + json_patch_instructions = [convert_model(x) for x in json_patch_instructions] + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='update_data_product_release', + ) + headers.update(sdk_headers) + + data = json.dumps(json_patch_instructions) + headers['content-type'] = 'application/json-patch+json' + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'release_id'] + path_param_values = self.encode_path_vars(data_product_id, release_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}'.format(**path_param_dict) + request = self.prepare_request( + method='PATCH', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def get_release_contract_terms_document( + self, + data_product_id: str, + release_id: str, + contract_terms_id: str, + document_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Get a contract document. + + If the document has a completed attachment, the response contains the `url` to + download the attachment.

If the document does not have an attachment, + the response contains the `url` which was submitted at document + creation.

If the document has an incomplete attachment, an error is + returned to prompt the user to upload the document file to complete the + attachment. Use '-' for the `data_product_id` to skip specifying the data product + ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str release_id: Data product release id. + :param str contract_terms_id: Contract terms id. + :param str document_id: Document id. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `ContractTermsDocument` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not release_id: + raise ValueError('release_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + if not document_id: + raise ValueError('document_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_release_contract_terms_document', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'release_id', 'contract_terms_id', 'document_id'] + path_param_values = self.encode_path_vars(data_product_id, release_id, contract_terms_id, document_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}/contract_terms/{contract_terms_id}/documents/{document_id}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + def get_published_data_product_draft_contract_terms( + self, + data_product_id: str, + release_id: str, + contract_terms_id: str, + *, + accept: Optional[str] = None, + include_contract_documents: Optional[bool] = None, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a published data product contract terms identified by id. + + Retrieve a published data product contract terms identified by id. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str release_id: Data product release id. + :param str contract_terms_id: Contract terms id. + :param str accept: (optional) The type of the response: + application/odcs+yaml or application/json. + :param bool include_contract_documents: (optional) Set to false to exclude + external contract documents (e.g., Terms and Conditions URLs) from the + response. By default, these are included. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `BinaryIO` result + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not release_id: + raise ValueError('release_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + headers = { + 'Accept': accept, + } + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_published_data_product_draft_contract_terms', + ) + headers.update(sdk_headers) + + params = { + 'include_contract_documents': include_contract_documents, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + + path_param_keys = ['data_product_id', 'release_id', 'contract_terms_id'] + path_param_values = self.encode_path_vars(data_product_id, release_id, contract_terms_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}/contract_terms/{contract_terms_id}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def list_data_product_releases( + self, + data_product_id: str, + *, + asset_container_id: Optional[str] = None, + state: Optional[List[str]] = None, + version: Optional[str] = None, + limit: Optional[int] = None, + start: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a list of data product releases. + + Retrieve a list of data product releases. Use '-' for the `data_product_id` to + skip specifying the data product ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str asset_container_id: (optional) Filter the list of data product + releases by container id. + :param List[str] state: (optional) Filter the list of data product versions + by state. States are: available and retired. Default is + "available","retired". + :param str version: (optional) Filter the list of data product releases by + version number. + :param int limit: (optional) Limit the number of data product releases in + the results. The maximum is 200. + :param str start: (optional) Start token for pagination. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductReleaseCollection` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='list_data_product_releases', + ) + headers.update(sdk_headers) + + params = { + 'asset.container.id': asset_container_id, + 'state': convert_list(state), + 'version': version, + 'limit': limit, + 'start': start, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id'] + path_param_values = self.encode_path_vars(data_product_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/releases'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def retire_data_product_release( + self, + data_product_id: str, + release_id: str, + *, + revoke_access: Optional[bool] = None, + start_at: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Retire a release of an existing data product. + + Retire a release of an existing data product. Use '-' for the `data_product_id` to + skip specifying the data product ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str release_id: Data product release id. + :param bool revoke_access: (optional) Revoke's Access from all the + Subscriptions of the Data Product. No user's can able to see the subscribed + assets anymore. + :param str start_at: (optional) The date and time when the revoke access + operation should start (ISO 8601 format, e.g., 2025-09-24T06:55:29Z). If + not provided, the operation starts immediately. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductRelease` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not release_id: + raise ValueError('release_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='retire_data_product_release', + ) + headers.update(sdk_headers) + + params = { + 'revoke_access': revoke_access, + 'start_at': start_at, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'release_id'] + path_param_values = self.encode_path_vars(data_product_id, release_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}/retire'.format(**path_param_dict) + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def create_revoke_access_process( + self, + data_product_id: str, + release_id: str, + *, + body: Optional[BinaryIO] = None, + content_type: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Revoke access from Data Product subscriptions. + + Revoke's access from Subscriptions of the data product id passed in the path + parameter. Optionally specify a future date-time when the revoke access operation + should start using the start_at field in ISO 8601 format (e.g., + 2025-09-24T06:55:29Z). If start_at is not provided, the revoke access operation + starts immediately. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str release_id: The unique identifier of the data product release. + :param BinaryIO body: (optional) Request parameters to handle revoke access + from subscriptions. The start_at field can be used to schedule the revoke + access operation for a future date-time. + :param str content_type: (optional) The type of the input. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `RevokeAccessResponse` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not release_id: + raise ValueError('release_id must be provided') + headers = { + 'Content-Type': content_type, + } + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='create_revoke_access_process', + ) + headers.update(sdk_headers) + + data = body + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'release_id'] + path_param_values = self.encode_path_vars(data_product_id, release_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}/revoke_access'.format(**path_param_dict) + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + ######################### + # Data Product Contract Templates + ######################### + + def list_data_product_contract_template( + self, + *, + container_id: Optional[str] = None, + contract_template_name: Optional[str] = None, + domain_ids: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a list of data product contract templates. + + Retrieve a list of data product contract templates. + + :param str container_id: (optional) Container ID of the data product + catalog. If not supplied, the data product catalog is looked up by using + the uid of the default data product catalog. + :param str contract_template_name: (optional) Name of the data product + contract template. If not supplied, the data product templates within the + catalog will returned. + :param str domain_ids: (optional) Comma-separated domain IDs to filter data + product contract templates. If not supplied, the data product templates + within the catalog will returned. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductContractTemplateCollection` object + """ + + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='list_data_product_contract_template', + ) + headers.update(sdk_headers) + + params = { + 'container.id': container_id, + 'contract_template.name': contract_template_name, + 'domain.ids': domain_ids, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/contract_templates' + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def create_contract_template( + self, + container: 'ContainerReference', + *, + id: Optional[str] = None, + creator_id: Optional[str] = None, + created_at: Optional[str] = None, + name: Optional[str] = None, + error: Optional['ErrorMessage'] = None, + contract_terms: Optional['ContractTerms'] = None, + container_id: Optional[str] = None, + contract_template_name: Optional[str] = None, + domain_ids: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Create new data product contract template. + + Create new data product contract template. + + :param ContainerReference container: Container reference. + :param str id: (optional) The identifier of the data product contract + template. + :param str creator_id: (optional) The identifier of the user who created + the data product contract template. + :param str created_at: (optional) The timestamp when the data product + contract template was created. + :param str name: (optional) The name of the contract template. + :param ErrorMessage error: (optional) Contains the code and details. + :param ContractTerms contract_terms: (optional) Defines the complete + structure of a contract terms. + :param str container_id: (optional) Container ID of the data product + catalog. If not supplied, the data product catalog is looked up by using + the uid of the default data product catalog. + :param str contract_template_name: (optional) Name of the data product + contract template. If not supplied, the data product templates within the + catalog will returned. + :param str domain_ids: (optional) Comma-separated domain IDs to filter data + product contract templates. If not supplied, the data product templates + within the catalog will returned. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductContractTemplate` object + """ + + if container is None: + raise ValueError('container must be provided') + container = convert_model(container) + if error is not None: + error = convert_model(error) + if contract_terms is not None: + contract_terms = convert_model(contract_terms) + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='create_contract_template', + ) + headers.update(sdk_headers) + + params = { + 'container.id': container_id, + 'contract_template.name': contract_template_name, + 'domain.ids': domain_ids, + } + + data = { + 'container': container, + 'id': id, + 'creator_id': creator_id, + 'created_at': created_at, + 'name': name, + 'error': error, + 'contract_terms': contract_terms, + } + data = {k: v for (k, v) in data.items() if v is not None} + data = json.dumps(data) + headers['content-type'] = CONTENT_TYPE_JSON + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/contract_templates' + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + params=params, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def get_contract_template( + self, + contract_template_id: str, + container_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a data product contract template identified by id. + + Retrieve a data product contract template identified by id. + + :param str contract_template_id: Data Product Contract Template id. + :param str container_id: Container ID of the data product catalog. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductContractTemplate` object + """ + + if not contract_template_id: + raise ValueError('contract_template_id must be provided') + if not container_id: + raise ValueError('container_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_contract_template', + ) + headers.update(sdk_headers) + + params = { + 'container.id': container_id, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['contract_template_id'] + path_param_values = self.encode_path_vars(contract_template_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/contract_templates/{contract_template_id}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def delete_data_product_contract_template( + self, + contract_template_id: str, + container_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Delete a data product contract template identified by id. + + Delete a data product contract template identified by id. + + :param str contract_template_id: Data Product Contract Template id. + :param str container_id: Container ID of the data product catalog. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse + """ + + if not contract_template_id: + raise ValueError('contract_template_id must be provided') + if not container_id: + raise ValueError('container_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='delete_data_product_contract_template', + ) + headers.update(sdk_headers) + + params = { + 'container.id': container_id, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + + path_param_keys = ['contract_template_id'] + path_param_values = self.encode_path_vars(contract_template_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/contract_templates/{contract_template_id}'.format(**path_param_dict) + request = self.prepare_request( + method='DELETE', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def update_data_product_contract_template( + self, + contract_template_id: str, + container_id: str, + json_patch_instructions: List['JsonPatchOperation'], + **kwargs, + ) -> DetailedResponse: + """ + Update the data product contract template identified by ID. + + Use this API to update the properties of a data product contract template + identified by a valid ID.

Specify patch operations using + http://jsonpatch.com/ syntax.

Supported patch operations + include:

- Update the name of a data product contract template

- + Update the contract terms of data product contract template

. + + :param str contract_template_id: Data Product Contract Template id. + :param str container_id: Container ID of the data product catalog. + :param List[JsonPatchOperation] json_patch_instructions: A set of patch + operations as defined in RFC 6902. See http://jsonpatch.com/ for more + information. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductContractTemplate` object + """ + + if not contract_template_id: + raise ValueError('contract_template_id must be provided') + if not container_id: + raise ValueError('container_id must be provided') + if json_patch_instructions is None: + raise ValueError('json_patch_instructions must be provided') + json_patch_instructions = [convert_model(x) for x in json_patch_instructions] + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='update_data_product_contract_template', + ) + headers.update(sdk_headers) + + params = { + 'container.id': container_id, + } + + data = json.dumps(json_patch_instructions) + headers['content-type'] = 'application/json-patch+json' + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['contract_template_id'] + path_param_values = self.encode_path_vars(contract_template_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/contract_templates/{contract_template_id}'.format(**path_param_dict) + request = self.prepare_request( + method='PATCH', + url=url, + headers=headers, + params=params, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + ######################### + # Data Product Domains + ######################### + + def list_data_product_domains( + self, + *, + container_id: Optional[str] = None, + include_subdomains: Optional[bool] = None, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a list of data product domains. + + Retrieve a list of data product domains. + + :param str container_id: (optional) Container ID of the data product + catalog. If not supplied, the data product catalog is looked up by using + the uid of the default data product catalog. + :param bool include_subdomains: (optional) Include subdomains in the + response. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductDomainCollection` object + """ + + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='list_data_product_domains', + ) + headers.update(sdk_headers) + + params = { + 'container.id': container_id, + 'include_subdomains': include_subdomains, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/domains' + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def create_data_product_domain( + self, + container: 'ContainerReference', + *, + trace: Optional[str] = None, + errors: Optional[List['ErrorModelResource']] = None, + name: Optional[str] = None, + description: Optional[str] = None, + id: Optional[str] = None, + created_by: Optional[str] = None, + member_roles: Optional['MemberRolesSchema'] = None, + properties: Optional['PropertiesSchema'] = None, + sub_domains: Optional[List['InitializeSubDomain']] = None, + sub_container: Optional['ContainerIdentity'] = None, + link_to_subcontainers: Optional[bool] = None, + **kwargs, + ) -> DetailedResponse: + """ + Create new data product domain. + + Create new data product domain. + + :param ContainerReference container: Container reference. + :param str trace: (optional) The id to trace the failed domain creations. + :param List[ErrorModelResource] errors: (optional) Set of errors on the sub + domain creation. + :param str name: (optional) The name of the data product domain. + :param str description: (optional) The description of the data product + domain. + :param str id: (optional) The identifier of the data product domain. + :param str created_by: (optional) The identifier of the creator of the data + product domain. + :param MemberRolesSchema member_roles: (optional) Member roles of a + corresponding asset. + :param PropertiesSchema properties: (optional) Properties of the + corresponding asset. + :param List[InitializeSubDomain] sub_domains: (optional) List of sub + domains to be added within a domain. + :param ContainerIdentity sub_container: (optional) The identity schema for + a IBM knowledge catalog container (catalog/project/space). + :param bool link_to_subcontainers: (optional) Link domains to + subcontainers. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductDomain` object + """ + + if container is None: + raise ValueError('container must be provided') + container = convert_model(container) + if errors is not None: + errors = [convert_model(x) for x in errors] + if member_roles is not None: + member_roles = convert_model(member_roles) + if properties is not None: + properties = convert_model(properties) + if sub_domains is not None: + sub_domains = [convert_model(x) for x in sub_domains] + if sub_container is not None: + sub_container = convert_model(sub_container) + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='create_data_product_domain', + ) + headers.update(sdk_headers) + + params = { + 'link_to_subcontainers': link_to_subcontainers, + } + + data = { + 'container': container, + 'trace': trace, + 'errors': errors, + 'name': name, + 'description': description, + 'id': id, + 'created_by': created_by, + 'member_roles': member_roles, + 'properties': properties, + 'sub_domains': sub_domains, + 'sub_container': sub_container, + } + data = {k: v for (k, v) in data.items() if v is not None} + data = json.dumps(data) + headers['content-type'] = CONTENT_TYPE_JSON + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/domains' + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + params=params, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def create_data_product_subdomain( + self, + domain_id: str, + container_id: str, + *, + name: Optional[str] = None, + id: Optional[str] = None, + description: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Create data product subdomains for a specific domain identified by id. + + Create data product subdomains for a specific domain identified by id. + + :param str domain_id: Domain id. + :param str container_id: Container ID of the data product catalog. + :param str name: (optional) The name of the data product subdomain. + :param str id: (optional) The identifier of the data product subdomain. + :param str description: (optional) The description of the data product + subdomain. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `InitializeSubDomain` object + """ + + if not domain_id: + raise ValueError('domain_id must be provided') + if not container_id: + raise ValueError('container_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='create_data_product_subdomain', + ) + headers.update(sdk_headers) + + params = { + 'container.id': container_id, + } + + data = { + 'name': name, + 'id': id, + 'description': description, + } + data = {k: v for (k, v) in data.items() if v is not None} + data = json.dumps(data) + headers['content-type'] = CONTENT_TYPE_JSON + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['domain_id'] + path_param_values = self.encode_path_vars(domain_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/domains/{domain_id}/subdomains'.format(**path_param_dict) + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + params=params, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def get_domain( + self, + domain_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a data product domain or subdomain identified by id. + + Retrieve a data product domain or subdomain identified by id. + + :param str domain_id: Domain id. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductDomain` object + """ + + if not domain_id: + raise ValueError('domain_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_domain', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['domain_id'] + path_param_values = self.encode_path_vars(domain_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/domains/{domain_id}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + def delete_domain( + self, + domain_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Delete a data product domain identified by id. + + Delete a data product domain identified by id. + + :param str domain_id: Domain id. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse + """ + + if not domain_id: + raise ValueError('domain_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='delete_domain', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + + path_param_keys = ['domain_id'] + path_param_values = self.encode_path_vars(domain_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/domains/{domain_id}'.format(**path_param_dict) + request = self.prepare_request( + method='DELETE', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + def update_data_product_domain( + self, + domain_id: str, + container_id: str, + json_patch_instructions: List['JsonPatchOperation'], + **kwargs, + ) -> DetailedResponse: + """ + Update the data product domain identified by ID. + + Use this API to update the properties of a data product domain identified by a + valid ID.

Specify patch operations using http://jsonpatch.com/ + syntax.

Supported patch operations include:

- Update the name of + a data product domain

- Update the description of a data product + domain

- Update the rov of a data product domain

. + + :param str domain_id: Domain id. + :param str container_id: Container ID of the data product catalog. + :param List[JsonPatchOperation] json_patch_instructions: A set of patch + operations as defined in RFC 6902. See http://jsonpatch.com/ for more + information. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductDomain` object + """ + + if not domain_id: + raise ValueError('domain_id must be provided') + if not container_id: + raise ValueError('container_id must be provided') + if json_patch_instructions is None: + raise ValueError('json_patch_instructions must be provided') + json_patch_instructions = [convert_model(x) for x in json_patch_instructions] + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='update_data_product_domain', + ) + headers.update(sdk_headers) + + params = { + 'container.id': container_id, + } + + data = json.dumps(json_patch_instructions) + headers['content-type'] = 'application/json-patch+json' + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['domain_id'] + path_param_values = self.encode_path_vars(domain_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/domains/{domain_id}'.format(**path_param_dict) + request = self.prepare_request( + method='PATCH', + url=url, + headers=headers, + params=params, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def get_data_product_by_domain( + self, + domain_id: str, + container_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve all data products in a domain specified by id or any of it's subdomains. + + Retrieve all the data products tagged to the domain identified by id or any of + it's subdomains. + + :param str domain_id: Domain id. + :param str container_id: Container ID of the data product catalog. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductVersionCollection` object + """ + + if not domain_id: + raise ValueError('domain_id must be provided') + if not container_id: + raise ValueError('container_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_data_product_by_domain', + ) + headers.update(sdk_headers) + + params = { + 'container.id': container_id, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['domain_id'] + path_param_values = self.encode_path_vars(domain_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/domains/{domain_id}/data_products'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + ######################### + # Bucket Services + ######################### + + def create_s3_bucket( + self, + is_shared: bool, + **kwargs, + ) -> DetailedResponse: + """ + Create a new Bucket. + + Use this API to create a new S3 Bucket on an AWS Hosting. + + :param bool is_shared: Flag to specify whether the bucket is dedicated or + shared. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `BucketResponse` object + """ + + if is_shared is None: + raise ValueError('is_shared must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='create_s3_bucket', + ) + headers.update(sdk_headers) + + params = { + 'is_shared': is_shared, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/bucket' + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def get_s3_bucket_validation( + self, + bucket_name: str, + **kwargs, + ) -> DetailedResponse: + """ + Validate the Bucket Existence. + + Use this API to validate the bucket existence on an AWS hosting. + + :param str bucket_name: Name of the bucket to validate. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `BucketValidationResponse` object + """ + + if not bucket_name: + raise ValueError('bucket_name must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_s3_bucket_validation', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['bucket_name'] + path_param_values = self.encode_path_vars(bucket_name) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/bucket/validate/{bucket_name}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + ######################### + # Data Product Revoke Access Job Runs + ######################### + + def get_revoke_access_process_state( + self, + release_id: str, + *, + limit: Optional[int] = None, + start: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Access revoke status of the subscriptions against the data product release id. + + Retrieves the status of revoke access requests. + + :param str release_id: Pass the data product release version id to retrieve + job runs state for that specific DPV ID. + :param int limit: (optional) Limit the number of tracking assets in the + results. The maximum is 200. + :param str start: (optional) Start token for pagination. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `RevokeAccessStateResponse` object + """ + + if not release_id: + raise ValueError('release_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_revoke_access_process_state', + ) + headers.update(sdk_headers) + + params = { + 'release_id': release_id, + 'limit': limit, + 'start': start, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/data_product_revoke_access/job_runs' + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + +class GetDataProductDraftContractTermsEnums: + """ + Enums for get_data_product_draft_contract_terms parameters. + """ + + class Accept(str, Enum): + """ + The type of the response: application/json or application/odcs+yaml. + """ + + APPLICATION_JSON = CONTENT_TYPE_JSON + APPLICATION_ODCS_YAML = 'application/odcs+yaml' + + +class GetContractTermsInSpecifiedFormatEnums: + """ + Enums for get_contract_terms_in_specified_format parameters. + """ + + class Accept(str, Enum): + """ + The type of the response: application/odcs+yaml or application/json. + """ + + APPLICATION_ODCS_YAML = 'application/odcs+yaml' + APPLICATION_JSON = CONTENT_TYPE_JSON + + +class GetPublishedDataProductDraftContractTermsEnums: + """ + Enums for get_published_data_product_draft_contract_terms parameters. + """ + + class Accept(str, Enum): + """ + The type of the response: application/odcs+yaml or application/json. + """ + + APPLICATION_ODCS_YAML = 'application/odcs+yaml' + APPLICATION_JSON = CONTENT_TYPE_JSON + + +class ListDataProductReleasesEnums: + """ + Enums for list_data_product_releases parameters. + """ + + class State(str, Enum): + """ + Filter the list of data product versions by state. States are: available and + retired. Default is "available","retired". + """ + + AVAILABLE = 'available' + RETIRED = 'retired' + + +############################################################################## +# Models +############################################################################## + + +class Asset: + """ + Asset. + + :param dict metadata: (optional) + :param dict entity: (optional) + """ + + def __init__( + self, + *, + metadata: Optional[dict] = None, + entity: Optional[dict] = None, + ) -> None: + """ + Initialize a Asset object. + + :param dict metadata: (optional) + :param dict entity: (optional) + """ + self.metadata = metadata + self.entity = entity + + @classmethod + def from_dict(cls, _dict: Dict) -> 'Asset': + """Initialize a Asset object from a json dictionary.""" + args = {} + if (metadata := _dict.get('metadata')) is not None: + args['metadata'] = metadata + if (entity := _dict.get('entity')) is not None: + args['entity'] = entity + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a Asset object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'metadata') and self.metadata is not None: + _dict['metadata'] = self.metadata + if hasattr(self, 'entity') and self.entity is not None: + _dict['entity'] = self.entity + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this Asset object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'Asset') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'Asset') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class AssetListAccessControl: + """ + Access control object. + + :param str owner: (optional) The owner of the asset. + """ + + def __init__( + self, + *, + owner: Optional[str] = None, + ) -> None: + """ + Initialize a AssetListAccessControl object. + + :param str owner: (optional) The owner of the asset. + """ + self.owner = owner + + @classmethod + def from_dict(cls, _dict: Dict) -> 'AssetListAccessControl': + """Initialize a AssetListAccessControl object from a json dictionary.""" + args = {} + if (owner := _dict.get('owner')) is not None: + args['owner'] = owner + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a AssetListAccessControl object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'owner') and self.owner is not None: + _dict['owner'] = self.owner + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this AssetListAccessControl object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'AssetListAccessControl') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'AssetListAccessControl') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class AssetPartReference: + """ + The asset represented in this part. + + :param str id: (optional) The unique identifier of the asset. + :param str name: (optional) Asset name. + :param ContainerReference container: Container reference. + :param str type: (optional) The type of the asset. + """ + + def __init__( + self, + container: 'ContainerReference', + *, + id: Optional[str] = None, + name: Optional[str] = None, + type: Optional[str] = None, + ) -> None: + """ + Initialize a AssetPartReference object. + + :param ContainerReference container: Container reference. + :param str id: (optional) The unique identifier of the asset. + :param str name: (optional) Asset name. + :param str type: (optional) The type of the asset. + """ + self.id = id + self.name = name + self.container = container + self.type = type + + @classmethod + def from_dict(cls, _dict: Dict) -> 'AssetPartReference': + """Initialize a AssetPartReference object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + if (name := _dict.get('name')) is not None: + args['name'] = name + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in AssetPartReference JSON') + if (type := _dict.get('type')) is not None: + args['type'] = type + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a AssetPartReference object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + if hasattr(self, 'type') and self.type is not None: + _dict['type'] = self.type + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this AssetPartReference object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'AssetPartReference') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'AssetPartReference') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class AssetPrototype: + """ + New asset input properties. + + :param str id: (optional) The unique identifier of the asset. + :param ContainerIdentity container: The identity schema for a IBM knowledge + catalog container (catalog/project/space). + """ + + def __init__( + self, + container: 'ContainerIdentity', + *, + id: Optional[str] = None, + ) -> None: + """ + Initialize a AssetPrototype object. + + :param ContainerIdentity container: The identity schema for a IBM knowledge + catalog container (catalog/project/space). + :param str id: (optional) The unique identifier of the asset. + """ + self.id = id + self.container = container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'AssetPrototype': + """Initialize a AssetPrototype object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + if (container := _dict.get('container')) is not None: + args['container'] = ContainerIdentity.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in AssetPrototype JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a AssetPrototype object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this AssetPrototype object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'AssetPrototype') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'AssetPrototype') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class AssetReference: + """ + The reference schema for a asset in a container. + + :param str id: (optional) The unique identifier of the asset. + :param str name: (optional) Asset name. + :param ContainerReference container: Container reference. + """ + + def __init__( + self, + container: 'ContainerReference', + *, + id: Optional[str] = None, + name: Optional[str] = None, + ) -> None: + """ + Initialize a AssetReference object. + + :param ContainerReference container: Container reference. + :param str id: (optional) The unique identifier of the asset. + :param str name: (optional) Asset name. + """ + self.id = id + self.name = name + self.container = container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'AssetReference': + """Initialize a AssetReference object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + if (name := _dict.get('name')) is not None: + args['name'] = name + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in AssetReference JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a AssetReference object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this AssetReference object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'AssetReference') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'AssetReference') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class BucketResponse: + """ + BucketResponse to hold the Bucket response data. + + :param str bucket_name: (optional) Name of the Bucket. + :param str bucket_location: (optional) Location of the Bucket stored. + :param str role_arn: (optional) Role ARN. + :param str bucket_type: (optional) Bucket Type. + :param bool shared: (optional) Is Shared Bucket. + """ + + def __init__( + self, + *, + bucket_name: Optional[str] = None, + bucket_location: Optional[str] = None, + role_arn: Optional[str] = None, + bucket_type: Optional[str] = None, + shared: Optional[bool] = None, + ) -> None: + """ + Initialize a BucketResponse object. + + :param str bucket_name: (optional) Name of the Bucket. + :param str bucket_location: (optional) Location of the Bucket stored. + :param str role_arn: (optional) Role ARN. + :param str bucket_type: (optional) Bucket Type. + :param bool shared: (optional) Is Shared Bucket. + """ + self.bucket_name = bucket_name + self.bucket_location = bucket_location + self.role_arn = role_arn + self.bucket_type = bucket_type + self.shared = shared + + @classmethod + def from_dict(cls, _dict: Dict) -> 'BucketResponse': + """Initialize a BucketResponse object from a json dictionary.""" + args = {} + if (bucket_name := _dict.get('bucket_name')) is not None: + args['bucket_name'] = bucket_name + if (bucket_location := _dict.get('bucket_location')) is not None: + args['bucket_location'] = bucket_location + if (role_arn := _dict.get('role_arn')) is not None: + args['role_arn'] = role_arn + if (bucket_type := _dict.get('bucket_type')) is not None: + args['bucket_type'] = bucket_type + if (shared := _dict.get('shared')) is not None: + args['shared'] = shared + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a BucketResponse object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'bucket_name') and self.bucket_name is not None: + _dict['bucket_name'] = self.bucket_name + if hasattr(self, 'bucket_location') and self.bucket_location is not None: + _dict['bucket_location'] = self.bucket_location + if hasattr(self, 'role_arn') and self.role_arn is not None: + _dict['role_arn'] = self.role_arn + if hasattr(self, 'bucket_type') and self.bucket_type is not None: + _dict['bucket_type'] = self.bucket_type + if hasattr(self, 'shared') and self.shared is not None: + _dict['shared'] = self.shared + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this BucketResponse object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'BucketResponse') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'BucketResponse') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class BucketValidationResponse: + """ + BucketValidationResponse to hold the bucket validation data. + + :param bool bucket_exists: (optional) Flag of bucket existence. + """ + + def __init__( + self, + *, + bucket_exists: Optional[bool] = None, + ) -> None: + """ + Initialize a BucketValidationResponse object. + + :param bool bucket_exists: (optional) Flag of bucket existence. + """ + self.bucket_exists = bucket_exists + + @classmethod + def from_dict(cls, _dict: Dict) -> 'BucketValidationResponse': + """Initialize a BucketValidationResponse object from a json dictionary.""" + args = {} + if (bucket_exists := _dict.get('bucket_exists')) is not None: + args['bucket_exists'] = bucket_exists + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a BucketValidationResponse object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'bucket_exists') and self.bucket_exists is not None: + _dict['bucket_exists'] = self.bucket_exists + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this BucketValidationResponse object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'BucketValidationResponse') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'BucketValidationResponse') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContainerIdentity: + """ + The identity schema for a IBM knowledge catalog container (catalog/project/space). + + :param str id: Container identifier. + """ + + def __init__( + self, + id: str, + ) -> None: + """ + Initialize a ContainerIdentity object. + + :param str id: Container identifier. + """ + self.id = id + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContainerIdentity': + """Initialize a ContainerIdentity object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in ContainerIdentity JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContainerIdentity object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContainerIdentity object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContainerIdentity') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContainerIdentity') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContainerReference: + """ + Container reference. + + :param str id: Container identifier. + :param str type: Container type. + """ + + def __init__( + self, + id: str, + type: str, + ) -> None: + """ + Initialize a ContainerReference object. + + :param str id: Container identifier. + :param str type: Container type. + """ + self.id = id + self.type = type + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContainerReference': + """Initialize a ContainerReference object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in ContainerReference JSON') + if (type := _dict.get('type')) is not None: + args['type'] = type + else: + raise ValueError('Required property \'type\' not present in ContainerReference JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContainerReference object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'type') and self.type is not None: + _dict['type'] = self.type + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContainerReference object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContainerReference') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContainerReference') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class TypeEnum(str, Enum): + """ + Container type. + """ + + CATALOG = 'catalog' + PROJECT = 'project' + + + +class ContractAsset: + """ + Defines a data asset name and id. + + :param str id: (optional) ID of the data asset. + :param str name: (optional) Name of the data asset. + """ + + def __init__( + self, + *, + id: Optional[str] = None, + name: Optional[str] = None, + ) -> None: + """ + Initialize a ContractAsset object. + + :param str id: (optional) ID of the data asset. + :param str name: (optional) Name of the data asset. + """ + self.id = id + self.name = name + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractAsset': + """Initialize a ContractAsset object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + if (name := _dict.get('name')) is not None: + args['name'] = name + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractAsset object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractAsset object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractAsset') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractAsset') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractQualityRule: + """ + Defines a quality rule for validating data assets. + + :param str type: The type of the quality rule: 'text', 'library', or 'sql'. + :param str description: (optional) A descriptive explanation of the quality + rule. + :param str rule: (optional) The name or identifier of the library-based quality + rule to be applied. + :param str implementation: (optional) A text (non-parsed) block of code required + for the third-party DQ engine to run. + :param str engine: (optional) Required for custom DQ rule: name of the + third-party engine being used. Common values include soda, greatExpectations, + montecarlo, etc. + :param str must_be_less_than: (optional) The threshold value that the quality + check result must be less than. + :param str must_be_less_or_equal_to: (optional) The threshold value that the + quality check result must be less than or equal to. + :param str must_be_greater_than: (optional) The threshold value that the quality + check result must be greater than. + :param str must_be_greater_or_equal_to: (optional) The threshold value that the + quality check result must be greater than or equal to. + :param List[str] must_be_between: (optional) Inclusive range (min and max) for + the quality check result. + :param List[str] must_not_be_between: (optional) Inclusive range (min and max) + the quality check must not fall within. + :param str must_be: (optional) The exact value(s) the quality check result must + match. + :param str must_not_be: (optional) The exact value(s) the quality check result + must not match. + :param str name: (optional) User-friendly name for the quality rule. + :param str unit: (optional) Unit used for evaluating the quality rule (e.g., + rows, records). + :param str query: (optional) The SQL query to execute for validating quality in + case of a 'sql' rule type. + """ + + def __init__( + self, + type: str, + *, + description: Optional[str] = None, + rule: Optional[str] = None, + implementation: Optional[str] = None, + engine: Optional[str] = None, + must_be_less_than: Optional[str] = None, + must_be_less_or_equal_to: Optional[str] = None, + must_be_greater_than: Optional[str] = None, + must_be_greater_or_equal_to: Optional[str] = None, + must_be_between: Optional[List[str]] = None, + must_not_be_between: Optional[List[str]] = None, + must_be: Optional[str] = None, + must_not_be: Optional[str] = None, + name: Optional[str] = None, + unit: Optional[str] = None, + query: Optional[str] = None, + ) -> None: + """ + Initialize a ContractQualityRule object. + + :param str type: The type of the quality rule: 'text', 'library', or 'sql'. + :param str description: (optional) A descriptive explanation of the quality + rule. + :param str rule: (optional) The name or identifier of the library-based + quality rule to be applied. + :param str implementation: (optional) A text (non-parsed) block of code + required for the third-party DQ engine to run. + :param str engine: (optional) Required for custom DQ rule: name of the + third-party engine being used. Common values include soda, + greatExpectations, montecarlo, etc. + :param str must_be_less_than: (optional) The threshold value that the + quality check result must be less than. + :param str must_be_less_or_equal_to: (optional) The threshold value that + the quality check result must be less than or equal to. + :param str must_be_greater_than: (optional) The threshold value that the + quality check result must be greater than. + :param str must_be_greater_or_equal_to: (optional) The threshold value that + the quality check result must be greater than or equal to. + :param List[str] must_be_between: (optional) Inclusive range (min and max) + for the quality check result. + :param List[str] must_not_be_between: (optional) Inclusive range (min and + max) the quality check must not fall within. + :param str must_be: (optional) The exact value(s) the quality check result + must match. + :param str must_not_be: (optional) The exact value(s) the quality check + result must not match. + :param str name: (optional) User-friendly name for the quality rule. + :param str unit: (optional) Unit used for evaluating the quality rule + (e.g., rows, records). + :param str query: (optional) The SQL query to execute for validating + quality in case of a 'sql' rule type. + """ + self.type = type + self.description = description + self.rule = rule + self.implementation = implementation + self.engine = engine + self.must_be_less_than = must_be_less_than + self.must_be_less_or_equal_to = must_be_less_or_equal_to + self.must_be_greater_than = must_be_greater_than + self.must_be_greater_or_equal_to = must_be_greater_or_equal_to + self.must_be_between = must_be_between + self.must_not_be_between = must_not_be_between + self.must_be = must_be + self.must_not_be = must_not_be + self.name = name + self.unit = unit + self.query = query + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractQualityRule': + """Initialize a ContractQualityRule object from a json dictionary.""" + args = {} + if (type := _dict.get('type')) is not None: + args['type'] = type + else: + raise ValueError('Required property \'type\' not present in ContractQualityRule JSON') + if (description := _dict.get('description')) is not None: + args['description'] = description + if (rule := _dict.get('rule')) is not None: + args['rule'] = rule + if (implementation := _dict.get('implementation')) is not None: + args['implementation'] = implementation + if (engine := _dict.get('engine')) is not None: + args['engine'] = engine + if (must_be_less_than := _dict.get('must_be_less_than')) is not None: + args['must_be_less_than'] = must_be_less_than + if (must_be_less_or_equal_to := _dict.get('must_be_less_or_equal_to')) is not None: + args['must_be_less_or_equal_to'] = must_be_less_or_equal_to + if (must_be_greater_than := _dict.get('must_be_greater_than')) is not None: + args['must_be_greater_than'] = must_be_greater_than + if (must_be_greater_or_equal_to := _dict.get('must_be_greater_or_equal_to')) is not None: + args['must_be_greater_or_equal_to'] = must_be_greater_or_equal_to + if (must_be_between := _dict.get('must_be_between')) is not None: + args['must_be_between'] = must_be_between + if (must_not_be_between := _dict.get('must_not_be_between')) is not None: + args['must_not_be_between'] = must_not_be_between + if (must_be := _dict.get('must_be')) is not None: + args['must_be'] = must_be + if (must_not_be := _dict.get('must_not_be')) is not None: + args['must_not_be'] = must_not_be + if (name := _dict.get('name')) is not None: + args['name'] = name + if (unit := _dict.get('unit')) is not None: + args['unit'] = unit + if (query := _dict.get('query')) is not None: + args['query'] = query + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractQualityRule object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'type') and self.type is not None: + _dict['type'] = self.type + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'rule') and self.rule is not None: + _dict['rule'] = self.rule + if hasattr(self, 'implementation') and self.implementation is not None: + _dict['implementation'] = self.implementation + if hasattr(self, 'engine') and self.engine is not None: + _dict['engine'] = self.engine + if hasattr(self, 'must_be_less_than') and self.must_be_less_than is not None: + _dict['must_be_less_than'] = self.must_be_less_than + if hasattr(self, 'must_be_less_or_equal_to') and self.must_be_less_or_equal_to is not None: + _dict['must_be_less_or_equal_to'] = self.must_be_less_or_equal_to + if hasattr(self, 'must_be_greater_than') and self.must_be_greater_than is not None: + _dict['must_be_greater_than'] = self.must_be_greater_than + if hasattr(self, 'must_be_greater_or_equal_to') and self.must_be_greater_or_equal_to is not None: + _dict['must_be_greater_or_equal_to'] = self.must_be_greater_or_equal_to + if hasattr(self, 'must_be_between') and self.must_be_between is not None: + _dict['must_be_between'] = self.must_be_between + if hasattr(self, 'must_not_be_between') and self.must_not_be_between is not None: + _dict['must_not_be_between'] = self.must_not_be_between + if hasattr(self, 'must_be') and self.must_be is not None: + _dict['must_be'] = self.must_be + if hasattr(self, 'must_not_be') and self.must_not_be is not None: + _dict['must_not_be'] = self.must_not_be + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'unit') and self.unit is not None: + _dict['unit'] = self.unit + if hasattr(self, 'query') and self.query is not None: + _dict['query'] = self.query + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractQualityRule object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractQualityRule') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractQualityRule') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractSchema: + """ + Schema definition of the data asset. + + :param str asset_id: Id of the data asset whose schema information is stored. + :param str connection_id: Connection Id of the data asset whose schema + information is stored. + :param str name: (optional) Name of the schema or data asset part. + :param str description: (optional) Description of the schema. + :param str connection_path: (optional) Connection path of the asset. + :param str physical_type: (optional) MIME type or physical type. + :param List[ContractSchemaProperty] properties: (optional) List of properties. + :param List[ContractQualityRule] quality: (optional) List of quality rules + defined for the asset. + """ + + def __init__( + self, + asset_id: str, + connection_id: str, + *, + name: Optional[str] = None, + description: Optional[str] = None, + connection_path: Optional[str] = None, + physical_type: Optional[str] = None, + properties: Optional[List['ContractSchemaProperty']] = None, + quality: Optional[List['ContractQualityRule']] = None, + ) -> None: + """ + Initialize a ContractSchema object. + + :param str asset_id: Id of the data asset whose schema information is + stored. + :param str connection_id: Connection Id of the data asset whose schema + information is stored. + :param str name: (optional) Name of the schema or data asset part. + :param str description: (optional) Description of the schema. + :param str connection_path: (optional) Connection path of the asset. + :param str physical_type: (optional) MIME type or physical type. + :param List[ContractSchemaProperty] properties: (optional) List of + properties. + :param List[ContractQualityRule] quality: (optional) List of quality rules + defined for the asset. + """ + self.asset_id = asset_id + self.connection_id = connection_id + self.name = name + self.description = description + self.connection_path = connection_path + self.physical_type = physical_type + self.properties = properties + self.quality = quality + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractSchema': + """Initialize a ContractSchema object from a json dictionary.""" + args = {} + if (asset_id := _dict.get('asset_id')) is not None: + args['asset_id'] = asset_id + else: + raise ValueError('Required property \'asset_id\' not present in ContractSchema JSON') + if (connection_id := _dict.get('connection_id')) is not None: + args['connection_id'] = connection_id + else: + raise ValueError('Required property \'connection_id\' not present in ContractSchema JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + if (description := _dict.get('description')) is not None: + args['description'] = description + if (connection_path := _dict.get('connection_path')) is not None: + args['connection_path'] = connection_path + if (physical_type := _dict.get('physical_type')) is not None: + args['physical_type'] = physical_type + if (properties := _dict.get('properties')) is not None: + args['properties'] = [ContractSchemaProperty.from_dict(v) for v in properties] + if (quality := _dict.get('quality')) is not None: + args['quality'] = [ContractQualityRule.from_dict(v) for v in quality] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractSchema object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'asset_id') and self.asset_id is not None: + _dict['asset_id'] = self.asset_id + if hasattr(self, 'connection_id') and self.connection_id is not None: + _dict['connection_id'] = self.connection_id + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'connection_path') and self.connection_path is not None: + _dict['connection_path'] = self.connection_path + if hasattr(self, 'physical_type') and self.physical_type is not None: + _dict['physical_type'] = self.physical_type + if hasattr(self, 'properties') and self.properties is not None: + properties_list = [] + for v in self.properties: + if isinstance(v, dict): + properties_list.append(v) + else: + properties_list.append(v.to_dict()) + _dict['properties'] = properties_list + if hasattr(self, 'quality') and self.quality is not None: + quality_list = [] + for v in self.quality: + if isinstance(v, dict): + quality_list.append(v) + else: + quality_list.append(v.to_dict()) + _dict['quality'] = quality_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractSchema object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractSchema') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractSchema') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractSchemaProperty: + """ + Defines a property inside the schema. + + :param str name: Property name. + :param ContractSchemaPropertyType type: (optional) Detailed type definition of a + schema property. + :param List[ContractQualityRule] quality: (optional) List of quality rules + defined for the column. + """ + + def __init__( + self, + name: str, + *, + type: Optional['ContractSchemaPropertyType'] = None, + quality: Optional[List['ContractQualityRule']] = None, + ) -> None: + """ + Initialize a ContractSchemaProperty object. + + :param str name: Property name. + :param ContractSchemaPropertyType type: (optional) Detailed type definition + of a schema property. + :param List[ContractQualityRule] quality: (optional) List of quality rules + defined for the column. + """ + self.name = name + self.type = type + self.quality = quality + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractSchemaProperty': + """Initialize a ContractSchemaProperty object from a json dictionary.""" + args = {} + if (name := _dict.get('name')) is not None: + args['name'] = name + else: + raise ValueError('Required property \'name\' not present in ContractSchemaProperty JSON') + if (type := _dict.get('type')) is not None: + args['type'] = ContractSchemaPropertyType.from_dict(type) + if (quality := _dict.get('quality')) is not None: + args['quality'] = [ContractQualityRule.from_dict(v) for v in quality] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractSchemaProperty object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'type') and self.type is not None: + if isinstance(self.type, dict): + _dict['type'] = self.type + else: + _dict['type'] = self.type.to_dict() + if hasattr(self, 'quality') and self.quality is not None: + quality_list = [] + for v in self.quality: + if isinstance(v, dict): + quality_list.append(v) + else: + quality_list.append(v.to_dict()) + _dict['quality'] = quality_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractSchemaProperty object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractSchemaProperty') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractSchemaProperty') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractSchemaPropertyType: + """ + Detailed type definition of a schema property. + + :param str type: (optional) Type of the field. + :param str length: (optional) Length of the field as string. + :param str scale: (optional) Scale of the field as string. + :param str nullable: (optional) Is field nullable? true/false as string. + :param str signed: (optional) Is field signed? true/false as string. + :param str native_type: (optional) Native type of the field. + """ + + def __init__( + self, + *, + type: Optional[str] = None, + length: Optional[str] = None, + scale: Optional[str] = None, + nullable: Optional[str] = None, + signed: Optional[str] = None, + native_type: Optional[str] = None, + ) -> None: + """ + Initialize a ContractSchemaPropertyType object. + + :param str type: (optional) Type of the field. + :param str length: (optional) Length of the field as string. + :param str scale: (optional) Scale of the field as string. + :param str nullable: (optional) Is field nullable? true/false as string. + :param str signed: (optional) Is field signed? true/false as string. + :param str native_type: (optional) Native type of the field. + """ + self.type = type + self.length = length + self.scale = scale + self.nullable = nullable + self.signed = signed + self.native_type = native_type + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractSchemaPropertyType': + """Initialize a ContractSchemaPropertyType object from a json dictionary.""" + args = {} + if (type := _dict.get('type')) is not None: + args['type'] = type + if (length := _dict.get('length')) is not None: + args['length'] = length + if (scale := _dict.get('scale')) is not None: + args['scale'] = scale + if (nullable := _dict.get('nullable')) is not None: + args['nullable'] = nullable + if (signed := _dict.get('signed')) is not None: + args['signed'] = signed + if (native_type := _dict.get('native_type')) is not None: + args['native_type'] = native_type + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractSchemaPropertyType object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'type') and self.type is not None: + _dict['type'] = self.type + if hasattr(self, 'length') and self.length is not None: + _dict['length'] = self.length + if hasattr(self, 'scale') and self.scale is not None: + _dict['scale'] = self.scale + if hasattr(self, 'nullable') and self.nullable is not None: + _dict['nullable'] = self.nullable + if hasattr(self, 'signed') and self.signed is not None: + _dict['signed'] = self.signed + if hasattr(self, 'native_type') and self.native_type is not None: + _dict['native_type'] = self.native_type + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractSchemaPropertyType object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractSchemaPropertyType') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractSchemaPropertyType') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractServer: + """ + Schema definition of a server configuration for the asset. + + :param str server: Name of the server. + :param ContractAsset asset: (optional) Defines a data asset name and id. + :param str connection_id: (optional) ID of the data source associated with data + asset. + :param str type: (optional) Type of the server. + :param str description: (optional) Description of the server. + :param str environment: (optional) Environment in which the server operates. + :param str account: (optional) Account used by the server. + :param str catalog: (optional) Catalog name. + :param str database: (optional) Database name. + :param str dataset: (optional) Dataset name. + :param str delimiter: (optional) Delimiter. + :param str endpoint_url: (optional) Server endpoint URL. + :param str format: (optional) File format. + :param str host: (optional) Host name or IP address. + :param str location: (optional) Location URL. + :param str path: (optional) Relative or absolute path to the data. + :param str port: (optional) Port to the server. + :param str project: (optional) Project name. + :param str region: (optional) Cloud region. + :param str region_name: (optional) Region name. + :param str schema: (optional) Schema name. + :param str service_name: (optional) Service name. + :param str staging_dir: (optional) Staging directory. + :param str stream: (optional) Data stream name. + :param str warehouse: (optional) Warehouse or cluster name. + :param List[str] roles: (optional) List of roles for the server. + :param List[ContractTemplateCustomProperty] custom_properties: (optional) List + of custom properties for the server. + """ + + def __init__( + self, + server: str, + *, + asset: Optional['ContractAsset'] = None, + connection_id: Optional[str] = None, + type: Optional[str] = None, + description: Optional[str] = None, + environment: Optional[str] = None, + account: Optional[str] = None, + catalog: Optional[str] = None, + database: Optional[str] = None, + dataset: Optional[str] = None, + delimiter: Optional[str] = None, + endpoint_url: Optional[str] = None, + format: Optional[str] = None, + host: Optional[str] = None, + location: Optional[str] = None, + path: Optional[str] = None, + port: Optional[str] = None, + project: Optional[str] = None, + region: Optional[str] = None, + region_name: Optional[str] = None, + schema: Optional[str] = None, + service_name: Optional[str] = None, + staging_dir: Optional[str] = None, + stream: Optional[str] = None, + warehouse: Optional[str] = None, + roles: Optional[List[str]] = None, + custom_properties: Optional[List['ContractTemplateCustomProperty']] = None, + ) -> None: + """ + Initialize a ContractServer object. + + :param str server: Name of the server. + :param ContractAsset asset: (optional) Defines a data asset name and id. + :param str connection_id: (optional) ID of the data source associated with + data asset. + :param str type: (optional) Type of the server. + :param str description: (optional) Description of the server. + :param str environment: (optional) Environment in which the server + operates. + :param str account: (optional) Account used by the server. + :param str catalog: (optional) Catalog name. + :param str database: (optional) Database name. + :param str dataset: (optional) Dataset name. + :param str delimiter: (optional) Delimiter. + :param str endpoint_url: (optional) Server endpoint URL. + :param str format: (optional) File format. + :param str host: (optional) Host name or IP address. + :param str location: (optional) Location URL. + :param str path: (optional) Relative or absolute path to the data. + :param str port: (optional) Port to the server. + :param str project: (optional) Project name. + :param str region: (optional) Cloud region. + :param str region_name: (optional) Region name. + :param str schema: (optional) Schema name. + :param str service_name: (optional) Service name. + :param str staging_dir: (optional) Staging directory. + :param str stream: (optional) Data stream name. + :param str warehouse: (optional) Warehouse or cluster name. + :param List[str] roles: (optional) List of roles for the server. + :param List[ContractTemplateCustomProperty] custom_properties: (optional) + List of custom properties for the server. + """ + self.server = server + self.asset = asset + self.connection_id = connection_id + self.type = type + self.description = description + self.environment = environment + self.account = account + self.catalog = catalog + self.database = database + self.dataset = dataset + self.delimiter = delimiter + self.endpoint_url = endpoint_url + self.format = format + self.host = host + self.location = location + self.path = path + self.port = port + self.project = project + self.region = region + self.region_name = region_name + self.schema = schema + self.service_name = service_name + self.staging_dir = staging_dir + self.stream = stream + self.warehouse = warehouse + self.roles = roles + self.custom_properties = custom_properties + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractServer': + """Initialize a ContractServer object from a json dictionary.""" + args = {} + if (server := _dict.get('server')) is not None: + args['server'] = server + else: + raise ValueError('Required property \'server\' not present in ContractServer JSON') + if (asset := _dict.get('asset')) is not None: + args['asset'] = ContractAsset.from_dict(asset) + if (connection_id := _dict.get('connection_id')) is not None: + args['connection_id'] = connection_id + if (type := _dict.get('type')) is not None: + args['type'] = type + if (description := _dict.get('description')) is not None: + args['description'] = description + if (environment := _dict.get('environment')) is not None: + args['environment'] = environment + if (account := _dict.get('account')) is not None: + args['account'] = account + if (catalog := _dict.get('catalog')) is not None: + args['catalog'] = catalog + if (database := _dict.get('database')) is not None: + args['database'] = database + if (dataset := _dict.get('dataset')) is not None: + args['dataset'] = dataset + if (delimiter := _dict.get('delimiter')) is not None: + args['delimiter'] = delimiter + if (endpoint_url := _dict.get('endpoint_url')) is not None: + args['endpoint_url'] = endpoint_url + if (format := _dict.get('format')) is not None: + args['format'] = format + if (host := _dict.get('host')) is not None: + args['host'] = host + if (location := _dict.get('location')) is not None: + args['location'] = location + if (path := _dict.get('path')) is not None: + args['path'] = path + if (port := _dict.get('port')) is not None: + args['port'] = port + if (project := _dict.get('project')) is not None: + args['project'] = project + if (region := _dict.get('region')) is not None: + args['region'] = region + if (region_name := _dict.get('region_name')) is not None: + args['region_name'] = region_name + if (schema := _dict.get('schema')) is not None: + args['schema'] = schema + if (service_name := _dict.get('service_name')) is not None: + args['service_name'] = service_name + if (staging_dir := _dict.get('staging_dir')) is not None: + args['staging_dir'] = staging_dir + if (stream := _dict.get('stream')) is not None: + args['stream'] = stream + if (warehouse := _dict.get('warehouse')) is not None: + args['warehouse'] = warehouse + if (roles := _dict.get('roles')) is not None: + args['roles'] = roles + if (custom_properties := _dict.get('custom_properties')) is not None: + args['custom_properties'] = [ContractTemplateCustomProperty.from_dict(v) for v in custom_properties] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractServer object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'server') and self.server is not None: + _dict['server'] = self.server + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + if hasattr(self, 'connection_id') and self.connection_id is not None: + _dict['connection_id'] = self.connection_id + if hasattr(self, 'type') and self.type is not None: + _dict['type'] = self.type + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'environment') and self.environment is not None: + _dict['environment'] = self.environment + if hasattr(self, 'account') and self.account is not None: + _dict['account'] = self.account + if hasattr(self, 'catalog') and self.catalog is not None: + _dict['catalog'] = self.catalog + if hasattr(self, 'database') and self.database is not None: + _dict['database'] = self.database + if hasattr(self, 'dataset') and self.dataset is not None: + _dict['dataset'] = self.dataset + if hasattr(self, 'delimiter') and self.delimiter is not None: + _dict['delimiter'] = self.delimiter + if hasattr(self, 'endpoint_url') and self.endpoint_url is not None: + _dict['endpoint_url'] = self.endpoint_url + if hasattr(self, 'format') and self.format is not None: + _dict['format'] = self.format + if hasattr(self, 'host') and self.host is not None: + _dict['host'] = self.host + if hasattr(self, 'location') and self.location is not None: + _dict['location'] = self.location + if hasattr(self, 'path') and self.path is not None: + _dict['path'] = self.path + if hasattr(self, 'port') and self.port is not None: + _dict['port'] = self.port + if hasattr(self, 'project') and self.project is not None: + _dict['project'] = self.project + if hasattr(self, 'region') and self.region is not None: + _dict['region'] = self.region + if hasattr(self, 'region_name') and self.region_name is not None: + _dict['region_name'] = self.region_name + if hasattr(self, 'schema') and self.schema is not None: + _dict['schema'] = self.schema + if hasattr(self, 'service_name') and self.service_name is not None: + _dict['service_name'] = self.service_name + if hasattr(self, 'staging_dir') and self.staging_dir is not None: + _dict['staging_dir'] = self.staging_dir + if hasattr(self, 'stream') and self.stream is not None: + _dict['stream'] = self.stream + if hasattr(self, 'warehouse') and self.warehouse is not None: + _dict['warehouse'] = self.warehouse + if hasattr(self, 'roles') and self.roles is not None: + _dict['roles'] = self.roles + if hasattr(self, 'custom_properties') and self.custom_properties is not None: + custom_properties_list = [] + for v in self.custom_properties: + if isinstance(v, dict): + custom_properties_list.append(v) + else: + custom_properties_list.append(v.to_dict()) + _dict['custom_properties'] = custom_properties_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractServer object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractServer') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractServer') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractTemplateCustomProperty: + """ + Represents a custom property within the contract. + + :param str key: The name of the key. Names should be in camel case–the same as + if they were permanent properties in the contract. + :param str value: The value of the key. + """ + + def __init__( + self, + key: str, + value: str, + ) -> None: + """ + Initialize a ContractTemplateCustomProperty object. + + :param str key: The name of the key. Names should be in camel case–the same + as if they were permanent properties in the contract. + :param str value: The value of the key. + """ + self.key = key + self.value = value + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTemplateCustomProperty': + """Initialize a ContractTemplateCustomProperty object from a json dictionary.""" + args = {} + if (key := _dict.get('key')) is not None: + args['key'] = key + else: + raise ValueError('Required property \'key\' not present in ContractTemplateCustomProperty JSON') + if (value := _dict.get('value')) is not None: + args['value'] = value + else: + raise ValueError('Required property \'value\' not present in ContractTemplateCustomProperty JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTemplateCustomProperty object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'key') and self.key is not None: + _dict['key'] = self.key + if hasattr(self, 'value') and self.value is not None: + _dict['value'] = self.value + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTemplateCustomProperty object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTemplateCustomProperty') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTemplateCustomProperty') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractTemplateOrganization: + """ + Represents a user and their role in the contract. + + :param str user_id: The user ID associated with the contract. + :param str role: The role of the user in the contract. + """ + + def __init__( + self, + user_id: str, + role: str, + ) -> None: + """ + Initialize a ContractTemplateOrganization object. + + :param str user_id: The user ID associated with the contract. + :param str role: The role of the user in the contract. + """ + self.user_id = user_id + self.role = role + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTemplateOrganization': + """Initialize a ContractTemplateOrganization object from a json dictionary.""" + args = {} + if (user_id := _dict.get('user_id')) is not None: + args['user_id'] = user_id + else: + raise ValueError('Required property \'user_id\' not present in ContractTemplateOrganization JSON') + if (role := _dict.get('role')) is not None: + args['role'] = role + else: + raise ValueError('Required property \'role\' not present in ContractTemplateOrganization JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTemplateOrganization object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'user_id') and self.user_id is not None: + _dict['user_id'] = self.user_id + if hasattr(self, 'role') and self.role is not None: + _dict['role'] = self.role + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTemplateOrganization object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTemplateOrganization') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTemplateOrganization') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractTemplateSLA: + """ + Represents the SLA details of the contract. + + :param str default_element: (optional) The default SLA element. + :param List[ContractTemplateSLAProperty] properties: (optional) List of SLA + properties and their values. + """ + + def __init__( + self, + *, + default_element: Optional[str] = None, + properties: Optional[List['ContractTemplateSLAProperty']] = None, + ) -> None: + """ + Initialize a ContractTemplateSLA object. + + :param str default_element: (optional) The default SLA element. + :param List[ContractTemplateSLAProperty] properties: (optional) List of SLA + properties and their values. + """ + self.default_element = default_element + self.properties = properties + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTemplateSLA': + """Initialize a ContractTemplateSLA object from a json dictionary.""" + args = {} + if (default_element := _dict.get('default_element')) is not None: + args['default_element'] = default_element + if (properties := _dict.get('properties')) is not None: + args['properties'] = [ContractTemplateSLAProperty.from_dict(v) for v in properties] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTemplateSLA object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'default_element') and self.default_element is not None: + _dict['default_element'] = self.default_element + if hasattr(self, 'properties') and self.properties is not None: + properties_list = [] + for v in self.properties: + if isinstance(v, dict): + properties_list.append(v) + else: + properties_list.append(v.to_dict()) + _dict['properties'] = properties_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTemplateSLA object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTemplateSLA') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTemplateSLA') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractTemplateSLAProperty: + """ + Represents an SLA property and its value. + + :param str property: The SLA property name. + :param str value: The value associated with the SLA property. + """ + + def __init__( + self, + property: str, + value: str, + ) -> None: + """ + Initialize a ContractTemplateSLAProperty object. + + :param str property: The SLA property name. + :param str value: The value associated with the SLA property. + """ + self.property = property + self.value = value + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTemplateSLAProperty': + """Initialize a ContractTemplateSLAProperty object from a json dictionary.""" + args = {} + if (property := _dict.get('property')) is not None: + args['property'] = property + else: + raise ValueError('Required property \'property\' not present in ContractTemplateSLAProperty JSON') + if (value := _dict.get('value')) is not None: + args['value'] = value + else: + raise ValueError('Required property \'value\' not present in ContractTemplateSLAProperty JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTemplateSLAProperty object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'property') and self.property is not None: + _dict['property'] = self.property + if hasattr(self, 'value') and self.value is not None: + _dict['value'] = self.value + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTemplateSLAProperty object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTemplateSLAProperty') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTemplateSLAProperty') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractTemplateSupportAndCommunication: + """ + Represents a support and communication channel for the contract. + + :param str channel: The communication channel. + :param str url: The URL associated with the communication channel. + """ + + def __init__( + self, + channel: str, + url: str, + ) -> None: + """ + Initialize a ContractTemplateSupportAndCommunication object. + + :param str channel: The communication channel. + :param str url: The URL associated with the communication channel. + """ + self.channel = channel + self.url = url + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTemplateSupportAndCommunication': + """Initialize a ContractTemplateSupportAndCommunication object from a json dictionary.""" + args = {} + if (channel := _dict.get('channel')) is not None: + args['channel'] = channel + else: + raise ValueError('Required property \'channel\' not present in ContractTemplateSupportAndCommunication JSON') + if (url := _dict.get('url')) is not None: + args['url'] = url + else: + raise ValueError('Required property \'url\' not present in ContractTemplateSupportAndCommunication JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTemplateSupportAndCommunication object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'channel') and self.channel is not None: + _dict['channel'] = self.channel + if hasattr(self, 'url') and self.url is not None: + _dict['url'] = self.url + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTemplateSupportAndCommunication object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTemplateSupportAndCommunication') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTemplateSupportAndCommunication') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractTerms: + """ + Defines the complete structure of a contract terms. + + :param AssetReference asset: (optional) The reference schema for a asset in a + container. + :param str id: (optional) ID of the contract terms. + :param List[ContractTermsDocument] documents: (optional) Collection of contract + terms documents. + :param str error_msg: (optional) An error message, if existing, relating to the + contract terms. + :param Overview overview: (optional) Overview details of a data contract. + :param Description description: (optional) Description details of a data + contract. + :param List[ContractTemplateOrganization] organization: (optional) List of sub + domains to be added within a domain. + :param List[Roles] roles: (optional) List of roles associated with the contract. + :param Pricing price: (optional) Represents the pricing details of the contract. + :param List[ContractTemplateSLA] sla: (optional) Service Level Agreement + details. + :param List[ContractTemplateSupportAndCommunication] support_and_communication: + (optional) Support and communication details for the contract. + :param List[ContractTemplateCustomProperty] custom_properties: (optional) Custom + properties that are not part of the standard contract. + :param ContractTest contract_test: (optional) Contains the contract test status + and related metadata. + :param List[ContractServer] servers: (optional) List of server definitions. + :param List[ContractSchema] schema: (optional) Schema details of the data asset. + """ + + def __init__( + self, + *, + asset: Optional['AssetReference'] = None, + id: Optional[str] = None, + documents: Optional[List['ContractTermsDocument']] = None, + error_msg: Optional[str] = None, + overview: Optional['Overview'] = None, + description: Optional['Description'] = None, + organization: Optional[List['ContractTemplateOrganization']] = None, + roles: Optional[List['Roles']] = None, + price: Optional['Pricing'] = None, + sla: Optional[List['ContractTemplateSLA']] = None, + support_and_communication: Optional[List['ContractTemplateSupportAndCommunication']] = None, + custom_properties: Optional[List['ContractTemplateCustomProperty']] = None, + contract_test: Optional['ContractTest'] = None, + servers: Optional[List['ContractServer']] = None, + schema: Optional[List['ContractSchema']] = None, + ) -> None: + """ + Initialize a ContractTerms object. + + :param AssetReference asset: (optional) The reference schema for a asset in + a container. + :param str id: (optional) ID of the contract terms. + :param List[ContractTermsDocument] documents: (optional) Collection of + contract terms documents. + :param str error_msg: (optional) An error message, if existing, relating to + the contract terms. + :param Overview overview: (optional) Overview details of a data contract. + :param Description description: (optional) Description details of a data + contract. + :param List[ContractTemplateOrganization] organization: (optional) List of + sub domains to be added within a domain. + :param List[Roles] roles: (optional) List of roles associated with the + contract. + :param Pricing price: (optional) Represents the pricing details of the + contract. + :param List[ContractTemplateSLA] sla: (optional) Service Level Agreement + details. + :param List[ContractTemplateSupportAndCommunication] + support_and_communication: (optional) Support and communication details for + the contract. + :param List[ContractTemplateCustomProperty] custom_properties: (optional) + Custom properties that are not part of the standard contract. + :param ContractTest contract_test: (optional) Contains the contract test + status and related metadata. + :param List[ContractServer] servers: (optional) List of server definitions. + :param List[ContractSchema] schema: (optional) Schema details of the data + asset. + """ + self.asset = asset + self.id = id + self.documents = documents + self.error_msg = error_msg + self.overview = overview + self.description = description + self.organization = organization + self.roles = roles + self.price = price + self.sla = sla + self.support_and_communication = support_and_communication + self.custom_properties = custom_properties + self.contract_test = contract_test + self.servers = servers + self.schema = schema + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTerms': + """Initialize a ContractTerms object from a json dictionary.""" + args = {} + if (asset := _dict.get('asset')) is not None: + args['asset'] = AssetReference.from_dict(asset) + if (id := _dict.get('id')) is not None: + args['id'] = id + if (documents := _dict.get('documents')) is not None: + args['documents'] = [ContractTermsDocument.from_dict(v) for v in documents] + if (error_msg := _dict.get('error_msg')) is not None: + args['error_msg'] = error_msg + if (overview := _dict.get('overview')) is not None: + args['overview'] = Overview.from_dict(overview) + if (description := _dict.get('description')) is not None: + args['description'] = Description.from_dict(description) + if (organization := _dict.get('organization')) is not None: + args['organization'] = [ContractTemplateOrganization.from_dict(v) for v in organization] + if (roles := _dict.get('roles')) is not None: + args['roles'] = [Roles.from_dict(v) for v in roles] + if (price := _dict.get('price')) is not None: + args['price'] = Pricing.from_dict(price) + if (sla := _dict.get('sla')) is not None: + args['sla'] = [ContractTemplateSLA.from_dict(v) for v in sla] + if (support_and_communication := _dict.get('support_and_communication')) is not None: + args['support_and_communication'] = [ContractTemplateSupportAndCommunication.from_dict(v) for v in support_and_communication] + if (custom_properties := _dict.get('custom_properties')) is not None: + args['custom_properties'] = [ContractTemplateCustomProperty.from_dict(v) for v in custom_properties] + if (contract_test := _dict.get('contract_test')) is not None: + args['contract_test'] = ContractTest.from_dict(contract_test) + if (servers := _dict.get('servers')) is not None: + args['servers'] = [ContractServer.from_dict(v) for v in servers] + if (schema := _dict.get('schema')) is not None: + args['schema'] = [ContractSchema.from_dict(v) for v in schema] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTerms object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'documents') and self.documents is not None: + documents_list = [] + for v in self.documents: + if isinstance(v, dict): + documents_list.append(v) + else: + documents_list.append(v.to_dict()) + _dict['documents'] = documents_list + if hasattr(self, 'error_msg') and self.error_msg is not None: + _dict['error_msg'] = self.error_msg + if hasattr(self, 'overview') and self.overview is not None: + if isinstance(self.overview, dict): + _dict['overview'] = self.overview + else: + _dict['overview'] = self.overview.to_dict() + if hasattr(self, 'description') and self.description is not None: + if isinstance(self.description, dict): + _dict['description'] = self.description + else: + _dict['description'] = self.description.to_dict() + if hasattr(self, 'organization') and self.organization is not None: + organization_list = [] + for v in self.organization: + if isinstance(v, dict): + organization_list.append(v) + else: + organization_list.append(v.to_dict()) + _dict['organization'] = organization_list + if hasattr(self, 'roles') and self.roles is not None: + roles_list = [] + for v in self.roles: + if isinstance(v, dict): + roles_list.append(v) + else: + roles_list.append(v.to_dict()) + _dict['roles'] = roles_list + if hasattr(self, 'price') and self.price is not None: + if isinstance(self.price, dict): + _dict['price'] = self.price + else: + _dict['price'] = self.price.to_dict() + if hasattr(self, 'sla') and self.sla is not None: + sla_list = [] + for v in self.sla: + if isinstance(v, dict): + sla_list.append(v) + else: + sla_list.append(v.to_dict()) + _dict['sla'] = sla_list + if hasattr(self, 'support_and_communication') and self.support_and_communication is not None: + support_and_communication_list = [] + for v in self.support_and_communication: + if isinstance(v, dict): + support_and_communication_list.append(v) + else: + support_and_communication_list.append(v.to_dict()) + _dict['support_and_communication'] = support_and_communication_list + if hasattr(self, 'custom_properties') and self.custom_properties is not None: + custom_properties_list = [] + for v in self.custom_properties: + if isinstance(v, dict): + custom_properties_list.append(v) + else: + custom_properties_list.append(v.to_dict()) + _dict['custom_properties'] = custom_properties_list + if hasattr(self, 'contract_test') and self.contract_test is not None: + if isinstance(self.contract_test, dict): + _dict['contract_test'] = self.contract_test + else: + _dict['contract_test'] = self.contract_test.to_dict() + if hasattr(self, 'servers') and self.servers is not None: + servers_list = [] + for v in self.servers: + if isinstance(v, dict): + servers_list.append(v) + else: + servers_list.append(v.to_dict()) + _dict['servers'] = servers_list + if hasattr(self, 'schema') and self.schema is not None: + schema_list = [] + for v in self.schema: + if isinstance(v, dict): + schema_list.append(v) + else: + schema_list.append(v.to_dict()) + _dict['schema'] = schema_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTerms object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTerms') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTerms') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractTermsDocument: + """ + Standard contract terms document, which is used for get and list contract terms + responses. + + :param str url: (optional) URL that can be used to retrieve the contract + document. + :param str type: Type of the contract document. + :param str name: Name of the contract document. + :param str id: Id uniquely identifying this document within the contract terms + instance. + :param ContractTermsDocumentAttachment attachment: (optional) Attachment + associated witht the document. + :param str upload_url: (optional) URL which can be used to upload document file. + """ + + def __init__( + self, + type: str, + name: str, + id: str, + *, + url: Optional[str] = None, + attachment: Optional['ContractTermsDocumentAttachment'] = None, + upload_url: Optional[str] = None, + ) -> None: + """ + Initialize a ContractTermsDocument object. + + :param str type: Type of the contract document. + :param str name: Name of the contract document. + :param str id: Id uniquely identifying this document within the contract + terms instance. + :param str url: (optional) URL that can be used to retrieve the contract + document. + :param ContractTermsDocumentAttachment attachment: (optional) Attachment + associated witht the document. + :param str upload_url: (optional) URL which can be used to upload document + file. + """ + self.url = url + self.type = type + self.name = name + self.id = id + self.attachment = attachment + self.upload_url = upload_url + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTermsDocument': + """Initialize a ContractTermsDocument object from a json dictionary.""" + args = {} + if (url := _dict.get('url')) is not None: + args['url'] = url + if (type := _dict.get('type')) is not None: + args['type'] = type + else: + raise ValueError('Required property \'type\' not present in ContractTermsDocument JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + else: + raise ValueError('Required property \'name\' not present in ContractTermsDocument JSON') + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in ContractTermsDocument JSON') + if (attachment := _dict.get('attachment')) is not None: + args['attachment'] = ContractTermsDocumentAttachment.from_dict(attachment) + if (upload_url := _dict.get('upload_url')) is not None: + args['upload_url'] = upload_url + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTermsDocument object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'url') and self.url is not None: + _dict['url'] = self.url + if hasattr(self, 'type') and self.type is not None: + _dict['type'] = self.type + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'attachment') and self.attachment is not None: + if isinstance(self.attachment, dict): + _dict['attachment'] = self.attachment + else: + _dict['attachment'] = self.attachment.to_dict() + if hasattr(self, 'upload_url') and self.upload_url is not None: + _dict['upload_url'] = self.upload_url + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTermsDocument object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTermsDocument') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTermsDocument') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class TypeEnum(str, Enum): + """ + Type of the contract document. + """ + + TERMS_AND_CONDITIONS = 'terms_and_conditions' + SLA = 'sla' + + + +class ContractTermsDocumentAttachment: + """ + Attachment associated witht the document. + + :param str id: (optional) Id representing the corresponding attachment. + """ + + def __init__( + self, + *, + id: Optional[str] = None, + ) -> None: + """ + Initialize a ContractTermsDocumentAttachment object. + + :param str id: (optional) Id representing the corresponding attachment. + """ + self.id = id + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTermsDocumentAttachment': + """Initialize a ContractTermsDocumentAttachment object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTermsDocumentAttachment object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTermsDocumentAttachment object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTermsDocumentAttachment') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTermsDocumentAttachment') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractTermsMoreInfo: + """ + List of links to sources that provide more details on the dataset. + + :param str type: Type of Source Link. + :param str url: Link to source that provide more details on the dataset. + """ + + def __init__( + self, + type: str, + url: str, + ) -> None: + """ + Initialize a ContractTermsMoreInfo object. + + :param str type: Type of Source Link. + :param str url: Link to source that provide more details on the dataset. + """ + self.type = type + self.url = url + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTermsMoreInfo': + """Initialize a ContractTermsMoreInfo object from a json dictionary.""" + args = {} + if (type := _dict.get('type')) is not None: + args['type'] = type + else: + raise ValueError('Required property \'type\' not present in ContractTermsMoreInfo JSON') + if (url := _dict.get('url')) is not None: + args['url'] = url + else: + raise ValueError('Required property \'url\' not present in ContractTermsMoreInfo JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTermsMoreInfo object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'type') and self.type is not None: + _dict['type'] = self.type + if hasattr(self, 'url') and self.url is not None: + _dict['url'] = self.url + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTermsMoreInfo object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTermsMoreInfo') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTermsMoreInfo') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractTest: + """ + Contains the contract test status and related metadata. + + :param str status: Status of the contract test (pass or fail). + :param str last_tested_time: Timestamp of when the contract was last tested. + :param str message: (optional) Optional message or details about the contract + test. + """ + + def __init__( + self, + status: str, + last_tested_time: str, + *, + message: Optional[str] = None, + ) -> None: + """ + Initialize a ContractTest object. + + :param str status: Status of the contract test (pass or fail). + :param str last_tested_time: Timestamp of when the contract was last + tested. + :param str message: (optional) Optional message or details about the + contract test. + """ + self.status = status + self.last_tested_time = last_tested_time + self.message = message + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTest': + """Initialize a ContractTest object from a json dictionary.""" + args = {} + if (status := _dict.get('status')) is not None: + args['status'] = status + else: + raise ValueError('Required property \'status\' not present in ContractTest JSON') + if (last_tested_time := _dict.get('last_tested_time')) is not None: + args['last_tested_time'] = last_tested_time + else: + raise ValueError('Required property \'last_tested_time\' not present in ContractTest JSON') + if (message := _dict.get('message')) is not None: + args['message'] = message + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTest object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'status') and self.status is not None: + _dict['status'] = self.status + if hasattr(self, 'last_tested_time') and self.last_tested_time is not None: + _dict['last_tested_time'] = self.last_tested_time + if hasattr(self, 'message') and self.message is not None: + _dict['message'] = self.message + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTest object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTest') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTest') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class StatusEnum(str, Enum): + """ + Status of the contract test (pass or fail). + """ + + PASS = 'pass' + FAIL = 'fail' + + + +class DataAssetRelationship: + """ + Data members for visualization process. + + :param Visualization visualization: (optional) Data members for visualization. + :param AssetReference asset: The reference schema for a asset in a container. + :param AssetReference related_asset: The reference schema for a asset in a + container. + :param ErrorMessage error: (optional) Contains the code and details. + """ + + def __init__( + self, + asset: 'AssetReference', + related_asset: 'AssetReference', + *, + visualization: Optional['Visualization'] = None, + error: Optional['ErrorMessage'] = None, + ) -> None: + """ + Initialize a DataAssetRelationship object. + + :param AssetReference asset: The reference schema for a asset in a + container. + :param AssetReference related_asset: The reference schema for a asset in a + container. + :param Visualization visualization: (optional) Data members for + visualization. + :param ErrorMessage error: (optional) Contains the code and details. + """ + self.visualization = visualization + self.asset = asset + self.related_asset = related_asset + self.error = error + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataAssetRelationship': + """Initialize a DataAssetRelationship object from a json dictionary.""" + args = {} + if (visualization := _dict.get('visualization')) is not None: + args['visualization'] = Visualization.from_dict(visualization) + if (asset := _dict.get('asset')) is not None: + args['asset'] = AssetReference.from_dict(asset) + else: + raise ValueError('Required property \'asset\' not present in DataAssetRelationship JSON') + if (related_asset := _dict.get('related_asset')) is not None: + args['related_asset'] = AssetReference.from_dict(related_asset) + else: + raise ValueError('Required property \'related_asset\' not present in DataAssetRelationship JSON') + if (error := _dict.get('error')) is not None: + args['error'] = ErrorMessage.from_dict(error) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataAssetRelationship object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'visualization') and self.visualization is not None: + if isinstance(self.visualization, dict): + _dict['visualization'] = self.visualization + else: + _dict['visualization'] = self.visualization.to_dict() + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + if hasattr(self, 'related_asset') and self.related_asset is not None: + if isinstance(self.related_asset, dict): + _dict['related_asset'] = self.related_asset + else: + _dict['related_asset'] = self.related_asset.to_dict() + if hasattr(self, 'error') and self.error is not None: + if isinstance(self.error, dict): + _dict['error'] = self.error + else: + _dict['error'] = self.error.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataAssetRelationship object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataAssetRelationship') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataAssetRelationship') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataAssetVisualizationRes: + """ + Data relationships for the visualization process response. + + :param List[DataAssetRelationship] results: (optional) Data asset Ids and their + related asset Ids. + """ + + def __init__( + self, + *, + results: Optional[List['DataAssetRelationship']] = None, + ) -> None: + """ + Initialize a DataAssetVisualizationRes object. + + :param List[DataAssetRelationship] results: (optional) Data asset Ids and + their related asset Ids. + """ + self.results = results + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataAssetVisualizationRes': + """Initialize a DataAssetVisualizationRes object from a json dictionary.""" + args = {} + if (results := _dict.get('results')) is not None: + args['results'] = [DataAssetRelationship.from_dict(v) for v in results] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataAssetVisualizationRes object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'results') and self.results is not None: + results_list = [] + for v in self.results: + if isinstance(v, dict): + results_list.append(v) + else: + results_list.append(v.to_dict()) + _dict['results'] = results_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataAssetVisualizationRes object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataAssetVisualizationRes') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataAssetVisualizationRes') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProduct: + """ + Data Product. + + :param str id: Data product identifier. + :param DataProductDraftVersionRelease release: (optional) A data product draft + version object. + :param ContainerReference container: Container reference. + :param str name: (optional) Data product name. + :param DataProductVersionSummary latest_release: (optional) Summary of Data + Product Version object. + :param List[DataProductVersionSummary] drafts: (optional) List of draft + summaries of this data product. + """ + + def __init__( + self, + id: str, + container: 'ContainerReference', + *, + release: Optional['DataProductDraftVersionRelease'] = None, + name: Optional[str] = None, + latest_release: Optional['DataProductVersionSummary'] = None, + drafts: Optional[List['DataProductVersionSummary']] = None, + ) -> None: + """ + Initialize a DataProduct object. + + :param str id: Data product identifier. + :param ContainerReference container: Container reference. + :param DataProductDraftVersionRelease release: (optional) A data product + draft version object. + :param str name: (optional) Data product name. + :param DataProductVersionSummary latest_release: (optional) Summary of Data + Product Version object. + :param List[DataProductVersionSummary] drafts: (optional) List of draft + summaries of this data product. + """ + self.id = id + self.release = release + self.container = container + self.name = name + self.latest_release = latest_release + self.drafts = drafts + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProduct': + """Initialize a DataProduct object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProduct JSON') + if (release := _dict.get('release')) is not None: + args['release'] = DataProductDraftVersionRelease.from_dict(release) + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DataProduct JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + if (latest_release := _dict.get('latest_release')) is not None: + args['latest_release'] = DataProductVersionSummary.from_dict(latest_release) + if (drafts := _dict.get('drafts')) is not None: + args['drafts'] = [DataProductVersionSummary.from_dict(v) for v in drafts] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProduct object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'release') and self.release is not None: + if isinstance(self.release, dict): + _dict['release'] = self.release + else: + _dict['release'] = self.release.to_dict() + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'latest_release') and self.latest_release is not None: + if isinstance(self.latest_release, dict): + _dict['latest_release'] = self.latest_release + else: + _dict['latest_release'] = self.latest_release.to_dict() + if hasattr(self, 'drafts') and self.drafts is not None: + drafts_list = [] + for v in self.drafts: + if isinstance(v, dict): + drafts_list.append(v) + else: + drafts_list.append(v.to_dict()) + _dict['drafts'] = drafts_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProduct object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProduct') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProduct') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductCollection: + """ + A collection of data product summaries. + + :param int limit: Set a limit on the number of results returned. + :param FirstPage first: First page in the collection. + :param NextPage next: (optional) Next page in the collection. + :param int total_results: (optional) Indicates the total number of results + returned. + :param List[DataProductSummary] data_products: Collection of data product + summaries. + """ + + def __init__( + self, + limit: int, + first: 'FirstPage', + data_products: List['DataProductSummary'], + *, + next: Optional['NextPage'] = None, + total_results: Optional[int] = None, + ) -> None: + """ + Initialize a DataProductCollection object. + + :param int limit: Set a limit on the number of results returned. + :param FirstPage first: First page in the collection. + :param List[DataProductSummary] data_products: Collection of data product + summaries. + :param NextPage next: (optional) Next page in the collection. + :param int total_results: (optional) Indicates the total number of results + returned. + """ + self.limit = limit + self.first = first + self.next = next + self.total_results = total_results + self.data_products = data_products + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductCollection': + """Initialize a DataProductCollection object from a json dictionary.""" + args = {} + if (limit := _dict.get('limit')) is not None: + args['limit'] = limit + else: + raise ValueError('Required property \'limit\' not present in DataProductCollection JSON') + if (first := _dict.get('first')) is not None: + args['first'] = FirstPage.from_dict(first) + else: + raise ValueError('Required property \'first\' not present in DataProductCollection JSON') + if (next := _dict.get('next')) is not None: + args['next'] = NextPage.from_dict(next) + if (total_results := _dict.get('total_results')) is not None: + args['total_results'] = total_results + if (data_products := _dict.get('data_products')) is not None: + args['data_products'] = [DataProductSummary.from_dict(v) for v in data_products] + else: + raise ValueError('Required property \'data_products\' not present in DataProductCollection JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductCollection object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'limit') and self.limit is not None: + _dict['limit'] = self.limit + if hasattr(self, 'first') and self.first is not None: + if isinstance(self.first, dict): + _dict['first'] = self.first + else: + _dict['first'] = self.first.to_dict() + if hasattr(self, 'next') and self.next is not None: + if isinstance(self.next, dict): + _dict['next'] = self.next + else: + _dict['next'] = self.next.to_dict() + if hasattr(self, 'total_results') and self.total_results is not None: + _dict['total_results'] = self.total_results + if hasattr(self, 'data_products') and self.data_products is not None: + data_products_list = [] + for v in self.data_products: + if isinstance(v, dict): + data_products_list.append(v) + else: + data_products_list.append(v.to_dict()) + _dict['data_products'] = data_products_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductCollection object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductCollection') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductCollection') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductContractTemplate: + """ + Defines the complete structure of a contract template. + + :param ContainerReference container: Container reference. + :param str id: (optional) The identifier of the data product contract template. + :param str creator_id: (optional) The identifier of the user who created the + data product contract template. + :param str created_at: (optional) The timestamp when the data product contract + template was created. + :param str name: (optional) The name of the contract template. + :param ErrorMessage error: (optional) Contains the code and details. + :param ContractTerms contract_terms: (optional) Defines the complete structure + of a contract terms. + """ + + def __init__( + self, + container: 'ContainerReference', + *, + id: Optional[str] = None, + creator_id: Optional[str] = None, + created_at: Optional[str] = None, + name: Optional[str] = None, + error: Optional['ErrorMessage'] = None, + contract_terms: Optional['ContractTerms'] = None, + ) -> None: + """ + Initialize a DataProductContractTemplate object. + + :param ContainerReference container: Container reference. + :param str id: (optional) The identifier of the data product contract + template. + :param str creator_id: (optional) The identifier of the user who created + the data product contract template. + :param str created_at: (optional) The timestamp when the data product + contract template was created. + :param str name: (optional) The name of the contract template. + :param ErrorMessage error: (optional) Contains the code and details. + :param ContractTerms contract_terms: (optional) Defines the complete + structure of a contract terms. + """ + self.container = container + self.id = id + self.creator_id = creator_id + self.created_at = created_at + self.name = name + self.error = error + self.contract_terms = contract_terms + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductContractTemplate': + """Initialize a DataProductContractTemplate object from a json dictionary.""" + args = {} + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DataProductContractTemplate JSON') + if (id := _dict.get('id')) is not None: + args['id'] = id + if (creator_id := _dict.get('creator_id')) is not None: + args['creator_id'] = creator_id + if (created_at := _dict.get('created_at')) is not None: + args['created_at'] = created_at + if (name := _dict.get('name')) is not None: + args['name'] = name + if (error := _dict.get('error')) is not None: + args['error'] = ErrorMessage.from_dict(error) + if (contract_terms := _dict.get('contract_terms')) is not None: + args['contract_terms'] = ContractTerms.from_dict(contract_terms) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductContractTemplate object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'creator_id') and self.creator_id is not None: + _dict['creator_id'] = self.creator_id + if hasattr(self, 'created_at') and self.created_at is not None: + _dict['created_at'] = self.created_at + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'error') and self.error is not None: + if isinstance(self.error, dict): + _dict['error'] = self.error + else: + _dict['error'] = self.error.to_dict() + if hasattr(self, 'contract_terms') and self.contract_terms is not None: + if isinstance(self.contract_terms, dict): + _dict['contract_terms'] = self.contract_terms + else: + _dict['contract_terms'] = self.contract_terms.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductContractTemplate object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductContractTemplate') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductContractTemplate') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductContractTemplateCollection: + """ + A collection of data product contract templates. + + :param List[DataProductContractTemplate] contract_templates: Collection of data + product contract templates. + """ + + def __init__( + self, + contract_templates: List['DataProductContractTemplate'], + ) -> None: + """ + Initialize a DataProductContractTemplateCollection object. + + :param List[DataProductContractTemplate] contract_templates: Collection of + data product contract templates. + """ + self.contract_templates = contract_templates + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductContractTemplateCollection': + """Initialize a DataProductContractTemplateCollection object from a json dictionary.""" + args = {} + if (contract_templates := _dict.get('contract_templates')) is not None: + args['contract_templates'] = [DataProductContractTemplate.from_dict(v) for v in contract_templates] + else: + raise ValueError('Required property \'contract_templates\' not present in DataProductContractTemplateCollection JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductContractTemplateCollection object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'contract_templates') and self.contract_templates is not None: + contract_templates_list = [] + for v in self.contract_templates: + if isinstance(v, dict): + contract_templates_list.append(v) + else: + contract_templates_list.append(v.to_dict()) + _dict['contract_templates'] = contract_templates_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductContractTemplateCollection object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductContractTemplateCollection') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductContractTemplateCollection') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductCustomWorkflowDefinition: + """ + A custom workflow definition to be used to create a workflow to approve a data product + subscription. + + :param str id: (optional) ID of a workflow definition. + """ + + def __init__( + self, + *, + id: Optional[str] = None, + ) -> None: + """ + Initialize a DataProductCustomWorkflowDefinition object. + + :param str id: (optional) ID of a workflow definition. + """ + self.id = id + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductCustomWorkflowDefinition': + """Initialize a DataProductCustomWorkflowDefinition object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductCustomWorkflowDefinition object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductCustomWorkflowDefinition object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductCustomWorkflowDefinition') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductCustomWorkflowDefinition') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductDomain: + """ + The data product domain. + + :param ContainerReference container: Container reference. + :param str trace: (optional) The id to trace the failed domain creations. + :param List[ErrorModelResource] errors: (optional) Set of errors on the sub + domain creation. + :param str name: (optional) The name of the data product domain. + :param str description: (optional) The description of the data product domain. + :param str id: (optional) The identifier of the data product domain. + :param str created_by: (optional) The identifier of the creator of the data + product domain. + :param MemberRolesSchema member_roles: (optional) Member roles of a + corresponding asset. + :param PropertiesSchema properties: (optional) Properties of the corresponding + asset. + :param List[InitializeSubDomain] sub_domains: (optional) List of sub domains to + be added within a domain. + :param ContainerIdentity sub_container: (optional) The identity schema for a IBM + knowledge catalog container (catalog/project/space). + """ + + def __init__( + self, + container: 'ContainerReference', + *, + trace: Optional[str] = None, + errors: Optional[List['ErrorModelResource']] = None, + name: Optional[str] = None, + description: Optional[str] = None, + id: Optional[str] = None, + created_by: Optional[str] = None, + member_roles: Optional['MemberRolesSchema'] = None, + properties: Optional['PropertiesSchema'] = None, + sub_domains: Optional[List['InitializeSubDomain']] = None, + sub_container: Optional['ContainerIdentity'] = None, + ) -> None: + """ + Initialize a DataProductDomain object. + + :param ContainerReference container: Container reference. + :param str trace: (optional) The id to trace the failed domain creations. + :param List[ErrorModelResource] errors: (optional) Set of errors on the sub + domain creation. + :param str name: (optional) The name of the data product domain. + :param str description: (optional) The description of the data product + domain. + :param str id: (optional) The identifier of the data product domain. + :param str created_by: (optional) The identifier of the creator of the data + product domain. + :param MemberRolesSchema member_roles: (optional) Member roles of a + corresponding asset. + :param PropertiesSchema properties: (optional) Properties of the + corresponding asset. + :param List[InitializeSubDomain] sub_domains: (optional) List of sub + domains to be added within a domain. + :param ContainerIdentity sub_container: (optional) The identity schema for + a IBM knowledge catalog container (catalog/project/space). + """ + self.container = container + self.trace = trace + self.errors = errors + self.name = name + self.description = description + self.id = id + self.created_by = created_by + self.member_roles = member_roles + self.properties = properties + self.sub_domains = sub_domains + self.sub_container = sub_container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductDomain': + """Initialize a DataProductDomain object from a json dictionary.""" + args = {} + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DataProductDomain JSON') + if (trace := _dict.get('trace')) is not None: + args['trace'] = trace + if (errors := _dict.get('errors')) is not None: + args['errors'] = [ErrorModelResource.from_dict(v) for v in errors] + if (name := _dict.get('name')) is not None: + args['name'] = name + if (description := _dict.get('description')) is not None: + args['description'] = description + if (id := _dict.get('id')) is not None: + args['id'] = id + if (created_by := _dict.get('created_by')) is not None: + args['created_by'] = created_by + if (member_roles := _dict.get('member_roles')) is not None: + args['member_roles'] = MemberRolesSchema.from_dict(member_roles) + if (properties := _dict.get('properties')) is not None: + args['properties'] = PropertiesSchema.from_dict(properties) + if (sub_domains := _dict.get('sub_domains')) is not None: + args['sub_domains'] = [InitializeSubDomain.from_dict(v) for v in sub_domains] + if (sub_container := _dict.get('sub_container')) is not None: + args['sub_container'] = ContainerIdentity.from_dict(sub_container) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductDomain object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + if hasattr(self, 'trace') and self.trace is not None: + _dict['trace'] = self.trace + if hasattr(self, 'errors') and self.errors is not None: + errors_list = [] + for v in self.errors: + if isinstance(v, dict): + errors_list.append(v) + else: + errors_list.append(v.to_dict()) + _dict['errors'] = errors_list + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'created_by') and self.created_by is not None: + _dict['created_by'] = self.created_by + if hasattr(self, 'member_roles') and self.member_roles is not None: + if isinstance(self.member_roles, dict): + _dict['member_roles'] = self.member_roles + else: + _dict['member_roles'] = self.member_roles.to_dict() + if hasattr(self, 'properties') and self.properties is not None: + if isinstance(self.properties, dict): + _dict['properties'] = self.properties + else: + _dict['properties'] = self.properties.to_dict() + if hasattr(self, 'sub_domains') and self.sub_domains is not None: + sub_domains_list = [] + for v in self.sub_domains: + if isinstance(v, dict): + sub_domains_list.append(v) + else: + sub_domains_list.append(v.to_dict()) + _dict['sub_domains'] = sub_domains_list + if hasattr(self, 'sub_container') and self.sub_container is not None: + if isinstance(self.sub_container, dict): + _dict['sub_container'] = self.sub_container + else: + _dict['sub_container'] = self.sub_container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductDomain object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductDomain') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductDomain') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductDomainCollection: + """ + A collection of data product domains. + + :param List[DataProductDomain] domains: Collection of data product domains. + """ + + def __init__( + self, + domains: List['DataProductDomain'], + ) -> None: + """ + Initialize a DataProductDomainCollection object. + + :param List[DataProductDomain] domains: Collection of data product domains. + """ + self.domains = domains + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductDomainCollection': + """Initialize a DataProductDomainCollection object from a json dictionary.""" + args = {} + if (domains := _dict.get('domains')) is not None: + args['domains'] = [DataProductDomain.from_dict(v) for v in domains] + else: + raise ValueError('Required property \'domains\' not present in DataProductDomainCollection JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductDomainCollection object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'domains') and self.domains is not None: + domains_list = [] + for v in self.domains: + if isinstance(v, dict): + domains_list.append(v) + else: + domains_list.append(v.to_dict()) + _dict['domains'] = domains_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductDomainCollection object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductDomainCollection') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductDomainCollection') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductDraft: + """ + Data Product version draft. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductDraftDataProduct data_product: Data product reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable and + reusable. + :param str description: The description of the data product version. + :param List[str] tags: Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated with + the data product version. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param Domain domain: Domain that the data product version belongs to. If this + is the first version of a data product, this field is required. If this is a new + version of an existing data product, the domain will default to the domain of + the previous version of the data product. + :param List[DataProductPart] parts_out: The outgoing parts of this data product + version to be delivered to consumers. If this is the first version of a data + product, this field defaults to an empty list. If this is a new version of an + existing data product, the data product parts will default to the parts list + from the previous version of the data product. + :param DataProductWorkflows workflows: (optional) The workflows associated with + the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided either + at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for a IBM + knowledge catalog container (catalog/project/space). + :param bool is_restricted: Indicates whether the data product is restricted or + not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a container. + :param str published_by: (optional) The user who published this data product + version. + :param datetime published_at: (optional) The time when this data product version + was published. + :param str created_by: The creator of this data product version. + :param datetime created_at: The time when this data product version was created. + :param dict properties: (optional) Metadata properties on data products. + :param List[DataAssetRelationship] visualization_errors: (optional) Errors + encountered during the visualization creation process. + """ + + def __init__( + self, + version: str, + state: str, + data_product: 'DataProductDraftDataProduct', + name: str, + description: str, + tags: List[str], + types: List[str], + contract_terms: List['ContractTerms'], + domain: 'Domain', + parts_out: List['DataProductPart'], + is_restricted: bool, + id: str, + asset: 'AssetReference', + created_by: str, + created_at: datetime, + *, + use_cases: Optional[List['UseCase']] = None, + workflows: Optional['DataProductWorkflows'] = None, + dataview_enabled: Optional[bool] = None, + comments: Optional[str] = None, + access_control: Optional['AssetListAccessControl'] = None, + last_updated_at: Optional[datetime] = None, + sub_container: Optional['ContainerIdentity'] = None, + published_by: Optional[str] = None, + published_at: Optional[datetime] = None, + properties: Optional[dict] = None, + visualization_errors: Optional[List['DataAssetRelationship']] = None, + ) -> None: + """ + Initialize a DataProductDraft object. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductDraftDataProduct data_product: Data product reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable + and reusable. + :param str description: The description of the data product version. + :param List[str] tags: Tags on the data product. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param Domain domain: Domain that the data product version belongs to. If + this is the first version of a data product, this field is required. If + this is a new version of an existing data product, the domain will default + to the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: The outgoing parts of this data + product version to be delivered to consumers. If this is the first version + of a data product, this field defaults to an empty list. If this is a new + version of an existing data product, the data product parts will default to + the parts list from the previous version of the data product. + :param bool is_restricted: Indicates whether the data product is restricted + or not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a + container. + :param str created_by: The creator of this data product version. + :param datetime created_at: The time when this data product version was + created. + :param List[UseCase] use_cases: (optional) A list of use cases associated + with the data product version. + :param DataProductWorkflows workflows: (optional) The workflows associated + with the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided + either at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control + object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for + a IBM knowledge catalog container (catalog/project/space). + :param str published_by: (optional) The user who published this data + product version. + :param datetime published_at: (optional) The time when this data product + version was published. + :param dict properties: (optional) Metadata properties on data products. + :param List[DataAssetRelationship] visualization_errors: (optional) Errors + encountered during the visualization creation process. + """ + self.version = version + self.state = state + self.data_product = data_product + self.name = name + self.description = description + self.tags = tags + self.use_cases = use_cases + self.types = types + self.contract_terms = contract_terms + self.domain = domain + self.parts_out = parts_out + self.workflows = workflows + self.dataview_enabled = dataview_enabled + self.comments = comments + self.access_control = access_control + self.last_updated_at = last_updated_at + self.sub_container = sub_container + self.is_restricted = is_restricted + self.id = id + self.asset = asset + self.published_by = published_by + self.published_at = published_at + self.created_by = created_by + self.created_at = created_at + self.properties = properties + self.visualization_errors = visualization_errors + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductDraft': + """Initialize a DataProductDraft object from a json dictionary.""" + args = {} + if (version := _dict.get('version')) is not None: + args['version'] = version + else: + raise ValueError('Required property \'version\' not present in DataProductDraft JSON') + if (state := _dict.get('state')) is not None: + args['state'] = state + else: + raise ValueError('Required property \'state\' not present in DataProductDraft JSON') + if (data_product := _dict.get('data_product')) is not None: + args['data_product'] = DataProductDraftDataProduct.from_dict(data_product) + else: + raise ValueError('Required property \'data_product\' not present in DataProductDraft JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + else: + raise ValueError('Required property \'name\' not present in DataProductDraft JSON') + if (description := _dict.get('description')) is not None: + args['description'] = description + else: + raise ValueError('Required property \'description\' not present in DataProductDraft JSON') + if (tags := _dict.get('tags')) is not None: + args['tags'] = tags + else: + raise ValueError('Required property \'tags\' not present in DataProductDraft JSON') + if (use_cases := _dict.get('use_cases')) is not None: + args['use_cases'] = [UseCase.from_dict(v) for v in use_cases] + if (types := _dict.get('types')) is not None: + args['types'] = types + else: + raise ValueError('Required property \'types\' not present in DataProductDraft JSON') + if (contract_terms := _dict.get('contract_terms')) is not None: + args['contract_terms'] = [ContractTerms.from_dict(v) for v in contract_terms] + else: + raise ValueError('Required property \'contract_terms\' not present in DataProductDraft JSON') + if (domain := _dict.get('domain')) is not None: + args['domain'] = Domain.from_dict(domain) + else: + raise ValueError('Required property \'domain\' not present in DataProductDraft JSON') + if (parts_out := _dict.get('parts_out')) is not None: + args['parts_out'] = [DataProductPart.from_dict(v) for v in parts_out] + else: + raise ValueError('Required property \'parts_out\' not present in DataProductDraft JSON') + if (workflows := _dict.get('workflows')) is not None: + args['workflows'] = DataProductWorkflows.from_dict(workflows) + if (dataview_enabled := _dict.get('dataview_enabled')) is not None: + args['dataview_enabled'] = dataview_enabled + if (comments := _dict.get('comments')) is not None: + args['comments'] = comments + if (access_control := _dict.get('access_control')) is not None: + args['access_control'] = AssetListAccessControl.from_dict(access_control) + if (last_updated_at := _dict.get('last_updated_at')) is not None: + args['last_updated_at'] = string_to_datetime(last_updated_at) + if (sub_container := _dict.get('sub_container')) is not None: + args['sub_container'] = ContainerIdentity.from_dict(sub_container) + if (is_restricted := _dict.get('is_restricted')) is not None: + args['is_restricted'] = is_restricted + else: + raise ValueError('Required property \'is_restricted\' not present in DataProductDraft JSON') + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductDraft JSON') + if (asset := _dict.get('asset')) is not None: + args['asset'] = AssetReference.from_dict(asset) + else: + raise ValueError('Required property \'asset\' not present in DataProductDraft JSON') + if (published_by := _dict.get('published_by')) is not None: + args['published_by'] = published_by + if (published_at := _dict.get('published_at')) is not None: + args['published_at'] = string_to_datetime(published_at) + if (created_by := _dict.get('created_by')) is not None: + args['created_by'] = created_by + else: + raise ValueError('Required property \'created_by\' not present in DataProductDraft JSON') + if (created_at := _dict.get('created_at')) is not None: + args['created_at'] = string_to_datetime(created_at) + else: + raise ValueError('Required property \'created_at\' not present in DataProductDraft JSON') + if (properties := _dict.get('properties')) is not None: + args['properties'] = properties + if (visualization_errors := _dict.get('visualization_errors')) is not None: + args['visualization_errors'] = [DataAssetRelationship.from_dict(v) for v in visualization_errors] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductDraft object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'version') and self.version is not None: + _dict['version'] = self.version + if hasattr(self, 'state') and self.state is not None: + _dict['state'] = self.state + if hasattr(self, 'data_product') and self.data_product is not None: + if isinstance(self.data_product, dict): + _dict['data_product'] = self.data_product + else: + _dict['data_product'] = self.data_product.to_dict() + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'tags') and self.tags is not None: + _dict['tags'] = self.tags + if hasattr(self, 'use_cases') and self.use_cases is not None: + use_cases_list = [] + for v in self.use_cases: + if isinstance(v, dict): + use_cases_list.append(v) + else: + use_cases_list.append(v.to_dict()) + _dict['use_cases'] = use_cases_list + if hasattr(self, 'types') and self.types is not None: + _dict['types'] = self.types + if hasattr(self, 'contract_terms') and self.contract_terms is not None: + contract_terms_list = [] + for v in self.contract_terms: + if isinstance(v, dict): + contract_terms_list.append(v) + else: + contract_terms_list.append(v.to_dict()) + _dict['contract_terms'] = contract_terms_list + if hasattr(self, 'domain') and self.domain is not None: + if isinstance(self.domain, dict): + _dict['domain'] = self.domain + else: + _dict['domain'] = self.domain.to_dict() + if hasattr(self, 'parts_out') and self.parts_out is not None: + parts_out_list = [] + for v in self.parts_out: + if isinstance(v, dict): + parts_out_list.append(v) + else: + parts_out_list.append(v.to_dict()) + _dict['parts_out'] = parts_out_list + if hasattr(self, 'workflows') and self.workflows is not None: + if isinstance(self.workflows, dict): + _dict['workflows'] = self.workflows + else: + _dict['workflows'] = self.workflows.to_dict() + if hasattr(self, 'dataview_enabled') and self.dataview_enabled is not None: + _dict['dataview_enabled'] = self.dataview_enabled + if hasattr(self, 'comments') and self.comments is not None: + _dict['comments'] = self.comments + if hasattr(self, 'access_control') and self.access_control is not None: + if isinstance(self.access_control, dict): + _dict['access_control'] = self.access_control + else: + _dict['access_control'] = self.access_control.to_dict() + if hasattr(self, 'last_updated_at') and self.last_updated_at is not None: + _dict['last_updated_at'] = datetime_to_string(self.last_updated_at) + if hasattr(self, 'sub_container') and self.sub_container is not None: + if isinstance(self.sub_container, dict): + _dict['sub_container'] = self.sub_container + else: + _dict['sub_container'] = self.sub_container.to_dict() + if hasattr(self, 'is_restricted') and self.is_restricted is not None: + _dict['is_restricted'] = self.is_restricted + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + if hasattr(self, 'published_by') and self.published_by is not None: + _dict['published_by'] = self.published_by + if hasattr(self, 'published_at') and self.published_at is not None: + _dict['published_at'] = datetime_to_string(self.published_at) + if hasattr(self, 'created_by') and self.created_by is not None: + _dict['created_by'] = self.created_by + if hasattr(self, 'created_at') and self.created_at is not None: + _dict['created_at'] = datetime_to_string(self.created_at) + if hasattr(self, 'properties') and self.properties is not None: + _dict['properties'] = self.properties + if hasattr(self, 'visualization_errors') and self.visualization_errors is not None: + visualization_errors_list = [] + for v in self.visualization_errors: + if isinstance(v, dict): + visualization_errors_list.append(v) + else: + visualization_errors_list.append(v.to_dict()) + _dict['visualization_errors'] = visualization_errors_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductDraft object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductDraft') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductDraft') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class StateEnum(str, Enum): + """ + The state of the data product version. + """ + + DRAFT = 'draft' + AVAILABLE = 'available' + RETIRED = 'retired' + + + class TypesEnum(str, Enum): + """ + types. + """ + + DATA = 'data' + CODE = 'code' + + + +class DataProductDraftCollection: + """ + A collection of data product draft summaries. + + :param int limit: Set a limit on the number of results returned. + :param FirstPage first: First page in the collection. + :param NextPage next: (optional) Next page in the collection. + :param int total_results: (optional) Indicates the total number of results + returned. + :param List[DataProductDraftSummary] drafts: Collection of data product drafts. + """ + + def __init__( + self, + limit: int, + first: 'FirstPage', + drafts: List['DataProductDraftSummary'], + *, + next: Optional['NextPage'] = None, + total_results: Optional[int] = None, + ) -> None: + """ + Initialize a DataProductDraftCollection object. + + :param int limit: Set a limit on the number of results returned. + :param FirstPage first: First page in the collection. + :param List[DataProductDraftSummary] drafts: Collection of data product + drafts. + :param NextPage next: (optional) Next page in the collection. + :param int total_results: (optional) Indicates the total number of results + returned. + """ + self.limit = limit + self.first = first + self.next = next + self.total_results = total_results + self.drafts = drafts + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductDraftCollection': + """Initialize a DataProductDraftCollection object from a json dictionary.""" + args = {} + if (limit := _dict.get('limit')) is not None: + args['limit'] = limit + else: + raise ValueError('Required property \'limit\' not present in DataProductDraftCollection JSON') + if (first := _dict.get('first')) is not None: + args['first'] = FirstPage.from_dict(first) + else: + raise ValueError('Required property \'first\' not present in DataProductDraftCollection JSON') + if (next := _dict.get('next')) is not None: + args['next'] = NextPage.from_dict(next) + if (total_results := _dict.get('total_results')) is not None: + args['total_results'] = total_results + if (drafts := _dict.get('drafts')) is not None: + args['drafts'] = [DataProductDraftSummary.from_dict(v) for v in drafts] + else: + raise ValueError('Required property \'drafts\' not present in DataProductDraftCollection JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductDraftCollection object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'limit') and self.limit is not None: + _dict['limit'] = self.limit + if hasattr(self, 'first') and self.first is not None: + if isinstance(self.first, dict): + _dict['first'] = self.first + else: + _dict['first'] = self.first.to_dict() + if hasattr(self, 'next') and self.next is not None: + if isinstance(self.next, dict): + _dict['next'] = self.next + else: + _dict['next'] = self.next.to_dict() + if hasattr(self, 'total_results') and self.total_results is not None: + _dict['total_results'] = self.total_results + if hasattr(self, 'drafts') and self.drafts is not None: + drafts_list = [] + for v in self.drafts: + if isinstance(v, dict): + drafts_list.append(v) + else: + drafts_list.append(v.to_dict()) + _dict['drafts'] = drafts_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductDraftCollection object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductDraftCollection') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductDraftCollection') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductDraftDataProduct: + """ + Data product reference. + + :param str id: Data product identifier. + :param DataProductDraftVersionRelease release: (optional) A data product draft + version object. + :param ContainerReference container: Container reference. + """ + + def __init__( + self, + id: str, + container: 'ContainerReference', + *, + release: Optional['DataProductDraftVersionRelease'] = None, + ) -> None: + """ + Initialize a DataProductDraftDataProduct object. + + :param str id: Data product identifier. + :param ContainerReference container: Container reference. + :param DataProductDraftVersionRelease release: (optional) A data product + draft version object. + """ + self.id = id + self.release = release + self.container = container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductDraftDataProduct': + """Initialize a DataProductDraftDataProduct object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductDraftDataProduct JSON') + if (release := _dict.get('release')) is not None: + args['release'] = DataProductDraftVersionRelease.from_dict(release) + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DataProductDraftDataProduct JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductDraftDataProduct object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'release') and self.release is not None: + if isinstance(self.release, dict): + _dict['release'] = self.release + else: + _dict['release'] = self.release.to_dict() + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductDraftDataProduct object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductDraftDataProduct') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductDraftDataProduct') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductDraftPrototype: + """ + New data product version input properties. + + :param str version: (optional) The data product version number. + :param str state: (optional) The state of the data product version. If not + specified, the data product version will be created in `draft` state. + :param DataProductIdentity data_product: (optional) Data product identifier. + :param str name: (optional) The name that refers to the new data product + version. If this is a new data product, this value must be specified. If this is + a new version of an existing data product, the name will default to the name of + the previous data product version. A name can contain letters, numbers, + understores, dashes, spaces or periods. A name must contain at least one + non-space character. + :param str description: (optional) Description of the data product version. If + this is a new version of an existing data product, the description will default + to the description of the previous version of the data product. + :param List[str] tags: (optional) Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated with + the data product version. + :param List[str] types: (optional) Types of parts on the data product. + :param List[ContractTerms] contract_terms: (optional) Contract terms binding + various aspects of the data product. + :param Domain domain: (optional) Domain that the data product version belongs + to. If this is the first version of a data product, this field is required. If + this is a new version of an existing data product, the domain will default to + the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: (optional) The outgoing parts of this + data product version to be delivered to consumers. If this is the first version + of a data product, this field defaults to an empty list. If this is a new + version of an existing data product, the data product parts will default to the + parts list from the previous version of the data product. + :param DataProductWorkflows workflows: (optional) The workflows associated with + the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided either + at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for a IBM + knowledge catalog container (catalog/project/space). + :param bool is_restricted: (optional) Indicates whether the data product is + restricted or not. A restricted data product indicates that orders of the data + product requires explicit approval before data is delivered. + :param AssetPrototype asset: New asset input properties. + """ + + def __init__( + self, + asset: 'AssetPrototype', + *, + version: Optional[str] = None, + state: Optional[str] = None, + data_product: Optional['DataProductIdentity'] = None, + name: Optional[str] = None, + description: Optional[str] = None, + tags: Optional[List[str]] = None, + use_cases: Optional[List['UseCase']] = None, + types: Optional[List[str]] = None, + contract_terms: Optional[List['ContractTerms']] = None, + domain: Optional['Domain'] = None, + parts_out: Optional[List['DataProductPart']] = None, + workflows: Optional['DataProductWorkflows'] = None, + dataview_enabled: Optional[bool] = None, + comments: Optional[str] = None, + access_control: Optional['AssetListAccessControl'] = None, + last_updated_at: Optional[datetime] = None, + sub_container: Optional['ContainerIdentity'] = None, + is_restricted: Optional[bool] = None, + ) -> None: + """ + Initialize a DataProductDraftPrototype object. + + :param AssetPrototype asset: New asset input properties. + :param str version: (optional) The data product version number. + :param str state: (optional) The state of the data product version. If not + specified, the data product version will be created in `draft` state. + :param DataProductIdentity data_product: (optional) Data product + identifier. + :param str name: (optional) The name that refers to the new data product + version. If this is a new data product, this value must be specified. If + this is a new version of an existing data product, the name will default to + the name of the previous data product version. A name can contain letters, + numbers, understores, dashes, spaces or periods. A name must contain at + least one non-space character. + :param str description: (optional) Description of the data product version. + If this is a new version of an existing data product, the description will + default to the description of the previous version of the data product. + :param List[str] tags: (optional) Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated + with the data product version. + :param List[str] types: (optional) Types of parts on the data product. + :param List[ContractTerms] contract_terms: (optional) Contract terms + binding various aspects of the data product. + :param Domain domain: (optional) Domain that the data product version + belongs to. If this is the first version of a data product, this field is + required. If this is a new version of an existing data product, the domain + will default to the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: (optional) The outgoing parts of + this data product version to be delivered to consumers. If this is the + first version of a data product, this field defaults to an empty list. If + this is a new version of an existing data product, the data product parts + will default to the parts list from the previous version of the data + product. + :param DataProductWorkflows workflows: (optional) The workflows associated + with the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided + either at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control + object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for + a IBM knowledge catalog container (catalog/project/space). + :param bool is_restricted: (optional) Indicates whether the data product is + restricted or not. A restricted data product indicates that orders of the + data product requires explicit approval before data is delivered. + """ + self.version = version + self.state = state + self.data_product = data_product + self.name = name + self.description = description + self.tags = tags + self.use_cases = use_cases + self.types = types + self.contract_terms = contract_terms + self.domain = domain + self.parts_out = parts_out + self.workflows = workflows + self.dataview_enabled = dataview_enabled + self.comments = comments + self.access_control = access_control + self.last_updated_at = last_updated_at + self.sub_container = sub_container + self.is_restricted = is_restricted + self.asset = asset + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductDraftPrototype': + """Initialize a DataProductDraftPrototype object from a json dictionary.""" + args = {} + if (version := _dict.get('version')) is not None: + args['version'] = version + if (state := _dict.get('state')) is not None: + args['state'] = state + if (data_product := _dict.get('data_product')) is not None: + args['data_product'] = DataProductIdentity.from_dict(data_product) + if (name := _dict.get('name')) is not None: + args['name'] = name + if (description := _dict.get('description')) is not None: + args['description'] = description + if (tags := _dict.get('tags')) is not None: + args['tags'] = tags + if (use_cases := _dict.get('use_cases')) is not None: + args['use_cases'] = [UseCase.from_dict(v) for v in use_cases] + if (types := _dict.get('types')) is not None: + args['types'] = types + if (contract_terms := _dict.get('contract_terms')) is not None: + args['contract_terms'] = [ContractTerms.from_dict(v) for v in contract_terms] + if (domain := _dict.get('domain')) is not None: + args['domain'] = Domain.from_dict(domain) + if (parts_out := _dict.get('parts_out')) is not None: + args['parts_out'] = [DataProductPart.from_dict(v) for v in parts_out] + if (workflows := _dict.get('workflows')) is not None: + args['workflows'] = DataProductWorkflows.from_dict(workflows) + if (dataview_enabled := _dict.get('dataview_enabled')) is not None: + args['dataview_enabled'] = dataview_enabled + if (comments := _dict.get('comments')) is not None: + args['comments'] = comments + if (access_control := _dict.get('access_control')) is not None: + args['access_control'] = AssetListAccessControl.from_dict(access_control) + if (last_updated_at := _dict.get('last_updated_at')) is not None: + args['last_updated_at'] = string_to_datetime(last_updated_at) + if (sub_container := _dict.get('sub_container')) is not None: + args['sub_container'] = ContainerIdentity.from_dict(sub_container) + if (is_restricted := _dict.get('is_restricted')) is not None: + args['is_restricted'] = is_restricted + if (asset := _dict.get('asset')) is not None: + args['asset'] = AssetPrototype.from_dict(asset) + else: + raise ValueError('Required property \'asset\' not present in DataProductDraftPrototype JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductDraftPrototype object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'version') and self.version is not None: + _dict['version'] = self.version + if hasattr(self, 'state') and self.state is not None: + _dict['state'] = self.state + if hasattr(self, 'data_product') and self.data_product is not None: + if isinstance(self.data_product, dict): + _dict['data_product'] = self.data_product + else: + _dict['data_product'] = self.data_product.to_dict() + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'tags') and self.tags is not None: + _dict['tags'] = self.tags + if hasattr(self, 'use_cases') and self.use_cases is not None: + use_cases_list = [] + for v in self.use_cases: + if isinstance(v, dict): + use_cases_list.append(v) + else: + use_cases_list.append(v.to_dict()) + _dict['use_cases'] = use_cases_list + if hasattr(self, 'types') and self.types is not None: + _dict['types'] = self.types + if hasattr(self, 'contract_terms') and self.contract_terms is not None: + contract_terms_list = [] + for v in self.contract_terms: + if isinstance(v, dict): + contract_terms_list.append(v) + else: + contract_terms_list.append(v.to_dict()) + _dict['contract_terms'] = contract_terms_list + if hasattr(self, 'domain') and self.domain is not None: + if isinstance(self.domain, dict): + _dict['domain'] = self.domain + else: + _dict['domain'] = self.domain.to_dict() + if hasattr(self, 'parts_out') and self.parts_out is not None: + parts_out_list = [] + for v in self.parts_out: + if isinstance(v, dict): + parts_out_list.append(v) + else: + parts_out_list.append(v.to_dict()) + _dict['parts_out'] = parts_out_list + if hasattr(self, 'workflows') and self.workflows is not None: + if isinstance(self.workflows, dict): + _dict['workflows'] = self.workflows + else: + _dict['workflows'] = self.workflows.to_dict() + if hasattr(self, 'dataview_enabled') and self.dataview_enabled is not None: + _dict['dataview_enabled'] = self.dataview_enabled + if hasattr(self, 'comments') and self.comments is not None: + _dict['comments'] = self.comments + if hasattr(self, 'access_control') and self.access_control is not None: + if isinstance(self.access_control, dict): + _dict['access_control'] = self.access_control + else: + _dict['access_control'] = self.access_control.to_dict() + if hasattr(self, 'last_updated_at') and self.last_updated_at is not None: + _dict['last_updated_at'] = datetime_to_string(self.last_updated_at) + if hasattr(self, 'sub_container') and self.sub_container is not None: + if isinstance(self.sub_container, dict): + _dict['sub_container'] = self.sub_container + else: + _dict['sub_container'] = self.sub_container.to_dict() + if hasattr(self, 'is_restricted') and self.is_restricted is not None: + _dict['is_restricted'] = self.is_restricted + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductDraftPrototype object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductDraftPrototype') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductDraftPrototype') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class StateEnum(str, Enum): + """ + The state of the data product version. If not specified, the data product version + will be created in `draft` state. + """ + + DRAFT = 'draft' + AVAILABLE = 'available' + RETIRED = 'retired' + + + class TypesEnum(str, Enum): + """ + types. + """ + + DATA = 'data' + CODE = 'code' + + + +class DataProductDraftSummary: + """ + Summary of Data Product Version object. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductDraftSummaryDataProduct data_product: Data product reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable and + reusable. + :param str description: The description of the data product version. + :param List[str] tags: Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated with + the data product version. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param Domain domain: Domain that the data product version belongs to. If this + is the first version of a data product, this field is required. If this is a new + version of an existing data product, the domain will default to the domain of + the previous version of the data product. + :param List[DataProductPart] parts_out: The outgoing parts of this data product + version to be delivered to consumers. If this is the first version of a data + product, this field defaults to an empty list. If this is a new version of an + existing data product, the data product parts will default to the parts list + from the previous version of the data product. + :param DataProductWorkflows workflows: (optional) The workflows associated with + the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided either + at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for a IBM + knowledge catalog container (catalog/project/space). + :param bool is_restricted: Indicates whether the data product is restricted or + not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a container. + """ + + def __init__( + self, + version: str, + state: str, + data_product: 'DataProductDraftSummaryDataProduct', + name: str, + description: str, + tags: List[str], + types: List[str], + contract_terms: List['ContractTerms'], + domain: 'Domain', + parts_out: List['DataProductPart'], + is_restricted: bool, + id: str, + asset: 'AssetReference', + *, + use_cases: Optional[List['UseCase']] = None, + workflows: Optional['DataProductWorkflows'] = None, + dataview_enabled: Optional[bool] = None, + comments: Optional[str] = None, + access_control: Optional['AssetListAccessControl'] = None, + last_updated_at: Optional[datetime] = None, + sub_container: Optional['ContainerIdentity'] = None, + ) -> None: + """ + Initialize a DataProductDraftSummary object. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductDraftSummaryDataProduct data_product: Data product + reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable + and reusable. + :param str description: The description of the data product version. + :param List[str] tags: Tags on the data product. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param Domain domain: Domain that the data product version belongs to. If + this is the first version of a data product, this field is required. If + this is a new version of an existing data product, the domain will default + to the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: The outgoing parts of this data + product version to be delivered to consumers. If this is the first version + of a data product, this field defaults to an empty list. If this is a new + version of an existing data product, the data product parts will default to + the parts list from the previous version of the data product. + :param bool is_restricted: Indicates whether the data product is restricted + or not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a + container. + :param List[UseCase] use_cases: (optional) A list of use cases associated + with the data product version. + :param DataProductWorkflows workflows: (optional) The workflows associated + with the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided + either at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control + object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for + a IBM knowledge catalog container (catalog/project/space). + """ + self.version = version + self.state = state + self.data_product = data_product + self.name = name + self.description = description + self.tags = tags + self.use_cases = use_cases + self.types = types + self.contract_terms = contract_terms + self.domain = domain + self.parts_out = parts_out + self.workflows = workflows + self.dataview_enabled = dataview_enabled + self.comments = comments + self.access_control = access_control + self.last_updated_at = last_updated_at + self.sub_container = sub_container + self.is_restricted = is_restricted + self.id = id + self.asset = asset + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductDraftSummary': + """Initialize a DataProductDraftSummary object from a json dictionary.""" + args = {} + if (version := _dict.get('version')) is not None: + args['version'] = version + else: + raise ValueError('Required property \'version\' not present in DataProductDraftSummary JSON') + if (state := _dict.get('state')) is not None: + args['state'] = state + else: + raise ValueError('Required property \'state\' not present in DataProductDraftSummary JSON') + if (data_product := _dict.get('data_product')) is not None: + args['data_product'] = DataProductDraftSummaryDataProduct.from_dict(data_product) + else: + raise ValueError('Required property \'data_product\' not present in DataProductDraftSummary JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + else: + raise ValueError('Required property \'name\' not present in DataProductDraftSummary JSON') + if (description := _dict.get('description')) is not None: + args['description'] = description + else: + raise ValueError('Required property \'description\' not present in DataProductDraftSummary JSON') + if (tags := _dict.get('tags')) is not None: + args['tags'] = tags + else: + raise ValueError('Required property \'tags\' not present in DataProductDraftSummary JSON') + if (use_cases := _dict.get('use_cases')) is not None: + args['use_cases'] = [UseCase.from_dict(v) for v in use_cases] + if (types := _dict.get('types')) is not None: + args['types'] = types + else: + raise ValueError('Required property \'types\' not present in DataProductDraftSummary JSON') + if (contract_terms := _dict.get('contract_terms')) is not None: + args['contract_terms'] = [ContractTerms.from_dict(v) for v in contract_terms] + else: + raise ValueError('Required property \'contract_terms\' not present in DataProductDraftSummary JSON') + if (domain := _dict.get('domain')) is not None: + args['domain'] = Domain.from_dict(domain) + else: + raise ValueError('Required property \'domain\' not present in DataProductDraftSummary JSON') + if (parts_out := _dict.get('parts_out')) is not None: + args['parts_out'] = [DataProductPart.from_dict(v) for v in parts_out] + else: + raise ValueError('Required property \'parts_out\' not present in DataProductDraftSummary JSON') + if (workflows := _dict.get('workflows')) is not None: + args['workflows'] = DataProductWorkflows.from_dict(workflows) + if (dataview_enabled := _dict.get('dataview_enabled')) is not None: + args['dataview_enabled'] = dataview_enabled + if (comments := _dict.get('comments')) is not None: + args['comments'] = comments + if (access_control := _dict.get('access_control')) is not None: + args['access_control'] = AssetListAccessControl.from_dict(access_control) + if (last_updated_at := _dict.get('last_updated_at')) is not None: + args['last_updated_at'] = string_to_datetime(last_updated_at) + if (sub_container := _dict.get('sub_container')) is not None: + args['sub_container'] = ContainerIdentity.from_dict(sub_container) + if (is_restricted := _dict.get('is_restricted')) is not None: + args['is_restricted'] = is_restricted + else: + raise ValueError('Required property \'is_restricted\' not present in DataProductDraftSummary JSON') + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductDraftSummary JSON') + if (asset := _dict.get('asset')) is not None: + args['asset'] = AssetReference.from_dict(asset) + else: + raise ValueError('Required property \'asset\' not present in DataProductDraftSummary JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductDraftSummary object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'version') and self.version is not None: + _dict['version'] = self.version + if hasattr(self, 'state') and self.state is not None: + _dict['state'] = self.state + if hasattr(self, 'data_product') and self.data_product is not None: + if isinstance(self.data_product, dict): + _dict['data_product'] = self.data_product + else: + _dict['data_product'] = self.data_product.to_dict() + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'tags') and self.tags is not None: + _dict['tags'] = self.tags + if hasattr(self, 'use_cases') and self.use_cases is not None: + use_cases_list = [] + for v in self.use_cases: + if isinstance(v, dict): + use_cases_list.append(v) + else: + use_cases_list.append(v.to_dict()) + _dict['use_cases'] = use_cases_list + if hasattr(self, 'types') and self.types is not None: + _dict['types'] = self.types + if hasattr(self, 'contract_terms') and self.contract_terms is not None: + contract_terms_list = [] + for v in self.contract_terms: + if isinstance(v, dict): + contract_terms_list.append(v) + else: + contract_terms_list.append(v.to_dict()) + _dict['contract_terms'] = contract_terms_list + if hasattr(self, 'domain') and self.domain is not None: + if isinstance(self.domain, dict): + _dict['domain'] = self.domain + else: + _dict['domain'] = self.domain.to_dict() + if hasattr(self, 'parts_out') and self.parts_out is not None: + parts_out_list = [] + for v in self.parts_out: + if isinstance(v, dict): + parts_out_list.append(v) + else: + parts_out_list.append(v.to_dict()) + _dict['parts_out'] = parts_out_list + if hasattr(self, 'workflows') and self.workflows is not None: + if isinstance(self.workflows, dict): + _dict['workflows'] = self.workflows + else: + _dict['workflows'] = self.workflows.to_dict() + if hasattr(self, 'dataview_enabled') and self.dataview_enabled is not None: + _dict['dataview_enabled'] = self.dataview_enabled + if hasattr(self, 'comments') and self.comments is not None: + _dict['comments'] = self.comments + if hasattr(self, 'access_control') and self.access_control is not None: + if isinstance(self.access_control, dict): + _dict['access_control'] = self.access_control + else: + _dict['access_control'] = self.access_control.to_dict() + if hasattr(self, 'last_updated_at') and self.last_updated_at is not None: + _dict['last_updated_at'] = datetime_to_string(self.last_updated_at) + if hasattr(self, 'sub_container') and self.sub_container is not None: + if isinstance(self.sub_container, dict): + _dict['sub_container'] = self.sub_container + else: + _dict['sub_container'] = self.sub_container.to_dict() + if hasattr(self, 'is_restricted') and self.is_restricted is not None: + _dict['is_restricted'] = self.is_restricted + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductDraftSummary object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductDraftSummary') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductDraftSummary') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class StateEnum(str, Enum): + """ + The state of the data product version. + """ + + DRAFT = 'draft' + AVAILABLE = 'available' + RETIRED = 'retired' + + + class TypesEnum(str, Enum): + """ + types. + """ + + DATA = 'data' + CODE = 'code' + + + +class DataProductDraftSummaryDataProduct: + """ + Data product reference. + + :param str id: Data product identifier. + :param DataProductDraftVersionRelease release: (optional) A data product draft + version object. + :param ContainerReference container: Container reference. + """ + + def __init__( + self, + id: str, + container: 'ContainerReference', + *, + release: Optional['DataProductDraftVersionRelease'] = None, + ) -> None: + """ + Initialize a DataProductDraftSummaryDataProduct object. + + :param str id: Data product identifier. + :param ContainerReference container: Container reference. + :param DataProductDraftVersionRelease release: (optional) A data product + draft version object. + """ + self.id = id + self.release = release + self.container = container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductDraftSummaryDataProduct': + """Initialize a DataProductDraftSummaryDataProduct object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductDraftSummaryDataProduct JSON') + if (release := _dict.get('release')) is not None: + args['release'] = DataProductDraftVersionRelease.from_dict(release) + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DataProductDraftSummaryDataProduct JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductDraftSummaryDataProduct object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'release') and self.release is not None: + if isinstance(self.release, dict): + _dict['release'] = self.release + else: + _dict['release'] = self.release.to_dict() + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductDraftSummaryDataProduct object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductDraftSummaryDataProduct') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductDraftSummaryDataProduct') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductDraftVersionRelease: + """ + A data product draft version object. + + :param str id: (optional) ID of a draft version of data product. + """ + + def __init__( + self, + *, + id: Optional[str] = None, + ) -> None: + """ + Initialize a DataProductDraftVersionRelease object. + + :param str id: (optional) ID of a draft version of data product. + """ + self.id = id + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductDraftVersionRelease': + """Initialize a DataProductDraftVersionRelease object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductDraftVersionRelease object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductDraftVersionRelease object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductDraftVersionRelease') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductDraftVersionRelease') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductIdentity: + """ + Data product identifier. + + :param str id: Data product identifier. + :param DataProductDraftVersionRelease release: (optional) A data product draft + version object. + """ + + def __init__( + self, + id: str, + *, + release: Optional['DataProductDraftVersionRelease'] = None, + ) -> None: + """ + Initialize a DataProductIdentity object. + + :param str id: Data product identifier. + :param DataProductDraftVersionRelease release: (optional) A data product + draft version object. + """ + self.id = id + self.release = release + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductIdentity': + """Initialize a DataProductIdentity object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductIdentity JSON') + if (release := _dict.get('release')) is not None: + args['release'] = DataProductDraftVersionRelease.from_dict(release) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductIdentity object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'release') and self.release is not None: + if isinstance(self.release, dict): + _dict['release'] = self.release + else: + _dict['release'] = self.release.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductIdentity object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductIdentity') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductIdentity') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductOrderAccessRequest: + """ + The approval workflows associated with the data product version. + + :param List[str] task_assignee_users: (optional) The workflow approvers + associated with the data product version. + :param List[str] pre_approved_users: (optional) The list of users or groups + whose request will get pre-approved associated with the data product version. + :param DataProductCustomWorkflowDefinition custom_workflow_definition: + (optional) A custom workflow definition to be used to create a workflow to + approve a data product subscription. + """ + + def __init__( + self, + *, + task_assignee_users: Optional[List[str]] = None, + pre_approved_users: Optional[List[str]] = None, + custom_workflow_definition: Optional['DataProductCustomWorkflowDefinition'] = None, + ) -> None: + """ + Initialize a DataProductOrderAccessRequest object. + + :param List[str] task_assignee_users: (optional) The workflow approvers + associated with the data product version. + :param List[str] pre_approved_users: (optional) The list of users or groups + whose request will get pre-approved associated with the data product + version. + :param DataProductCustomWorkflowDefinition custom_workflow_definition: + (optional) A custom workflow definition to be used to create a workflow to + approve a data product subscription. + """ + self.task_assignee_users = task_assignee_users + self.pre_approved_users = pre_approved_users + self.custom_workflow_definition = custom_workflow_definition + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductOrderAccessRequest': + """Initialize a DataProductOrderAccessRequest object from a json dictionary.""" + args = {} + if (task_assignee_users := _dict.get('task_assignee_users')) is not None: + args['task_assignee_users'] = task_assignee_users + if (pre_approved_users := _dict.get('pre_approved_users')) is not None: + args['pre_approved_users'] = pre_approved_users + if (custom_workflow_definition := _dict.get('custom_workflow_definition')) is not None: + args['custom_workflow_definition'] = DataProductCustomWorkflowDefinition.from_dict(custom_workflow_definition) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductOrderAccessRequest object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'task_assignee_users') and self.task_assignee_users is not None: + _dict['task_assignee_users'] = self.task_assignee_users + if hasattr(self, 'pre_approved_users') and self.pre_approved_users is not None: + _dict['pre_approved_users'] = self.pre_approved_users + if hasattr(self, 'custom_workflow_definition') and self.custom_workflow_definition is not None: + if isinstance(self.custom_workflow_definition, dict): + _dict['custom_workflow_definition'] = self.custom_workflow_definition + else: + _dict['custom_workflow_definition'] = self.custom_workflow_definition.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductOrderAccessRequest object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductOrderAccessRequest') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductOrderAccessRequest') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductPart: + """ + Data Product Part. + + :param AssetPartReference asset: The asset represented in this part. + :param List[DeliveryMethod] delivery_methods: (optional) Delivery methods + describing the delivery options available for this part. + """ + + def __init__( + self, + asset: 'AssetPartReference', + *, + delivery_methods: Optional[List['DeliveryMethod']] = None, + ) -> None: + """ + Initialize a DataProductPart object. + + :param AssetPartReference asset: The asset represented in this part. + :param List[DeliveryMethod] delivery_methods: (optional) Delivery methods + describing the delivery options available for this part. + """ + self.asset = asset + self.delivery_methods = delivery_methods + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductPart': + """Initialize a DataProductPart object from a json dictionary.""" + args = {} + if (asset := _dict.get('asset')) is not None: + args['asset'] = AssetPartReference.from_dict(asset) + else: + raise ValueError('Required property \'asset\' not present in DataProductPart JSON') + if (delivery_methods := _dict.get('delivery_methods')) is not None: + args['delivery_methods'] = [DeliveryMethod.from_dict(v) for v in delivery_methods] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductPart object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + if hasattr(self, 'delivery_methods') and self.delivery_methods is not None: + delivery_methods_list = [] + for v in self.delivery_methods: + if isinstance(v, dict): + delivery_methods_list.append(v) + else: + delivery_methods_list.append(v.to_dict()) + _dict['delivery_methods'] = delivery_methods_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductPart object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductPart') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductPart') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductRelease: + """ + Data Product version release. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductReleaseDataProduct data_product: Data product reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable and + reusable. + :param str description: The description of the data product version. + :param List[str] tags: Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated with + the data product version. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param Domain domain: Domain that the data product version belongs to. If this + is the first version of a data product, this field is required. If this is a new + version of an existing data product, the domain will default to the domain of + the previous version of the data product. + :param List[DataProductPart] parts_out: The outgoing parts of this data product + version to be delivered to consumers. If this is the first version of a data + product, this field defaults to an empty list. If this is a new version of an + existing data product, the data product parts will default to the parts list + from the previous version of the data product. + :param DataProductWorkflows workflows: (optional) The workflows associated with + the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided either + at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for a IBM + knowledge catalog container (catalog/project/space). + :param bool is_restricted: Indicates whether the data product is restricted or + not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a container. + :param str published_by: (optional) The user who published this data product + version. + :param datetime published_at: (optional) The time when this data product version + was published. + :param str created_by: The creator of this data product version. + :param datetime created_at: The time when this data product version was created. + :param dict properties: (optional) Metadata properties on data products. + :param List[DataAssetRelationship] visualization_errors: (optional) Errors + encountered during the visualization creation process. + """ + + def __init__( + self, + version: str, + state: str, + data_product: 'DataProductReleaseDataProduct', + name: str, + description: str, + tags: List[str], + types: List[str], + contract_terms: List['ContractTerms'], + domain: 'Domain', + parts_out: List['DataProductPart'], + is_restricted: bool, + id: str, + asset: 'AssetReference', + created_by: str, + created_at: datetime, + *, + use_cases: Optional[List['UseCase']] = None, + workflows: Optional['DataProductWorkflows'] = None, + dataview_enabled: Optional[bool] = None, + comments: Optional[str] = None, + access_control: Optional['AssetListAccessControl'] = None, + last_updated_at: Optional[datetime] = None, + sub_container: Optional['ContainerIdentity'] = None, + published_by: Optional[str] = None, + published_at: Optional[datetime] = None, + properties: Optional[dict] = None, + visualization_errors: Optional[List['DataAssetRelationship']] = None, + ) -> None: + """ + Initialize a DataProductRelease object. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductReleaseDataProduct data_product: Data product reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable + and reusable. + :param str description: The description of the data product version. + :param List[str] tags: Tags on the data product. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param Domain domain: Domain that the data product version belongs to. If + this is the first version of a data product, this field is required. If + this is a new version of an existing data product, the domain will default + to the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: The outgoing parts of this data + product version to be delivered to consumers. If this is the first version + of a data product, this field defaults to an empty list. If this is a new + version of an existing data product, the data product parts will default to + the parts list from the previous version of the data product. + :param bool is_restricted: Indicates whether the data product is restricted + or not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a + container. + :param str created_by: The creator of this data product version. + :param datetime created_at: The time when this data product version was + created. + :param List[UseCase] use_cases: (optional) A list of use cases associated + with the data product version. + :param DataProductWorkflows workflows: (optional) The workflows associated + with the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided + either at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control + object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for + a IBM knowledge catalog container (catalog/project/space). + :param str published_by: (optional) The user who published this data + product version. + :param datetime published_at: (optional) The time when this data product + version was published. + :param dict properties: (optional) Metadata properties on data products. + :param List[DataAssetRelationship] visualization_errors: (optional) Errors + encountered during the visualization creation process. + """ + self.version = version + self.state = state + self.data_product = data_product + self.name = name + self.description = description + self.tags = tags + self.use_cases = use_cases + self.types = types + self.contract_terms = contract_terms + self.domain = domain + self.parts_out = parts_out + self.workflows = workflows + self.dataview_enabled = dataview_enabled + self.comments = comments + self.access_control = access_control + self.last_updated_at = last_updated_at + self.sub_container = sub_container + self.is_restricted = is_restricted + self.id = id + self.asset = asset + self.published_by = published_by + self.published_at = published_at + self.created_by = created_by + self.created_at = created_at + self.properties = properties + self.visualization_errors = visualization_errors + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductRelease': + """Initialize a DataProductRelease object from a json dictionary.""" + args = {} + if (version := _dict.get('version')) is not None: + args['version'] = version + else: + raise ValueError('Required property \'version\' not present in DataProductRelease JSON') + if (state := _dict.get('state')) is not None: + args['state'] = state + else: + raise ValueError('Required property \'state\' not present in DataProductRelease JSON') + if (data_product := _dict.get('data_product')) is not None: + args['data_product'] = DataProductReleaseDataProduct.from_dict(data_product) + else: + raise ValueError('Required property \'data_product\' not present in DataProductRelease JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + else: + raise ValueError('Required property \'name\' not present in DataProductRelease JSON') + if (description := _dict.get('description')) is not None: + args['description'] = description + else: + raise ValueError('Required property \'description\' not present in DataProductRelease JSON') + if (tags := _dict.get('tags')) is not None: + args['tags'] = tags + else: + raise ValueError('Required property \'tags\' not present in DataProductRelease JSON') + if (use_cases := _dict.get('use_cases')) is not None: + args['use_cases'] = [UseCase.from_dict(v) for v in use_cases] + if (types := _dict.get('types')) is not None: + args['types'] = types + else: + raise ValueError('Required property \'types\' not present in DataProductRelease JSON') + if (contract_terms := _dict.get('contract_terms')) is not None: + args['contract_terms'] = [ContractTerms.from_dict(v) for v in contract_terms] + else: + raise ValueError('Required property \'contract_terms\' not present in DataProductRelease JSON') + if (domain := _dict.get('domain')) is not None: + args['domain'] = Domain.from_dict(domain) + else: + raise ValueError('Required property \'domain\' not present in DataProductRelease JSON') + if (parts_out := _dict.get('parts_out')) is not None: + args['parts_out'] = [DataProductPart.from_dict(v) for v in parts_out] + else: + raise ValueError('Required property \'parts_out\' not present in DataProductRelease JSON') + if (workflows := _dict.get('workflows')) is not None: + args['workflows'] = DataProductWorkflows.from_dict(workflows) + if (dataview_enabled := _dict.get('dataview_enabled')) is not None: + args['dataview_enabled'] = dataview_enabled + if (comments := _dict.get('comments')) is not None: + args['comments'] = comments + if (access_control := _dict.get('access_control')) is not None: + args['access_control'] = AssetListAccessControl.from_dict(access_control) + if (last_updated_at := _dict.get('last_updated_at')) is not None: + args['last_updated_at'] = string_to_datetime(last_updated_at) + if (sub_container := _dict.get('sub_container')) is not None: + args['sub_container'] = ContainerIdentity.from_dict(sub_container) + if (is_restricted := _dict.get('is_restricted')) is not None: + args['is_restricted'] = is_restricted + else: + raise ValueError('Required property \'is_restricted\' not present in DataProductRelease JSON') + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductRelease JSON') + if (asset := _dict.get('asset')) is not None: + args['asset'] = AssetReference.from_dict(asset) + else: + raise ValueError('Required property \'asset\' not present in DataProductRelease JSON') + if (published_by := _dict.get('published_by')) is not None: + args['published_by'] = published_by + if (published_at := _dict.get('published_at')) is not None: + args['published_at'] = string_to_datetime(published_at) + if (created_by := _dict.get('created_by')) is not None: + args['created_by'] = created_by + else: + raise ValueError('Required property \'created_by\' not present in DataProductRelease JSON') + if (created_at := _dict.get('created_at')) is not None: + args['created_at'] = string_to_datetime(created_at) + else: + raise ValueError('Required property \'created_at\' not present in DataProductRelease JSON') + if (properties := _dict.get('properties')) is not None: + args['properties'] = properties + if (visualization_errors := _dict.get('visualization_errors')) is not None: + args['visualization_errors'] = [DataAssetRelationship.from_dict(v) for v in visualization_errors] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductRelease object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'version') and self.version is not None: + _dict['version'] = self.version + if hasattr(self, 'state') and self.state is not None: + _dict['state'] = self.state + if hasattr(self, 'data_product') and self.data_product is not None: + if isinstance(self.data_product, dict): + _dict['data_product'] = self.data_product + else: + _dict['data_product'] = self.data_product.to_dict() + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'tags') and self.tags is not None: + _dict['tags'] = self.tags + if hasattr(self, 'use_cases') and self.use_cases is not None: + use_cases_list = [] + for v in self.use_cases: + if isinstance(v, dict): + use_cases_list.append(v) + else: + use_cases_list.append(v.to_dict()) + _dict['use_cases'] = use_cases_list + if hasattr(self, 'types') and self.types is not None: + _dict['types'] = self.types + if hasattr(self, 'contract_terms') and self.contract_terms is not None: + contract_terms_list = [] + for v in self.contract_terms: + if isinstance(v, dict): + contract_terms_list.append(v) + else: + contract_terms_list.append(v.to_dict()) + _dict['contract_terms'] = contract_terms_list + if hasattr(self, 'domain') and self.domain is not None: + if isinstance(self.domain, dict): + _dict['domain'] = self.domain + else: + _dict['domain'] = self.domain.to_dict() + if hasattr(self, 'parts_out') and self.parts_out is not None: + parts_out_list = [] + for v in self.parts_out: + if isinstance(v, dict): + parts_out_list.append(v) + else: + parts_out_list.append(v.to_dict()) + _dict['parts_out'] = parts_out_list + if hasattr(self, 'workflows') and self.workflows is not None: + if isinstance(self.workflows, dict): + _dict['workflows'] = self.workflows + else: + _dict['workflows'] = self.workflows.to_dict() + if hasattr(self, 'dataview_enabled') and self.dataview_enabled is not None: + _dict['dataview_enabled'] = self.dataview_enabled + if hasattr(self, 'comments') and self.comments is not None: + _dict['comments'] = self.comments + if hasattr(self, 'access_control') and self.access_control is not None: + if isinstance(self.access_control, dict): + _dict['access_control'] = self.access_control + else: + _dict['access_control'] = self.access_control.to_dict() + if hasattr(self, 'last_updated_at') and self.last_updated_at is not None: + _dict['last_updated_at'] = datetime_to_string(self.last_updated_at) + if hasattr(self, 'sub_container') and self.sub_container is not None: + if isinstance(self.sub_container, dict): + _dict['sub_container'] = self.sub_container + else: + _dict['sub_container'] = self.sub_container.to_dict() + if hasattr(self, 'is_restricted') and self.is_restricted is not None: + _dict['is_restricted'] = self.is_restricted + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + if hasattr(self, 'published_by') and self.published_by is not None: + _dict['published_by'] = self.published_by + if hasattr(self, 'published_at') and self.published_at is not None: + _dict['published_at'] = datetime_to_string(self.published_at) + if hasattr(self, 'created_by') and self.created_by is not None: + _dict['created_by'] = self.created_by + if hasattr(self, 'created_at') and self.created_at is not None: + _dict['created_at'] = datetime_to_string(self.created_at) + if hasattr(self, 'properties') and self.properties is not None: + _dict['properties'] = self.properties + if hasattr(self, 'visualization_errors') and self.visualization_errors is not None: + visualization_errors_list = [] + for v in self.visualization_errors: + if isinstance(v, dict): + visualization_errors_list.append(v) + else: + visualization_errors_list.append(v.to_dict()) + _dict['visualization_errors'] = visualization_errors_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductRelease object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductRelease') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductRelease') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class StateEnum(str, Enum): + """ + The state of the data product version. + """ + + DRAFT = 'draft' + AVAILABLE = 'available' + RETIRED = 'retired' + + + class TypesEnum(str, Enum): + """ + types. + """ + + DATA = 'data' + CODE = 'code' + + + +class DataProductReleaseCollection: + """ + A collection of data product release summaries. + + :param int limit: Set a limit on the number of results returned. + :param FirstPage first: First page in the collection. + :param NextPage next: (optional) Next page in the collection. + :param int total_results: (optional) Indicates the total number of results + returned. + :param List[DataProductReleaseSummary] releases: Collection of data product + releases. + """ + + def __init__( + self, + limit: int, + first: 'FirstPage', + releases: List['DataProductReleaseSummary'], + *, + next: Optional['NextPage'] = None, + total_results: Optional[int] = None, + ) -> None: + """ + Initialize a DataProductReleaseCollection object. + + :param int limit: Set a limit on the number of results returned. + :param FirstPage first: First page in the collection. + :param List[DataProductReleaseSummary] releases: Collection of data product + releases. + :param NextPage next: (optional) Next page in the collection. + :param int total_results: (optional) Indicates the total number of results + returned. + """ + self.limit = limit + self.first = first + self.next = next + self.total_results = total_results + self.releases = releases + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductReleaseCollection': + """Initialize a DataProductReleaseCollection object from a json dictionary.""" + args = {} + if (limit := _dict.get('limit')) is not None: + args['limit'] = limit + else: + raise ValueError('Required property \'limit\' not present in DataProductReleaseCollection JSON') + if (first := _dict.get('first')) is not None: + args['first'] = FirstPage.from_dict(first) + else: + raise ValueError('Required property \'first\' not present in DataProductReleaseCollection JSON') + if (next := _dict.get('next')) is not None: + args['next'] = NextPage.from_dict(next) + if (total_results := _dict.get('total_results')) is not None: + args['total_results'] = total_results + if (releases := _dict.get('releases')) is not None: + args['releases'] = [DataProductReleaseSummary.from_dict(v) for v in releases] + else: + raise ValueError('Required property \'releases\' not present in DataProductReleaseCollection JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductReleaseCollection object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'limit') and self.limit is not None: + _dict['limit'] = self.limit + if hasattr(self, 'first') and self.first is not None: + if isinstance(self.first, dict): + _dict['first'] = self.first + else: + _dict['first'] = self.first.to_dict() + if hasattr(self, 'next') and self.next is not None: + if isinstance(self.next, dict): + _dict['next'] = self.next + else: + _dict['next'] = self.next.to_dict() + if hasattr(self, 'total_results') and self.total_results is not None: + _dict['total_results'] = self.total_results + if hasattr(self, 'releases') and self.releases is not None: + releases_list = [] + for v in self.releases: + if isinstance(v, dict): + releases_list.append(v) + else: + releases_list.append(v.to_dict()) + _dict['releases'] = releases_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductReleaseCollection object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductReleaseCollection') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductReleaseCollection') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductReleaseDataProduct: + """ + Data product reference. + + :param str id: Data product identifier. + :param DataProductDraftVersionRelease release: (optional) A data product draft + version object. + :param ContainerReference container: Container reference. + """ + + def __init__( + self, + id: str, + container: 'ContainerReference', + *, + release: Optional['DataProductDraftVersionRelease'] = None, + ) -> None: + """ + Initialize a DataProductReleaseDataProduct object. + + :param str id: Data product identifier. + :param ContainerReference container: Container reference. + :param DataProductDraftVersionRelease release: (optional) A data product + draft version object. + """ + self.id = id + self.release = release + self.container = container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductReleaseDataProduct': + """Initialize a DataProductReleaseDataProduct object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductReleaseDataProduct JSON') + if (release := _dict.get('release')) is not None: + args['release'] = DataProductDraftVersionRelease.from_dict(release) + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DataProductReleaseDataProduct JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductReleaseDataProduct object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'release') and self.release is not None: + if isinstance(self.release, dict): + _dict['release'] = self.release + else: + _dict['release'] = self.release.to_dict() + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductReleaseDataProduct object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductReleaseDataProduct') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductReleaseDataProduct') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductReleaseSummary: + """ + Summary of Data Product Version object. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductReleaseSummaryDataProduct data_product: Data product + reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable and + reusable. + :param str description: The description of the data product version. + :param List[str] tags: (optional) Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated with + the data product version. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param Domain domain: (optional) Domain that the data product version belongs + to. If this is the first version of a data product, this field is required. If + this is a new version of an existing data product, the domain will default to + the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: (optional) The outgoing parts of this + data product version to be delivered to consumers. If this is the first version + of a data product, this field defaults to an empty list. If this is a new + version of an existing data product, the data product parts will default to the + parts list from the previous version of the data product. + :param DataProductWorkflows workflows: (optional) The workflows associated with + the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided either + at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for a IBM + knowledge catalog container (catalog/project/space). + :param bool is_restricted: Indicates whether the data product is restricted or + not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a container. + """ + + def __init__( + self, + version: str, + state: str, + data_product: 'DataProductReleaseSummaryDataProduct', + name: str, + description: str, + types: List[str], + contract_terms: List['ContractTerms'], + is_restricted: bool, + id: str, + asset: 'AssetReference', + *, + tags: Optional[List[str]] = None, + use_cases: Optional[List['UseCase']] = None, + domain: Optional['Domain'] = None, + parts_out: Optional[List['DataProductPart']] = None, + workflows: Optional['DataProductWorkflows'] = None, + dataview_enabled: Optional[bool] = None, + comments: Optional[str] = None, + access_control: Optional['AssetListAccessControl'] = None, + last_updated_at: Optional[datetime] = None, + sub_container: Optional['ContainerIdentity'] = None, + ) -> None: + """ + Initialize a DataProductReleaseSummary object. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductReleaseSummaryDataProduct data_product: Data product + reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable + and reusable. + :param str description: The description of the data product version. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param bool is_restricted: Indicates whether the data product is restricted + or not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a + container. + :param List[str] tags: (optional) Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated + with the data product version. + :param Domain domain: (optional) Domain that the data product version + belongs to. If this is the first version of a data product, this field is + required. If this is a new version of an existing data product, the domain + will default to the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: (optional) The outgoing parts of + this data product version to be delivered to consumers. If this is the + first version of a data product, this field defaults to an empty list. If + this is a new version of an existing data product, the data product parts + will default to the parts list from the previous version of the data + product. + :param DataProductWorkflows workflows: (optional) The workflows associated + with the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided + either at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control + object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for + a IBM knowledge catalog container (catalog/project/space). + """ + self.version = version + self.state = state + self.data_product = data_product + self.name = name + self.description = description + self.tags = tags + self.use_cases = use_cases + self.types = types + self.contract_terms = contract_terms + self.domain = domain + self.parts_out = parts_out + self.workflows = workflows + self.dataview_enabled = dataview_enabled + self.comments = comments + self.access_control = access_control + self.last_updated_at = last_updated_at + self.sub_container = sub_container + self.is_restricted = is_restricted + self.id = id + self.asset = asset + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductReleaseSummary': + """Initialize a DataProductReleaseSummary object from a json dictionary.""" + args = {} + if (version := _dict.get('version')) is not None: + args['version'] = version + else: + raise ValueError('Required property \'version\' not present in DataProductReleaseSummary JSON') + if (state := _dict.get('state')) is not None: + args['state'] = state + else: + raise ValueError('Required property \'state\' not present in DataProductReleaseSummary JSON') + if (data_product := _dict.get('data_product')) is not None: + args['data_product'] = DataProductReleaseSummaryDataProduct.from_dict(data_product) + else: + raise ValueError('Required property \'data_product\' not present in DataProductReleaseSummary JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + else: + raise ValueError('Required property \'name\' not present in DataProductReleaseSummary JSON') + if (description := _dict.get('description')) is not None: + args['description'] = description + else: + raise ValueError('Required property \'description\' not present in DataProductReleaseSummary JSON') + if (tags := _dict.get('tags')) is not None: + args['tags'] = tags + if (use_cases := _dict.get('use_cases')) is not None: + args['use_cases'] = [UseCase.from_dict(v) for v in use_cases] + if (types := _dict.get('types')) is not None: + args['types'] = types + else: + raise ValueError('Required property \'types\' not present in DataProductReleaseSummary JSON') + if (contract_terms := _dict.get('contract_terms')) is not None: + args['contract_terms'] = [ContractTerms.from_dict(v) for v in contract_terms] + else: + raise ValueError('Required property \'contract_terms\' not present in DataProductReleaseSummary JSON') + if (domain := _dict.get('domain')) is not None: + args['domain'] = Domain.from_dict(domain) + if (parts_out := _dict.get('parts_out')) is not None: + args['parts_out'] = [DataProductPart.from_dict(v) for v in parts_out] + if (workflows := _dict.get('workflows')) is not None: + args['workflows'] = DataProductWorkflows.from_dict(workflows) + if (dataview_enabled := _dict.get('dataview_enabled')) is not None: + args['dataview_enabled'] = dataview_enabled + if (comments := _dict.get('comments')) is not None: + args['comments'] = comments + if (access_control := _dict.get('access_control')) is not None: + args['access_control'] = AssetListAccessControl.from_dict(access_control) + if (last_updated_at := _dict.get('last_updated_at')) is not None: + args['last_updated_at'] = string_to_datetime(last_updated_at) + if (sub_container := _dict.get('sub_container')) is not None: + args['sub_container'] = ContainerIdentity.from_dict(sub_container) + if (is_restricted := _dict.get('is_restricted')) is not None: + args['is_restricted'] = is_restricted + else: + raise ValueError('Required property \'is_restricted\' not present in DataProductReleaseSummary JSON') + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductReleaseSummary JSON') + if (asset := _dict.get('asset')) is not None: + args['asset'] = AssetReference.from_dict(asset) + else: + raise ValueError('Required property \'asset\' not present in DataProductReleaseSummary JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductReleaseSummary object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'version') and self.version is not None: + _dict['version'] = self.version + if hasattr(self, 'state') and self.state is not None: + _dict['state'] = self.state + if hasattr(self, 'data_product') and self.data_product is not None: + if isinstance(self.data_product, dict): + _dict['data_product'] = self.data_product + else: + _dict['data_product'] = self.data_product.to_dict() + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'tags') and self.tags is not None: + _dict['tags'] = self.tags + if hasattr(self, 'use_cases') and self.use_cases is not None: + use_cases_list = [] + for v in self.use_cases: + if isinstance(v, dict): + use_cases_list.append(v) + else: + use_cases_list.append(v.to_dict()) + _dict['use_cases'] = use_cases_list + if hasattr(self, 'types') and self.types is not None: + _dict['types'] = self.types + if hasattr(self, 'contract_terms') and self.contract_terms is not None: + contract_terms_list = [] + for v in self.contract_terms: + if isinstance(v, dict): + contract_terms_list.append(v) + else: + contract_terms_list.append(v.to_dict()) + _dict['contract_terms'] = contract_terms_list + if hasattr(self, 'domain') and self.domain is not None: + if isinstance(self.domain, dict): + _dict['domain'] = self.domain + else: + _dict['domain'] = self.domain.to_dict() + if hasattr(self, 'parts_out') and self.parts_out is not None: + parts_out_list = [] + for v in self.parts_out: + if isinstance(v, dict): + parts_out_list.append(v) + else: + parts_out_list.append(v.to_dict()) + _dict['parts_out'] = parts_out_list + if hasattr(self, 'workflows') and self.workflows is not None: + if isinstance(self.workflows, dict): + _dict['workflows'] = self.workflows + else: + _dict['workflows'] = self.workflows.to_dict() + if hasattr(self, 'dataview_enabled') and self.dataview_enabled is not None: + _dict['dataview_enabled'] = self.dataview_enabled + if hasattr(self, 'comments') and self.comments is not None: + _dict['comments'] = self.comments + if hasattr(self, 'access_control') and self.access_control is not None: + if isinstance(self.access_control, dict): + _dict['access_control'] = self.access_control + else: + _dict['access_control'] = self.access_control.to_dict() + if hasattr(self, 'last_updated_at') and self.last_updated_at is not None: + _dict['last_updated_at'] = datetime_to_string(self.last_updated_at) + if hasattr(self, 'sub_container') and self.sub_container is not None: + if isinstance(self.sub_container, dict): + _dict['sub_container'] = self.sub_container + else: + _dict['sub_container'] = self.sub_container.to_dict() + if hasattr(self, 'is_restricted') and self.is_restricted is not None: + _dict['is_restricted'] = self.is_restricted + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductReleaseSummary object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductReleaseSummary') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductReleaseSummary') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class StateEnum(str, Enum): + """ + The state of the data product version. + """ + + DRAFT = 'draft' + AVAILABLE = 'available' + RETIRED = 'retired' + + + class TypesEnum(str, Enum): + """ + types. + """ + + DATA = 'data' + CODE = 'code' + + + +class DataProductReleaseSummaryDataProduct: + """ + Data product reference. + + :param str id: Data product identifier. + :param DataProductDraftVersionRelease release: (optional) A data product draft + version object. + :param ContainerReference container: Container reference. + """ + + def __init__( + self, + id: str, + container: 'ContainerReference', + *, + release: Optional['DataProductDraftVersionRelease'] = None, + ) -> None: + """ + Initialize a DataProductReleaseSummaryDataProduct object. + + :param str id: Data product identifier. + :param ContainerReference container: Container reference. + :param DataProductDraftVersionRelease release: (optional) A data product + draft version object. + """ + self.id = id + self.release = release + self.container = container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductReleaseSummaryDataProduct': + """Initialize a DataProductReleaseSummaryDataProduct object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductReleaseSummaryDataProduct JSON') + if (release := _dict.get('release')) is not None: + args['release'] = DataProductDraftVersionRelease.from_dict(release) + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DataProductReleaseSummaryDataProduct JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductReleaseSummaryDataProduct object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'release') and self.release is not None: + if isinstance(self.release, dict): + _dict['release'] = self.release + else: + _dict['release'] = self.release.to_dict() + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductReleaseSummaryDataProduct object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductReleaseSummaryDataProduct') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductReleaseSummaryDataProduct') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductSummary: + """ + Data Product Summary. + + :param str id: Data product identifier. + :param DataProductDraftVersionRelease release: (optional) A data product draft + version object. + :param ContainerReference container: Container reference. + :param str name: (optional) Data product name. + """ + + def __init__( + self, + id: str, + container: 'ContainerReference', + *, + release: Optional['DataProductDraftVersionRelease'] = None, + name: Optional[str] = None, + ) -> None: + """ + Initialize a DataProductSummary object. + + :param str id: Data product identifier. + :param ContainerReference container: Container reference. + :param DataProductDraftVersionRelease release: (optional) A data product + draft version object. + :param str name: (optional) Data product name. + """ + self.id = id + self.release = release + self.container = container + self.name = name + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductSummary': + """Initialize a DataProductSummary object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductSummary JSON') + if (release := _dict.get('release')) is not None: + args['release'] = DataProductDraftVersionRelease.from_dict(release) + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DataProductSummary JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductSummary object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'release') and self.release is not None: + if isinstance(self.release, dict): + _dict['release'] = self.release + else: + _dict['release'] = self.release.to_dict() + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductSummary object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductSummary') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductSummary') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductVersionCollection: + """ + A collection of data product version summaries. + + :param int limit: Set a limit on the number of results returned. + :param FirstPage first: First page in the collection. + :param NextPage next: (optional) Next page in the collection. + :param int total_results: (optional) Indicates the total number of results + returned. + :param List[DataProductVersionSummary] data_product_versions: Collection of data + product versions. + """ + + def __init__( + self, + limit: int, + first: 'FirstPage', + data_product_versions: List['DataProductVersionSummary'], + *, + next: Optional['NextPage'] = None, + total_results: Optional[int] = None, + ) -> None: + """ + Initialize a DataProductVersionCollection object. + + :param int limit: Set a limit on the number of results returned. + :param FirstPage first: First page in the collection. + :param List[DataProductVersionSummary] data_product_versions: Collection of + data product versions. + :param NextPage next: (optional) Next page in the collection. + :param int total_results: (optional) Indicates the total number of results + returned. + """ + self.limit = limit + self.first = first + self.next = next + self.total_results = total_results + self.data_product_versions = data_product_versions + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductVersionCollection': + """Initialize a DataProductVersionCollection object from a json dictionary.""" + args = {} + if (limit := _dict.get('limit')) is not None: + args['limit'] = limit + else: + raise ValueError('Required property \'limit\' not present in DataProductVersionCollection JSON') + if (first := _dict.get('first')) is not None: + args['first'] = FirstPage.from_dict(first) + else: + raise ValueError('Required property \'first\' not present in DataProductVersionCollection JSON') + if (next := _dict.get('next')) is not None: + args['next'] = NextPage.from_dict(next) + if (total_results := _dict.get('total_results')) is not None: + args['total_results'] = total_results + if (data_product_versions := _dict.get('data_product_versions')) is not None: + args['data_product_versions'] = [DataProductVersionSummary.from_dict(v) for v in data_product_versions] + else: + raise ValueError('Required property \'data_product_versions\' not present in DataProductVersionCollection JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductVersionCollection object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'limit') and self.limit is not None: + _dict['limit'] = self.limit + if hasattr(self, 'first') and self.first is not None: + if isinstance(self.first, dict): + _dict['first'] = self.first + else: + _dict['first'] = self.first.to_dict() + if hasattr(self, 'next') and self.next is not None: + if isinstance(self.next, dict): + _dict['next'] = self.next + else: + _dict['next'] = self.next.to_dict() + if hasattr(self, 'total_results') and self.total_results is not None: + _dict['total_results'] = self.total_results + if hasattr(self, 'data_product_versions') and self.data_product_versions is not None: + data_product_versions_list = [] + for v in self.data_product_versions: + if isinstance(v, dict): + data_product_versions_list.append(v) + else: + data_product_versions_list.append(v.to_dict()) + _dict['data_product_versions'] = data_product_versions_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductVersionCollection object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductVersionCollection') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductVersionCollection') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductVersionSummary: + """ + Summary of Data Product Version object. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductVersionSummaryDataProduct data_product: Data product + reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable and + reusable. + :param str description: The description of the data product version. + :param List[str] tags: (optional) Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated with + the data product version. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param Domain domain: (optional) Domain that the data product version belongs + to. If this is the first version of a data product, this field is required. If + this is a new version of an existing data product, the domain will default to + the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: (optional) The outgoing parts of this + data product version to be delivered to consumers. If this is the first version + of a data product, this field defaults to an empty list. If this is a new + version of an existing data product, the data product parts will default to the + parts list from the previous version of the data product. + :param DataProductWorkflows workflows: (optional) The workflows associated with + the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided either + at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for a IBM + knowledge catalog container (catalog/project/space). + :param bool is_restricted: Indicates whether the data product is restricted or + not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a container. + """ + + def __init__( + self, + version: str, + state: str, + data_product: 'DataProductVersionSummaryDataProduct', + name: str, + description: str, + types: List[str], + contract_terms: List['ContractTerms'], + is_restricted: bool, + id: str, + asset: 'AssetReference', + *, + tags: Optional[List[str]] = None, + use_cases: Optional[List['UseCase']] = None, + domain: Optional['Domain'] = None, + parts_out: Optional[List['DataProductPart']] = None, + workflows: Optional['DataProductWorkflows'] = None, + dataview_enabled: Optional[bool] = None, + comments: Optional[str] = None, + access_control: Optional['AssetListAccessControl'] = None, + last_updated_at: Optional[datetime] = None, + sub_container: Optional['ContainerIdentity'] = None, + ) -> None: + """ + Initialize a DataProductVersionSummary object. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductVersionSummaryDataProduct data_product: Data product + reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable + and reusable. + :param str description: The description of the data product version. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param bool is_restricted: Indicates whether the data product is restricted + or not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a + container. + :param List[str] tags: (optional) Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated + with the data product version. + :param Domain domain: (optional) Domain that the data product version + belongs to. If this is the first version of a data product, this field is + required. If this is a new version of an existing data product, the domain + will default to the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: (optional) The outgoing parts of + this data product version to be delivered to consumers. If this is the + first version of a data product, this field defaults to an empty list. If + this is a new version of an existing data product, the data product parts + will default to the parts list from the previous version of the data + product. + :param DataProductWorkflows workflows: (optional) The workflows associated + with the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided + either at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control + object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for + a IBM knowledge catalog container (catalog/project/space). + """ + self.version = version + self.state = state + self.data_product = data_product + self.name = name + self.description = description + self.tags = tags + self.use_cases = use_cases + self.types = types + self.contract_terms = contract_terms + self.domain = domain + self.parts_out = parts_out + self.workflows = workflows + self.dataview_enabled = dataview_enabled + self.comments = comments + self.access_control = access_control + self.last_updated_at = last_updated_at + self.sub_container = sub_container + self.is_restricted = is_restricted + self.id = id + self.asset = asset + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductVersionSummary': + """Initialize a DataProductVersionSummary object from a json dictionary.""" + args = {} + if (version := _dict.get('version')) is not None: + args['version'] = version + else: + raise ValueError('Required property \'version\' not present in DataProductVersionSummary JSON') + if (state := _dict.get('state')) is not None: + args['state'] = state + else: + raise ValueError('Required property \'state\' not present in DataProductVersionSummary JSON') + if (data_product := _dict.get('data_product')) is not None: + args['data_product'] = DataProductVersionSummaryDataProduct.from_dict(data_product) + else: + raise ValueError('Required property \'data_product\' not present in DataProductVersionSummary JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + else: + raise ValueError('Required property \'name\' not present in DataProductVersionSummary JSON') + if (description := _dict.get('description')) is not None: + args['description'] = description + else: + raise ValueError('Required property \'description\' not present in DataProductVersionSummary JSON') + if (tags := _dict.get('tags')) is not None: + args['tags'] = tags + if (use_cases := _dict.get('use_cases')) is not None: + args['use_cases'] = [UseCase.from_dict(v) for v in use_cases] + if (types := _dict.get('types')) is not None: + args['types'] = types + else: + raise ValueError('Required property \'types\' not present in DataProductVersionSummary JSON') + if (contract_terms := _dict.get('contract_terms')) is not None: + args['contract_terms'] = [ContractTerms.from_dict(v) for v in contract_terms] + else: + raise ValueError('Required property \'contract_terms\' not present in DataProductVersionSummary JSON') + if (domain := _dict.get('domain')) is not None: + args['domain'] = Domain.from_dict(domain) + if (parts_out := _dict.get('parts_out')) is not None: + args['parts_out'] = [DataProductPart.from_dict(v) for v in parts_out] + if (workflows := _dict.get('workflows')) is not None: + args['workflows'] = DataProductWorkflows.from_dict(workflows) + if (dataview_enabled := _dict.get('dataview_enabled')) is not None: + args['dataview_enabled'] = dataview_enabled + if (comments := _dict.get('comments')) is not None: + args['comments'] = comments + if (access_control := _dict.get('access_control')) is not None: + args['access_control'] = AssetListAccessControl.from_dict(access_control) + if (last_updated_at := _dict.get('last_updated_at')) is not None: + args['last_updated_at'] = string_to_datetime(last_updated_at) + if (sub_container := _dict.get('sub_container')) is not None: + args['sub_container'] = ContainerIdentity.from_dict(sub_container) + if (is_restricted := _dict.get('is_restricted')) is not None: + args['is_restricted'] = is_restricted + else: + raise ValueError('Required property \'is_restricted\' not present in DataProductVersionSummary JSON') + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductVersionSummary JSON') + if (asset := _dict.get('asset')) is not None: + args['asset'] = AssetReference.from_dict(asset) + else: + raise ValueError('Required property \'asset\' not present in DataProductVersionSummary JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductVersionSummary object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'version') and self.version is not None: + _dict['version'] = self.version + if hasattr(self, 'state') and self.state is not None: + _dict['state'] = self.state + if hasattr(self, 'data_product') and self.data_product is not None: + if isinstance(self.data_product, dict): + _dict['data_product'] = self.data_product + else: + _dict['data_product'] = self.data_product.to_dict() + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'tags') and self.tags is not None: + _dict['tags'] = self.tags + if hasattr(self, 'use_cases') and self.use_cases is not None: + use_cases_list = [] + for v in self.use_cases: + if isinstance(v, dict): + use_cases_list.append(v) + else: + use_cases_list.append(v.to_dict()) + _dict['use_cases'] = use_cases_list + if hasattr(self, 'types') and self.types is not None: + _dict['types'] = self.types + if hasattr(self, 'contract_terms') and self.contract_terms is not None: + contract_terms_list = [] + for v in self.contract_terms: + if isinstance(v, dict): + contract_terms_list.append(v) + else: + contract_terms_list.append(v.to_dict()) + _dict['contract_terms'] = contract_terms_list + if hasattr(self, 'domain') and self.domain is not None: + if isinstance(self.domain, dict): + _dict['domain'] = self.domain + else: + _dict['domain'] = self.domain.to_dict() + if hasattr(self, 'parts_out') and self.parts_out is not None: + parts_out_list = [] + for v in self.parts_out: + if isinstance(v, dict): + parts_out_list.append(v) + else: + parts_out_list.append(v.to_dict()) + _dict['parts_out'] = parts_out_list + if hasattr(self, 'workflows') and self.workflows is not None: + if isinstance(self.workflows, dict): + _dict['workflows'] = self.workflows + else: + _dict['workflows'] = self.workflows.to_dict() + if hasattr(self, 'dataview_enabled') and self.dataview_enabled is not None: + _dict['dataview_enabled'] = self.dataview_enabled + if hasattr(self, 'comments') and self.comments is not None: + _dict['comments'] = self.comments + if hasattr(self, 'access_control') and self.access_control is not None: + if isinstance(self.access_control, dict): + _dict['access_control'] = self.access_control + else: + _dict['access_control'] = self.access_control.to_dict() + if hasattr(self, 'last_updated_at') and self.last_updated_at is not None: + _dict['last_updated_at'] = datetime_to_string(self.last_updated_at) + if hasattr(self, 'sub_container') and self.sub_container is not None: + if isinstance(self.sub_container, dict): + _dict['sub_container'] = self.sub_container + else: + _dict['sub_container'] = self.sub_container.to_dict() + if hasattr(self, 'is_restricted') and self.is_restricted is not None: + _dict['is_restricted'] = self.is_restricted + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductVersionSummary object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductVersionSummary') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductVersionSummary') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class StateEnum(str, Enum): + """ + The state of the data product version. + """ + + DRAFT = 'draft' + AVAILABLE = 'available' + RETIRED = 'retired' + + + class TypesEnum(str, Enum): + """ + types. + """ + + DATA = 'data' + CODE = 'code' + + + +class DataProductVersionSummaryDataProduct: + """ + Data product reference. + + :param str id: Data product identifier. + :param DataProductDraftVersionRelease release: (optional) A data product draft + version object. + :param ContainerReference container: Container reference. + """ + + def __init__( + self, + id: str, + container: 'ContainerReference', + *, + release: Optional['DataProductDraftVersionRelease'] = None, + ) -> None: + """ + Initialize a DataProductVersionSummaryDataProduct object. + + :param str id: Data product identifier. + :param ContainerReference container: Container reference. + :param DataProductDraftVersionRelease release: (optional) A data product + draft version object. + """ + self.id = id + self.release = release + self.container = container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductVersionSummaryDataProduct': + """Initialize a DataProductVersionSummaryDataProduct object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductVersionSummaryDataProduct JSON') + if (release := _dict.get('release')) is not None: + args['release'] = DataProductDraftVersionRelease.from_dict(release) + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DataProductVersionSummaryDataProduct JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductVersionSummaryDataProduct object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'release') and self.release is not None: + if isinstance(self.release, dict): + _dict['release'] = self.release + else: + _dict['release'] = self.release.to_dict() + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductVersionSummaryDataProduct object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductVersionSummaryDataProduct') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductVersionSummaryDataProduct') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductWorkflows: + """ + The workflows associated with the data product version. + + :param DataProductOrderAccessRequest order_access_request: (optional) The + approval workflows associated with the data product version. + """ + + def __init__( + self, + *, + order_access_request: Optional['DataProductOrderAccessRequest'] = None, + ) -> None: + """ + Initialize a DataProductWorkflows object. + + :param DataProductOrderAccessRequest order_access_request: (optional) The + approval workflows associated with the data product version. + """ + self.order_access_request = order_access_request + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductWorkflows': + """Initialize a DataProductWorkflows object from a json dictionary.""" + args = {} + if (order_access_request := _dict.get('order_access_request')) is not None: + args['order_access_request'] = DataProductOrderAccessRequest.from_dict(order_access_request) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductWorkflows object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'order_access_request') and self.order_access_request is not None: + if isinstance(self.order_access_request, dict): + _dict['order_access_request'] = self.order_access_request + else: + _dict['order_access_request'] = self.order_access_request.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductWorkflows object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductWorkflows') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductWorkflows') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DeliveryMethod: + """ + DeliveryMethod. + + :param str id: The ID of the delivery method. + :param ContainerReference container: Container reference. + :param DeliveryMethodPropertiesModel getproperties: (optional) The propertiess + of the delivery method. + """ + + def __init__( + self, + id: str, + container: 'ContainerReference', + *, + getproperties: Optional['DeliveryMethodPropertiesModel'] = None, + ) -> None: + """ + Initialize a DeliveryMethod object. + + :param str id: The ID of the delivery method. + :param ContainerReference container: Container reference. + :param DeliveryMethodPropertiesModel getproperties: (optional) The + propertiess of the delivery method. + """ + self.id = id + self.container = container + self.getproperties = getproperties + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DeliveryMethod': + """Initialize a DeliveryMethod object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DeliveryMethod JSON') + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DeliveryMethod JSON') + if (getproperties := _dict.get('getproperties')) is not None: + args['getproperties'] = DeliveryMethodPropertiesModel.from_dict(getproperties) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DeliveryMethod object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + if hasattr(self, 'getproperties') and self.getproperties is not None: + if isinstance(self.getproperties, dict): + _dict['getproperties'] = self.getproperties + else: + _dict['getproperties'] = self.getproperties.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DeliveryMethod object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DeliveryMethod') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DeliveryMethod') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DeliveryMethodPropertiesModel: + """ + The propertiess of the delivery method. + + :param ProducerInputModel producer_input: (optional) Parameters for delivery + that are set by a data product producer. + """ + + def __init__( + self, + *, + producer_input: Optional['ProducerInputModel'] = None, + ) -> None: + """ + Initialize a DeliveryMethodPropertiesModel object. + + :param ProducerInputModel producer_input: (optional) Parameters for + delivery that are set by a data product producer. + """ + self.producer_input = producer_input + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DeliveryMethodPropertiesModel': + """Initialize a DeliveryMethodPropertiesModel object from a json dictionary.""" + args = {} + if (producer_input := _dict.get('producer_input')) is not None: + args['producer_input'] = ProducerInputModel.from_dict(producer_input) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DeliveryMethodPropertiesModel object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'producer_input') and self.producer_input is not None: + if isinstance(self.producer_input, dict): + _dict['producer_input'] = self.producer_input + else: + _dict['producer_input'] = self.producer_input.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DeliveryMethodPropertiesModel object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DeliveryMethodPropertiesModel') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DeliveryMethodPropertiesModel') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class Description: + """ + Description details of a data contract. + + :param str purpose: (optional) Intended purpose for the provided data. + :param str limitations: (optional) Technical, compliance, and legal limitations + for data use. + :param str usage: (optional) Recommended usage of the data. + :param List[ContractTermsMoreInfo] more_info: (optional) List of links to + sources that provide more details on the dataset. + :param str custom_properties: (optional) Custom properties that are not part of + the standard. + """ + + def __init__( + self, + *, + purpose: Optional[str] = None, + limitations: Optional[str] = None, + usage: Optional[str] = None, + more_info: Optional[List['ContractTermsMoreInfo']] = None, + custom_properties: Optional[str] = None, + ) -> None: + """ + Initialize a Description object. + + :param str purpose: (optional) Intended purpose for the provided data. + :param str limitations: (optional) Technical, compliance, and legal + limitations for data use. + :param str usage: (optional) Recommended usage of the data. + :param List[ContractTermsMoreInfo] more_info: (optional) List of links to + sources that provide more details on the dataset. + :param str custom_properties: (optional) Custom properties that are not + part of the standard. + """ + self.purpose = purpose + self.limitations = limitations + self.usage = usage + self.more_info = more_info + self.custom_properties = custom_properties + + @classmethod + def from_dict(cls, _dict: Dict) -> 'Description': + """Initialize a Description object from a json dictionary.""" + args = {} + if (purpose := _dict.get('purpose')) is not None: + args['purpose'] = purpose + if (limitations := _dict.get('limitations')) is not None: + args['limitations'] = limitations + if (usage := _dict.get('usage')) is not None: + args['usage'] = usage + if (more_info := _dict.get('more_info')) is not None: + args['more_info'] = [ContractTermsMoreInfo.from_dict(v) for v in more_info] + if (custom_properties := _dict.get('custom_properties')) is not None: + args['custom_properties'] = custom_properties + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a Description object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'purpose') and self.purpose is not None: + _dict['purpose'] = self.purpose + if hasattr(self, 'limitations') and self.limitations is not None: + _dict['limitations'] = self.limitations + if hasattr(self, 'usage') and self.usage is not None: + _dict['usage'] = self.usage + if hasattr(self, 'more_info') and self.more_info is not None: + more_info_list = [] + for v in self.more_info: + if isinstance(v, dict): + more_info_list.append(v) + else: + more_info_list.append(v.to_dict()) + _dict['more_info'] = more_info_list + if hasattr(self, 'custom_properties') and self.custom_properties is not None: + _dict['custom_properties'] = self.custom_properties + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this Description object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'Description') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'Description') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class Domain: + """ + Domain that the data product version belongs to. If this is the first version of a + data product, this field is required. If this is a new version of an existing data + product, the domain will default to the domain of the previous version of the data + product. + + :param str id: The ID of the domain. + :param str name: (optional) The display name of the domain. + :param ContainerReference container: (optional) Container reference. + """ + + def __init__( + self, + id: str, + *, + name: Optional[str] = None, + container: Optional['ContainerReference'] = None, + ) -> None: + """ + Initialize a Domain object. + + :param str id: The ID of the domain. + :param str name: (optional) The display name of the domain. + :param ContainerReference container: (optional) Container reference. + """ + self.id = id + self.name = name + self.container = container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'Domain': + """Initialize a Domain object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in Domain JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a Domain object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this Domain object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'Domain') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'Domain') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class EngineDetailsModel: + """ + Engine details as defined by the data product producer. + + :param str display_name: (optional) The name of the engine defined by the data + product producer. + :param str engine_id: (optional) The id of the engine defined by the data + product producer. + :param str engine_port: (optional) The port of the engine defined by the data + product producer. + :param str engine_host: (optional) The host of the engine defined by the data + product producer. + :param str engine_type: The type of the engine (eg: Presto/Spark). + :param List[str] associated_catalogs: (optional) The list of associated + catalogs. + """ + + def __init__( + self, + engine_type: str, + *, + display_name: Optional[str] = None, + engine_id: Optional[str] = None, + engine_port: Optional[str] = None, + engine_host: Optional[str] = None, + associated_catalogs: Optional[List[str]] = None, + ) -> None: + """ + Initialize a EngineDetailsModel object. + + :param str engine_type: The type of the engine (eg: Presto/Spark). + :param str display_name: (optional) The name of the engine defined by the + data product producer. + :param str engine_id: (optional) The id of the engine defined by the data + product producer. + :param str engine_port: (optional) The port of the engine defined by the + data product producer. + :param str engine_host: (optional) The host of the engine defined by the + data product producer. + :param List[str] associated_catalogs: (optional) The list of associated + catalogs. + """ + self.display_name = display_name + self.engine_id = engine_id + self.engine_port = engine_port + self.engine_host = engine_host + self.engine_type = engine_type + self.associated_catalogs = associated_catalogs + + @classmethod + def from_dict(cls, _dict: Dict) -> 'EngineDetailsModel': + """Initialize a EngineDetailsModel object from a json dictionary.""" + args = {} + if (display_name := _dict.get('display_name')) is not None: + args['display_name'] = display_name + if (engine_id := _dict.get('engine_id')) is not None: + args['engine_id'] = engine_id + if (engine_port := _dict.get('engine_port')) is not None: + args['engine_port'] = engine_port + if (engine_host := _dict.get('engine_host')) is not None: + args['engine_host'] = engine_host + if (engine_type := _dict.get('engine_type')) is not None: + args['engine_type'] = engine_type + else: + raise ValueError('Required property \'engine_type\' not present in EngineDetailsModel JSON') + if (associated_catalogs := _dict.get('associated_catalogs')) is not None: + args['associated_catalogs'] = associated_catalogs + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a EngineDetailsModel object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'display_name') and self.display_name is not None: + _dict['display_name'] = self.display_name + if hasattr(self, 'engine_id') and self.engine_id is not None: + _dict['engine_id'] = self.engine_id + if hasattr(self, 'engine_port') and self.engine_port is not None: + _dict['engine_port'] = self.engine_port + if hasattr(self, 'engine_host') and self.engine_host is not None: + _dict['engine_host'] = self.engine_host + if hasattr(self, 'engine_type') and self.engine_type is not None: + _dict['engine_type'] = self.engine_type + if hasattr(self, 'associated_catalogs') and self.associated_catalogs is not None: + _dict['associated_catalogs'] = self.associated_catalogs + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this EngineDetailsModel object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'EngineDetailsModel') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'EngineDetailsModel') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class EngineTypeEnum(str, Enum): + """ + The type of the engine (eg: Presto/Spark). + """ + + SPARK = 'spark' + PRESTO = 'presto' + + + +class ErrorExtraResource: + """ + Detailed error information. + + :param str id: (optional) Error id. + :param datetime timestamp: (optional) Timestamp of the error. + :param str environment_name: (optional) Environment where the error occurred. + :param int http_status: (optional) Http status code. + :param int source_cluster: (optional) Source cluster of the error. + :param int source_component: (optional) Source component of the error. + :param int transaction_id: (optional) Transaction id of the request. + """ + + def __init__( + self, + *, + id: Optional[str] = None, + timestamp: Optional[datetime] = None, + environment_name: Optional[str] = None, + http_status: Optional[int] = None, + source_cluster: Optional[int] = None, + source_component: Optional[int] = None, + transaction_id: Optional[int] = None, + ) -> None: + """ + Initialize a ErrorExtraResource object. + + :param str id: (optional) Error id. + :param datetime timestamp: (optional) Timestamp of the error. + :param str environment_name: (optional) Environment where the error + occurred. + :param int http_status: (optional) Http status code. + :param int source_cluster: (optional) Source cluster of the error. + :param int source_component: (optional) Source component of the error. + :param int transaction_id: (optional) Transaction id of the request. + """ + self.id = id + self.timestamp = timestamp + self.environment_name = environment_name + self.http_status = http_status + self.source_cluster = source_cluster + self.source_component = source_component + self.transaction_id = transaction_id + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ErrorExtraResource': + """Initialize a ErrorExtraResource object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + if (timestamp := _dict.get('timestamp')) is not None: + args['timestamp'] = string_to_datetime(timestamp) + if (environment_name := _dict.get('environment_name')) is not None: + args['environment_name'] = environment_name + if (http_status := _dict.get('http_status')) is not None: + args['http_status'] = http_status + if (source_cluster := _dict.get('source_cluster')) is not None: + args['source_cluster'] = source_cluster + if (source_component := _dict.get('source_component')) is not None: + args['source_component'] = source_component + if (transaction_id := _dict.get('transaction_id')) is not None: + args['transaction_id'] = transaction_id + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ErrorExtraResource object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'timestamp') and self.timestamp is not None: + _dict['timestamp'] = datetime_to_string(self.timestamp) + if hasattr(self, 'environment_name') and self.environment_name is not None: + _dict['environment_name'] = self.environment_name + if hasattr(self, 'http_status') and self.http_status is not None: + _dict['http_status'] = self.http_status + if hasattr(self, 'source_cluster') and self.source_cluster is not None: + _dict['source_cluster'] = self.source_cluster + if hasattr(self, 'source_component') and self.source_component is not None: + _dict['source_component'] = self.source_component + if hasattr(self, 'transaction_id') and self.transaction_id is not None: + _dict['transaction_id'] = self.transaction_id + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ErrorExtraResource object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ErrorExtraResource') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ErrorExtraResource') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ErrorMessage: + """ + Contains the code and details. + + :param str code: The error code. + :param str message: The error details. + """ + + def __init__( + self, + code: str, + message: str, + ) -> None: + """ + Initialize a ErrorMessage object. + + :param str code: The error code. + :param str message: The error details. + """ + self.code = code + self.message = message + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ErrorMessage': + """Initialize a ErrorMessage object from a json dictionary.""" + args = {} + if (code := _dict.get('code')) is not None: + args['code'] = code + else: + raise ValueError('Required property \'code\' not present in ErrorMessage JSON') + if (message := _dict.get('message')) is not None: + args['message'] = message + else: + raise ValueError('Required property \'message\' not present in ErrorMessage JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ErrorMessage object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'code') and self.code is not None: + _dict['code'] = self.code + if hasattr(self, 'message') and self.message is not None: + _dict['message'] = self.message + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ErrorMessage object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ErrorMessage') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ErrorMessage') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ErrorModelResource: + """ + Detailed error information. + + :param str code: Error code. + :param str message: (optional) Error message. + :param ErrorExtraResource extra: (optional) Detailed error information. + :param str more_info: (optional) More info message. + """ + + def __init__( + self, + code: str, + *, + message: Optional[str] = None, + extra: Optional['ErrorExtraResource'] = None, + more_info: Optional[str] = None, + ) -> None: + """ + Initialize a ErrorModelResource object. + + :param str code: Error code. + :param str message: (optional) Error message. + :param ErrorExtraResource extra: (optional) Detailed error information. + :param str more_info: (optional) More info message. + """ + self.code = code + self.message = message + self.extra = extra + self.more_info = more_info + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ErrorModelResource': + """Initialize a ErrorModelResource object from a json dictionary.""" + args = {} + if (code := _dict.get('code')) is not None: + args['code'] = code + else: + raise ValueError('Required property \'code\' not present in ErrorModelResource JSON') + if (message := _dict.get('message')) is not None: + args['message'] = message + if (extra := _dict.get('extra')) is not None: + args['extra'] = ErrorExtraResource.from_dict(extra) + if (more_info := _dict.get('more_info')) is not None: + args['more_info'] = more_info + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ErrorModelResource object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'code') and self.code is not None: + _dict['code'] = self.code + if hasattr(self, 'message') and self.message is not None: + _dict['message'] = self.message + if hasattr(self, 'extra') and self.extra is not None: + if isinstance(self.extra, dict): + _dict['extra'] = self.extra + else: + _dict['extra'] = self.extra.to_dict() + if hasattr(self, 'more_info') and self.more_info is not None: + _dict['more_info'] = self.more_info + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ErrorModelResource object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ErrorModelResource') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ErrorModelResource') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class CodeEnum(str, Enum): + """ + Error code. + """ + + REQUEST_BODY_ERROR = 'request_body_error' + MISSING_REQUIRED_VALUE = 'missing_required_value' + INVALID_PARAMETER = 'invalid_parameter' + DOES_NOT_EXIST = 'does_not_exist' + ALREADY_EXISTS = 'already_exists' + NOT_AUTHENTICATED = 'not_authenticated' + NOT_AUTHORIZED = 'not_authorized' + FORBIDDEN = 'forbidden' + CONFLICT = 'conflict' + CREATE_ERROR = 'create_error' + FETCH_ERROR = 'fetch_error' + UPDATE_ERROR = 'update_error' + DELETE_ERROR = 'delete_error' + PATCH_ERROR = 'patch_error' + DATA_ERROR = 'data_error' + DATABASE_ERROR = 'database_error' + DATABASE_QUERY_ERROR = 'database_query_error' + CONSTRAINT_VIOLATION = 'constraint_violation' + UNABLE_TO_PERFORM = 'unable_to_perform' + TOO_MANY_REQUESTS = 'too_many_requests' + DEPENDENT_SERVICE_ERROR = 'dependent_service_error' + CONFIGURATION_ERROR = 'configuration_error' + UNEXPECTED_EXCEPTION = 'unexpected_exception' + GOVERNANCE_POLICY_DENIAL = 'governance_policy_denial' + DATABASE_USAGE_LIMITS = 'database_usage_limits' + INACTIVE_USER = 'inactive_user' + ENTITLEMENT_ENFORCEMENT = 'entitlement_enforcement' + DELETED = 'deleted' + NOT_IMPLEMENTED = 'not_implemented' + FEATURE_NOT_ENABLED = 'feature_not_enabled' + MISSING_ASSET_DETAILS = 'missing_asset_details' + + + +class FirstPage: + """ + First page in the collection. + + :param str href: Link to the first page in the collection. + """ + + def __init__( + self, + href: str, + ) -> None: + """ + Initialize a FirstPage object. + + :param str href: Link to the first page in the collection. + """ + self.href = href + + @classmethod + def from_dict(cls, _dict: Dict) -> 'FirstPage': + """Initialize a FirstPage object from a json dictionary.""" + args = {} + if (href := _dict.get('href')) is not None: + args['href'] = href + else: + raise ValueError('Required property \'href\' not present in FirstPage JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a FirstPage object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'href') and self.href is not None: + _dict['href'] = self.href + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this FirstPage object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'FirstPage') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'FirstPage') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class InitializeResource: + """ + Resource defining initialization parameters. + + :param ContainerReference container: (optional) Container reference. + :param str href: (optional) Link to monitor the status of the initialize + operation. + :param str status: Status of the initialize operation. + :param str trace: (optional) The id to trace the failed initialization + operation. + :param List[ErrorModelResource] errors: (optional) Set of errors on the latest + initialization request. + :param datetime last_started_at: (optional) Start time of the last + initialization. + :param datetime last_finished_at: (optional) End time of the last + initialization. + :param List[InitializedOption] initialized_options: (optional) Initialized + options. + :param ProvidedCatalogWorkflows workflows: (optional) Resource defining provided + workflow definitions. + """ + + def __init__( + self, + status: str, + *, + container: Optional['ContainerReference'] = None, + href: Optional[str] = None, + trace: Optional[str] = None, + errors: Optional[List['ErrorModelResource']] = None, + last_started_at: Optional[datetime] = None, + last_finished_at: Optional[datetime] = None, + initialized_options: Optional[List['InitializedOption']] = None, + workflows: Optional['ProvidedCatalogWorkflows'] = None, + ) -> None: + """ + Initialize a InitializeResource object. + + :param str status: Status of the initialize operation. + :param ContainerReference container: (optional) Container reference. + :param str href: (optional) Link to monitor the status of the initialize + operation. + :param str trace: (optional) The id to trace the failed initialization + operation. + :param List[ErrorModelResource] errors: (optional) Set of errors on the + latest initialization request. + :param datetime last_started_at: (optional) Start time of the last + initialization. + :param datetime last_finished_at: (optional) End time of the last + initialization. + :param List[InitializedOption] initialized_options: (optional) Initialized + options. + :param ProvidedCatalogWorkflows workflows: (optional) Resource defining + provided workflow definitions. + """ + self.container = container + self.href = href + self.status = status + self.trace = trace + self.errors = errors + self.last_started_at = last_started_at + self.last_finished_at = last_finished_at + self.initialized_options = initialized_options + self.workflows = workflows + + @classmethod + def from_dict(cls, _dict: Dict) -> 'InitializeResource': + """Initialize a InitializeResource object from a json dictionary.""" + args = {} + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + if (href := _dict.get('href')) is not None: + args['href'] = href + if (status := _dict.get('status')) is not None: + args['status'] = status + else: + raise ValueError('Required property \'status\' not present in InitializeResource JSON') + if (trace := _dict.get('trace')) is not None: + args['trace'] = trace + if (errors := _dict.get('errors')) is not None: + args['errors'] = [ErrorModelResource.from_dict(v) for v in errors] + if (last_started_at := _dict.get('last_started_at')) is not None: + args['last_started_at'] = string_to_datetime(last_started_at) + if (last_finished_at := _dict.get('last_finished_at')) is not None: + args['last_finished_at'] = string_to_datetime(last_finished_at) + if (initialized_options := _dict.get('initialized_options')) is not None: + args['initialized_options'] = [InitializedOption.from_dict(v) for v in initialized_options] + if (workflows := _dict.get('workflows')) is not None: + args['workflows'] = ProvidedCatalogWorkflows.from_dict(workflows) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a InitializeResource object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + if hasattr(self, 'href') and self.href is not None: + _dict['href'] = self.href + if hasattr(self, 'status') and self.status is not None: + _dict['status'] = self.status + if hasattr(self, 'trace') and self.trace is not None: + _dict['trace'] = self.trace + if hasattr(self, 'errors') and self.errors is not None: + errors_list = [] + for v in self.errors: + if isinstance(v, dict): + errors_list.append(v) + else: + errors_list.append(v.to_dict()) + _dict['errors'] = errors_list + if hasattr(self, 'last_started_at') and self.last_started_at is not None: + _dict['last_started_at'] = datetime_to_string(self.last_started_at) + if hasattr(self, 'last_finished_at') and self.last_finished_at is not None: + _dict['last_finished_at'] = datetime_to_string(self.last_finished_at) + if hasattr(self, 'initialized_options') and self.initialized_options is not None: + initialized_options_list = [] + for v in self.initialized_options: + if isinstance(v, dict): + initialized_options_list.append(v) + else: + initialized_options_list.append(v.to_dict()) + _dict['initialized_options'] = initialized_options_list + if hasattr(self, 'workflows') and self.workflows is not None: + if isinstance(self.workflows, dict): + _dict['workflows'] = self.workflows + else: + _dict['workflows'] = self.workflows.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this InitializeResource object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'InitializeResource') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'InitializeResource') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class StatusEnum(str, Enum): + """ + Status of the initialize operation. + """ + + NOT_STARTED = 'not_started' + IN_PROGRESS = 'in_progress' + SUCCEEDED = 'succeeded' + FAILED = 'failed' + + + +class InitializeSubDomain: + """ + The subdomain for a data product domain. + + :param str name: (optional) The name of the data product subdomain. + :param str id: (optional) The identifier of the data product subdomain. + :param str description: (optional) The description of the data product + subdomain. + """ + + def __init__( + self, + *, + name: Optional[str] = None, + id: Optional[str] = None, + description: Optional[str] = None, + ) -> None: + """ + Initialize a InitializeSubDomain object. + + :param str name: (optional) The name of the data product subdomain. + :param str id: (optional) The identifier of the data product subdomain. + :param str description: (optional) The description of the data product + subdomain. + """ + self.name = name + self.id = id + self.description = description + + @classmethod + def from_dict(cls, _dict: Dict) -> 'InitializeSubDomain': + """Initialize a InitializeSubDomain object from a json dictionary.""" + args = {} + if (name := _dict.get('name')) is not None: + args['name'] = name + if (id := _dict.get('id')) is not None: + args['id'] = id + if (description := _dict.get('description')) is not None: + args['description'] = description + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a InitializeSubDomain object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this InitializeSubDomain object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'InitializeSubDomain') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'InitializeSubDomain') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class InitializedOption: + """ + List of options successfully initialized. + + :param str name: (optional) The name of the option. + :param int version: (optional) The version of the option. + """ + + def __init__( + self, + *, + name: Optional[str] = None, + version: Optional[int] = None, + ) -> None: + """ + Initialize a InitializedOption object. + + :param str name: (optional) The name of the option. + :param int version: (optional) The version of the option. + """ + self.name = name + self.version = version + + @classmethod + def from_dict(cls, _dict: Dict) -> 'InitializedOption': + """Initialize a InitializedOption object from a json dictionary.""" + args = {} + if (name := _dict.get('name')) is not None: + args['name'] = name + if (version := _dict.get('version')) is not None: + args['version'] = version + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a InitializedOption object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'version') and self.version is not None: + _dict['version'] = self.version + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this InitializedOption object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'InitializedOption') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'InitializedOption') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class JsonPatchOperation: + """ + This model represents an individual patch operation to be performed on a JSON + document, as defined by RFC 6902. + + :param str op: The operation to be performed. + :param str path: The JSON Pointer that identifies the field that is the target + of the operation. + :param str from_: (optional) The JSON Pointer that identifies the field that is + the source of the operation. + :param object value: (optional) The value to be used within the operation. + """ + + def __init__( + self, + op: str, + path: str, + *, + from_: Optional[str] = None, + value: Optional[object] = None, + ) -> None: + """ + Initialize a JsonPatchOperation object. + + :param str op: The operation to be performed. + :param str path: The JSON Pointer that identifies the field that is the + target of the operation. + :param str from_: (optional) The JSON Pointer that identifies the field + that is the source of the operation. + :param object value: (optional) The value to be used within the operation. + """ + self.op = op + self.path = path + self.from_ = from_ + self.value = value + + @classmethod + def from_dict(cls, _dict: Dict) -> 'JsonPatchOperation': + """Initialize a JsonPatchOperation object from a json dictionary.""" + args = {} + if (op := _dict.get('op')) is not None: + args['op'] = op + else: + raise ValueError('Required property \'op\' not present in JsonPatchOperation JSON') + if (path := _dict.get('path')) is not None: + args['path'] = path + else: + raise ValueError('Required property \'path\' not present in JsonPatchOperation JSON') + if (from_ := _dict.get('from')) is not None: + args['from_'] = from_ + if (value := _dict.get('value')) is not None: + args['value'] = value + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a JsonPatchOperation object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'op') and self.op is not None: + _dict['op'] = self.op + if hasattr(self, 'path') and self.path is not None: + _dict['path'] = self.path + if hasattr(self, 'from_') and self.from_ is not None: + _dict['from'] = self.from_ + if hasattr(self, 'value') and self.value is not None: + _dict['value'] = self.value + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this JsonPatchOperation object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'JsonPatchOperation') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'JsonPatchOperation') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class OpEnum(str, Enum): + """ + The operation to be performed. + """ + + ADD = 'add' + REMOVE = 'remove' + REPLACE = 'replace' + MOVE = 'move' + COPY = 'copy' + TEST = 'test' + + + +class MemberRolesSchema: + """ + Member roles of a corresponding asset. + + :param str user_iam_id: (optional) User id. + :param List[str] roles: (optional) Roles of the given user. + """ + + def __init__( + self, + *, + user_iam_id: Optional[str] = None, + roles: Optional[List[str]] = None, + ) -> None: + """ + Initialize a MemberRolesSchema object. + + :param str user_iam_id: (optional) User id. + :param List[str] roles: (optional) Roles of the given user. + """ + self.user_iam_id = user_iam_id + self.roles = roles + + @classmethod + def from_dict(cls, _dict: Dict) -> 'MemberRolesSchema': + """Initialize a MemberRolesSchema object from a json dictionary.""" + args = {} + if (user_iam_id := _dict.get('user_iam_id')) is not None: + args['user_iam_id'] = user_iam_id + if (roles := _dict.get('roles')) is not None: + args['roles'] = roles + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a MemberRolesSchema object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'user_iam_id') and self.user_iam_id is not None: + _dict['user_iam_id'] = self.user_iam_id + if hasattr(self, 'roles') and self.roles is not None: + _dict['roles'] = self.roles + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this MemberRolesSchema object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'MemberRolesSchema') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'MemberRolesSchema') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class NextPage: + """ + Next page in the collection. + + :param str href: Link to the next page in the collection. + :param str start: Start token for pagination to the next page in the collection. + """ + + def __init__( + self, + href: str, + start: str, + ) -> None: + """ + Initialize a NextPage object. + + :param str href: Link to the next page in the collection. + :param str start: Start token for pagination to the next page in the + collection. + """ + self.href = href + self.start = start + + @classmethod + def from_dict(cls, _dict: Dict) -> 'NextPage': + """Initialize a NextPage object from a json dictionary.""" + args = {} + if (href := _dict.get('href')) is not None: + args['href'] = href + else: + raise ValueError('Required property \'href\' not present in NextPage JSON') + if (start := _dict.get('start')) is not None: + args['start'] = start + else: + raise ValueError('Required property \'start\' not present in NextPage JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a NextPage object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'href') and self.href is not None: + _dict['href'] = self.href + if hasattr(self, 'start') and self.start is not None: + _dict['start'] = self.start + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this NextPage object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'NextPage') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'NextPage') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class Overview: + """ + Overview details of a data contract. + + :param str api_version: (optional) The API version of the contract. + :param str kind: (optional) The kind of contract. + :param str name: (optional) The name of the contract. + :param str version: The version of the contract. + :param Domain domain: Domain that the data product version belongs to. If this + is the first version of a data product, this field is required. If this is a new + version of an existing data product, the domain will default to the domain of + the previous version of the data product. + :param str more_info: (optional) Additional information links about the + contract. + """ + + def __init__( + self, + version: str, + domain: 'Domain', + *, + api_version: Optional[str] = None, + kind: Optional[str] = None, + name: Optional[str] = None, + more_info: Optional[str] = None, + ) -> None: + """ + Initialize a Overview object. + + :param str version: The version of the contract. + :param Domain domain: Domain that the data product version belongs to. If + this is the first version of a data product, this field is required. If + this is a new version of an existing data product, the domain will default + to the domain of the previous version of the data product. + :param str api_version: (optional) The API version of the contract. + :param str kind: (optional) The kind of contract. + :param str name: (optional) The name of the contract. + :param str more_info: (optional) Additional information links about the + contract. + """ + self.api_version = api_version + self.kind = kind + self.name = name + self.version = version + self.domain = domain + self.more_info = more_info + + @classmethod + def from_dict(cls, _dict: Dict) -> 'Overview': + """Initialize a Overview object from a json dictionary.""" + args = {} + if (api_version := _dict.get('api_version')) is not None: + args['api_version'] = api_version + if (kind := _dict.get('kind')) is not None: + args['kind'] = kind + if (name := _dict.get('name')) is not None: + args['name'] = name + if (version := _dict.get('version')) is not None: + args['version'] = version + else: + raise ValueError('Required property \'version\' not present in Overview JSON') + if (domain := _dict.get('domain')) is not None: + args['domain'] = Domain.from_dict(domain) + else: + raise ValueError('Required property \'domain\' not present in Overview JSON') + if (more_info := _dict.get('more_info')) is not None: + args['more_info'] = more_info + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a Overview object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'api_version') and self.api_version is not None: + _dict['api_version'] = self.api_version + if hasattr(self, 'kind') and self.kind is not None: + _dict['kind'] = self.kind + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'version') and self.version is not None: + _dict['version'] = self.version + if hasattr(self, 'domain') and self.domain is not None: + if isinstance(self.domain, dict): + _dict['domain'] = self.domain + else: + _dict['domain'] = self.domain.to_dict() + if hasattr(self, 'more_info') and self.more_info is not None: + _dict['more_info'] = self.more_info + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this Overview object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'Overview') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'Overview') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class Pricing: + """ + Represents the pricing details of the contract. + + :param str amount: (optional) The amount for the contract pricing. + :param str currency: (optional) The currency for the pricing amount. + :param str unit: (optional) The unit associated with the pricing. + """ + + def __init__( + self, + *, + amount: Optional[str] = None, + currency: Optional[str] = None, + unit: Optional[str] = None, + ) -> None: + """ + Initialize a Pricing object. + + :param str amount: (optional) The amount for the contract pricing. + :param str currency: (optional) The currency for the pricing amount. + :param str unit: (optional) The unit associated with the pricing. + """ + self.amount = amount + self.currency = currency + self.unit = unit + + @classmethod + def from_dict(cls, _dict: Dict) -> 'Pricing': + """Initialize a Pricing object from a json dictionary.""" + args = {} + if (amount := _dict.get('amount')) is not None: + args['amount'] = amount + if (currency := _dict.get('currency')) is not None: + args['currency'] = currency + if (unit := _dict.get('unit')) is not None: + args['unit'] = unit + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a Pricing object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'amount') and self.amount is not None: + _dict['amount'] = self.amount + if hasattr(self, 'currency') and self.currency is not None: + _dict['currency'] = self.currency + if hasattr(self, 'unit') and self.unit is not None: + _dict['unit'] = self.unit + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this Pricing object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'Pricing') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'Pricing') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ProducerInputModel: + """ + Parameters for delivery that are set by a data product producer. + + :param EngineDetailsModel engine_details: (optional) Engine details as defined + by the data product producer. + :param List[EngineDetailsModel] engines: (optional) List of engines defined by + the data product producer. + """ + + def __init__( + self, + *, + engine_details: Optional['EngineDetailsModel'] = None, + engines: Optional[List['EngineDetailsModel']] = None, + ) -> None: + """ + Initialize a ProducerInputModel object. + + :param EngineDetailsModel engine_details: (optional) Engine details as + defined by the data product producer. + :param List[EngineDetailsModel] engines: (optional) List of engines defined + by the data product producer. + """ + self.engine_details = engine_details + self.engines = engines + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ProducerInputModel': + """Initialize a ProducerInputModel object from a json dictionary.""" + args = {} + if (engine_details := _dict.get('engine_details')) is not None: + args['engine_details'] = EngineDetailsModel.from_dict(engine_details) + if (engines := _dict.get('engines')) is not None: + args['engines'] = [EngineDetailsModel.from_dict(v) for v in engines] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ProducerInputModel object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'engine_details') and self.engine_details is not None: + if isinstance(self.engine_details, dict): + _dict['engine_details'] = self.engine_details + else: + _dict['engine_details'] = self.engine_details.to_dict() + if hasattr(self, 'engines') and self.engines is not None: + engines_list = [] + for v in self.engines: + if isinstance(v, dict): + engines_list.append(v) + else: + engines_list.append(v.to_dict()) + _dict['engines'] = engines_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ProducerInputModel object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ProducerInputModel') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ProducerInputModel') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class PropertiesSchema: + """ + Properties of the corresponding asset. + + :param str value: (optional) Value of the property object. + """ + + def __init__( + self, + *, + value: Optional[str] = None, + ) -> None: + """ + Initialize a PropertiesSchema object. + + :param str value: (optional) Value of the property object. + """ + self.value = value + + @classmethod + def from_dict(cls, _dict: Dict) -> 'PropertiesSchema': + """Initialize a PropertiesSchema object from a json dictionary.""" + args = {} + if (value := _dict.get('value')) is not None: + args['value'] = value + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a PropertiesSchema object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'value') and self.value is not None: + _dict['value'] = self.value + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this PropertiesSchema object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'PropertiesSchema') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'PropertiesSchema') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ProvidedCatalogWorkflows: + """ + Resource defining provided workflow definitions. + + :param ProvidedWorkflowResource data_access: (optional) A reference to a + workflow definition. + :param ProvidedWorkflowResource request_new_product: (optional) A reference to a + workflow definition. + """ + + def __init__( + self, + *, + data_access: Optional['ProvidedWorkflowResource'] = None, + request_new_product: Optional['ProvidedWorkflowResource'] = None, + ) -> None: + """ + Initialize a ProvidedCatalogWorkflows object. + + :param ProvidedWorkflowResource data_access: (optional) A reference to a + workflow definition. + :param ProvidedWorkflowResource request_new_product: (optional) A reference + to a workflow definition. + """ + self.data_access = data_access + self.request_new_product = request_new_product + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ProvidedCatalogWorkflows': + """Initialize a ProvidedCatalogWorkflows object from a json dictionary.""" + args = {} + if (data_access := _dict.get('data_access')) is not None: + args['data_access'] = ProvidedWorkflowResource.from_dict(data_access) + if (request_new_product := _dict.get('request_new_product')) is not None: + args['request_new_product'] = ProvidedWorkflowResource.from_dict(request_new_product) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ProvidedCatalogWorkflows object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'data_access') and self.data_access is not None: + if isinstance(self.data_access, dict): + _dict['data_access'] = self.data_access + else: + _dict['data_access'] = self.data_access.to_dict() + if hasattr(self, 'request_new_product') and self.request_new_product is not None: + if isinstance(self.request_new_product, dict): + _dict['request_new_product'] = self.request_new_product + else: + _dict['request_new_product'] = self.request_new_product.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ProvidedCatalogWorkflows object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ProvidedCatalogWorkflows') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ProvidedCatalogWorkflows') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ProvidedWorkflowResource: + """ + A reference to a workflow definition. + + :param WorkflowDefinitionReference definition: (optional) Reference to a + workflow definition. + """ + + def __init__( + self, + *, + definition: Optional['WorkflowDefinitionReference'] = None, + ) -> None: + """ + Initialize a ProvidedWorkflowResource object. + + :param WorkflowDefinitionReference definition: (optional) Reference to a + workflow definition. + """ + self.definition = definition + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ProvidedWorkflowResource': + """Initialize a ProvidedWorkflowResource object from a json dictionary.""" + args = {} + if (definition := _dict.get('definition')) is not None: + args['definition'] = WorkflowDefinitionReference.from_dict(definition) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ProvidedWorkflowResource object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'definition') and self.definition is not None: + if isinstance(self.definition, dict): + _dict['definition'] = self.definition + else: + _dict['definition'] = self.definition.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ProvidedWorkflowResource object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ProvidedWorkflowResource') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ProvidedWorkflowResource') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class RevokeAccessResponse: + """ + This class holds the response message from the revoke access operation. + + :param str message: (optional) Response message of revoke access. + """ + + def __init__( + self, + *, + message: Optional[str] = None, + ) -> None: + """ + Initialize a RevokeAccessResponse object. + + :param str message: (optional) Response message of revoke access. + """ + self.message = message + + @classmethod + def from_dict(cls, _dict: Dict) -> 'RevokeAccessResponse': + """Initialize a RevokeAccessResponse object from a json dictionary.""" + args = {} + if (message := _dict.get('message')) is not None: + args['message'] = message + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a RevokeAccessResponse object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'message') and self.message is not None: + _dict['message'] = self.message + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this RevokeAccessResponse object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'RevokeAccessResponse') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'RevokeAccessResponse') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class RevokeAccessStateResponse: + """ + Revoke access states with pagination support. + + :param List[Asset] results: (optional) Holds revoke access state. + :param int total_count: (optional) Total number of rows available. + :param SearchAssetPaginationInfo next: (optional) Pagination information for the + next page of results. + """ + + def __init__( + self, + *, + results: Optional[List['Asset']] = None, + total_count: Optional[int] = None, + next: Optional['SearchAssetPaginationInfo'] = None, + ) -> None: + """ + Initialize a RevokeAccessStateResponse object. + + :param List[Asset] results: (optional) Holds revoke access state. + :param int total_count: (optional) Total number of rows available. + :param SearchAssetPaginationInfo next: (optional) Pagination information + for the next page of results. + """ + self.results = results + self.total_count = total_count + self.next = next + + @classmethod + def from_dict(cls, _dict: Dict) -> 'RevokeAccessStateResponse': + """Initialize a RevokeAccessStateResponse object from a json dictionary.""" + args = {} + if (results := _dict.get('results')) is not None: + args['results'] = [Asset.from_dict(v) for v in results] + if (total_count := _dict.get('total_count')) is not None: + args['total_count'] = total_count + if (next := _dict.get('next')) is not None: + args['next'] = SearchAssetPaginationInfo.from_dict(next) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a RevokeAccessStateResponse object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'results') and self.results is not None: + results_list = [] + for v in self.results: + if isinstance(v, dict): + results_list.append(v) + else: + results_list.append(v.to_dict()) + _dict['results'] = results_list + if hasattr(self, 'total_count') and self.total_count is not None: + _dict['total_count'] = self.total_count + if hasattr(self, 'next') and self.next is not None: + if isinstance(self.next, dict): + _dict['next'] = self.next + else: + _dict['next'] = self.next.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this RevokeAccessStateResponse object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'RevokeAccessStateResponse') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'RevokeAccessStateResponse') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class Roles: + """ + Represents a role associated with the contract. + + :param str role: (optional) The role associated with the contract. + """ + + def __init__( + self, + *, + role: Optional[str] = None, + ) -> None: + """ + Initialize a Roles object. + + :param str role: (optional) The role associated with the contract. + """ + self.role = role + + @classmethod + def from_dict(cls, _dict: Dict) -> 'Roles': + """Initialize a Roles object from a json dictionary.""" + args = {} + if (role := _dict.get('role')) is not None: + args['role'] = role + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a Roles object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'role') and self.role is not None: + _dict['role'] = self.role + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this Roles object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'Roles') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'Roles') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class SearchAssetPaginationInfo: + """ + Pagination information for the next page of results. + + :param str query: (optional) Search query for filtering results. + :param int limit: (optional) Number of items per page. + :param str bookmark: (optional) Bookmark for pagination. + :param str include: (optional) What to include in the results. + :param int skip: (optional) Number of items to skip. + """ + + def __init__( + self, + *, + query: Optional[str] = None, + limit: Optional[int] = None, + bookmark: Optional[str] = None, + include: Optional[str] = None, + skip: Optional[int] = None, + ) -> None: + """ + Initialize a SearchAssetPaginationInfo object. + + :param str query: (optional) Search query for filtering results. + :param int limit: (optional) Number of items per page. + :param str bookmark: (optional) Bookmark for pagination. + :param str include: (optional) What to include in the results. + :param int skip: (optional) Number of items to skip. + """ + self.query = query + self.limit = limit + self.bookmark = bookmark + self.include = include + self.skip = skip + + @classmethod + def from_dict(cls, _dict: Dict) -> 'SearchAssetPaginationInfo': + """Initialize a SearchAssetPaginationInfo object from a json dictionary.""" + args = {} + if (query := _dict.get('query')) is not None: + args['query'] = query + if (limit := _dict.get('limit')) is not None: + args['limit'] = limit + if (bookmark := _dict.get('bookmark')) is not None: + args['bookmark'] = bookmark + if (include := _dict.get('include')) is not None: + args['include'] = include + if (skip := _dict.get('skip')) is not None: + args['skip'] = skip + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a SearchAssetPaginationInfo object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'query') and self.query is not None: + _dict['query'] = self.query + if hasattr(self, 'limit') and self.limit is not None: + _dict['limit'] = self.limit + if hasattr(self, 'bookmark') and self.bookmark is not None: + _dict['bookmark'] = self.bookmark + if hasattr(self, 'include') and self.include is not None: + _dict['include'] = self.include + if hasattr(self, 'skip') and self.skip is not None: + _dict['skip'] = self.skip + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this SearchAssetPaginationInfo object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'SearchAssetPaginationInfo') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'SearchAssetPaginationInfo') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ServiceIdCredentials: + """ + Service id credentials. + + :param str name: (optional) Name of the api key of the service id. + :param datetime created_at: (optional) Created date of the api key of the + service id. + """ + + def __init__( + self, + *, + name: Optional[str] = None, + created_at: Optional[datetime] = None, + ) -> None: + """ + Initialize a ServiceIdCredentials object. + + :param str name: (optional) Name of the api key of the service id. + :param datetime created_at: (optional) Created date of the api key of the + service id. + """ + self.name = name + self.created_at = created_at + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ServiceIdCredentials': + """Initialize a ServiceIdCredentials object from a json dictionary.""" + args = {} + if (name := _dict.get('name')) is not None: + args['name'] = name + if (created_at := _dict.get('created_at')) is not None: + args['created_at'] = string_to_datetime(created_at) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ServiceIdCredentials object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'created_at') and self.created_at is not None: + _dict['created_at'] = datetime_to_string(self.created_at) + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ServiceIdCredentials object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ServiceIdCredentials') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ServiceIdCredentials') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class UseCase: + """ + UseCase. + + :param str id: The id of the use case associated with the data product. + :param str name: (optional) The display name of the use case associated with the + data product. + :param ContainerReference container: (optional) Container reference. + """ + + def __init__( + self, + id: str, + *, + name: Optional[str] = None, + container: Optional['ContainerReference'] = None, + ) -> None: + """ + Initialize a UseCase object. + + :param str id: The id of the use case associated with the data product. + :param str name: (optional) The display name of the use case associated + with the data product. + :param ContainerReference container: (optional) Container reference. + """ + self.id = id + self.name = name + self.container = container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'UseCase': + """Initialize a UseCase object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in UseCase JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a UseCase object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this UseCase object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'UseCase') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'UseCase') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class Visualization: + """ + Data members for visualization. + + :param str id: (optional) Visualization identifier. + :param str name: (optional) Visualization name. + """ + + def __init__( + self, + *, + id: Optional[str] = None, + name: Optional[str] = None, + ) -> None: + """ + Initialize a Visualization object. + + :param str id: (optional) Visualization identifier. + :param str name: (optional) Visualization name. + """ + self.id = id + self.name = name + + @classmethod + def from_dict(cls, _dict: Dict) -> 'Visualization': + """Initialize a Visualization object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + if (name := _dict.get('name')) is not None: + args['name'] = name + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a Visualization object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this Visualization object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'Visualization') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'Visualization') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class WorkflowDefinitionReference: + """ + Reference to a workflow definition. + + :param str id: (optional) ID of a workflow definition. + """ + + def __init__( + self, + *, + id: Optional[str] = None, + ) -> None: + """ + Initialize a WorkflowDefinitionReference object. + + :param str id: (optional) ID of a workflow definition. + """ + self.id = id + + @classmethod + def from_dict(cls, _dict: Dict) -> 'WorkflowDefinitionReference': + """Initialize a WorkflowDefinitionReference object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a WorkflowDefinitionReference object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this WorkflowDefinitionReference object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'WorkflowDefinitionReference') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'WorkflowDefinitionReference') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + +############################################################################## +# Pagers +############################################################################## + + +class DataProductsPager: + """ + DataProductsPager can be used to simplify the use of the "list_data_products" method. + """ + + def __init__( + self, + *, + client: DphV1, + limit: int = None, + ) -> None: + """ + Initialize a DataProductsPager object. + :param int limit: (optional) Limit the number of data products in the + results. The maximum limit is 200. + """ + self._has_next = True + self._client = client + self._page_context = {'next': None} + self._limit = limit + + def has_next(self) -> bool: + """ + Returns true if there are potentially more results to be retrieved. + """ + return self._has_next + + def get_next(self) -> List[dict]: + """ + Returns the next page of results. + :return: A List[dict], where each element is a dict that represents an instance of DataProductSummary. + :rtype: List[dict] + """ + if not self.has_next(): + raise StopIteration(message='No more results available') + + result = self._client.list_data_products( + limit=self._limit, + start=self._page_context.get('next'), + ).get_result() + + next = None + next_page_link = result.get('next') + if next_page_link is not None: + next = next_page_link.get('start') + self._page_context['next'] = next + if next is None: + self._has_next = False + + return result.get('data_products') + + def get_all(self) -> List[dict]: + """ + Returns all results by invoking get_next() repeatedly + until all pages of results have been retrieved. + :return: A List[dict], where each element is a dict that represents an instance of DataProductSummary. + :rtype: List[dict] + """ + results = [] + while self.has_next(): + next_page = self.get_next() + results.extend(next_page) + return results + + +class DataProductDraftsPager: + """ + DataProductDraftsPager can be used to simplify the use of the "list_data_product_drafts" method. + """ + + def __init__( + self, + *, + client: DphV1, + data_product_id: str, + asset_container_id: str = None, + version: str = None, + limit: int = None, + ) -> None: + """ + Initialize a DataProductDraftsPager object. + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str asset_container_id: (optional) Filter the list of data product + drafts by container id. + :param str version: (optional) Filter the list of data product drafts by + version number. + :param int limit: (optional) Limit the number of data product drafts in the + results. The maximum limit is 200. + """ + self._has_next = True + self._client = client + self._page_context = {'next': None} + self._data_product_id = data_product_id + self._asset_container_id = asset_container_id + self._version = version + self._limit = limit + + def has_next(self) -> bool: + """ + Returns true if there are potentially more results to be retrieved. + """ + return self._has_next + + def get_next(self) -> List[dict]: + """ + Returns the next page of results. + :return: A List[dict], where each element is a dict that represents an instance of DataProductDraftSummary. + :rtype: List[dict] + """ + if not self.has_next(): + raise StopIteration(message='No more results available') + + result = self._client.list_data_product_drafts( + data_product_id=self._data_product_id, + asset_container_id=self._asset_container_id, + version=self._version, + limit=self._limit, + start=self._page_context.get('next'), + ).get_result() + + next = None + next_page_link = result.get('next') + if next_page_link is not None: + next = next_page_link.get('start') + self._page_context['next'] = next + if next is None: + self._has_next = False + + return result.get('drafts') + + def get_all(self) -> List[dict]: + """ + Returns all results by invoking get_next() repeatedly + until all pages of results have been retrieved. + :return: A List[dict], where each element is a dict that represents an instance of DataProductDraftSummary. + :rtype: List[dict] + """ + results = [] + while self.has_next(): + next_page = self.get_next() + results.extend(next_page) + return results + + +class DataProductReleasesPager: + """ + DataProductReleasesPager can be used to simplify the use of the "list_data_product_releases" method. + """ + + def __init__( + self, + *, + client: DphV1, + data_product_id: str, + asset_container_id: str = None, + state: List[str] = None, + version: str = None, + limit: int = None, + ) -> None: + """ + Initialize a DataProductReleasesPager object. + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str asset_container_id: (optional) Filter the list of data product + releases by container id. + :param List[str] state: (optional) Filter the list of data product versions + by state. States are: available and retired. Default is + "available","retired". + :param str version: (optional) Filter the list of data product releases by + version number. + :param int limit: (optional) Limit the number of data product releases in + the results. The maximum is 200. + """ + self._has_next = True + self._client = client + self._page_context = {'next': None} + self._data_product_id = data_product_id + self._asset_container_id = asset_container_id + self._state = state + self._version = version + self._limit = limit + + def has_next(self) -> bool: + """ + Returns true if there are potentially more results to be retrieved. + """ + return self._has_next + + def get_next(self) -> List[dict]: + """ + Returns the next page of results. + :return: A List[dict], where each element is a dict that represents an instance of DataProductReleaseSummary. + :rtype: List[dict] + """ + if not self.has_next(): + raise StopIteration(message='No more results available') + + result = self._client.list_data_product_releases( + data_product_id=self._data_product_id, + asset_container_id=self._asset_container_id, + state=self._state, + version=self._version, + limit=self._limit, + start=self._page_context.get('next'), + ).get_result() + + next = None + next_page_link = result.get('next') + if next_page_link is not None: + next = next_page_link.get('start') + self._page_context['next'] = next + if next is None: + self._has_next = False + + return result.get('releases') + + def get_all(self) -> List[dict]: + """ + Returns all results by invoking get_next() repeatedly + until all pages of results have been retrieved. + :return: A List[dict], where each element is a dict that represents an instance of DataProductReleaseSummary. + :rtype: List[dict] + """ + results = [] + while self.has_next(): + next_page = self.get_next() + results.extend(next_page) + return results diff --git a/src/wxdi/dph_services/version.py b/src/wxdi/dph_services/version.py new file mode 100644 index 0000000..6061e76 --- /dev/null +++ b/src/wxdi/dph_services/version.py @@ -0,0 +1,20 @@ +# coding: utf-8 + +# (C) Copyright IBM Corp. 2019, 2020. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Version of dph_services +""" +__version__ = '1.0.0' diff --git a/src/wxdi/dq_validator/issue_reporting.py b/src/wxdi/dq_validator/issue_reporting.py index aae69c5..a067481 100644 --- a/src/wxdi/dq_validator/issue_reporting.py +++ b/src/wxdi/dq_validator/issue_reporting.py @@ -209,24 +209,32 @@ def get_check_id( def create_check( self, asset_id: str, - column_name: str, check_obj: BaseCheck, + column_name: Optional[str] = None, project_id: Optional[str] = None, - catalog_id: Optional[str] = None - ) -> str: + catalog_id: Optional[str] = None, + parent_id: Optional[str] = None, + ) -> dict: """ Create a data quality check. Args: - cams_asset_id: Data asset ID - column_name: Name of the column + asset_id: Data asset ID + column_name: Name of the column (required if parent_id is provided) check_obj: BaseCheck instance to extract check details from project_id: Project ID (optional) catalog_id: Catalog ID (optional) + parent_id: Parent check ID (optional). If provided, native_id includes column details Returns: - str: The created check ID + dict: The full check response body from the API + + Raises: + ValueError: If parent_id is provided but column_name is None """ + # Validate: column_name is required when parent_id is provided + if parent_id is not None and column_name is None: + raise ValueError("column_name is required when parent_id is provided") # Extract check details from check object check_name = check_obj.get_check_name() dimension_name = check_obj.get_dimension().name @@ -243,7 +251,7 @@ def create_check( # Special handling for comparison check to get operator native_id_suffix = "" - if check_name == "comparison_check": + if check_name == "comparison_check" and parent_id is not None: from .checks.comparison_check import ComparisonCheck if isinstance(check_obj, ComparisonCheck): @@ -263,21 +271,232 @@ def create_check( # Get dimension ID from dimension name dimension_id = self.dimension_provider.search_dimension(dimension_name) - # Build native_id with the suffix - column_name_lower = column_name.lower() - native_id = f"{asset_id}/{check_type}/{column_name_lower}/{native_id_suffix}" + # Build native_id based on whether parent_id is provided + if parent_id is not None: + # With parent: detailed format with column name and suffix + # column_name is guaranteed to be not None due to validation at line 236 + assert column_name is not None + column_name_lower = column_name.lower() + native_id = f"{asset_id}/{check_type}/{column_name_lower}/{native_id_suffix}" + else: + # No parent: simple format with dimension name + native_id = f"{asset_id}/{check_type}/{dimension_name.capitalize()}" - # Create the check and return the check_id - check_id = self.check_provider.create_check( + # Create the check and return the full check body + check_body = self.check_provider._create_check_full( name=cpd_name, dimension_id=dimension_id, native_id=native_id, check_type=check_type, project_id=project_id, - catalog_id=catalog_id + catalog_id=catalog_id, + parent_check_id=parent_id ) - return check_id + return check_body + + def handle_parent( + self, + asset_id: str, + check_obj: BaseCheck, + project_id: Optional[str] = None, + catalog_id: Optional[str] = None + ) -> dict: + """ + Search for parent check using search_dq_check method. + If not found, create the parent check. + + Args: + asset_id: Data asset ID + check_obj: BaseCheck instance to extract check details from + project_id: Project ID (optional) + catalog_id: Catalog ID (optional) + + Returns: + dict: The full parent check body (found or created) + + Raises: + Exception: If parent check creation fails (not search failure, but actual creation failure) + """ + # Extract check details from check object + check_name = check_obj.get_check_name() + dimension_name = check_obj.get_dimension().name + + # Map check_name to check_type + check_type = self.map_check_name_to_check_type(check_name) or check_name + + # Construct native_id + native_id = f"{asset_id}/{check_type}/{dimension_name.capitalize()}" + + try: + # Search for the check using search_dq_check + check_response = self.search_provider.search_dq_check( + native_id=native_id, + check_type=check_type, + project_id=project_id, + catalog_id=catalog_id, + include_children=False + ) + + # Extract and return the check ID + return check_response + except Exception: + # Check not found during search - this is expected, so we'll try to create it + # Now attempt to create the parent check + try: + parent_check = self.create_check( + asset_id=asset_id, + column_name=None, + check_obj=check_obj, + project_id=project_id, + catalog_id=catalog_id, + parent_id=None + ) + # Mark that this check was newly created + parent_check["_newly_created"] = True + return parent_check + except Exception as creation_error: + # Parent check creation failed - raise a more specific exception + raise RuntimeError( + f"Failed to create parent check for asset_id='{asset_id}', " + f"check_type='{check_type}', dimension='{dimension_name}'. " + f"Original error: {str(creation_error)}" + ) from creation_error + + def create_bulk_issues( + self, + parent_check: dict, + child_check: dict, + column_name: str, + assets_map: Dict[str, Dict], + number_of_occurrences: int, + total_records: int, + project_id: str + ) -> dict: + """ + Create bulk issues for parent and child checks in a single API call. + + Args: + parent_check: Parent check body (table-level) + child_check: Child check body (column-level) + column_name: Name of the column + assets_map: Map of asset names to full asset objects (includes both data_asset and columns) + number_of_occurrences: Number of failed occurrences + total_records: Total number of records + project_id: Project ID + + Returns: + dict: Response from the bulk issue creation API + """ + # Fetch column asset from map + column_asset = assets_map.get(column_name) + if not column_asset: + raise ValueError(f"Column asset not found for column: {column_name}") + + # Get parent asset ID from column asset + parent_asset_id = column_asset.get("parent", {}).get("id") + if not parent_asset_id: + raise ValueError(f"Parent asset ID not found in column asset for column: {column_name}") + + # Find parent asset in the map by searching for asset with matching ID + parent_asset = None + for asset_name, asset_body in assets_map.items(): + if asset_body.get("id") == parent_asset_id: + parent_asset = asset_body + break + + if not parent_asset: + raise ValueError(f"Parent asset not found in assets_map for ID: {parent_asset_id}") + + # Extract native_id from parent asset + parent_native_id = parent_asset.get("native_id") + if not parent_native_id: + raise ValueError("Parent asset native_id not found") + + # Build issues array + issues = [ + { + "check": { + "native_id": parent_check.get("native_id"), + "type": parent_check.get("type") + }, + "reported_for": { + "native_id": parent_native_id, + "type": "data_asset" + }, + "number_of_occurrences": number_of_occurrences, + "number_of_tested_records": total_records, + "status": "aggregation", + "ignored": False + }, + { + "check": { + "native_id": child_check.get("native_id"), + "type": child_check.get("type") + }, + "reported_for": { + "native_id": column_asset.get("native_id"), + "type": "column" + }, + "number_of_occurrences": number_of_occurrences, + "number_of_tested_records": total_records, + "status": "actual", + "ignored": False + } + ] + + # Build assets array + assets = [ + { + "name": parent_asset.get("name"), + "type": "data_asset", + "native_id": parent_native_id, + "weight": parent_asset.get("weight", 1) + }, + { + "name": column_asset.get("name"), + "type": "column", + "native_id": column_asset.get("native_id"), + "parent": { + "native_id": parent_native_id, + "type": "data_asset" + }, + "weight": column_asset.get("weight", 1) + } + ] + + # Build existing_checks array + existing_checks = [ + { + "native_id": parent_check.get("native_id"), + "type": parent_check.get("type") + }, + { + "native_id": child_check.get("native_id"), + "type": child_check.get("type") + } + ] + + # Construct the bulk payload + bulk_payload = { + "issues": issues, + "assets": assets, + "existing_checks": existing_checks + } + + # Call the bulk issue creation API + try: + response = self.issues_provider.create_issues_bulk( + payload=bulk_payload, + project_id=project_id, + incremental_reporting=False, + refresh_assets=False + ) + print(f"Bulk issues created successfully: {len(issues)} issues") + return response + except Exception as e: + print(f"Failed to create bulk issues: {str(e)}") + raise def _validate_and_prepare_check_data( self, @@ -285,7 +504,7 @@ def _validate_and_prepare_check_data( check_name: str, stats: Dict[str, int], data_asset_entity, - column_id_map: Dict[str, str], + assets_map: Dict[str, Dict], validator: Validator ) -> Optional[Tuple[str, str, BaseCheck, int, int]]: """ @@ -296,7 +515,7 @@ def _validate_and_prepare_check_data( check_name: Name of the check stats: Statistics dictionary with 'failed' and 'total' keys data_asset_entity: Data asset entity from CAMS - column_id_map: Map of column names to column asset IDs + assets_map: Map of asset names to full column asset objects validator: Validator instance Returns: @@ -319,8 +538,13 @@ def _validate_and_prepare_check_data( if not (data_asset_entity.column_info and column_name in data_asset_entity.column_info): return None - # Guard: Skip if column asset ID not found - column_id = column_id_map.get(column_name) + # Guard: Skip if column asset not found + column_asset = assets_map.get(column_name) + if not column_asset: + return None + + # Extract column ID from asset + column_id = column_asset.get('id') if not column_id: return None @@ -331,6 +555,138 @@ def _validate_and_prepare_check_data( return (check_type, column_id, check_obj, number_of_occurrences, total_records) + def _find_existing_check( + self, + column_id: str, + check_type: str, + project_id: str + ) -> Optional[Tuple[str, Optional[str]]]: + """ + Find an existing check by column ID and check type. + + Args: + column_id: Column asset ID + check_type: Type of check (e.g., "format", "completeness") + project_id: Project ID + + Returns: + Tuple of (check_id, native_id) if found, None otherwise + """ + try: + checks = self.check_provider.get_checks( + dq_asset_id=column_id, + check_type=check_type, + project_id=project_id + ) + + for check in checks: + if check.get("type") == check_type: + return (check.get("id"), check.get("native_id")) + + return None + except ValueError as e: + print(f"Warning: Failed to get existing checks: {e}") + return None + + def _update_existing_check_metrics( + self, + existing_check_native_id: Optional[str], + number_of_occurrences: int, + total_records: int, + column_name: str, + check_type: str, + project_id: str + ) -> bool: + """ + Update metrics for an existing check. + + Args: + existing_check_native_id: Native ID of the existing check + number_of_occurrences: Number of failed occurrences + total_records: Total number of records + column_name: Name of the column + check_type: Type of check + project_id: Project ID + + Returns: + True if update successful, False otherwise + """ + try: + self.issues_provider.update_issue_metrics( + occurrences=number_of_occurrences, + tested_records=total_records, + column_name=column_name, + check_type=check_type, + project_id=project_id, + asset_type="column", + operation="add", + check_native_id=existing_check_native_id + ) + return True + except ValueError: + return False + + def _handle_409_conflict( + self, + column_id: str, + check_type: str, + number_of_occurrences: int, + total_records: int, + column_name: str, + check_name: str, + asset_id: str, + project_id: str + ) -> bool: + """ + Handle 409 conflict by finding and updating existing check. + + Args: + column_id: Column asset ID + check_type: Type of check + number_of_occurrences: Number of failed occurrences + total_records: Total number of records + column_name: Name of the column + check_name: Name of the check + asset_id: CAMS asset ID + project_id: Project ID + + Returns: + True if conflict handled successfully, False otherwise + """ + existing_check = self._find_existing_check(column_id, check_type, project_id) + + if not existing_check: + print(f"Warning: Check already exists but could not be found for column '{column_name}', check '{check_name}'") + return False + + existing_check_id, existing_check_native_id = existing_check + + update_success = self._update_existing_check_metrics( + existing_check_native_id, + number_of_occurrences, + total_records, + column_name, + check_type, + project_id + ) + + if update_success: + return True + + # If update fails, try to create the issue + self._handle_update_failure( + ValueError("Update failed"), + asset_id, + check_type, + column_name, + column_id, + number_of_occurrences, + total_records, + project_id, + dq_check_id=existing_check_id + ) + return True + def _create_check_and_issue( self, asset_id: str, @@ -340,7 +696,8 @@ def _create_check_and_issue( check_obj: BaseCheck, number_of_occurrences: int, total_records: int, - project_id: str + project_id: str, + assets_map: Dict[str, Dict] ) -> bool: """ Create a check and its associated issue. @@ -355,6 +712,7 @@ def _create_check_and_issue( number_of_occurrences: Number of failed occurrences total_records: Total number of records project_id: Project ID + assets_map: Map of asset names to full asset objects Returns: True if successful, False if creation failed @@ -366,88 +724,84 @@ def _create_check_and_issue( return False try: - check_id = self.create_check( + # Get parent check - may raise exception if parent creation fails + parent_check = self.handle_parent( asset_id=asset_id, - column_name=column_name, check_obj=check_obj, project_id=project_id ) - # Create the issue directly using the issues provider - self.issues_provider.create_issue( - check_id=check_id, - reported_for_id=column_id, - number_of_occurrences=number_of_occurrences, - number_of_tested_records=total_records, + + # Extract parent ID from parent check + parent_id = parent_check.get("id") + + # Check if parent was newly created + parent_was_created = parent_check.get("_newly_created", False) + + # Create child check + check = self.create_check( + asset_id=asset_id, + column_name=column_name, + check_obj=check_obj, project_id=project_id, - catalog_id=None + parent_id=parent_id ) + check_id = check.get("id") + if not check_id: + raise ValueError("Check ID not found in response") + + # If parent was newly created, use bulk issue creation + if parent_was_created and parent_check: + self.create_bulk_issues( + parent_check=parent_check, + child_check=check, + column_name=column_name, + assets_map=assets_map, + number_of_occurrences=number_of_occurrences, + total_records=total_records, + project_id=project_id + ) + else: + # Create the issue directly using the issues provider + self.issues_provider.create_issue( + dq_check_id=check_id, + reported_for_id=column_id, + number_of_occurrences=number_of_occurrences, + number_of_tested_records=total_records, + project_id=project_id, + catalog_id=None + ) return True - except ValueError as e: + except Exception as e: error_msg = str(e) # If 409 conflict (check already exists), try to find and update it if "409" in error_msg or "already exists" in error_msg: - try: - # Get existing checks for this column asset filtered by check type - checks = self.check_provider.get_checks( - asset_id=column_id, - check_type=check_type, - project_id=project_id - ) - - # Find the check with matching type - existing_check_id = None - existing_check_native_id = None - for check in checks: - if check.get("type") == check_type: - existing_check_id = check.get("id") - existing_check_native_id = check.get("native_id") - break - - if existing_check_id: - # Update issue metrics for the existing check - try: - self.issues_provider.update_issue_metrics( - occurrences=number_of_occurrences, - tested_records=total_records, - column_name=column_name, - check_type=check_type, - project_id=project_id, - asset_type="column", - operation="add", - check_native_id=existing_check_native_id - ) - return True - except ValueError as update_error: - # If update fails, try to create the issue - self._handle_update_failure( - update_error, asset_id, existing_check_id, check_type, - column_name, check_name, column_id, - number_of_occurrences, total_records, project_id - ) - return True - else: - print(f"Warning: Check already exists but could not be found for column '{column_name}', check '{check_name}'") - return False - except ValueError as get_error: - print(f"Warning: Failed to get existing checks for column '{column_name}', check '{check_name}': {get_error}") - return False + return self._handle_409_conflict( + column_id=column_id, + check_type=check_type, + number_of_occurrences=number_of_occurrences, + total_records=total_records, + column_name=column_name, + check_name=check_name, + asset_id=asset_id, + project_id=project_id + ) else: - # Different error - log and return False - print(f"Warning: Failed to create check for column '{column_name}', check '{check_name}': {e}") + # Any other error (including parent check creation failure) - log and return False + print(f"Error: Failed to create check and issue for column '{column_name}', check '{check_name}': {e}") return False def _handle_update_failure( self, error: ValueError, asset_id: str, - cams_check_id: str, check_type: str, column_name: str, - check_name: str, column_id: str, number_of_occurrences: int, total_records: int, - project_id: str + project_id: str, + check_id: Optional[str] = None, + dq_check_id: Optional[str] = None ) -> bool: """ Handle failure when updating issue metrics. @@ -455,32 +809,42 @@ def _handle_update_failure( Args: error: The ValueError that was raised asset_id: CAMS asset ID - cams_check_id: CAMS check ID check_type: Type of the check column_name: Name of the column - check_name: Name of the check column_id: Column asset ID number_of_occurrences: Number of failed occurrences total_records: Total number of records project_id: Project ID + check_id: Check ID (optional) + dq_check_id: DQ check ID (optional, if provided will be used directly) Returns: True (always, to indicate error was handled) """ + # Validate that at least one check ID is provided + if not dq_check_id and not check_id: + print(f"Warning: Neither dq_check_id nor check_id provided for column '{column_name}', of check_type '{check_type}'. Cannot handle update failure.") + return False + error_msg = str(error) # If issue not found, try to create it if "Issue not found" in error_msg or "Issue ID not found" in error_msg: - check_id = self.get_check_id( - check_native_id=f"{asset_id}/{cams_check_id}", - check_type=check_type, - project_id=project_id - ) + # Use provided dq_check_id or fetch it + check_id_to_use = dq_check_id - if check_id: + if not check_id_to_use: + # Only fetch check_id if dq_check_id is not provided + check_id_to_use = self.get_check_id( + check_native_id=f"{asset_id}/{check_id}", + check_type=check_type, + project_id=project_id + ) + + if check_id_to_use: # Create the issue directly using the issues provider self.issues_provider.create_issue( - check_id=cams_check_id, + dq_check_id=check_id_to_use, reported_for_id=column_id, number_of_occurrences=number_of_occurrences, number_of_tested_records=total_records, @@ -488,10 +852,10 @@ def _handle_update_failure( catalog_id=None ) else: - print(f"Warning: Could not find check_id for column '{column_name}', check '{check_name}'. Issue not created.") + print(f"Warning: Could not find check_id for column '{column_name}', of check_type '{check_type}'. Issue not created.") else: # Different error - log and continue - print(f"Warning: Failed to update issue metrics for column '{column_name}', check '{check_name}': {error}") + print(f"Warning: Failed to update issue metrics for column '{column_name}', of check_type '{check_type}': {error}") return True @@ -501,7 +865,6 @@ def _handle_existing_check( check_type: str, asset_id: str, column_name: str, - check_name: str, column_id: str, number_of_occurrences: int, total_records: int, @@ -528,15 +891,15 @@ def _handle_existing_check( if check.metadata.type != check_type: continue - cams_check_id = check.metadata.check_id - if not cams_check_id: + check_id = check.metadata.check_id + if not check_id: continue # Try to update existing issue metrics try: self.issues_provider.update_issue_metrics( - cams_asset_id=asset_id, - cams_check_id=cams_check_id, + asset_id=asset_id, + check_id=check_id, occurrences=number_of_occurrences, tested_records=total_records, column_name=column_name, @@ -548,9 +911,15 @@ def _handle_existing_check( return True except ValueError as e: self._handle_update_failure( - e, asset_id, cams_check_id, check_type, - column_name, check_name, column_id, - number_of_occurrences, total_records, project_id + e, + asset_id, + check_type, + column_name, + column_id, + number_of_occurrences, + total_records, + project_id, + check_id=check_id ) return True @@ -607,14 +976,14 @@ def report_issues( data_asset_entity = data_asset.entity # Fetch all column assets once and build a lookup map for efficiency - assets_response = self.asset_provider.get_assets(project_id=project_id, asset_type="column") - column_id_map = {asset['name']: asset['id'] for asset in assets_response.get('assets', [])} + assets_response = self.asset_provider.get_assets(project_id=project_id) + assets_map = {asset['name']: asset for asset in assets_response.get('assets', [])} # Iterate over combined statistics for (column_name, check_name), individual_stats in combined_stats.items(): # Validate and prepare data validated_data = self._validate_and_prepare_check_data( - column_name, check_name, individual_stats, data_asset_entity, column_id_map, validator + column_name, check_name, individual_stats, data_asset_entity, assets_map, validator ) if not validated_data: continue @@ -627,19 +996,21 @@ def report_issues( if not column_info.column_checks: self._create_check_and_issue( asset_id, column_name, column_id, check_name, - check_obj, number_of_occurrences, total_records, project_id + check_obj, number_of_occurrences, total_records, project_id, + assets_map ) continue # Try to handle existing check check_handled = self._handle_existing_check( - column_info, check_type, asset_id, column_name, check_name, - column_id, number_of_occurrences, total_records, project_id + column_info, check_type, asset_id, column_name, column_id, + number_of_occurrences, total_records, project_id ) # If check not found, create it if not check_handled: self._create_check_and_issue( asset_id, column_name, column_id, check_name, - check_obj, number_of_occurrences, total_records, project_id + check_obj, number_of_occurrences, total_records, project_id, + assets_map ) diff --git a/src/wxdi/dq_validator/provider/checks.py b/src/wxdi/dq_validator/provider/checks.py index 24c4392..f573955 100644 --- a/src/wxdi/dq_validator/provider/checks.py +++ b/src/wxdi/dq_validator/provider/checks.py @@ -18,12 +18,16 @@ """ import json -from typing import Optional +from typing import Optional, Dict, Any from .base_provider import BaseProvider from .config import ProviderConfig from ..utils import get_request_headers +# Error message constants +_ERR_MISSING_PROJECT_OR_CATALOG = "Either project_id or catalog_id must be provided" +_ERR_BOTH_PROJECT_AND_CATALOG = "Only one of project_id or catalog_id should be provided, not both" + class ChecksProvider(BaseProvider): """Provider for managing data quality checks. @@ -66,11 +70,15 @@ def create_check( native_id: str, check_type: Optional[str] = None, project_id: Optional[str] = None, - catalog_id: Optional[str] = None + catalog_id: Optional[str] = None, + parent_check_id: Optional[str] = None ) -> str: """ Create a new check for a data asset. + Note: Table-level checks are created without a parent_check_id, while column-level checks + require the table-level check ID as parent_check_id to establish the hierarchical relationship. + Args: name: Name of the check (e.g., "check_uniqueness_of_id") dimension_id: The dimension ID for the check @@ -78,9 +86,11 @@ def create_check( check_type: Type of check (optional, defaults to the check name if not provided) project_id (str, optional): The project ID containing the check catalog_id (str, optional): The catalog ID containing the check + parent_check_id (str, optional): The parent check ID. Required for column-level checks + (use table-level check ID). Omit for table-level checks. Returns: - str: The check_id of the created check + str: The check ID from the created check Raises: ValueError: If the API request fails or returns an error status, or if neither @@ -94,14 +104,86 @@ def create_check( ... project_id="project-123" ... ) '6be18374-573a-4cf8-8ab7-e428506e428b' + >>> # With parent parameter + >>> provider.create_check( + ... name="Format check", + ... dimension_id="ec453723-669c-48bb-82c1-11b69b3b8c93", + ... native_id="ba23145a-6d0a-46db-b314-41526b1e465f/format/sample3", + ... project_id="project-123", + ... parent_check_id="848aaddc-7401-4a43-ad2b-96a0946d4674" + ... ) + '7be18374-573a-4cf8-8ab7-e428506e428c' + """ + # Call _create_check_full and extract just the ID + result = self._create_check_full( + name=name, + dimension_id=dimension_id, + native_id=native_id, + check_type=check_type, + project_id=project_id, + catalog_id=catalog_id, + parent_check_id=parent_check_id + ) + return result["id"] + + def _create_check_full( + self, + name: str, + dimension_id: str, + native_id: str, + check_type: Optional[str] = None, + project_id: Optional[str] = None, + catalog_id: Optional[str] = None, + parent_check_id: Optional[str] = None + ) -> Dict[str, Any]: + """ + Create a new check and return the full check body. + + This method creates a check and returns the complete check object including + all properties like id, name, type, native_id, parent, etc. + + Note: Table-level checks are created without a parent_check_id, while column-level checks + require the table-level check ID as parent_check_id to establish the hierarchical relationship. + + Args: + name: Name of the check + dimension_id: ID of the dimension this check belongs to + native_id: Native identifier for the check + check_type: Type of check (defaults to name if not provided) + project_id: Project ID (mutually exclusive with catalog_id) + catalog_id: Catalog ID (mutually exclusive with project_id) + parent_check_id: Optional parent check ID for hierarchical checks. Required for column-level + checks (use table-level check ID). Omit for table-level checks. + + Returns: + Dict containing the full check body with all properties + + Raises: + ValueError: If neither or both project_id and catalog_id are provided, + or if the API request fails + + Example: + >>> from wxdi.dq_validator.provider import ProviderConfig, ChecksProvider + >>> config = ProviderConfig(url="https://example.com", auth_token="token") + >>> provider = ChecksProvider(config) + >>> check = provider.create_check_full( + ... name="Format check", + ... dimension_id="ec453723-669c-48bb-82c1-11b69b3b8c93", + ... native_id="ba23145a-6d0a-46db-b314-41526b1e465f/format/sample3", + ... project_id="project-123", + ... parent_check_id="848aaddc-7401-4a43-ad2b-96a0946d4674" + ... ) + >>> print(check) + {'id': '7be18374-573a-4cf8-8ab7-e428506e428c', 'name': 'Format check', + 'type': 'format', 'native_id': '...', 'parent': {'id': '...'}, ...} """ from ..utils import get_url_with_query_params # Validate that exactly one of project_id or catalog_id is provided if project_id is None and catalog_id is None: - raise ValueError("Either project_id or catalog_id must be provided") + raise ValueError(_ERR_MISSING_PROJECT_OR_CATALOG) if project_id is not None and catalog_id is not None: - raise ValueError("Only one of project_id or catalog_id should be provided, not both") + raise ValueError(_ERR_BOTH_PROJECT_AND_CATALOG) # Default check_type to name if not provided if check_type is None: @@ -129,6 +211,12 @@ def create_check( "details": '{"origin": "SDK"}' } + # Add parent only if provided + if parent_check_id is not None: + payload["parent"] = { + "id": parent_check_id + } + # Get request headers headers = get_request_headers(self.config.auth_token) @@ -147,17 +235,16 @@ def create_check( f"Response: {response.text}" ) - # Parse response and extract check_id + # Parse response and return full check body result = json.loads(response.text) - check_id = result.get("id") - if not check_id: + if not result.get("id"): raise ValueError("Check ID not found in response") - return check_id + return result def get_checks( self, - asset_id: str, + dq_asset_id: str, check_type: str, project_id: Optional[str] = None, catalog_id: Optional[str] = None, @@ -167,7 +254,7 @@ def get_checks( Get all checks for a specific asset filtered by check type. Args: - asset_id: The data quality asset identifier (column asset ID) + dq_asset_id: The data quality asset identifier (column asset ID) check_type: Type of check to filter by (e.g., "case", "completeness", "comparison") project_id (str, optional): The project ID containing the checks catalog_id (str, optional): The catalog ID containing the checks @@ -182,7 +269,7 @@ def get_checks( Example: >>> provider.get_checks( - ... asset_id="column-asset-123", + ... dq_asset_id="column-asset-123", ... check_type="case", ... project_id="project-123" ... ) @@ -192,16 +279,16 @@ def get_checks( # Validate that exactly one of project_id or catalog_id is provided if project_id is None and catalog_id is None: - raise ValueError("Either project_id or catalog_id must be provided") + raise ValueError(_ERR_MISSING_PROJECT_OR_CATALOG) if project_id is not None and catalog_id is not None: - raise ValueError("Only one of project_id or catalog_id should be provided, not both") + raise ValueError(_ERR_BOTH_PROJECT_AND_CATALOG) # Build the URL for checks API url = f"{self.config.url}/data_quality/v4/checks" # Add query parameters params = { - "asset.id": asset_id, + "asset.id": dq_asset_id, "type": check_type, "include_children": str(include_children).lower() } diff --git a/src/wxdi/dq_validator/provider/constraint_model.py b/src/wxdi/dq_validator/provider/constraint_model.py index a834cd9..72c949a 100644 --- a/src/wxdi/dq_validator/provider/constraint_model.py +++ b/src/wxdi/dq_validator/provider/constraint_model.py @@ -17,7 +17,6 @@ Pydantic models for Data Quality Constraints """ -from enum import StrEnum from typing import List, Optional, Any, Dict from datetime import datetime from pydantic import BaseModel @@ -33,6 +32,13 @@ from wxdi.dq_validator.checks.valid_values_check import ValidValuesCheck from wxdi.dq_validator.datatypes import DataType +try: + from enum import StrEnum +except ImportError: + # Remove when support for Python 3.10 is removed + from enum import Enum + class StrEnum(str, Enum): + pass class CheckType(StrEnum): """Enumeration of data quality check types""" diff --git a/src/wxdi/dq_validator/provider/issues.py b/src/wxdi/dq_validator/provider/issues.py index addfc4a..f8b1155 100644 --- a/src/wxdi/dq_validator/provider/issues.py +++ b/src/wxdi/dq_validator/provider/issues.py @@ -39,6 +39,10 @@ class IssuesProvider(BaseProvider): >>> provider = IssuesProvider(config) >>> provider.update_issue_values("issue-123", "project-123", occurrences=10, tested_records=100) """ + + # Error message constants + _ERR_MISSING_PROJECT_OR_CATALOG = "Either project_id or catalog_id must be provided" + _ERR_BOTH_PROJECT_AND_CATALOG = "Only one of project_id or catalog_id should be provided, not both" def __init__(self, config: ProviderConfig): """Initialize the IssuesProvider with configuration. @@ -78,9 +82,9 @@ def _patch_issue_field( # Validate that exactly one of project_id or catalog_id is provided if project_id is None and catalog_id is None: - raise ValueError("Either project_id or catalog_id must be provided") + raise ValueError(self._ERR_MISSING_PROJECT_OR_CATALOG) if project_id is not None and catalog_id is not None: - raise ValueError("Only one of project_id or catalog_id should be provided, not both") + raise ValueError(self._ERR_BOTH_PROJECT_AND_CATALOG) url = f"{self.config.url}/data_quality/v4/issues/{issue_id}" @@ -171,9 +175,9 @@ def update_issue_values( # Validate that exactly one of project_id or catalog_id is provided if project_id is None and catalog_id is None: - raise ValueError("Either project_id or catalog_id must be provided") + raise ValueError(self._ERR_MISSING_PROJECT_OR_CATALOG) if project_id is not None and catalog_id is not None: - raise ValueError("Only one of project_id or catalog_id should be provided, not both") + raise ValueError(self._ERR_BOTH_PROJECT_AND_CATALOG) # Build patch operations list patch_operations = [] @@ -227,7 +231,7 @@ def update_issue_values( def get_issue( self, reported_for_id: str, - check_id: str, + dq_check_id: str, project_id: Optional[str] = None, catalog_id: Optional[str] = None ) -> dict: @@ -238,7 +242,7 @@ def get_issue( Args: reported_for_id (str): The DQ asset ID to search for - check_id (str): The check ID to search for + dq_check_id (str): The check ID to search for project_id (str, optional): The project ID containing the issue catalog_id (str, optional): The catalog ID containing the issue @@ -253,13 +257,13 @@ def get_issue( >>> # Using project_id >>> provider.get_issue( ... reported_for_id="1488a413-99f9-4bed-906d-c33b505d5728", - ... check_id="ad277842-dea7-44ef-8e4b-d940df0f79aa", + ... dq_check_id="ad277842-dea7-44ef-8e4b-d940df0f79aa", ... project_id="24419069-d649-45cb-a2c1-64d6eed650d5" ... ) >>> # Using catalog_id >>> provider.get_issue( ... reported_for_id="1488a413-99f9-4bed-906d-c33b505d5728", - ... check_id="ad277842-dea7-44ef-8e4b-d940df0f79aa", + ... dq_check_id="ad277842-dea7-44ef-8e4b-d940df0f79aa", ... catalog_id="catalog-123" ... ) { @@ -272,16 +276,16 @@ def get_issue( # Validate that exactly one of project_id or catalog_id is provided if project_id is None and catalog_id is None: - raise ValueError("Either project_id or catalog_id must be provided") + raise ValueError(self._ERR_MISSING_PROJECT_OR_CATALOG) if project_id is not None and catalog_id is not None: - raise ValueError("Only one of project_id or catalog_id should be provided, not both") + raise ValueError(self._ERR_BOTH_PROJECT_AND_CATALOG) url = f"{self.config.url}/data_quality/v4/search_dq_issue" # Build query parameters params = { "reported_for.id": reported_for_id, - "check.id": check_id, + "check.id": dq_check_id, } # Add either project_id or catalog_id @@ -309,7 +313,7 @@ def get_issue( def get_issue_id( self, reported_for_id: str, - check_id: str, + dq_check_id: str, project_id: Optional[str] = None, catalog_id: Optional[str] = None ) -> str: @@ -320,7 +324,7 @@ def get_issue_id( Args: reported_for_id (str): The DQ asset ID to search for - check_id (str): The check ID to search for + dq_check_id (str): The check ID to search for project_id (str, optional): The project ID containing the issue catalog_id (str, optional): The catalog ID containing the issue @@ -335,23 +339,72 @@ def get_issue_id( >>> # Using project_id >>> provider.get_issue_id( ... reported_for_id="1488a413-99f9-4bed-906d-c33b505d5728", - ... check_id="ad277842-dea7-44ef-8e4b-d940df0f79aa", + ... dq_check_id="ad277842-dea7-44ef-8e4b-d940df0f79aa", ... project_id="24419069-d649-45cb-a2c1-64d6eed650d5" ... ) >>> # Using catalog_id >>> provider.get_issue_id( ... reported_for_id="1488a413-99f9-4bed-906d-c33b505d5728", - ... check_id="ad277842-dea7-44ef-8e4b-d940df0f79aa", + ... dq_check_id="ad277842-dea7-44ef-8e4b-d940df0f79aa", ... catalog_id="catalog-123" ... ) 'b8f4252b-cd35-4668-9b35-4635bfc6e2e0' """ - issue = self.get_issue(reported_for_id, check_id, project_id, catalog_id) + issue = self.get_issue(reported_for_id, dq_check_id, project_id, catalog_id) issue_id = issue.get("id") if issue_id is None: raise ValueError("Issue ID not found in response") return issue_id + def _validate_and_resolve_ids( + self, + asset_id: Optional[str], + check_id: Optional[str], + check_native_id: Optional[str] + ) -> tuple[str, str, str]: + """Validate and resolve asset_id, check_id, and check_native_id. + + Args: + asset_id: The CAMS data asset ID (optional) + check_id: The CAMS check ID (optional) + check_native_id: The check native_id (optional) + + Returns: + tuple: (asset_id, check_id, check_native_id) all resolved + + Raises: + ValueError: If validation fails or IDs cannot be resolved + """ + SEPARATOR = '/' + + # Validate that either (asset_id + check_id) or check_native_id is provided + has_cams_ids = asset_id is not None and check_id is not None + has_native_id = check_native_id is not None + + if not has_cams_ids and not has_native_id: + raise ValueError("Either (asset_id and check_id) or check_native_id must be provided") + + # If check_native_id is provided without asset_id/check_id, extract them + if has_native_id and not has_cams_ids: + # Parse check_native_id: first part before first SEPARATOR is asset_id, rest is check_id + # Format: "/" where check_id can contain slashes + first_slash_index = check_native_id.find(SEPARATOR) + if first_slash_index == -1: + raise ValueError(f"Invalid check_native_id format (missing {SEPARATOR}): {check_native_id}") + asset_id = check_native_id[:first_slash_index] + check_id = check_native_id[first_slash_index + 1:] + elif not has_native_id: + # Construct check_native_id from asset_id and check_id + # At this point, has_cams_ids is guaranteed to be True + check_native_id = f"{asset_id}{SEPARATOR}{check_id}" + + # At this point, all IDs should be set + assert asset_id is not None, "asset_id should be set by now" + assert check_id is not None, "check_id should be set by now" + assert check_native_id is not None, "check_native_id should be set by now" + + return asset_id, check_id, check_native_id + def update_issue_metrics( self, occurrences: int, @@ -362,8 +415,8 @@ def update_issue_metrics( catalog_id: Optional[str] = None, asset_type: str = "column", operation: str = "add", - cams_asset_id: Optional[str] = None, - cams_check_id: Optional[str] = None, + asset_id: Optional[str] = None, + check_id: Optional[str] = None, check_native_id: Optional[str] = None ) -> dict: """Update issue metrics using CAMS asset and check IDs or check native_id. @@ -381,10 +434,10 @@ def update_issue_metrics( catalog_id (str, optional): The catalog ID containing the issue asset_type (str, optional): The type of asset ("column" or "table"). Default is "column" operation (str, optional): Operation for both metrics - "add" or "replace". Default is "add" - cams_asset_id (str, optional): The CAMS data asset ID (required if check_native_id not provided) - cams_check_id (str, optional): The CAMS check ID (required if check_native_id not provided) - check_native_id (str, optional): The check native_id (required if cams_asset_id and cams_check_id not provided). - Format: "/" where cams_check_id can contain slashes + asset_id (str, optional): The CAMS data asset ID (required if check_native_id not provided) + check_id (str, optional): The CAMS check ID (required if check_native_id not provided) + check_native_id (str, optional): The check native_id (required if asset_id and check_id not provided). + Format: "/" where check_id can contain slashes Returns: dict: The response from the API containing the updated issue data @@ -392,13 +445,13 @@ def update_issue_metrics( Raises: ValueError: If the API request fails or returns an error status, or if neither project_id nor catalog_id is provided, or if both are provided, or if neither - (cams_asset_id + cams_check_id) nor check_native_id is provided + (asset_id + check_id) nor check_native_id is provided Example: - >>> # Using cams_asset_id and cams_check_id with project_id + >>> # Using asset_id and check_id with project_id >>> provider.update_issue_metrics( - ... cams_asset_id="b2debda2-6ab9-4a39-8c23-17954e004dcf", - ... cams_check_id="7377e2cd-ac0e-4833-8760-fd0e8cb682aa", + ... asset_id="b2debda2-6ab9-4a39-8c23-17954e004dcf", + ... check_id="7377e2cd-ac0e-4833-8760-fd0e8cb682aa", ... occurrences=10, ... tested_records=100, ... column_name="RTN", @@ -426,39 +479,16 @@ def update_issue_metrics( ... ) {'issue_id': 'b8f4252b-cd35-4668-9b35-4635bfc6e2e0', 'number_of_occurrences': 10, ...} """ - # Separator used in native_id format - SEPARATOR = '/' - # Validate that exactly one of project_id or catalog_id is provided if project_id is None and catalog_id is None: - raise ValueError("Either project_id or catalog_id must be provided") + raise ValueError(self._ERR_MISSING_PROJECT_OR_CATALOG) if project_id is not None and catalog_id is not None: - raise ValueError("Only one of project_id or catalog_id should be provided, not both") - - # Validate that either (cams_asset_id + cams_check_id) or check_native_id is provided - has_cams_ids = cams_asset_id is not None and cams_check_id is not None - has_native_id = check_native_id is not None - - if not has_cams_ids and not has_native_id: - raise ValueError("Either (cams_asset_id and cams_check_id) or check_native_id must be provided") + raise ValueError(self._ERR_BOTH_PROJECT_AND_CATALOG) - # If check_native_id is provided, extract cams_asset_id and cams_check_id from it - if has_native_id and not has_cams_ids: - # Parse check_native_id: first part before first SEPARATOR is cams_asset_id, rest is cams_check_id - # Format: "/" where cams_check_id can contain slashes - first_slash_index = check_native_id.find(SEPARATOR) - if first_slash_index == -1: - raise ValueError(f"Invalid check_native_id format (missing {SEPARATOR}): {check_native_id}") - cams_asset_id = check_native_id[:first_slash_index] - cams_check_id = check_native_id[first_slash_index + 1:] - elif has_cams_ids and not has_native_id: - # Construct check_native_id from cams_asset_id and cams_check_id - check_native_id = f"{cams_asset_id}{SEPARATOR}{cams_check_id}" - - # At this point, cams_asset_id and cams_check_id should be set - assert cams_asset_id is not None, "cams_asset_id should be set by now" - assert cams_check_id is not None, "cams_check_id should be set by now" - assert check_native_id is not None, "check_native_id should be set by now" + # Validate and resolve asset_id, check_id, and check_native_id + asset_id, check_id, check_native_id = self._validate_and_resolve_ids( + asset_id, check_id, check_native_id + ) from .dq_search import DQSearchProvider @@ -466,10 +496,10 @@ def update_issue_metrics( search_provider = DQSearchProvider(self.config) # Build native IDs for searching - # For asset: (for table) or / (for column) - asset_native_id = cams_asset_id # For table type, just the asset ID + # For asset: (for table) or / (for column) + asset_native_id = asset_id # For table type, just the asset ID if asset_type == "column": - asset_native_id += SEPARATOR + column_name + asset_native_id += '/' + column_name # Search for the DQ asset asset_response = search_provider.search_dq_asset( @@ -480,20 +510,20 @@ def update_issue_metrics( ) dq_asset_id = asset_response.get("id") if not dq_asset_id: - raise ValueError(f"DQ asset not found for CAMS asset ID: {cams_asset_id}") + raise ValueError(f"DQ asset not found for CAMS asset ID: {asset_id}") - # Get the issue using get_issues with the cams_check_id + # Get the issue using get_issues with the check_id issue = self.get_issues( dq_asset_id=dq_asset_id, check_type=check_type, - check_id=cams_check_id, + check_id=check_id, project_id=project_id, catalog_id=catalog_id ) if not issue: raise ValueError( - f"Issue not found for CAMS check ID: {cams_check_id} " + f"Issue not found for CAMS check ID: {check_id} " f"with type: {check_type}" ) @@ -581,9 +611,9 @@ def get_issues( # Validate that exactly one of project_id or catalog_id is provided if project_id is None and catalog_id is None: - raise ValueError("Either project_id or catalog_id must be provided") + raise ValueError(self._ERR_MISSING_PROJECT_OR_CATALOG) if project_id is not None and catalog_id is not None: - raise ValueError("Only one of project_id or catalog_id should be provided, not both") + raise ValueError(self._ERR_BOTH_PROJECT_AND_CATALOG) url = f"{self.config.url}/data_quality/v4/issues" @@ -643,7 +673,7 @@ def get_issues( def create_issue( self, - check_id: str, + dq_check_id: str, reported_for_id: str, number_of_occurrences: int, number_of_tested_records: int, @@ -658,7 +688,7 @@ def create_issue( This method creates a new issue for a specific check and data asset. Args: - check_id: The ID of the check for which to create the issue + dq_check_id: The ID of the check for which to create the issue reported_for_id: The ID of the data asset being reported on number_of_occurrences: Number of issue occurrences number_of_tested_records: Total number of records tested @@ -676,7 +706,7 @@ def create_issue( Example: >>> provider.create_issue( - ... check_id="6be18374-573a-4cf8-8ab7-e428506e428b", + ... dq_check_id="6be18374-573a-4cf8-8ab7-e428506e428b", ... reported_for_id="894d01fd-bdfc-4a4f-b68b-62751e06e06a", ... number_of_occurrences=123, ... number_of_tested_records=456789, @@ -688,9 +718,9 @@ def create_issue( # Validate that exactly one of project_id or catalog_id is provided if project_id is None and catalog_id is None: - raise ValueError("Either project_id or catalog_id must be provided") + raise ValueError(self._ERR_MISSING_PROJECT_OR_CATALOG) if project_id is not None and catalog_id is not None: - raise ValueError("Only one of project_id or catalog_id should be provided, not both") + raise ValueError(self._ERR_BOTH_PROJECT_AND_CATALOG) # Build the URL for issue creation API url = f"{self.config.url}/data_quality/v4/issues" @@ -706,7 +736,7 @@ def create_issue( # Prepare the payload payload = { "check": { - "id": check_id + "id": dq_check_id }, "reported_for": { "id": reported_for_id @@ -742,3 +772,147 @@ def create_issue( raise ValueError("Issue ID not found in response") return issue_id + + def create_issues_bulk( + self, + payload: dict, + project_id: Optional[str] = None, + catalog_id: Optional[str] = None, + incremental_reporting: bool = False, + refresh_assets: bool = False + ) -> dict: + """ + Create multiple data quality issues in bulk. + + This method creates multiple issues, assets, and checks in a single API call + for better performance when reporting multiple related issues. + + Args: + payload: The bulk payload containing issues, assets, and existing_checks arrays. Expected structure is + + >>> { + ... "issues": [ + ... { + ... "check": {"native_id": str, "type": str}, + ... "reported_for": {"native_id": str, "type": str}, + ... "number_of_occurrences": int, + ... "number_of_tested_records": int, + ... "status": str, + ... "ignored": bool + ... }, + ... ... + ... ], + ... "assets": [ + ... { + ... "name": str, + ... "type": str, + ... "native_id": str, + ... "weight": int, + ... "parent": {"native_id": str, "type": str} (optional) + ... }, + ... ... + ... ], + ... "existing_checks": [ + ... {"native_id": str, "type": str}, + ... ... + ... ] + ... } + + project_id (str, optional): The project ID containing the issues + catalog_id (str, optional): The catalog ID containing the issues + incremental_reporting (bool): If true, adds archived issue counts to new issues + instead of replacing them. Default is False. + refresh_assets (bool): If true, assets will be refreshed and any assets not + present in the updated list will be deleted. Default is False. + + Returns: + dict: The response from the API containing the created issues data + + Raises: + ValueError: If the API request fails or returns an error status, or if neither + project_id nor catalog_id is provided, or if both are provided + + Example: + >>> payload = { + ... "issues": [ + ... { + ... "check": { + ... "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/Validity", + ... "type": "format" + ... }, + ... "reported_for": { + ... "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f", + ... "type": "data_asset" + ... }, + ... "number_of_occurrences": 200, + ... "number_of_tested_records": 1000, + ... "status": "aggregation", + ... "ignored": False + ... } + ... ], + ... "assets": [ + ... { + ... "name": "ACCOUNT_HOLDERS.csv", + ... "type": "data_asset", + ... "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f", + ... "weight": 1 + ... } + ... ], + ... "existing_checks": [ + ... { + ... "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/Validity", + ... "type": "format" + ... } + ... ] + ... } + >>> provider.create_issues_bulk( + ... payload=payload, + ... project_id="project-123", + ... incremental_reporting=True + ... ) + {'issues': [...], 'assets': [...], ...} + """ + from ..utils import get_url_with_query_params + + # Validate that exactly one of project_id or catalog_id is provided + if project_id is None and catalog_id is None: + raise ValueError(self._ERR_MISSING_PROJECT_OR_CATALOG) + if project_id is not None and catalog_id is not None: + raise ValueError(self._ERR_BOTH_PROJECT_AND_CATALOG) + + # Build the URL for bulk issue creation API + url = f"{self.config.url}/data_quality/v4/create_issues" + + # Build query parameters + params = {} + if project_id is not None: + params["project_id"] = project_id + elif catalog_id is not None: + params["catalog_id"] = catalog_id + + # Add boolean query parameters + params["incremental_reporting"] = str(incremental_reporting).lower() + params["refresh_assets"] = str(refresh_assets).lower() + + url = get_url_with_query_params(url, params) + + # Get request headers + headers = get_request_headers(self.config.auth_token) + + # Make POST request + response = self.session.post( + url, + headers=headers, + data=json.dumps(payload), + verify=False + ) + + if not response.ok: + raise ValueError( + f"Failed to create issues in bulk. " + f"Status: {response.status_code}, " + f"Response: {response.text}" + ) + + # Parse and return response + return json.loads(response.text) diff --git a/src/wxdi/dq_validator/provider/response_model.py b/src/wxdi/dq_validator/provider/response_model.py index 8a5e009..f2dba07 100644 --- a/src/wxdi/dq_validator/provider/response_model.py +++ b/src/wxdi/dq_validator/provider/response_model.py @@ -31,8 +31,10 @@ def to_iso(self, dt: datetime): version_id: str source_repository_id: str global_id: str + workflow_id: Optional[str] = None + draft_mode: Optional[str] = None is_target_draft: Optional[bool] = None - effective_start_date: datetime + effective_start_date: Optional[datetime] = None created_by: str created_at: datetime modified_by: str @@ -45,6 +47,7 @@ def to_iso(self, dt: datetime): tags: Optional[List[str]] = None steward_ids: Optional[List[str]] = None steward_group_ids: Optional[List[str]] = None + workflow_state: Optional[str] = None user_access: Optional[bool] = None @classmethod diff --git a/src/wxdi/odcs_generator/README-GENERATE-ODCS-SCRIPT.md b/src/wxdi/odcs_generator/README-GENERATE-ODCS-SCRIPT.md new file mode 100644 index 0000000..9491007 --- /dev/null +++ b/src/wxdi/odcs_generator/README-GENERATE-ODCS-SCRIPT.md @@ -0,0 +1,1095 @@ + + +# Generate ODCS - Script Documentation + +## Overview + +The `odcs_generator` module provides tools to automatically generate ODCS (Open Data Contract Standard) v3.1.0 compliant YAML files from data catalog metadata. It supports multiple data catalog sources: + +- **Collibra**: Extracts table/view definitions, column schemas, data types, classifications, and custom properties +- **Informatica CDGC**: Fetches asset metadata including table definitions, column schemas, and system attributes + +**Scripts:** +- `wxdi.odcs_generator.generate_odcs_from_collibra` - Generate ODCS from Collibra assets +- `wxdi.odcs_generator.generate_odcs_from_informatica` - Generate ODCS from Informatica CDGC assets + +--- + +# Table of Contents + +1. [Collibra Integration](#collibra-integration) +2. [Informatica Integration](#informatica-integration) +3. [Common Features](#common-features) +4. [Related Documentation](#related-documentation) + +--- + +# Collibra Integration + +## Location +`wxdi.odcs_generator.generate_odcs_from_collibra` + +## Features + +- ✅ **Automatic Metadata Extraction**: Fetches asset details, attributes, and relations from Collibra REST API +- ✅ **Column Discovery**: Automatically discovers columns through asset relations +- ✅ **Data Type Mapping**: Maps Collibra logical and technical data types to ODCS standards +- ✅ **Classification Support**: Extracts data classifications using Collibra GraphQL API +- ✅ **Tag Integration**: Includes Collibra tags at both asset and column levels +- ✅ **Custom Properties**: Preserves Collibra attributes as custom properties +- ✅ **ODCS v3.1.0 Compliance**: Generates fully compliant ODCS YAML files + +## Requirements + +### Python Dependencies + +Install all dependencies from the project root: + +```bash +pip install -r requirements.txt +``` + +**Required packages:** +- `requests` >= 2.32.4 - HTTP library for Collibra API calls +- `pyyaml` >= 5.4.0 - YAML file generation +- `urllib3` >= 2.6.3 - HTTP client library +- `python_dateutil` >= 2.5.3 - Date/time utilities +- `ibm_cloud_sdk_core` >= 3.16.7 - IBM Cloud SDK core + +### Collibra Access + +- Valid Collibra instance URL +- User account with read access to assets, attributes, and relations +- Network connectivity to Collibra REST API and GraphQL endpoints + +## Installation + +1. Clone the repository: + ```bash + git clone + cd data-product-python-sdk + ``` + +2. Install dependencies: + ```bash + pip install -r requirements.txt + ``` + +3. Set up environment variables (see Configuration section) + +## Directory Structure + +``` +data-product-python-sdk/ +├── odcs_generator/ +│ ├── __init__.py +│ ├── generate_odcs_from_collibra.py # Main script +│ └── README-GENERATE-ODCS-SCRIPT.md # This file +├── examples/ +│ ├── __init__.py +│ └── odcs_generator_example.py # Usage examples +├── requirements.txt # Python dependencies +└── ... +``` + +## Configuration + +### Environment Variables + +Set the following environment variables before running the script: + +```bash +export COLLIBRA_URL="https://your-instance.collibra.com" +export COLLIBRA_USERNAME="myuser" +export COLLIBRA_PASSWORD="mypassword" +``` + +**Alternative**: Pass credentials via command-line arguments (see Usage section) + +### Collibra Permissions Required + +The user account needs the following permissions: +- Read access to assets +- Read access to attributes +- Read access to relations +- Access to GraphQL API (for classifications) +- Read access to tags + +## Usage + +### Command-Line Usage + +Run the script directly from the `odcs_generator` directory: + +```bash +python odcs_generator/generate_odcs_from_collibra.py +``` + +**Example:** +```bash +python odcs_generator/generate_odcs_from_collibra.py 019a57f9-62d2-7aa0-9f22-4fa2cea1180b +``` + +This generates a file named `-odcs.yaml` in the current directory. + +### Programmatic Usage + +Import and use the module in your Python code: + +```python +from odcs_generator import CollibraClient, ODCSGenerator + +# Initialize client +client = CollibraClient( + base_url="https://your-instance.collibra.com", + username="your_username", + password="your_pswd" +) + +# Create generator +generator = ODCSGenerator(client) + +# Generate ODCS +odcs_data = generator.generate_odcs("asset_id") +``` + +See `examples/odcs_generator_example.py` for complete examples. + +### Command-Line Options + +```bash +python odcs_generator/generate_odcs_from_collibra.py [OPTIONS] +``` + +**Options:** + +| Option | Description | Default | +|--------|-------------|---------| +| `asset_id` | Collibra asset ID (required) | - | +| `-o, --output` | Output YAML file path | `-odcs.yaml` | +| `--url` | Collibra base URL | `$COLLIBRA_URL` | +| `-u, --username` | Collibra username | `$COLLIBRA_USERNAME` | +| `-p, --password` | Collibra password | `$COLLIBRA_PASSWORD` | +| `-h, --help` | Show help message | - | + +### Examples + +**Generate with custom output file:** +```bash +python odcs_generator/generate_odcs_from_collibra.py 019a57f9-62d2-7aa0-9f22-4fa2cea1180b -o my-contract.yaml +``` + +**Pass credentials via command line:** +```bash +python odcs_generator/generate_odcs_from_collibra.py 019a57f9-62d2-7aa0-9f22-4fa2cea1180b \ + --url https://acme.collibra.com \ + -u myuser \ + -p mypassword +``` + +**Using environment variables:** +```bash +export COLLIBRA_URL="https://acme.collibra.com" +export COLLIBRA_USERNAME="myuser" +export COLLIBRA_PASSWORD="mypassword" + +python odcs_generator/generate_odcs_from_collibra.py 019a57f9-62d2-7aa0-9f22-4fa2cea1180b +``` + +## Example Scripts + +The `examples/` directory contains comprehensive usage examples: + +### Basic Usage Example +```bash +python examples/odcs_generator_example.py +``` + +This example demonstrates: +- Connecting to Collibra using environment variables +- Generating ODCS from a single asset +- Saving the output to a YAML file + +### Custom Processing Example +Shows how to: +- Generate ODCS programmatically +- Customize contract metadata (dataProduct, version, name) +- Add quality rules +- Update server configuration + +### Batch Processing Example +Demonstrates: +- Processing multiple assets in a loop +- Error handling for failed assets +- Generating summary reports + +See `examples/odcs_generator_example.py` for complete code and additional examples. + +## How It Works + +### 1. Asset Metadata Extraction + +The script fetches the following from Collibra: + +- **Asset Details**: Name, display name, type, domain, creation date +- **Attributes**: All custom attributes (Description, Data Type, etc.) +- **Relations**: Source and target relations to discover columns +- **Tags**: Direct tags assigned to the asset +- **Classifications**: Data classifications via GraphQL API + +### 2. Column Discovery + +Columns are discovered through asset relations: +- Processes both source and target relations +- Identifies assets with "column" in their type name +- Fetches column attributes and classifications +- Deduplicates columns by name + +### 3. Data Type Mapping + +**Logical Types** (Collibra → ODCS): +- `text` → `string` +- `whole number` → `integer` +- `decimal number` → `number` +- `date time` → `timestamp` +- `true/false` → `boolean` +- `geographical` → `string` + +**Physical Types** (with size/precision/scale): +- `VARCHAR(255)` +- `DECIMAL(10,2)` +- `NUMBER(18,4)` + +### 4. ODCS Generation + +Creates a complete ODCS v3.1.0 structure: + +```yaml +id: +kind: DataContract +apiVersion: v3.1.0 +domain: +dataProduct: Sample data product +version: 1.0.0 +name: Sample contract +status: active +contractCreatedTs: +description: + authoritativeDefinitions: + - type: collibra-asset + url: +tags: [...] +schema: + - id: + name: + physicalName: + physicalType: table|view + description: + properties: [...] +servers: + - id: + server: CONFIGURE_SERVER_HOSTNAME # ⚠️ Manual config required + type: DEFINE_SERVER_TYPE # ⚠️ Manual config required +``` + +### 5. Manual Configuration Comments + +The script adds inline comments to guide manual configuration: + +```yaml +servers: + # ============================================ + # ⚠️ MANUAL CONFIGURATION REQUIRED + # ============================================ + # Please update the following fields: + - id: server-79f2fd7b + server: CONFIGURE_SERVER_HOSTNAME # ⚠️ UPDATE: e.g., prod.snowflake.acme.com + type: DEFINE_SERVER_TYPE # ⚠️ UPDATE: e.g., snowflake, postgres, bigquery, redshift +``` + +## Output Structure + +### Generated ODCS File + +The script generates a YAML file with the following structure: + +```yaml +id: +kind: DataContract +apiVersion: v3.1.0 +domain: +dataProduct: +version: 1.0.0 +name: +status: active +contractCreatedTs: +description: + authoritativeDefinitions: + - type: collibra-asset + url: +tags: [...] +schema: + - id: + name: + physicalName: + physicalType: table|view + description: + customProperties: [...] + properties: + - name: + physicalName: + logicalType: string|integer|number|... + physicalType: VARCHAR(255)|DECIMAL(10,2)|... + description: + required: true|false + primaryKey: true|false + classification: + tags: [...] +servers: + - id: + server: CONFIGURE_SERVER_HOSTNAME + type: DEFINE_SERVER_TYPE +``` + +### Console Output + +The script provides detailed progress information: + +``` +Connecting to Collibra at https://acme.collibra.com... +Generating ODCS for asset: 019a57f9-62d2-7aa0-9f22-4fa2cea1180b + +=== Processing asset: 019a57f9-62d2-7aa0-9f22-4fa2cea1180b === +Fetching asset details... +Fetching asset attributes... +Fetching asset relations (as source)... +Fetching asset relations (as target)... +Found 15 relations where asset is target +Extracting column information from all relations... +Total relations found: 20 + Found column (source): name='customer_id', type='Column' + Found column (target): name='transaction_date', type='Column' +Found 12 unique columns from relations + +Writing ODCS to customer-transactions-odcs.yaml... +✓ Successfully generated ODCS file: customer-transactions-odcs.yaml +``` + +## Collibra Attribute Mapping + +### Asset-Level Attributes + +| Collibra Attribute | ODCS Field | Notes | +|-------------------|------------|-------| +| Name | `schema.name` | Full asset name | +| Display Name | `schema.physicalName` | Physical name | +| Description | `schema.description` | Asset description | +| Domain | `domain` | Domain name | +| Created On | `contractCreatedTs` | ISO timestamp | +| Table Type | `schema.physicalType` | table or view | +| Tags | `tags` | Array of tag names | +| Other attributes | `schema.customProperties` | Preserved as custom properties | + +### Column-Level Attributes + +| Collibra Attribute | ODCS Field | Notes | +|-------------------|------------|-------| +| Name / Display Name | `properties.name` | Column name | +| Original Name / Physical Name | `properties.physicalName` | Physical column name | +| Description / Definition | `properties.description` | Column description | +| Data Type | `properties.logicalType` | Mapped to ODCS types | +| Technical Data Type | `properties.physicalType` | With size/precision/scale | +| Is Nullable / Nullable | `properties.required` | Boolean | +| Is Primary Key / Primary Key | `properties.primaryKey` | Boolean | +| Security Classification | `properties.classification` | Security class | +| PII / Personally Identifiable Information | `properties.tags` | Adds "PII" tag | +| Size / Length | `properties.physicalType` | Appended to type | +| Precision | `properties.physicalType` | For numeric types | +| Scale / Number Of Fractional Digits | `properties.physicalType` | For numeric types | +| Classifications (GraphQL) | `properties.tags` | As `data_classification:*` | +| Tags | `properties.tags` | Column-level tags | + +## Error Handling + +### Common Errors + +**1. Missing Environment Variables** +``` +Error: Collibra URL is required. Set COLLIBRA_URL environment variable or use --url +``` +**Solution**: Set required environment variables or pass via command line + +**2. Authentication Failure** +``` +Error: HTTP 401 - Unauthorized +``` +**Solution**: Verify username and password are correct + +**3. Asset Not Found** +``` +Error: HTTP 404 - Asset not found +``` +**Solution**: Verify the asset ID exists and you have access + +**4. Network Issues** +``` +Error: Connection refused +``` +**Solution**: Check network connectivity and Collibra URL + +### Warnings + +The script may display warnings for non-critical issues: + +``` +Warning: Could not fetch tags: +Warning: No columns found in Collibra. +``` + +These warnings indicate missing data but don't prevent ODCS generation. + +## Validation + +After generating the ODCS file, validate it against the ODCS specification: + +1. **Manual Review**: Check the generated YAML for completeness +2. **YAML Syntax**: Ensure valid YAML format +3. **ODCS Compliance**: Verify against ODCS v3.1.0 specification +4. **Required Fields**: Confirm all mandatory fields are present +5. **Server Configuration**: Complete fields marked with ⚠️ warnings + +## Limitations + +## Script-Specific Limitations + +1. **Server Connection Details**: Collibra doesn't store actual server hostnames or connection strings +2. **Server Type**: May need manual verification/correction +3. **Additional Server Parameters**: Account, environment, roles, etc. +4. **Contract Metadata**: Data product name, version, contract name (uses defaults) +5. **Quality Rules**: Not extracted from Collibra +6. **SLA Terms**: Not available in Collibra metadata +7. **Stakeholder Information**: Requires manual addition + +### Collibra-Specific Limitations + +- Column discovery depends on proper relation setup in Collibra +- Data type mapping may need adjustment for custom types +- Classification extraction requires GraphQL API access +- Some attributes may have different names in your Collibra instance + +## Customization + +### Modifying Attribute Mappings + +Edit the `_build_attribute_map()` method to change which Collibra attributes are extracted: + +```python +# Add custom attribute mapping +custom_attr = attr_map.get('Your Custom Attribute', '') +``` + +### Changing Data Type Mappings + +Update the `LOGICAL_TYPE_MAPPING` dictionary: + +```python +LOGICAL_TYPE_MAPPING = { + 'your_custom_type': 'odcs_type', + # ... existing mappings +} +``` + +### Adding Custom Properties + +Modify the `_extract_custom_properties()` method to include/exclude specific attributes: + +```python +EXCLUDED_ATTRIBUTES = {'Description', 'Your Excluded Attr'} +``` + +## Best Practices + +1. **Verify Asset ID**: Ensure you're using the correct Collibra asset ID +2. **Check Relations**: Verify columns are properly related to tables in Collibra +3. **Review Output**: Always review the generated YAML before using +4. **Manual Configuration**: Complete all fields marked with ⚠️ warnings +5. **Validate**: Validate the YAML before finalizing + +## Troubleshooting + +### No Columns Generated + +**Problem**: Schema has no properties (columns) + +**Possible Causes**: +- Columns not properly related to table in Collibra +- Relation types not recognized +- Column assets have incorrect type + +**Solution**: +1. Check Collibra relations for the asset + +--- + +# Informatica Integration + +## Overview + +`generate_odcs_from_informatica.py` is a Python script that automatically generates ODCS (Open Data Contract Standard) v3.1.0 compliant YAML files from Informatica CDGC (Cloud Data Governance and Catalog) asset metadata. It extracts table definitions, column schemas, data types, and system attributes from Informatica and transforms them into standardized data contracts. + +**Location**: `odcs_generator/generate_odcs_from_informatica.py` + +## Features + +- ✅ **Automatic Metadata Extraction**: Fetches asset details from Informatica CDGC REST API +- ✅ **Column Discovery**: Automatically discovers columns through asset hierarchy +- ✅ **Concurrent Processing**: Fetches column details in parallel for better performance +- ✅ **System Attributes**: Preserves Informatica system attributes as custom properties +- ✅ **Server Type Detection**: Automatically maps Informatica resource types to ODCS server types +- ✅ **ODCS v3.1.0 Compliance**: Generates fully compliant ODCS YAML files + +## Requirements + +### Python Dependencies + +```bash +pip install requests pyyaml +``` + +**Required packages:** +- `requests` >= 2.25.0 - HTTP library for Informatica API calls +- `pyyaml` >= 5.4.0 - YAML file generation + +### Informatica Access + +- Valid Informatica CDGC instance URL +- User account with read access to assets and metadata +- Network connectivity to Informatica CDGC REST API and Identity Service endpoints + +## Configuration + +### Environment Variables + +Set the following environment variables before running the script: + +```bash +export INFORMATICA_CDGC_URL="https://cdgc.dm-us.informaticacloud.com" +export INFORMATICA_USERNAME="myuser" +export INFORMATICA_PASSWORD="mypassword" +``` + +**Alternative**: Pass credentials via command-line arguments (see Usage section) + +### Informatica Permissions Required + +The user account needs the following permissions: +- Read access to CDGC assets +- Access to asset metadata and attributes +- Access to Identity Service for authentication +- Read access to asset hierarchy (columns) + +## Usage + +### Basic Usage + +```bash +python odcs_generator/generate_odcs_from_informatica.py +``` + +**Example:** +```bash +python odcs_generator/generate_odcs_from_informatica.py 1b5fc805-252d-4ba2-bd90-e943103e411b +``` + +This generates a file named `-odcs.yaml` in the current directory. + +### Command-Line Options + +```bash +python odcs_generator/generate_odcs_from_informatica.py [OPTIONS] +``` + +**Options:** + +| Option | Description | Default | +|--------|-------------|---------| +| `asset_id` | Informatica asset ID (required) | - | +| `-o, --output` | Output YAML file path | `-odcs.yaml` | +| `--cdgc-url` | Informatica CDGC URL | `$INFORMATICA_CDGC_URL` | +| `-u, --username` | Informatica username | `$INFORMATICA_USERNAME` | +| `-p, --password` | Informatica password | `$INFORMATICA_PASSWORD` | +| `-h, --help` | Show help message | - | + +### Examples + +**Generate with custom output file:** +```bash +python odcs_generator/generate_odcs_from_informatica.py 1b5fc805-252d-4ba2-bd90-e943103e411b -o my-contract.yaml +``` + +**Pass credentials via command line:** +```bash +python odcs_generator/generate_odcs_from_informatica.py 1b5fc805-252d-4ba2-bd90-e943103e411b \ + --cdgc-url https://cdgc.dm-us.informaticacloud.com \ + -u myuser \ + -p mypassword +``` + +**Using environment variables:** +```bash +export INFORMATICA_CDGC_URL="https://cdgc.dm-us.informaticacloud.com" +export INFORMATICA_USERNAME="myuser" +export INFORMATICA_PASSWORD="mypswrd" + +python odcs_generator/generate_odcs_from_informatica.py 1b5fc805-252d-4ba2-bd90-e943103e411b +``` + +## How It Works + +### 1. Authentication + +The script performs a two-step authentication process: + +1. **Session ID**: Obtains a session ID from Identity Service using username/password +2. **JWT Token**: Exchanges session ID for a JWT token used for API calls +3. **Token Caching**: Caches the JWT token to avoid repeated authentication + +### 2. Asset Metadata Extraction + +The script fetches the following from Informatica CDGC: + +- **Asset Details**: Name, business name, type, resource type +- **System Attributes**: Schema, catalog source, row count, origin, timestamps +- **Hierarchy**: Column IDs from asset hierarchy +- **Column Details**: Data types, length, scale, precision, nullable, primary key + +### 3. Column Discovery + +Columns are discovered through the asset hierarchy: +- Extracts column IDs from the `hierarchy` array +- Fetches detailed metadata for each column concurrently (up to 10 parallel requests) +- Sorts columns by position for correct ordering +- Handles missing or incomplete column data gracefully + +### 4. Data Type Mapping + +**Physical Types** (with size/precision/scale): +- `VARCHAR(255)` +- `DECIMAL(10,2)` +- `NUMBER(18,4)` +- `CHAR(10)` +- `TIMESTAMP` + +The script automatically constructs physical types based on: +- Base data type from `com.infa.odin.models.relational.Datatype` +- Length from `com.infa.odin.models.relational.DatatypeLength` +- Scale from `com.infa.odin.models.relational.DatatypeScale` + +### 5. Server Type Detection + +Automatically maps Informatica resource types to ODCS server types: + +| Informatica Resource Type | ODCS Server Type | +|---------------------------|------------------| +| SqlServer | sqlserver | +| Oracle | oracle | +| PostgreSQL | postgresql | +| MySQL | mysql | +| Snowflake | snowflake | +| Redshift | redshift | +| BigQuery | bigquery | +| Databricks | databricks | +| Synapse | synapse | +| DB2 | db2 | +| Hive | hive | +| Impala | impala | +| Teradata | custom | + +### 6. ODCS Generation + +Creates a complete ODCS v3.1.0 structure: + +```yaml +id: +kind: DataContract +apiVersion: v3.1.0 +domain: Sample Domain +dataProduct: Sample data product +version: 1.0.0 +name: Sample contract +status: active +contractCreatedTs: +description: + authoritativeDefinitions: + - type: informatica-asset + url: +schema: + - id: + name: + physicalName: / + physicalType: Table + description: + properties: [...] +customProperties: [...] +servers: + - id: + server: CONFIGURE_SERVER_HOSTNAME # ⚠️ Manual config may be required + type: # Auto-detected from Informatica + schema: # Extracted from Informatica +``` + +### 7. Manual Configuration Comments + +The script adds inline comments to guide manual configuration where needed: + +```yaml +servers: + # ============================================ + # ⚠️ MANUAL CONFIGURATION REQUIRED + # ============================================ + # Please add/update the required server info: + - id: server-1b5fc805 + server: CONFIGURE_SERVER_HOSTNAME # ⚠️ UPDATE: e.g., prod.snowflake.acme.com + type: snowflake # Auto-detected from Informatica + schema: PUBLIC # Extracted from Informatica +``` + +## Output Structure + +### Generated ODCS File + +The script generates a YAML file with the following structure: + +```yaml +id: +kind: DataContract +apiVersion: v3.1.0 +domain: +dataProduct: +version: 1.0.0 +name: +status: active +contractCreatedTs: +description: + authoritativeDefinitions: + - type: informatica-asset + url: +schema: + - id: + name: + physicalName: / + physicalType: Table + description: + properties: + - name: + physicalType: VARCHAR(255)|DECIMAL(10,2)|... + description: + required: true|false + primaryKey: true|false +customProperties: + - property: + value: +servers: + - id: + server: CONFIGURE_SERVER_HOSTNAME + type: + schema: +``` + +### Console Output + +The script provides detailed progress information: + +``` +Fetching asset details for 1b5fc805-252d-4ba2-bd90-e943103e411b... +Fetching column details... + Fetched column 1/12... + Fetched column 2/12... + ... + Fetched column 12/12... +Generating ODCS YAML... + +Writing ODCS to customer-transactions-odcs.yaml... +✓ Successfully generated ODCS file: customer-transactions-odcs.yaml +``` + +## Informatica Attribute Mapping + +### Asset-Level Attributes + +| Informatica Attribute | ODCS Field | Notes | +|----------------------|------------|-------| +| core.name | `schema.name` | Table name | +| core.businessName | `schema.name` | Preferred if available | +| core.description | `schema.description` | Asset description | +| com.infa.odin.models.relational.Owner | `schema.physicalName`, `servers.schema` | Schema name | +| core.resourceType | `servers.type` | Mapped to ODCS type | +| core.identity | `id`, `schema.id` | Asset ID | + +### Column-Level Attributes + +| Informatica Attribute | ODCS Field | Notes | +|----------------------|------------|-------| +| core.name | `properties.name` | Column name | +| com.infa.odin.models.relational.Datatype | `properties.physicalType` | Base data type | +| com.infa.odin.models.relational.DatatypeLength | `properties.physicalType` | Appended to type | +| com.infa.odin.models.relational.DatatypeScale | `properties.physicalType` | For numeric types | +| core.description | `properties.description` | Column description | +| com.infa.odin.models.relational.Nullable | `properties.required` | Inverted boolean | +| com.infa.odin.models.relational.PrimaryKeyColumn | `properties.primaryKey` | Boolean | +| core.Position | - | Used for sorting columns | + +### System Attributes (Custom Properties) + +| Informatica Attribute | Custom Property Name | Description | +|----------------------|---------------------|-------------| +| core.resourceName | Catalog Source Name | Source catalog name | +| com.infa.odin.models.relational.NumberOfRows | Number of rows | Row count | +| core.origin | Origin | Data origin | +| com.infa.odin.models.relational.Owner | Schema | Schema name | +| core.sourceCreatedBy | Source Created By | Creator | +| core.sourceCreatedOn | Source Created On | Creation timestamp | +| core.sourceModifiedBy | Source Modified By | Last modifier | +| core.sourceModifiedOn | Source Modified On | Last modified timestamp | + +## Error Handling + +### Common Errors + +**1. Missing Environment Variables** +``` +Error: Informatica CDGC URL is required. Set INFORMATICA_CDGC_URL environment variable or use --cdgc-url +Example: --cdgc-url https://cdgc.dm-us.informaticacloud.com +``` +**Solution**: Set required environment variables or pass via command line + +**2. Authentication Failure** +``` +✗ HTTP Error: 401 + Authentication failed. Please check your credentials. +``` +**Solution**: Verify username and password are correct + +**3. Asset Not Found** +``` +✗ HTTP Error: 404 + Asset 1b5fc805-252d-4ba2-bd90-e943103e411b not found. +``` +**Solution**: Verify the asset ID exists and you have access + +**4. Connection Issues** +``` +✗ Connection Error: Unable to connect to Informatica CDGC at https://cdgc.dm-us.informaticacloud.com + Please check your network connection and CDGC URL. +``` +**Solution**: Check network connectivity and CDGC URL format + +**5. Timeout Error** +``` +✗ Timeout Error: Request timed out + The server took too long to respond. Please try again. +``` +**Solution**: Retry the request or check server status + +**6. Data Structure Error** +``` +✗ Data Error: Missing expected field 'summary' + The asset data structure may be incomplete or invalid. +``` +**Solution**: Verify the asset has complete metadata in Informatica + +**7. Asset Type Validation Error** +``` +✗ Validation Error: Asset 'MY_SCHEMA' is not a type 'Table'. This script only processes table assets. Please provide a table asset ID. +``` +**Solution**: The provided asset ID is not a table (it may be a schema, database, or other asset type). Use the Informatica catalog to find the correct table asset ID. Only table and view assets are supported. + +### Warnings + +The script may display warnings for non-critical issues: + +``` +Warning: Failed to fetch column : +``` + +These warnings indicate missing column data but don't prevent ODCS generation. The script will continue processing other columns. + +## Customization + +### Modifying Resource Type Mappings + +Edit the `RESOURCE_TYPE_MAPPING` dictionary to add or change server type mappings: + +```python +RESOURCE_TYPE_MAPPING = { + 'YourCustomType': 'custom', + # ... existing mappings +} +``` + +### Changing System Attributes + +Update the `SYSTEM_ATTRIBUTES_MAPPING` dictionary to include/exclude specific attributes: + +```python +SYSTEM_ATTRIBUTES_MAPPING = { + 'your.custom.attribute': 'Custom Property Name', + # ... existing mappings +} +``` + +## Best Practices + +1. **Verify Asset ID**: Ensure you're using the correct Informatica asset ID +2. **Check Hierarchy**: Verify columns are properly associated with tables in Informatica +3. **Review Output**: Always review the generated YAML before using +4. **Manual Configuration**: Complete server hostname if not auto-detected +5. **Validate**: Validate the YAML before finalizing + +## Troubleshooting + +### No Columns Generated + +**Problem**: Schema has no properties (columns) + +**Possible Causes**: +- Asset hierarchy is empty in Informatica +- Columns not properly associated with table +- Insufficient permissions to read column metadata + +**Solution**: +1. Check asset hierarchy in Informatica CDGC +2. Verify column associations +3. Review console output for column fetch errors +4. Check user permissions + +### Incorrect Data Types + +**Problem**: Physical types don't match expectations + +**Solution**: +1. Check Informatica attribute values for data type, length, scale +2. Verify the asset has complete metadata +3. Manually correct in generated YAML if needed + +### Missing System Attributes + +**Problem**: Custom properties are empty or incomplete + +**Solution**: +1. Verify system attributes are populated in Informatica +2. Check if attributes are available for the asset type +3. Update `SYSTEM_ATTRIBUTES_MAPPING` if using custom attributes + +### Authentication Token Expiry + +**Problem**: Script fails midway with authentication errors + +**Solution**: +1. The script caches tokens - this shouldn't happen normally +2. If it does, re-run the script (it will get a fresh token) +3. Check if your session timeout is very short + +--- + +# Common Features + +Both Collibra and Informatica integrations share these common features: + +## ODCS v3.1.0 Compliance + +Both scripts generate fully compliant ODCS v3.1.0 YAML files with: +- Required metadata fields (id, kind, apiVersion, domain, etc.) +- Schema definitions with properties +- Custom properties preservation +- Server configuration sections +- Authoritative definitions linking back to source + +## Manual Configuration Guidance + +Both scripts add helpful comments to guide manual configuration: +- Server hostname configuration +- Server type specification (when not auto-detected) +- Schema/database name configuration + +## Error Handling + +Comprehensive error handling for: +- Authentication failures +- Network connectivity issues +- Missing or invalid assets +- Incomplete metadata + +## Validation Support + +Generated YAML files can be validated using standard YAML validators and ODCS schema validators. + +2. Verify column assets have "column" in their type name +3. Review console output for relation processing messages + +### Incorrect Data Types + +**Problem**: Logical or physical types don't match expectations + +**Solution**: +1. Check Collibra attribute values for "Data Type" and "Technical Data Type" +2. Update `LOGICAL_TYPE_MAPPING` if needed +3. Manually correct in generated YAML + +### Missing Classifications + +**Problem**: Data classifications not included + +**Solution**: +1. Verify GraphQL API access +2. Check if classifications are assigned in Collibra +3. Ensure classifications have "ACCEPTED" status + + +## Related Documentation + +- [ODCS Specification v3.1.0](https://github.com/bitol-io/open-data-contract-standard) +- [Project README](../README.md) - Main project documentation +- [Build Guide](../BUILD_GUIDE.md) - Build and development instructions +- [Examples](../examples/odcs_generator_example.py) - Usage examples + +## Project Structure + +This script is part of the `data-product-python-sdk` project: + +``` +data-product-python-sdk/ +├── dph_services/ # Data Product Hub services +├── odcs_generator/ # ODCS generator module (this script) +├── examples/ # Usage examples +├── test/ # Test suites +│ ├── integration/ # Integration tests +│ └── unit/ # Unit tests +├── requirements.txt # Python dependencies +└── setup.py # Package setup +``` + +## Support + +For issues or questions: + +1. **Check Console Output**: Review error messages and warnings +2. **Verify Configuration**: Ensure environment variables are set correctly +3. **Test Connectivity**: Verify access to Collibra API +4. **Review Examples**: Check `examples/odcs_generator_example.py` for usage patterns +5. **Review Logs**: Check for detailed error information +6. **Consult Documentation**: Review this README and related docs \ No newline at end of file diff --git a/src/wxdi/odcs_generator/__init__.py b/src/wxdi/odcs_generator/__init__.py new file mode 100644 index 0000000..1593d17 --- /dev/null +++ b/src/wxdi/odcs_generator/__init__.py @@ -0,0 +1,55 @@ +# coding: utf-8 +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""ODCS Generator - Generate ODCS YAML files from Collibra and Informatica assets""" + +from .generate_odcs_from_collibra import ( + CollibraClient, + ODCSGenerator, + parse_arguments as collibra_parse_arguments, + validate_arguments as collibra_validate_arguments, + determine_output_file as collibra_determine_output_file, + write_yaml_file as collibra_write_yaml_file, + main as collibra_main +) + +from .generate_odcs_from_informatica import ( + InformaticaClient, + parse_arguments as informatica_parse_arguments, + validate_arguments as informatica_validate_arguments, + determine_output_file as informatica_determine_output_file, + write_yaml_file as informatica_write_yaml_file, + main as informatica_main +) + +__all__ = [ + # Collibra exports + 'CollibraClient', + 'ODCSGenerator', + 'collibra_parse_arguments', + 'collibra_validate_arguments', + 'collibra_determine_output_file', + 'collibra_write_yaml_file', + 'collibra_main', + # Informatica exports + 'InformaticaClient', + 'informatica_parse_arguments', + 'informatica_validate_arguments', + 'informatica_determine_output_file', + 'informatica_write_yaml_file', + 'informatica_main' +] + +__version__ = '1.0.0' diff --git a/src/wxdi/odcs_generator/generate_odcs_from_collibra.py b/src/wxdi/odcs_generator/generate_odcs_from_collibra.py new file mode 100644 index 0000000..46e9470 --- /dev/null +++ b/src/wxdi/odcs_generator/generate_odcs_from_collibra.py @@ -0,0 +1,765 @@ +#!/usr/bin/env python3 +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Generate ODCS YAML file from Collibra Asset + +This script fetches asset metadata from Collibra and generates an ODCS v3 compliant YAML file. + +Usage: + python generate_odcs_from_collibra.py + python generate_odcs_from_collibra.py 019a57f9-62d2-7aa0-9f22-4fa2cea1180b + +Environment Variables: + COLLIBRA_URL: Collibra instance URL (required) + COLLIBRA_USERNAME: Collibra username (required) + COLLIBRA_PASSWORD: Collibra password (required) +""" + +import argparse +import os +import sys +import uuid +from datetime import datetime, timezone +from typing import Any, Dict, List, Optional, Tuple + +import requests +import yaml + + +class CollibraClient: + """Client for interacting with Collibra REST API""" + + HEADERS_JSON = {"Accept": "application/json"} + HEADERS_CONTENT_JSON = {"Content-Type": "application/json"} + DEFAULT_LIMIT = 1000 + + def __init__(self, base_url: str, username: str, password: str): + self.base_url = base_url.rstrip('/') + self.auth = (username, password) + self.session = requests.Session() + self.session.auth = self.auth + + def get_asset(self, asset_id: str) -> Dict[str, Any]: + """Fetch asset details from Collibra""" + url = f"{self.base_url}/rest/2.0/assets/{asset_id}" + response = self.session.get(url, headers=self.HEADERS_JSON) + response.raise_for_status() + return response.json() + + def get_asset_attributes(self, asset_id: str) -> List[Dict[str, Any]]: + """Fetch all attributes for an asset""" + url = f"{self.base_url}/rest/2.0/attributes" + params = {'assetId': asset_id, 'limit': self.DEFAULT_LIMIT} + response = self.session.get(url, params=params, headers=self.HEADERS_JSON) + response.raise_for_status() + return response.json().get('results', []) + + def get_asset_relations(self, asset_id: str, as_source: bool = True) -> List[Dict[str, Any]]: + """Fetch all relations for an asset + + Args: + asset_id: The asset ID + as_source: If True, fetch relations where asset is source; if False, where asset is target + """ + url = f"{self.base_url}/rest/2.0/relations" + param_key = 'sourceId' if as_source else 'targetId' + params = {param_key: asset_id, 'limit': self.DEFAULT_LIMIT} + response = self.session.get(url, params=params, headers=self.HEADERS_JSON) + response.raise_for_status() + return response.json().get('results', []) + + def get_asset_tags(self, asset_id: str) -> List[str]: + """Fetch tags directly assigned to an asset""" + url = f"{self.base_url}/rest/2.0/assets/{asset_id}/tags" + try: + response = self.session.get(url, headers=self.HEADERS_JSON) + response.raise_for_status() + results = response.json() + return [tag['name'] for tag in results if 'name' in tag] + except Exception as e: + print(f"Warning: Could not fetch tags: {e}") + return [] + + def get_asset_classifications(self, asset_id: str) -> List[str]: + """Fetch data classifications for an asset using GraphQL API""" + url = f"{self.base_url}/graphql" + + query = """ + query AssetClassification($id: ID!) { + api { + asset(id: $id) { + id + classesForAsset { + id + classificationId + label + percentage + status + } + } + } + } + """ + + payload = {"query": query, "variables": {"id": asset_id}} + + try: + response = self.session.post(url, json=payload, headers=self.HEADERS_CONTENT_JSON) + response.raise_for_status() + data = response.json() + + asset_data = data.get('data', {}).get('api', {}).get('asset', {}) + classes = asset_data.get('classesForAsset', []) + + return [ + cls['label'] + for cls in classes + if cls.get('status') == 'ACCEPTED' and cls.get('label') + ] + except Exception: + return [] + +class ODCSGenerator: + """Generate ODCS YAML from Collibra asset metadata""" + + UTC_TIMEZONE_SUFFIX = '+00:00' + EXCLUDED_ATTRIBUTES = {'Description'} + LOGICAL_TYPE_MAPPING = { + 'text': 'string', + 'whole number': 'integer', + 'decimal number': 'number', + 'date time': 'timestamp', + 'string': 'string', + 'integer': 'integer', + 'number': 'number', + 'date': 'date', + 'time': 'time', + 'object': 'object', + 'array': 'array', + 'geographical': 'string', + 'true/false': 'boolean', + 'n/a': None + } + NUMERIC_TYPES = ['DECIMAL', 'NUMERIC', 'NUMBER'] + + def __init__(self, collibra_client: CollibraClient): + self.client = collibra_client + + def generate_odcs(self, asset_id: str) -> Dict[str, Any]: + """Generate ODCS structure from a single Collibra asset + + Args: + asset_id: Collibra asset ID to include in the contract + + Returns: + ODCS data contract dictionary with single asset in schema array + + Raises: + ValueError: If asset is not a table + """ + if not asset_id: + raise ValueError("Asset ID is required") + + print(f"Generating ODCS for asset: {asset_id}") + print(f"\n=== Processing asset: {asset_id} ===") + + # Fetch asset details for contract-level metadata + print("Fetching asset details...") + asset = self.client.get_asset(asset_id) + + # Validate that the asset is a table + asset_type = asset.get('type', {}).get('name', '').lower() + print(f"Asset type: {asset_type}") + + if 'table' not in asset_type: + raise ValueError( + f"Asset '{asset.get('displayName', asset_id)}' is is not a type 'Table'. " + f"This script only processes table assets. " + f"Please provide a table asset ID." + ) + + domain_name = asset.get('domain', {}).get('name', '') + asset_display_name = asset.get('displayName', asset.get('name', 'asset')) + created_date = datetime.now(timezone.utc).isoformat().replace(self.UTC_TIMEZONE_SUFFIX, 'Z') + + odcs = { + 'id': asset_id, + 'kind': 'DataContract', + 'apiVersion': 'v3.1.0', + 'domain': domain_name, + 'dataProduct': 'Sample data product', + 'version': '1.0.0', + 'name': 'Sample contract', + 'status': 'active', + 'contractCreatedTs': created_date, + 'description': { + 'authoritativeDefinitions': [{ + 'type': 'collibra-asset', + 'url': f'{self.client.base_url}/asset/{asset_id}' + }] + }, + '_asset_display_name': asset_display_name + } + + # Add tags from Collibra + collibra_tags = self.client.get_asset_tags(asset_id) + odcs['tags'] = collibra_tags if collibra_tags else [] + + # Process the asset and build schema array + result = self._process_asset(asset_id) + if result: + schema_def, server = result + odcs['schema'] = [schema_def] + odcs['servers'] = [server] + else: + odcs['schema'] = [] + odcs['servers'] = [] + + return odcs + + @staticmethod + def _convert_timestamp(timestamp_ms: int) -> str: + """Convert millisecond timestamp to ISO format""" + utc_timezone_suffix = '+00:00' + if timestamp_ms: + return datetime.fromtimestamp(timestamp_ms / 1000, tz=timezone.utc).isoformat().replace(utc_timezone_suffix, 'Z') + return datetime.now(timezone.utc).isoformat().replace(utc_timezone_suffix, 'Z') + + @staticmethod + def _build_attribute_map(attributes: List[Dict[str, Any]]) -> Dict[str, Any]: + """Build a dictionary mapping attribute type names to values""" + attr_map = {} + for attr in attributes: + attr_type_name = attr.get('type', {}).get('name', '') + attr_value = attr.get('value') + if attr_type_name and attr_value: + attr_map[attr_type_name] = attr_value + return attr_map + + + def _process_asset(self, asset_id: str) -> Optional[Tuple[Dict[str, Any], Dict[str, Any]]]: + """Process asset and return its schema definition and server + + Args: + asset_id: Collibra asset ID to process + + Returns: + Tuple of (schema_def, server) or None if processing fails + """ + try: + # Fetch asset details + print("Fetching asset details...") + asset = self.client.get_asset(asset_id) + + print("Fetching asset attributes...") + attributes = self.client.get_asset_attributes(asset_id) + + print("Fetching asset relations (as source)...") + source_relations = self.client.get_asset_relations(asset_id, as_source=True) + + print("Fetching asset relations (as target)...") + target_relations = self.client.get_asset_relations(asset_id, as_source=False) + print(f"Found {len(target_relations)} relations where asset is target") + + all_relations = source_relations + target_relations + + # Extract asset metadata + display_name = asset.get('displayName', '') + asset_type = asset.get('type', {}).get('name', '') + + + attr_map = self._build_attribute_map(attributes) + description = attr_map.get('Description', '') + + # Extract schema name from relations + schema_name = self._extract_schema_from_relations(target_relations) + + # Build physical name as schema/table_name + physical_name = f"{schema_name}/{display_name}" if schema_name else display_name + + # Create server definition with schema + server = self._create_server_definition(schema_name) + + # Create schema definition + schema_def = { + 'id': asset_id, + 'name': display_name, + 'physicalName': physical_name, + 'physicalType': 'table' if asset_type.lower() == 'table' else 'view', + 'description': description, + 'properties': [] + } + + # Add custom properties + custom_properties = self._extract_custom_properties(attr_map) + if custom_properties: + schema_def['customProperties'] = custom_properties + + # Extract and add column properties + columns_found = self._extract_columns_from_relations(all_relations) + + if columns_found: + print(f"Found {len(columns_found)} unique columns from relations") + self._add_columns_to_schema(schema_def, columns_found) + + if not schema_def['properties']: + print("Warning: No columns found in Collibra.") + + return (schema_def, server) + + except Exception as e: + print(f"Error processing asset {asset_id}: {e}") + return None + + @staticmethod + def _create_server_definition(schema_name: str = '') -> Dict[str, Any]: + """Create server definition with placeholder values and schema""" + server_id = f"server-{uuid.uuid4().hex[:8]}" + return { + 'id': server_id, + 'server': 'CONFIGURE_SERVER_HOSTNAME', + 'type': 'DEFINE_SERVER_TYPE', + 'schema': schema_name if schema_name else 'CONFIGURE_SCHEMA_NAME' + } + + def _extract_schema_from_relations(self, relations: List[Dict[str, Any]]) -> str: + """Extract schema name from relations where table is the target + + Args: + relations: List of relations where the table is the target + + Returns: + Schema name if found, empty string otherwise + """ + print("Attempting to extract schema from relations...") + for relation in relations: + try: + relation_type = relation.get('type', {}).get('name', '') + print(f" Checking relation type: {relation_type}") + + # Check source of the relation (where table is target) + source = relation.get('source', {}) + if source and source.get('id'): + source_asset = self.client.get_asset(str(source.get('id'))) + source_type = source_asset.get('type', {}).get('name', '') + source_name = source_asset.get('displayName', source_asset.get('name', '')) + + print(f" Source: name='{source_name}', type='{source_type}'") + + # Check if the source is a schema/database schema + if 'schema' in source_type.lower(): + print(f" ✓ Found schema from relation: {source_name}") + return source_name + + # Also check target of the relation (where table is source) + target = relation.get('target', {}) + if target and target.get('id'): + target_asset = self.client.get_asset(str(target.get('id'))) + target_type = target_asset.get('type', {}).get('name', '') + target_name = target_asset.get('displayName', target_asset.get('name', '')) + + print(f" Target: name='{target_name}', type='{target_type}'") + + # Check if the target is a schema/database schema + if 'schema' in target_type.lower(): + print(f" ✓ Found schema from relation: {target_name}") + return target_name + + except Exception as e: + print(f" Warning: Error extracting schema from relation: {e}") + continue + + print(" No schema found in relations") + return '' + + def _extract_custom_properties(self, attr_map: Dict[str, Any]) -> List[Dict[str, str]]: + """Extract custom properties from attribute map""" + custom_properties = [] + for attr_name, attr_value in attr_map.items(): + if attr_name not in self.EXCLUDED_ATTRIBUTES and attr_value: + custom_prop_name = attr_name.lower().replace(' ', '_') + custom_properties.append({ + 'property': custom_prop_name, + 'value': attr_value + }) + return custom_properties + + def _extract_columns_from_relations(self, relations: List[Dict[str, Any]]) -> List[Dict[str, Any]]: + """Extract column information from asset relations""" + print("Extracting column information from all relations...") + print(f"Total relations found: {len(relations)}") + + columns_found = [] + seen_column_ids = set() + seen_column_names = set() + + for relation in relations: + # Process source + self._process_relation_endpoint( + relation.get('source', {}), + 'source', + seen_column_ids, + seen_column_names, + columns_found + ) + + # Process target + self._process_relation_endpoint( + relation.get('target', {}), + 'target', + seen_column_ids, + seen_column_names, + columns_found + ) + + return columns_found + + def _process_relation_endpoint( + self, + endpoint: Dict[str, Any], + endpoint_type: str, + seen_column_ids: set, + seen_column_names: set, + columns_found: List[Dict[str, Any]] + ) -> None: + """Process a single relation endpoint (source or target)""" + if not endpoint or not endpoint.get('id'): + return + + endpoint_id = endpoint.get('id') + if not endpoint_id or endpoint_id in seen_column_ids: + return + + try: + endpoint_asset = self.client.get_asset(str(endpoint_id)) + endpoint_type_name = endpoint_asset.get('type', {}).get('name', '') + endpoint_name = endpoint_asset.get('displayName', endpoint_asset.get('name', '')) + + if 'column' in endpoint_type_name.lower(): + print(f" Found column ({endpoint_type}): name='{endpoint_name}', type='{endpoint_type_name}'") + seen_column_ids.add(endpoint_id) + + if endpoint_name not in seen_column_names: + seen_column_names.add(endpoint_name) + col_attributes = self.client.get_asset_attributes(str(endpoint_id)) + col_classifications = self.client.get_asset_classifications(str(endpoint_id)) + columns_found.append({ + 'asset': endpoint_asset, + 'attributes': col_attributes, + 'classifications': col_classifications + }) + except Exception: + pass + + def _add_columns_to_schema(self, schema_def: Dict[str, Any], columns_found: List[Dict[str, Any]]) -> None: + """Add column properties to schema definition""" + added_column_names = set() + for col_data in columns_found: + property_def = self._create_property_from_collibra( + col_data['asset'], + col_data['attributes'], + col_data.get('classifications', []) + ) + col_name = property_def.get('name') + + if col_name not in added_column_names: + added_column_names.add(col_name) + schema_def['properties'].append(property_def) + else: + print(f"Skipping duplicate property: {col_name}") + + def _create_property_from_collibra( + self, + col_asset: Dict[str, Any], + col_attributes: List[Dict[str, Any]], + classifications: Optional[List[str]] = None + ) -> Dict[str, Any]: + """Create a property (column) definition from Collibra asset, attributes, and classifications""" + col_name = col_asset.get('displayName', col_asset.get('name', 'unknown_column')) + col_id = col_asset.get('id') + + attr_map = self._build_attribute_map(col_attributes) + classifications = classifications or [] + + # Extract data types + logical_type = self._normalize_logical_type(attr_map.get('Data Type', '')) + physical_type = self._build_physical_type(attr_map) + + # Extract metadata + description = attr_map.get('Description', attr_map.get('Definition', '')) + is_nullable = self._parse_boolean_value( + attr_map.get('Is Nullable', attr_map.get('Nullable', 'true')) + ) + is_primary_key = self._parse_boolean_value( + attr_map.get('Is Primary Key', attr_map.get('Primary Key', attr_map.get('IsPrimaryKey', ''))) + ) + physical_name = attr_map.get('Original Name', attr_map.get('Physical Name', col_name)) + security_class = attr_map.get('Security Classification', attr_map.get('Classification', '')) + + # Build tags + tags = self._build_column_tags(attr_map, classifications, col_id) + + # Build property definition + prop = { + 'name': col_name, + 'physicalName': physical_name, + 'description': description, + 'required': is_nullable, + 'tags': tags + } + + # Add optional fields + if logical_type is not None: + prop['logicalType'] = logical_type + if physical_type is not None: + prop['physicalType'] = physical_type + if is_primary_key: + prop['primaryKey'] = True + if security_class: + prop['classification'] = security_class + + return prop + + def _normalize_logical_type(self, logical_type: str) -> Optional[str]: + """Normalize logical type to ODCS standard types""" + if not logical_type: + return None + + normalized = logical_type.lower().strip() + return self.LOGICAL_TYPE_MAPPING.get(normalized, logical_type) + + def _build_physical_type(self, attr_map: Dict[str, Any]) -> Optional[str]: + """Build physical type string with size/precision/scale""" + technical_data_type = attr_map.get('Technical Data Type', '') + if not technical_data_type: + return None + + base_type = technical_data_type.upper() + + # Extract size-related attributes + size = self._to_int(attr_map.get('Size', attr_map.get('Length', ''))) + precision = self._to_int(attr_map.get('Precision', '')) + scale = self._to_int(attr_map.get('Scale', attr_map.get('Number Of Fractional Digits', ''))) + + # For numeric types, use Size as precision if Precision is not set + if base_type in self.NUMERIC_TYPES and precision is None and size is not None: + precision = size + + # Build type string with parameters + if scale is not None and precision is not None: + return f"{base_type}({precision},{scale})" + elif precision is not None: + return f"{base_type}({precision})" + elif size is not None: + return f"{base_type}({size})" + else: + return base_type + + @staticmethod + def _to_int(value: Any) -> Optional[int]: + """Convert decimal string to integer, handling empty strings""" + if not value: + return None + try: + return int(float(str(value))) + except (ValueError, TypeError): + return None + + @staticmethod + def _parse_boolean_value(value: Any) -> bool: + """Parse boolean value from various formats""" + if isinstance(value, bool): + return value + return str(value).lower() in ['true', 'yes', '1', 'y'] + + def _build_column_tags( + self, + attr_map: Dict[str, Any], + classifications: List[str], + col_id: Optional[str] + ) -> List[str]: + """Build tags list for a column""" + tags = [] + + # Add PII tag if applicable + pii_value = attr_map.get('Personally Identifiable Information', attr_map.get('PII', '')) + if pii_value and self._parse_boolean_value(pii_value): + tags.append('PII') + + # Add classification tags + for classification in classifications: + tags.append(f'data_classification:{classification}') + + # Add Collibra tags + if col_id: + try: + col_tags = self.client.get_asset_tags(col_id) + if col_tags: + tags.extend(col_tags) + except Exception: + pass + + return tags + + + +def parse_arguments() -> argparse.Namespace: + """Parse command line arguments""" + parser = argparse.ArgumentParser( + description='Generate ODCS YAML file from Collibra asset', + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=""" +Examples: + Generate ODCS from asset: + python generate_odcs_from_collibra.py 019a57f9-62d2-7aa0-9f22-4fa2cea1180b + + With custom output: + python generate_odcs_from_collibra.py -o custom-output.yaml + +Environment Variables: + COLLIBRA_URL Collibra instance URL (required) + COLLIBRA_USERNAME Collibra username (required) + COLLIBRA_PASSWORD Collibra password (required) + """ + ) + + parser.add_argument('asset_id', help='Collibra asset ID') + parser.add_argument('-o', '--output', help='Output YAML file path (default: -odcs.yaml)') + parser.add_argument('--url', help='Collibra base URL', default=os.getenv('COLLIBRA_URL')) + parser.add_argument('-u', '--username', help='Collibra username', default=os.getenv('COLLIBRA_USERNAME')) + parser.add_argument('-p', '--password', help='Collibra password', default=os.getenv('COLLIBRA_PASSWORD')) + + return parser.parse_args() + + +def validate_arguments(args: argparse.Namespace) -> None: + """Validate required arguments""" + if not args.url: + print("Error: Collibra URL is required. Set COLLIBRA_URL environment variable or use --url") + sys.exit(1) + + if not args.username: + print("Error: Collibra username is required. Set COLLIBRA_USERNAME environment variable or use --username") + sys.exit(1) + + if not args.password: + print("Error: Collibra password is required. Set COLLIBRA_PASSWORD environment variable or use --password") + sys.exit(1) + + +def determine_output_file(args: argparse.Namespace, odcs_data: Dict[str, Any]) -> str: + """Determine the output file path""" + if args.output: + return args.output + + # Use asset display name for filename if available, otherwise use contract name + asset_name = odcs_data.get('_asset_display_name') or odcs_data.get('name', 'asset') + asset_name = asset_name.lower().replace(' ', '-') + return f"{asset_name}-odcs.yaml" + + +def _add_server_warning_comments(modified_lines: List[str], line: str) -> None: + """Add warning comment block before server definition""" + separator = ' # ============================================' + modified_lines.append(separator) + modified_lines.append(' # ⚠️ MANUAL CONFIGURATION REQUIRED') + modified_lines.append(separator) + modified_lines.append(' # Please add/update the required server info:') + modified_lines.append(line) + + +def _add_inline_comment_if_needed(line: str) -> str: + """Add inline comment to server configuration fields if needed""" + if ' server:' in line and 'CONFIGURE_SERVER_HOSTNAME' in line: + return line + ' # ⚠️ UPDATE: e.g., prod.snowflake.acme.com' + elif ' type:' in line and 'DEFINE_SERVER_TYPE' in line: + return line + ' # ⚠️ UPDATE: e.g., snowflake, postgres, bigquery, redshift' + elif ' schema:' in line and 'CONFIGURE_SCHEMA_NAME' in line: + return line + ' # ⚠️ UPDATE: e.g., public, dbo, my_schema' + return line + + +def write_yaml_file(output_file: str, odcs_data: Dict[str, Any]) -> None: + """Write ODCS data to YAML file with manual configuration comments""" + print(f"\nWriting ODCS to {output_file}...") + + # Remove temporary field used for filename generation + odcs_data_copy = odcs_data.copy() + odcs_data_copy.pop('_asset_display_name', None) + + # First, write the YAML normally + yaml_content = yaml.dump(odcs_data_copy, default_flow_style=False, sort_keys=False, allow_unicode=True) + + # Add manual configuration comments to the servers section + lines = yaml_content.split('\n') + modified_lines = [] + in_servers_section = False + server_block_started = False + + for line in lines: + # Detect servers section + if line.startswith('servers:'): + in_servers_section = True + modified_lines.append(line) + continue + + # Detect start of a server block (first item in servers array) + if in_servers_section and line.strip().startswith('- id:') and not server_block_started: + server_block_started = True + _add_server_warning_comments(modified_lines, line) + continue + + # Add inline comments for server and type fields + if in_servers_section and server_block_started: + line = _add_inline_comment_if_needed(line) + + modified_lines.append(line) + + # Write the modified content + with open(output_file, 'w') as f: + f.write('\n'.join(modified_lines)) + + +def main(): + """Main entry point""" + args = parse_arguments() + validate_arguments(args) + + try: + print(f"Connecting to Collibra at {args.url}...") + client = CollibraClient(args.url, args.username, args.password) + + print("Fetching asset...") + generator = ODCSGenerator(client) + odcs_data = generator.generate_odcs(args.asset_id) + + output_file = determine_output_file(args, odcs_data) + write_yaml_file(output_file, odcs_data) + + print(f"✓ Successfully generated ODCS file: {output_file}") + + except requests.exceptions.HTTPError as e: + print(f"Error: HTTP {e.response.status_code} - {e.response.text}") + sys.exit(1) + except Exception as e: + print(f"Error: {str(e)}") + sys.exit(1) + + +if __name__ == '__main__': + main() \ No newline at end of file diff --git a/src/wxdi/odcs_generator/generate_odcs_from_informatica.py b/src/wxdi/odcs_generator/generate_odcs_from_informatica.py new file mode 100644 index 0000000..a5314c8 --- /dev/null +++ b/src/wxdi/odcs_generator/generate_odcs_from_informatica.py @@ -0,0 +1,500 @@ +#!/usr/bin/env python3 +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Generate ODCS YAML file from Informatica Asset + +This script fetches asset metadata from Informatica and generates an ODCS v3 compliant YAML file. + +Usage: + python odcs_generator/generate_odcs_from_informatica.py + python odcs_generator/generate_odcs_from_informatica.py 1b5fc805-252d-4ba2-bd90-e943103e411b --cdgc-url https://cdgc.dm-us.informaticacloud.com -u username -p password + +Environment Variables: + INFORMATICA_CDGC_URL: Informatica CDGC URL (required, e.g., https://cdgc.dm-us.informaticacloud.com) + INFORMATICA_USERNAME: Informatica username (required) + INFORMATICA_PASSWORD: Informatica password (required) +""" + +import argparse +import os +import sys +from typing import Any, Dict, List, Optional +from datetime import datetime, timezone +from concurrent.futures import ThreadPoolExecutor, as_completed +import requests +import yaml + +# Constants +CORE_NAME_ATTR = 'core.name' + +# Resource type mapping from Informatica to ODCS +RESOURCE_TYPE_MAPPING = { + 'SqlServer': 'sqlserver', + 'Oracle': 'oracle', + 'PostgreSQL': 'postgresql', + 'MySQL': 'mysql', + 'Snowflake': 'snowflake', + 'Redshift': 'redshift', + 'BigQuery': 'bigquery', + 'Databricks': 'databricks', + 'Synapse': 'synapse', + 'DB2': 'db2', + 'Teradata': 'custom', + 'Hive': 'hive', + 'Impala': 'impala' +} + +# System attributes mapping for custom properties +SYSTEM_ATTRIBUTES_MAPPING = { + 'core.resourceName': 'Catalog Source Name', + 'com.infa.odin.models.relational.NumberOfRows': 'Number of rows', + 'core.origin': 'Origin', + 'com.infa.odin.models.relational.Owner': 'Schema', + 'core.sourceCreatedBy': 'Source Created By', + 'core.sourceCreatedOn': 'Source Created On', + 'core.sourceModifiedBy': 'Source Modified By', + 'core.sourceModifiedOn': 'Source Modified On' +} + +class InformaticaClient: + + CONTENT_TYPE_JSON = "application/json" + HEADERS_JSON = {"Accept": CONTENT_TYPE_JSON} + HEADERS_CONTENT_JSON = {"Content-Type": CONTENT_TYPE_JSON} + + def __init__(self, base_url, username, password): + self.base_url = base_url.rstrip('/') + self.username = username + self.password = password + self.region = self._extract_region_from_url(base_url) + self.identity_url = f"https://{self.region}.informaticacloud.com" + self._auth_token: Optional[str] = None + + def _extract_region_from_url(self, base_url: str) -> str: + """Extract region identifier from base URL. + + Examples: + https://cdgc.dm-us.informaticacloud.com -> dm-us + https://cdgc.na1.informaticacloud.com -> na1 + """ + # Remove protocol and trailing slash + url = base_url.replace('https://', '').rstrip('/') + + # Extract hostname + hostname = url.split('/')[0] + + # Remove 'cdgc.' prefix if present + if hostname.startswith('cdgc.'): + hostname = hostname[5:] + + # Extract region (everything before .informaticacloud.com) + if '.informaticacloud.com' in hostname: + region = hostname.split('.informaticacloud.com')[0] + return region + + # Fallback: return the hostname as-is + return hostname + + def get_session_id(self) -> Dict[str, Any]: + url = f"{self.identity_url}/identity-service/api/v1/Login" + payload = {"username": self.username, "password": self.password} + response = requests.post(url, json=payload, headers=self.HEADERS_CONTENT_JSON) + response.raise_for_status() + return response.json() + + def get_auth_token(self) -> str: + """Get authentication token with caching to avoid repeated auth calls.""" + if self._auth_token is None: + session_id = self.get_session_id()["sessionId"] + url = f"{self.identity_url}/identity-service/api/v1/jwt/Token?client_id=idmc_api&nonce=1234" + response = requests.post(url, headers={"Accept": self.CONTENT_TYPE_JSON, "IDS-SESSION-ID": session_id, "Cookie": f"USER_SESSION={session_id}"}) + response.raise_for_status() + self._auth_token = response.json()["jwt_token"] + return str(self._auth_token) + + def _fetch_asset(self, asset_id: str) -> Dict[str, Any]: + """Fetch asset data from Informatica API.""" + auth_token = self.get_auth_token() + url = f"{self.base_url}/data360/search/v1/assets/{asset_id}?scheme=internal&segments=all" + response = requests.get(url, headers={"Authorization": f"Bearer {auth_token}"}) + response.raise_for_status() + return response.json() + + def get_asset_details(self, asset_id: str) -> Dict[str, Any]: + """Fetch asset details including table and column information.""" + return self._fetch_asset(asset_id) + + def validate_asset_is_table(self, asset_data: Dict[str, Any]) -> None: + """Validate that the asset is a table and not a schema or other type. + + Args: + asset_data: The asset data returned from Informatica API + + Raises: + ValueError: If the asset is not a table + """ + # Check the asset class type from systemAttributes (correct location) + asset_class = asset_data.get('systemAttributes', {}).get('core.classType', '') + + # Valid table class types in Informatica + valid_table_types = [ + 'com.infa.odin.models.relational.Table', + 'com.infa.odin.models.relational.View' + ] + + if asset_class not in valid_table_types: + asset_name = asset_data.get('summary', {}).get(CORE_NAME_ATTR, 'unknown') + raise ValueError( + f"Asset '{asset_name}' is not a type 'Table'. " + f"This script only processes table assets. Please provide a table asset ID." + ) + + def get_column_details(self, column_id: str) -> Dict[str, Any]: + """Fetch detailed information for a specific column.""" + return self._fetch_asset(column_id) + +def parse_arguments(): + parser = argparse.ArgumentParser( + description='Generate ODCS YAML file from Informatica asset', + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog="") + + parser.add_argument('asset_id', help='Informatica asset ID') + parser.add_argument('-o', '--output', help='Output YAML file path (default: -odcs.yaml)') + parser.add_argument('--cdgc-url', help='Informatica CDGC URL (e.g., https://cdgc.dm-us.informaticacloud.com)', + default=os.getenv('INFORMATICA_CDGC_URL')) + parser.add_argument('-u', '--username', help='Informatica username', default=os.getenv('INFORMATICA_USERNAME')) + parser.add_argument('-p', '--password', help='Informatica password', default=os.getenv('INFORMATICA_PASSWORD')) + return parser.parse_args() + +def validate_arguments(args): + if not args.cdgc_url: + print("Error: Informatica CDGC URL is required. Set INFORMATICA_CDGC_URL environment variable or use --cdgc-url") + print("Example: --cdgc-url https://cdgc.dm-us.informaticacloud.com") + sys.exit(1) + + if not args.username: + print("Error: Informatica username is required. Set INFORMATICA_USERNAME environment variable or use --username") + sys.exit(1) + + if not args.password: + print("Error: Informatica password is required. Set INFORMATICA_PASSWORD environment variable or use --password") + sys.exit(1) + +def extract_column_position(col_data: Dict[str, Any]) -> int: + """Extract column position from column data.""" + self_attrs = col_data.get('selfAttributes', {}) + position = self_attrs.get('core.Position') + + try: + return int(position) if position is not None else 999999 + except (ValueError, TypeError): + return 999999 + + +def build_physical_type(datatype: str, datatype_length: str, datatype_scale: str) -> str: + """Build physical type string with length/scale if available.""" + physical_type = datatype + if datatype_length: + if datatype_scale and datatype_scale != '0': + physical_type = f"{datatype}({datatype_length},{datatype_scale})" + else: + physical_type = f"{datatype}({datatype_length})" + return physical_type + + +def build_column_property(column_detail: Dict[str, Any]) -> Dict[str, Any]: + """Build a single column property from column detail data.""" + col_summary = column_detail.get('summary', {}) + col_self_attrs = column_detail.get('selfAttributes', {}) + + col_name = col_summary.get(CORE_NAME_ATTR) + datatype = col_self_attrs.get('com.infa.odin.models.relational.Datatype') + datatype_length = col_self_attrs.get('com.infa.odin.models.relational.DatatypeLength', '') + datatype_scale = col_self_attrs.get('com.infa.odin.models.relational.DatatypeScale', '') + + physical_type = build_physical_type(datatype, datatype_length, datatype_scale) + col_description = col_summary.get('core.description', '') + + prop = { + 'name': col_name, + 'physicalType': physical_type, + } + + if col_description: + prop['description'] = col_description + + # Check if column is nullable + nullable_str = col_self_attrs.get('com.infa.odin.models.relational.Nullable', 'true') + is_nullable = nullable_str.lower() == 'true' + prop['required'] = not is_nullable + + # Check if column is primary key + pk_str = col_self_attrs.get('com.infa.odin.models.relational.PrimaryKeyColumn', 'false') + is_primary_key = pk_str.lower() == 'true' + if is_primary_key: + prop['primaryKey'] = True + + return prop + + +def build_custom_properties(table_self_attrs: Dict[str, Any]) -> List[Dict[str, str]]: + """Build custom properties list from table attributes.""" + custom_properties = [] + + for key, ui_field_name in SYSTEM_ATTRIBUTES_MAPPING.items(): + value = table_self_attrs.get(key) + if value: + custom_properties.append({ + 'property': ui_field_name, + 'value': value + }) + + return custom_properties + + +def generate_odcs_yaml(asset_data: Dict[str, Any], column_details: List[Dict[str, Any]], base_url: str) -> Dict[str, Any]: + """Generate ODCS YAML structure from Informatica asset data.""" + + # Extract table information + table_id = asset_data['core.identity'] + name = asset_data['summary'][CORE_NAME_ATTR] + + # Extract actual table name from selfAttributes (physical table name) + table_self_attrs = asset_data.get('selfAttributes', {}) + table_name = table_self_attrs.get(CORE_NAME_ATTR, name) + + # Extract schema name for physical name construction + schema_name = table_self_attrs.get('com.infa.odin.models.relational.Owner', '') + + # Build physical name as schema/table_name + physical_name = f"{schema_name}/{table_name}" if schema_name else table_name + + created_ts = datetime.now(timezone.utc).isoformat().replace('+00:00', 'Z') + + # Extract table description if available + table_description = asset_data.get('summary', {}).get('core.description') + + # Extract resource information for server configuration + resource_type = table_self_attrs.get('core.resourceType', '') + + # Build custom properties using helper function + custom_properties = build_custom_properties(table_self_attrs) + + # Map resource type to ODCS server type + server_type = RESOURCE_TYPE_MAPPING.get(resource_type) + + # Build properties from column details using helper function + properties = [build_column_property(column_detail) for column_detail in column_details] + + # Build schema entry + schema_entry = { + 'id': table_id, + 'name': table_name, + 'physicalName': physical_name, + 'physicalType': 'Table', + 'properties': properties + } + + if table_description: + schema_entry['description'] = table_description + + # Build ODCS structure + odcs = { + 'id': table_id, + 'kind': 'DataContract', + 'apiVersion': 'v3.1.0', + 'domain': 'Sample Domain', + 'dataProduct': 'Sample data product', + 'version': '1.0.0', + 'name': 'Sample contract', + 'status': 'active', + 'contractCreatedTs': created_ts, + 'description': { + 'authoritativeDefinitions': [ + { + 'type': 'informatica-asset', + 'url': f"{base_url}/asset/{table_id}" + } + ] + }, + 'schema': [schema_entry], + 'customProperties': custom_properties, + 'servers': [ + { + 'id': 'server-' + table_id[:8], + 'server': 'CONFIGURE_SERVER_HOSTNAME', + 'type': server_type if server_type else 'CONFIGURE_SERVER_TYPE', + 'schema': schema_name if schema_name else 'CONFIGURE_SCHEMA_NAME' + } + ] + } + + return odcs + +def _add_server_warning_comments(modified_lines: List[str], line: str) -> None: + """Add warning comment block before server definition""" + separator = ' # ============================================' + modified_lines.append(separator) + modified_lines.append(' # ⚠️ MANUAL CONFIGURATION REQUIRED') + modified_lines.append(separator) + modified_lines.append(' # Please add/update the required server info:') + modified_lines.append(line) + + +def _add_inline_comment_if_needed(line: str) -> str: + """Add inline comment to server configuration fields if needed""" + if ' server:' in line and 'CONFIGURE_SERVER_HOSTNAME' in line: + return line + ' # ⚠️ UPDATE: e.g., prod.snowflake.acme.com' + elif ' type:' in line and 'CONFIGURE_SERVER_TYPE' in line: + return line + ' # ⚠️ UPDATE: e.g., snowflake, postgres, bigquery, redshift' + elif ' schema:' in line and 'CONFIGURE_SCHEMA_NAME' in line: + return line + ' # ⚠️ UPDATE: e.g., public, dbo, my_schema' + return line + + +def determine_output_file(args, asset_data: Dict[str, Any]) -> str: + """Determine the output file path using the asset name""" + if args.output: + return args.output + + # Try to get name from ODCS data first (for generated ODCS), then from asset summary + asset_name = asset_data.get('name') or asset_data.get('summary', {}).get(CORE_NAME_ATTR, 'asset') + # Sanitize the name for use as a filename + file_name = asset_name.lower().replace(' ', '-') + return f"{file_name}-odcs.yaml" + + +def write_yaml_file(output_file: str, odcs_data: Dict[str, Any]) -> None: + """Write ODCS data to YAML file with manual configuration comments""" + print(f"\nWriting ODCS to {output_file}...") + + # First, write the YAML normally + yaml_content = yaml.dump(odcs_data, default_flow_style=False, sort_keys=False, allow_unicode=True) + + # Add manual configuration comments to the servers section + lines = yaml_content.split('\n') + modified_lines = [] + in_servers_section = False + server_block_started = False + + for line in lines: + # Detect servers section + if line.startswith('servers:'): + in_servers_section = True + modified_lines.append(line) + continue + + # Detect start of a server block (first item in servers array) + if in_servers_section and line.strip().startswith('- id:') and not server_block_started: + server_block_started = True + _add_server_warning_comments(modified_lines, line) + continue + + # Add inline comments for server and type fields + if in_servers_section and server_block_started: + line = _add_inline_comment_if_needed(line) + + modified_lines.append(line) + + # Write the modified content + with open(output_file, 'w') as f: + f.write('\n'.join(modified_lines)) + + +def main(): + args = parse_arguments() + validate_arguments(args) + + try: + print(f"Fetching asset details for {args.asset_id}...") + client = InformaticaClient(args.cdgc_url, args.username, args.password) + + # Fetch table asset details + asset_data = client.get_asset_details(args.asset_id) + + # Validate that the asset is a table, not a schema or other type + client.validate_asset_is_table(asset_data) + + # Get column IDs from hierarchy + column_ids = [col['core.identity'] for col in asset_data.get('hierarchy', [])] + + print("Fetching column details...") + + # Fetch details for each column concurrently for better performance + column_details = [] + with ThreadPoolExecutor(max_workers=10) as executor: + # Submit all column detail requests + future_to_col_id = { + executor.submit(client.get_column_details, col_id): col_id + for col_id in column_ids + } + + # Collect results as they complete + completed = 0 + for future in as_completed(future_to_col_id): + col_id = future_to_col_id[future] + try: + col_data = future.result() + column_details.append(col_data) + completed += 1 + print(f" Fetched column {completed}/{len(column_ids)}...") + except Exception as e: + print(f" Warning: Failed to fetch column {col_id}: {e}") + + column_details.sort(key=extract_column_position) + + print("Generating ODCS YAML...") + odcs_data = generate_odcs_yaml(asset_data, column_details, client.base_url) + + output_file = determine_output_file(args, asset_data) + write_yaml_file(output_file, odcs_data) + + print(f"✓ Successfully generated ODCS file: {output_file}") + + except ValueError as e: + print(f"\n✗ Validation Error: {e}") + sys.exit(1) + except requests.exceptions.HTTPError as e: + print(f"\n✗ HTTP Error: {e}") + if e.response.status_code == 401: + print(" Authentication failed. Please check your credentials.") + elif e.response.status_code == 404: + print(f" Asset {args.asset_id} not found.") + else: + print(f" Status code: {e.response.status_code}") + sys.exit(1) + except requests.exceptions.ConnectionError: + print(f"\n✗ Connection Error: Unable to connect to Informatica CDGC at {args.cdgc_url}") + print(" Please check your network connection and CDGC URL.") + sys.exit(1) + except requests.exceptions.Timeout: + print("\n✗ Timeout Error: Request timed out") + print(" The server took too long to respond. Please try again.") + sys.exit(1) + except KeyError as e: + print(f"\n✗ Data Error: Missing expected field {e}") + print(" The asset data structure may be incomplete or invalid.") + sys.exit(1) + except Exception as e: + print(f"\n✗ Unexpected Error: {e}") + print(" An unexpected error occurred. Please check your inputs and try again.") + sys.exit(1) + +if __name__ == '__main__': + main() diff --git a/src/wxdi/version.py b/src/wxdi/version.py index 9ab7ad4..52a54fa 100644 --- a/src/wxdi/version.py +++ b/src/wxdi/version.py @@ -16,4 +16,4 @@ """ Version of IBM watsonx.data intelligence SDK """ -__version__ = '1.0.0' \ No newline at end of file +__version__ = '2.0.0' \ No newline at end of file diff --git a/tests/data/data_asset_response.json b/tests/data/data_asset_response.json new file mode 100644 index 0000000..b56b376 --- /dev/null +++ b/tests/data/data_asset_response.json @@ -0,0 +1,739 @@ +{ + "metadata": { + "project_id": "72d21c1d-499b-4784-a3c7-6f84507f9a20", + "sandbox_id": "72d21c1d-499b-4784-a3c7-6f84507f9a20", + "usage": { + "last_updated_at": "2026-01-27T05:50:24Z", + "last_updater_id": "1000330999", + "last_update_time": 1769493024580, + "last_accessed_at": "2026-01-27T05:50:24Z", + "last_access_time": 1769493024580, + "last_accessor_id": "1000330999", + "access_count": 0 + }, + "rov": { + "mode": 0, + "collaborator_ids": {}, + "member_roles": { + "1000330999": { + "user_iam_id": "1000330999", + "roles": [ + "OWNER" + ] + } + } + }, + "is_linked_with_sub_container": false, + "name": "DEPARTMENT", + "tags": [ + "connected-data" + ], + "asset_type": "data_asset", + "origin_country": "none", + "resource_key": "0000:0000:0000:0000:0000:FFFF:091E:D6B4|50000|sample:/DB2INST1/DEPARTMENT", + "rating": 0, + "total_ratings": 0, + "catalog_id": "92223cde-d54c-49c6-8476-af53a4c1d7b6", + "created": 1766630683099, + "created_at": "2025-12-25T02:44:43Z", + "owner_id": "1000330999", + "size": 0, + "version": 2, + "asset_state": "available", + "asset_attributes": [ + "data_asset", + "discovered_asset", + "metadata_enrichment_area_info", + "metadata_enrichment_info", + "data_profile", + "key_analyses", + "column_info", + "dataview_visualization", + "asset_data_quality_constraint" + ], + "asset_id": "6862f3ba-81f5-4122-8286-62bb4c5d6543", + "asset_category": "USER", + "creator_id": "1000330999", + "is_branched": true, + "is_managed_asset": false + }, + "entity": { + "data_asset": { + "columns": [ + { + "name": "DEPTNO", + "type": { + "type": "char", + "scale": 0, + "length": 3, + "signed": false, + "nullable": false, + "native_type": "CHAR" + }, + "columnProperties": {} + }, + { + "name": "DEPTNAME", + "type": { + "type": "varchar", + "scale": 0, + "length": 36, + "signed": false, + "nullable": false, + "native_type": "VARCHAR" + }, + "columnProperties": {} + }, + { + "name": "MGRNO", + "type": { + "type": "char", + "scale": 0, + "length": 6, + "signed": false, + "nullable": true, + "native_type": "CHAR" + }, + "columnProperties": {} + }, + { + "name": "ADMRDEPT", + "type": { + "type": "char", + "scale": 0, + "length": 3, + "signed": false, + "nullable": false, + "native_type": "CHAR" + }, + "columnProperties": {} + }, + { + "name": "LOCATION", + "type": { + "type": "char", + "scale": 0, + "length": 16, + "signed": false, + "nullable": true, + "native_type": "CHAR" + }, + "columnProperties": {} + } + ], + "dataset": true, + "mime_type": "application/x-ibm-rel-table", + "properties": [ + { + "name": "schema_name", + "value": "DB2INST1" + }, + { + "name": "table_name", + "value": "DEPARTMENT" + } + ] + }, + "column_info": { + "MGRNO": { + "data_class": { + "selected_data_class": { + "id": "U", + "name": "NoClassDetected", + "setByUser": false + } + }, + "column_checks": [ + { + "check": [ + { + "name": "formats", + "list_value": [ + "999999" + ] + } + ], + "origin": [], + "metadata": { + "type": "format", + "hidden": false, + "check_id": "977d185d-56db-4145-bbe3-fee2d5ee60c5", + "confirmed": false, + "dimension": "Validity", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether values have the required format.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + }, + { + "check": [ + { + "name": "unique", + "boolean_value": true + } + ], + "origin": [], + "metadata": { + "type": "uniqueness", + "hidden": false, + "check_id": "64abfcbf-80b9-40c8-8ff1-2e225589eac6", + "confirmed": false, + "dimension": "Uniqueness", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether values are unique.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + }, + { + "check": [ + { + "name": "range_type", + "value": "number" + }, + { + "name": "min", + "numeric_value": 0 + } + ], + "origin": [], + "metadata": { + "type": "range", + "hidden": false, + "check_id": "40a901eb-531a-4904-85f3-fc3fabbf3030", + "confirmed": false, + "dimension": "Validity", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether values are within the allowed range.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + } + ], + "inferred_type": { + "type": "INT8", + "scale": 0, + "length": 6, + "precision": 3, + "display_name": "tinyint" + }, + "rejected_checks": [], + "suggested_checks": [] + }, + "DEPTNO": { + "data_class": { + "suggested_classes": [ + { + "id": "588b9c94-34d4-411f-89b6-1d74927b0aef_fa8e90fa-d34b-5a19-a232-6c0557973af5", + "name": "ICD-10", + "confidence": 1 + }, + { + "id": "588b9c94-34d4-411f-89b6-1d74927b0aef_0b193af8-a07d-5964-80c1-d6b3273947e8", + "name": "Identifier", + "confidence": 1 + } + ], + "selected_data_class": { + "id": "588b9c94-34d4-411f-89b6-1d74927b0aef_fa8e90fa-d34b-5a19-a232-6c0557973af5", + "name": "ICD-10", + "setByUser": false, + "confidence": 1 + } + }, + "column_checks": [ + { + "check": [ + { + "name": "formats", + "list_value": [ + "A99" + ] + } + ], + "origin": [], + "metadata": { + "type": "format", + "hidden": false, + "check_id": "d50e784b-de8e-4f88-a121-d76f93ccd6b1", + "confirmed": false, + "dimension": "Validity", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether values have the required format.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + }, + { + "check": [ + { + "name": "unique", + "boolean_value": true + } + ], + "origin": [], + "metadata": { + "type": "uniqueness", + "hidden": false, + "check_id": "5a7f7f77-5ce1-4584-8929-1a7fc2d09f22", + "confirmed": false, + "dimension": "Uniqueness", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether values are unique.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + }, + { + "check": [ + { + "name": "missing_values_allowed", + "boolean_value": false + } + ], + "origin": [], + "metadata": { + "type": "completeness", + "hidden": false, + "check_id": "21641493-0233-4731-ab78-c8ad08d4cda5", + "confirmed": false, + "dimension": "Completeness", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether all rows have a value in this column.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + }, + { + "check": [ + { + "name": "data_class", + "value": "588b9c94-34d4-411f-89b6-1d74927b0aef_fa8e90fa-d34b-5a19-a232-6c0557973af5" + }, + { + "name": "data_class_name", + "value": "ICD-10" + } + ], + "origin": [], + "metadata": { + "type": "data_class", + "hidden": false, + "check_id": "816e3722-ae12-4040-910e-8d4f05c2684b", + "confirmed": false, + "dimension": "Validity", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether values match a specific data class.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + } + ], + "inferred_type": { + "type": "STRING", + "scale": 0, + "length": 3, + "precision": 0, + "display_name": "varchar(3)" + }, + "rejected_checks": [], + "suggested_checks": [] + }, + "ADMRDEPT": { + "data_class": { + "suggested_classes": [ + { + "id": "588b9c94-34d4-411f-89b6-1d74927b0aef_fa8e90fa-d34b-5a19-a232-6c0557973af5", + "name": "ICD-10", + "confidence": 1 + }, + { + "id": "588b9c94-34d4-411f-89b6-1d74927b0aef_9f81abbf-b98f-5555-91ed-883195e1823d", + "name": "Code", + "confidence": 1 + } + ], + "selected_data_class": { + "id": "588b9c94-34d4-411f-89b6-1d74927b0aef_fa8e90fa-d34b-5a19-a232-6c0557973af5", + "name": "ICD-10", + "setByUser": false, + "confidence": 1 + } + }, + "column_checks": [ + { + "check": [ + { + "name": "formats", + "list_value": [ + "A99" + ] + } + ], + "origin": [], + "metadata": { + "type": "format", + "hidden": false, + "check_id": "7224564a-a403-4c11-a386-52ac1fb8d6ca", + "confirmed": false, + "dimension": "Validity", + "created_at": "2026-01-27T05:50:23.284Z", + "description": "Check whether values have the required format.", + "modified_at": "2026-01-27T05:50:23.284Z", + "origin_type": "profiling/result" + } + }, + { + "check": [ + { + "name": "missing_values_allowed", + "boolean_value": false + } + ], + "origin": [], + "metadata": { + "type": "completeness", + "hidden": false, + "check_id": "97f7c29a-1c06-47b3-90d2-29c9f9acc01f", + "confirmed": false, + "dimension": "Completeness", + "created_at": "2026-01-27T05:50:23.284Z", + "description": "Check whether all rows have a value in this column.", + "modified_at": "2026-01-27T05:50:23.284Z", + "origin_type": "profiling/result" + } + }, + { + "check": [ + { + "name": "data_class", + "value": "588b9c94-34d4-411f-89b6-1d74927b0aef_fa8e90fa-d34b-5a19-a232-6c0557973af5" + }, + { + "name": "data_class_name", + "value": "ICD-10" + } + ], + "origin": [], + "metadata": { + "type": "data_class", + "hidden": false, + "check_id": "1bcaf011-8d50-40c9-9235-f4da7ed25639", + "confirmed": false, + "dimension": "Validity", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether values match a specific data class.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + } + ], + "inferred_type": { + "type": "STRING", + "scale": 0, + "length": 3, + "precision": 0, + "display_name": "varchar(3)" + }, + "rejected_checks": [], + "suggested_checks": [] + }, + "DEPTNAME": { + "data_class": { + "suggested_classes": [ + { + "id": "588b9c94-34d4-411f-89b6-1d74927b0aef_ab084a02-8cd4-57db-8035-96fe65aecaac", + "name": "Text", + "confidence": 1 + } + ], + "selected_data_class": { + "id": "588b9c94-34d4-411f-89b6-1d74927b0aef_ab084a02-8cd4-57db-8035-96fe65aecaac", + "name": "Text", + "setByUser": false, + "confidence": 1 + } + }, + "column_checks": [ + { + "check": [ + { + "name": "unique", + "boolean_value": true + } + ], + "origin": [], + "metadata": { + "type": "uniqueness", + "hidden": false, + "check_id": "604e4736-cd14-40f8-b36d-feeab6e72c04", + "confirmed": false, + "dimension": "Uniqueness", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether values are unique.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + }, + { + "check": [ + { + "name": "missing_values_allowed", + "boolean_value": false + } + ], + "origin": [], + "metadata": { + "type": "completeness", + "hidden": false, + "check_id": "391c525a-ecd6-467b-82c5-779ca5a308dd", + "confirmed": false, + "dimension": "Completeness", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether all rows have a value in this column.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + }, + { + "check": [ + { + "name": "case_type", + "value": "UpperCase" + } + ], + "origin": [], + "metadata": { + "type": "case", + "hidden": false, + "check_id": "404268af-86fc-49c6-bfee-4560c0105b51", + "confirmed": false, + "dimension": "Consistency", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether values match the selected capitalization style.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + } + ], + "inferred_type": { + "type": "STRING", + "scale": 0, + "length": 28, + "precision": 0, + "display_name": "varchar(28)" + }, + "rejected_checks": [], + "suggested_checks": [] + }, + "LOCATION": { + "data_class": { + "selected_data_class": { + "id": "U", + "name": "NoClassDetected", + "setByUser": false + } + }, + "column_checks": [], + "rejected_checks": [], + "suggested_checks": [] + } + }, + "data_profile": { + "attribute_classes": [ + "ICD-10", + "Identifier", + "Text", + "Code" + ], + "e092d07b-60ac-4fb1-9168-ef57fc886008": { + "href": "https://internal-nginx-svc.wkc.svc:12443/v2/data_profiles/e092d07b-60ac-4fb1-9168-ef57fc886008?project_id=72d21c1d-499b-4784-a3c7-6f84507f9a20&dataset_id=6862f3ba-81f5-4122-8286-62bb4c5d6543", + "entity": { + "columns": [], + "data_profile": { + "logs": [], + "columns": [], + "options": { + "enable_dqa": false, + "sample_type": "SEQUENTIAL", + "max_row_count": 1000, + "min_row_count": 0, + "disable_profiling": false, + "max_rows_per_batch": 10000, + "collect_data_values": false, + "key_analysis_options": { + "analysis_id": "6fc5fd21-c74c-4f30-9f86-c55dc8af1535", + "min_confidence": 0.8 + }, + "max_columns_per_task": 250, + "uniqueness_threshold": 0.95, + "max_distribution_size": 100, + "nullability_threshold": 0.05, + "classification_options": { + "disabled": false, + "ibm_class_codes": [], + "custom_class_codes": [], + "use_all_ibm_classes": true, + "use_all_custom_classes": true + }, + "max_numeric_stats_bins": 100, + "collect_historical_data": true, + "historical_retention_days": 30, + "enable_fast_classification": true, + "use_approximate_record_count": true, + "data_classification_confidence_threshold": 0.75, + "min_data_classification_confidence_threshold": 0.25 + }, + "execution": { + "bulk": true, + "run_by": "999", + "status": "finished", + "hb_task_id": "6f2379cc-d8aa-4713-b43c-6e61434a5968", + "create_time": "2026-01-27T05:49:34.034Z", + "is_supported": true, + "completed_date": "2026-01-27T05:50:20.020Z" + }, + "failed_assets": [] + } + }, + "metadata": { + "url": "https://internal-nginx-svc.wkc.svc:12443/v2/data_profiles/048b9a79-8161-40bb-9545-5ce955db28d7?project_id=72d21c1d-499b-4784-a3c7-6f84507f9a20&dataset_id=6862f3ba-81f5-4122-8286-62bb4c5d6543", + "guid": "e092d07b-60ac-4fb1-9168-ef57fc886008", + "owner": "admin", + "asset_id": "e092d07b-60ac-4fb1-9168-ef57fc886008", + "owner_id": "1000330999", + "created_at": "2026-01-27T05:49:33.033Z", + "dataset_id": "6862f3ba-81f5-4122-8286-62bb4c5d6543", + "project_id": "72d21c1d-499b-4784-a3c7-6f84507f9a20", + "accessed_at": "2026-01-27T05:49:33.033Z", + "dataset_ids": [], + "last_updater_id": "1000330999" + }, + "record_info": { + "computed": true, + "approximated": true, + "number_of_records": 14 + } + } + }, + "key_analyses": { + "fk_defined": 0, + "pk_defined": 0, + "fk_assigned": 0, + "pk_assigned": 0, + "fk_suggested": 0, + "pk_suggested": 2, + "primary_keys": [], + "fk_defined_as_pk": 0, + "overlap_assigned": 0, + "fk_assigned_as_pk": 0, + "overlap_suggested": 0, + "fk_suggested_as_pk": 0, + "key_analysis_area_id": "6fc5fd21-c74c-4f30-9f86-c55dc8af1535" + }, + "discovered_asset": { + "extended_metadata": [ + { + "name": "table_type", + "value": "TABLE" + } + ] + }, + "dataview_visualization": { + "jobs": { + "3f5c41cd-ea04-4eec-bf6b-6c4281e8901b": { + "job_type": "profile", + "execution": { + "status": "completed", + "last_updated_at": "2026-01-26T05:55:49.391629198Z", + "last_update_time": 1769406949391 + }, + "resultAttachmentNames": [ + "viz_job_3f5c41cd-ea04-4eec-bf6b-6c4281e8901b_0.json" + ] + } + } + }, + "metadata_enrichment_info": { + "MDE_instrumented": true + }, + "asset_data_quality_constraint": { + "asset_checks": [], + "rejected_checks": [], + "suggested_checks": [] + }, + "metadata_enrichment_area_info": { + "job_id": "8e10bfa2-ce48-4eb0-a6fa-241faf519502", + "area_id": "ed95a40c-d97b-4de4-a10a-eea2f425c8ff", + "added_date": 1766630744611, + "job_run_id": "087dcf60-d61c-44b3-b8d4-b61912301efc", + "last_enrichment_status": "finished", + "last_enrichment_timestamp": 1769493024511 + } + }, + "attachments": [ + { + "id": "3d11a882-968a-47c3-b854-28cc6ce689da", + "version": 2, + "asset_type": "data_asset", + "name": "DEPARTMENT", + "description": "remote", + "mime": "application/x-ibm-rel-table", + "connection_id": "8f2948af-9bb6-4bae-93b1-ab986a5a744c", + "connection_path": "/DB2INST1/DEPARTMENT", + "datasource_type": "8c1a4480-1c29-4b33-9086-9cb799d7b157", + "creator_id": "1000330999", + "create_time": 1766630683187, + "size": 0, + "is_remote": true, + "is_managed": false, + "is_referenced": false, + "is_object_key_read_only": false, + "is_user_provided_path_key": true, + "transfer_complete": true, + "is_partitioned": false, + "complete_time_ticks": 1766630683187, + "user_data": {}, + "test_doc": 0, + "usage": { + "access_count": 0, + "last_access_time": 1766630683187, + "last_accessor_id": "1000330999" + } + }, + { + "id": "f934e42c-986f-4f8c-85ae-5ac389c7f393", + "version": 2, + "asset_type": "dataview_visualization", + "name": "viz_job_3f5c41cd-ea04-4eec-bf6b-6c4281e8901b_0.json", + "mime": "application/json", + "datasource_type": "81bafdbd-b7c6-45c5-a4fd-6ec135f66f4e", + "creator_id": "1000330999", + "create_time": 1769406947754, + "size": 0, + "is_remote": false, + "is_managed": true, + "is_referenced": false, + "is_object_key_read_only": false, + "is_user_provided_path_key": false, + "transfer_complete": false, + "is_partitioned": false, + "complete_time_ticks": 0, + "user_data": {}, + "test_doc": 0, + "handle": { + "type": "assetfiles", + "key": "92223cde-d54c-49c6-8476-af53a4c1d7b6/6862f3ba-81f5-4122-8286-62bb4c5d6543/f934e42c-986f-4f8c-85ae-5ac389c7f393", + "upload_id": "aabacef5-ef7e-4bf3-aa6c-0f180bfa50de", + "max_part_num": 1 + }, + "usage": { + "access_count": 0, + "last_access_time": 1769406947754, + "last_accessor_id": "1000330999" + } + } + ], + "href": "/v2/assets/6862f3ba-81f5-4122-8286-62bb4c5d6543?project_id=72d21c1d-499b-4784-a3c7-6f84507f9a20" +} \ No newline at end of file diff --git a/tests/data/term_draft.json b/tests/data/term_draft.json new file mode 100644 index 0000000..96b1127 --- /dev/null +++ b/tests/data/term_draft.json @@ -0,0 +1,54 @@ +{ + "metadata": { + "artifact_type": "glossary_term", + "artifact_id": "30d1b847-0aa9-4840-a182-dd157fe977a0", + "version_id": "bdeef8cc-d9ab-4822-b3df-cef82b4de538_0", + "source_repository_id": "1be893e3-d512-4fd8-b449-c953abeedb6c", + "global_id": "1be893e3-d512-4fd8-b449-c953abeedb6c_30d1b847-0aa9-4840-a182-dd157fe977a0", + "workflow_id": "ce7f65d2-1e18-11f1-ac95-0a580afe2048", + "draft_mode": "import_create", + "is_target_draft": false, + "created_by": "1000330999", + "created_at": "2026-03-12T13:38:49.067Z", + "modified_by": "1000330999", + "modified_at": "2026-03-12T13:38:49.067Z", + "revision": "1", + "name": "mango", + "state": "DRAFT_HISTORY", + "tags": [], + "steward_ids": [], + "steward_group_ids": [], + "workflow_state": "Not started", + "read_only": false + }, + "entity": { + "abbreviations": [], + "state": "DRAFT_HISTORY", + "default_locale_id": "en-US", + "reference_copy": false, + "steward_ids": [], + "steward_group_ids": [], + "custom_attributes": [ + { + "custom_attribute_definition_id": "3d4ca4dd-8853-4575-b88b-adacf5423aba", + "name": "Data Type", + "values": [], + "description": "Indicates what is the specific data type associated with a particular property term.", + "read_only": false, + "searchable": false, + "required": false, + "type": "ENUM" + }, + { + "custom_attribute_definition_id": "8798e8a3-c495-4d1f-973a-b7f9c9d2e25a", + "name": "Cardinality", + "values": [], + "description": "Indicates what is the cardinality relating to an instance of a property or relationship term.", + "read_only": false, + "searchable": false, + "required": false, + "type": "ENUM" + } + ] + } +} \ No newline at end of file diff --git a/tests/data/term_latest_version.json b/tests/data/term_latest_version.json new file mode 100644 index 0000000..fae6d66 --- /dev/null +++ b/tests/data/term_latest_version.json @@ -0,0 +1,31 @@ +{ + "metadata": { + "artifact_type": "glossary_term", + "artifact_id": "30d1b847-0aa9-4840-a182-dd157fe977a0", + "version_id": "bdeef8cc-d9ab-4822-b3df-cef82b4de538_0", + "source_repository_id": "1be893e3-d512-4fd8-b449-c953abeedb6c", + "global_id": "1be893e3-d512-4fd8-b449-c953abeedb6c_30d1b847-0aa9-4840-a182-dd157fe977a0", + "is_target_draft": false, + "draft_ancestor_id": "90774b3c-16c6-4dcf-8c81-407d2e440baa", + "effective_start_date": "2026-01-09T09:01:15.648Z", + "created_by": "IBMid-0600012F5M", + "created_at": "2026-01-09T09:01:15.648Z", + "modified_by": "IBMid-0600012F5M", + "modified_at": "2026-01-09T09:01:15.648Z", + "revision": "0", + "name": "mango", + "state": "PUBLISHED", + "tags": [], + "steward_ids": [], + "steward_group_ids": [], + "read_only": false + }, + "entity": { + "abbreviations": [], + "state": "PUBLISHED", + "default_locale_id": "en-US", + "reference_copy": false, + "steward_ids": [], + "steward_group_ids": [] + } +} diff --git a/tests/data/term_response.json b/tests/data/term_response.json new file mode 100644 index 0000000..d5c7536 --- /dev/null +++ b/tests/data/term_response.json @@ -0,0 +1,117 @@ +{ + "metadata": { + "artifact_type": "glossary_term", + "artifact_id": "30d1b847-0aa9-4840-a182-dd157fe977a0", + "version_id": "bdeef8cc-d9ab-4822-b3df-cef82b4de538_0", + "source_repository_id": "1be893e3-d512-4fd8-b449-c953abeedb6c", + "global_id": "1be893e3-d512-4fd8-b449-c953abeedb6c_30d1b847-0aa9-4840-a182-dd157fe977a0", + "is_target_draft": false, + "draft_ancestor_id": "90774b3c-16c6-4dcf-8c81-407d2e440baa", + "effective_start_date": "2026-01-09T09:01:15.648Z", + "created_by": "IBMid-0600012F5M", + "created_at": "2026-01-09T09:01:15.648Z", + "modified_by": "IBMid-0600012F5M", + "modified_at": "2026-01-09T09:01:15.648Z", + "revision": "0", + "name": "mango", + "state": "PUBLISHED", + "tags": [], + "steward_ids": [], + "steward_group_ids": [], + "read_only": false + }, + "entity": { + "parent_category": { + "resources": [ + { + "metadata": { + "artifact_type": "relationship", + "artifact_id": "97b2a3df-94cf-446c-9338-ba8c664bc09a", + "version_id": "2728caaf-714b-4440-8558-b1d97077d34e_0", + "source_repository_id": "1be893e3-d512-4fd8-b449-c953abeedb6c", + "global_id": "1be893e3-d512-4fd8-b449-c953abeedb6c_97b2a3df-94cf-446c-9338-ba8c664bc09a", + "is_target_draft": false, + "effective_start_date": "2026-01-09T09:01:15.567Z", + "created_by": "IBMid-0600012F5M", + "created_at": "2026-01-09T09:01:15.567Z", + "modified_by": "IBMid-0600012F5M", + "modified_at": "2026-01-09T09:01:15.567Z", + "revision": "0", + "state": "PUBLISHED", + "read_only": false, + "user_access": true + }, + "entity": { + "child_href": "/v3/glossary_terms/30d1b847-0aa9-4840-a182-dd157fe977a0/versions?status=ACTIVE", + "parent_enabled": true, + "relationship_type": "parent_category", + "source_type": "glossary_term", + "target_type": "category", + "reference_copy": false, + "parent_href": "/v3/categories/e39ada11-8338-3704-90e3-681a71e7c839", + "parent_name": "[uncategorized]", + "parent_global_id": "1be893e3-d512-4fd8-b449-c953abeedb6c_e39ada11-8338-3704-90e3-681a71e7c839", + "parent_version_id": "0b9f1be1-e076-3203-9d65-a8df62cbdbeb_0", + "parent_id": "e39ada11-8338-3704-90e3-681a71e7c839", + "parent_description": "This is the system default if a standard category is not assigned.", + "child_id": "30d1b847-0aa9-4840-a182-dd157fe977a0", + "child_global_id": "1be893e3-d512-4fd8-b449-c953abeedb6c_30d1b847-0aa9-4840-a182-dd157fe977a0", + "child_name": "mango" + } + } + ], + "offset": 0, + "set_uri": false, + "limit": 5, + "count": 1 + }, + "custom_relationships": [], + "abbreviations": [], + "extended_attribute_groups": { + "dq_constraints": [ + { + "metadata": { + "type": "data_type", + "confirmed": false, + "hidden": false + }, + "origin": [], + "check": [ + { + "name": "data_type", + "value": "STRING" + }, + { + "name": "length", + "numeric_value": 80 + } + ] + }, + { + "metadata": { + "type": "length", + "confirmed": false, + "hidden": false + }, + "origin": [], + "check": [ + { + "name": "min", + "numeric_value": 3 + }, + { + "name": "max", + "numeric_value": 80 + } + ] + } + ] + }, + "state": "PUBLISHED", + "default_locale_id": "en-US", + "reference_copy": false, + "steward_ids": [], + "steward_group_ids": [], + "custom_attributes": [] + } +} \ No newline at end of file diff --git a/tests/src/__init__.py b/tests/src/__init__.py new file mode 100644 index 0000000..bf69929 --- /dev/null +++ b/tests/src/__init__.py @@ -0,0 +1,19 @@ +# coding: utf-8 +# Copyright 2026 IBM Corporation +# Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# See the LICENSE file in the project root for license information. + +"""Unit tests""" + +# This file is only here to get pylint to check the files in this directory diff --git a/tests/src/auth/__init__.py b/tests/src/auth/__init__.py new file mode 100644 index 0000000..c792a11 --- /dev/null +++ b/tests/src/auth/__init__.py @@ -0,0 +1,18 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for authentication module. +""" \ No newline at end of file diff --git a/tests/src/auth/test_auth_config.py b/tests/src/auth/test_auth_config.py new file mode 100644 index 0000000..02428de --- /dev/null +++ b/tests/src/auth/test_auth_config.py @@ -0,0 +1,275 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for AuthConfig module. + +Tests the AuthConfig class and EnvironmentType enum functionality. +""" + +import pytest +from wxdi.common.auth import AuthConfig, EnvironmentType + + +class TestEnvironmentType: + """Tests for EnvironmentType enum.""" + + def test_environment_type_values(self): + """Test that all environment types have correct values.""" + assert EnvironmentType.IBM_CLOUD.value == "ibm_cloud" + assert EnvironmentType.AWS_CLOUD.value == "aws_cloud" + assert EnvironmentType.GOV_CLOUD.value == "gov_cloud" + assert EnvironmentType.ON_PREM.value == "on_prem" + + def test_environment_type_members(self): + """Test that all expected environment types exist.""" + env_types = [e.name for e in EnvironmentType] + assert "IBM_CLOUD" in env_types + assert "AWS_CLOUD" in env_types + assert "GOV_CLOUD" in env_types + assert "ON_PREM" in env_types + + +class TestAuthConfigIBMCloud: + """Tests for AuthConfig with IBM_CLOUD environment.""" + + def test_ibm_cloud_with_api_key(self): + """Test IBM_CLOUD configuration with API key.""" + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-api-key' + ) + assert config.environment_type == EnvironmentType.IBM_CLOUD + assert config.api_key == 'test-api-key' + assert config.url == 'https://iam.cloud.ibm.com/identity/token' + + def test_ibm_cloud_with_custom_url(self): + """Test IBM_CLOUD with custom URL.""" + custom_url = 'https://custom.ibm.com/token' + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-key', + url=custom_url + ) + assert config.url == custom_url + + def test_ibm_cloud_missing_api_key(self): + """Test that IBM_CLOUD requires API key.""" + with pytest.raises(ValueError, match="API key must be provided"): + AuthConfig(environment_type=EnvironmentType.IBM_CLOUD) + + def test_ibm_cloud_trailing_slash_stripped(self): + """Test that trailing slashes are stripped from URL.""" + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-key', + url='https://custom.ibm.com/token///' + ) + assert config.url == 'https://custom.ibm.com/token' + + +class TestAuthConfigAWSCloud: + """Tests for AuthConfig with AWS_CLOUD environment.""" + + def test_aws_cloud_with_default_url(self): + """Test AWS_CLOUD with default URL.""" + config = AuthConfig( + environment_type=EnvironmentType.AWS_CLOUD, + api_key='test-key', + account_id='account-123' + ) + assert config.environment_type == EnvironmentType.AWS_CLOUD + assert config.url == 'https://account-iam.platform.saas.ibm.com' + assert config.account_id == 'account-123' + + def test_aws_cloud_with_custom_url(self): + """Test AWS_CLOUD with custom URL.""" + custom_url = 'https://custom-account-iam.example.com' + config = AuthConfig( + environment_type=EnvironmentType.AWS_CLOUD, + api_key='test-key', + account_id='account-123', + url=custom_url + ) + assert config.url == custom_url + assert config.account_id == 'account-123' + + def test_aws_cloud_missing_api_key(self): + """Test that AWS_CLOUD requires API key.""" + with pytest.raises(ValueError, match="API key must be provided"): + AuthConfig( + environment_type=EnvironmentType.AWS_CLOUD, + account_id='account-123' + ) + + def test_aws_cloud_missing_account_id(self): + """Test that AWS_CLOUD requires account ID.""" + with pytest.raises(ValueError, match="Account ID must be provided"): + AuthConfig( + environment_type=EnvironmentType.AWS_CLOUD, + api_key='test-key' + ) + + def test_aws_cloud_account_id_required(self): + """Test that account_id is stored correctly for AWS_CLOUD.""" + config = AuthConfig( + environment_type=EnvironmentType.AWS_CLOUD, + api_key='test-key', + account_id='my-account' + ) + assert config.account_id == 'my-account' + + +class TestAuthConfigGovCloud: + """Tests for AuthConfig with GOV_CLOUD environment.""" + + def test_gov_cloud_with_api_key(self): + """Test GOV_CLOUD configuration.""" + config = AuthConfig( + environment_type=EnvironmentType.GOV_CLOUD, + api_key='test-key' + ) + assert config.environment_type == EnvironmentType.GOV_CLOUD + assert config.url == 'https://dai.ibmforusgov.com/api/rest/mcsp/apikeys/token' + + def test_gov_cloud_missing_api_key(self): + """Test that GOV_CLOUD requires API key.""" + with pytest.raises(ValueError, match="API key must be provided"): + AuthConfig(environment_type=EnvironmentType.GOV_CLOUD) + + +class TestAuthConfigOnPrem: + """Tests for AuthConfig with ON_PREM environment.""" + + def test_on_prem_with_api_key(self): + """Test ON_PREM with username and API key.""" + config = AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com', + username='admin', + api_key='test-key' + ) + assert config.environment_type == EnvironmentType.ON_PREM + assert config.url == 'https://cpd.example.com/icp4d-api/v1/authorize' + assert config.username == 'admin' + assert config.api_key == 'test-key' + + def test_on_prem_with_password(self): + """Test ON_PREM with username and password.""" + config = AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com', + username='admin', + password='secret' + ) + assert config.url == 'https://cpd.example.com/icp4d-api/v1/authorize' + assert config.username == 'admin' + assert config.password == 'secret' + + def test_on_prem_url_already_has_path(self): + """Test that path is not duplicated if already present.""" + config = AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com/icp4d-api/v1/authorize', + username='admin', + api_key='test-key' + ) + assert config.url == 'https://cpd.example.com/icp4d-api/v1/authorize' + + def test_on_prem_missing_url(self): + """Test that ON_PREM requires URL.""" + with pytest.raises(ValueError, match="URL must be specified"): + AuthConfig( + environment_type=EnvironmentType.ON_PREM, + username='admin', + password='secret' + ) + + def test_on_prem_missing_username(self): + """Test that ON_PREM requires username.""" + with pytest.raises(ValueError, match="Username must be provided"): + AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com', + api_key='test-key' + ) + + def test_on_prem_missing_credentials(self): + """Test that ON_PREM requires either API key or password.""" + with pytest.raises(ValueError, match="Either api_key or password must be provided"): + AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com', + username='admin' + ) + + def test_on_prem_trailing_slash_stripped(self): + """Test that trailing slash is stripped before appending path.""" + config = AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com/', + username='admin', + api_key='test-key' + ) + assert config.url == 'https://cpd.example.com/icp4d-api/v1/authorize' + + +class TestAuthConfigValidation: + """Tests for AuthConfig validation.""" + + def test_invalid_environment_type(self): + """Test that invalid environment type raises TypeError.""" + with pytest.raises(TypeError, match="environment_type must be an instance of EnvironmentType"): + AuthConfig( + environment_type="invalid", # type: ignore + api_key='test-key' + ) + + def test_url_trailing_slash_stripping(self): + """Test that multiple trailing slashes are stripped.""" + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-key', + url='https://example.com///' + ) + assert config.url == 'https://example.com' + + def test_disable_ssl_verification_default_true(self): + """Test that disable_ssl_verification defaults to True.""" + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-key' + ) + assert config.disable_ssl_verification is True + + def test_disable_ssl_verification_can_be_set_false(self): + """Test that disable_ssl_verification can be set to False.""" + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-key', + disable_ssl_verification=False + ) + assert config.disable_ssl_verification is False + + def test_disable_ssl_verification_with_on_prem(self): + """Test disable_ssl_verification with ON_PREM environment.""" + config = AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com', + username='admin', + api_key='test-key', + disable_ssl_verification=False + ) + assert config.disable_ssl_verification is False \ No newline at end of file diff --git a/tests/src/auth/test_auth_provider.py b/tests/src/auth/test_auth_provider.py new file mode 100644 index 0000000..4d9d6f6 --- /dev/null +++ b/tests/src/auth/test_auth_provider.py @@ -0,0 +1,309 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for AuthProvider module. + +Tests the AuthProvider class and its integration with different authenticators. +""" + +import pytest +from unittest.mock import Mock, patch, MagicMock +from wxdi.common.auth import AuthProvider, AuthConfig, EnvironmentType + + +class TestAuthProviderIBMCloud: + """Tests for AuthProvider with IBM_CLOUD environment.""" + + @patch('wxdi.common.auth.auth_provider.IAMAuthenticator') + def test_ibm_cloud_authenticator_creation(self, mock_iam_auth): + """Test that IAMAuthenticator is created for IBM_CLOUD.""" + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-api-key' + ) + + provider = AuthProvider(config) + + mock_iam_auth.assert_called_once_with( + apikey='test-api-key', + url='https://iam.cloud.ibm.com/identity/token' + ) + assert provider.config == config + + @patch('wxdi.common.auth.auth_provider.IAMAuthenticator') + def test_ibm_cloud_get_token(self, mock_iam_auth): + """Test getting token from IBM_CLOUD authenticator.""" + mock_token_manager = Mock() + mock_token_manager.get_token.return_value = 'test-token-123' + mock_authenticator = Mock() + mock_authenticator.token_manager = mock_token_manager + mock_iam_auth.return_value = mock_authenticator + + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-api-key' + ) + provider = AuthProvider(config) + + token = provider.get_token() + + assert token == 'test-token-123' + mock_token_manager.get_token.assert_called_once() + + +class TestAuthProviderAWSCloud: + """Tests for AuthProvider with AWS_CLOUD environment.""" + + @patch('wxdi.common.auth.auth_provider.MCSPV2Authenticator') + def test_aws_cloud_authenticator_creation(self, mock_mcsp_auth): + """Test that MCSPV2Authenticator is created for AWS_CLOUD.""" + config = AuthConfig( + environment_type=EnvironmentType.AWS_CLOUD, + api_key='test-api-key', + account_id='account-123' + ) + + provider = AuthProvider(config) + + mock_mcsp_auth.assert_called_once_with( + apikey='test-api-key', + url='https://account-iam.platform.saas.ibm.com', + scope_collection_type='accounts', + scope_id='account-123' + ) + + @patch('wxdi.common.auth.auth_provider.MCSPV2Authenticator') + def test_aws_cloud_get_token(self, mock_mcsp_auth): + """Test getting token from AWS_CLOUD authenticator.""" + mock_token_manager = Mock() + mock_token_manager.get_token.return_value = 'aws-token-456' + mock_authenticator = Mock() + mock_authenticator.token_manager = mock_token_manager + mock_mcsp_auth.return_value = mock_authenticator + + config = AuthConfig( + environment_type=EnvironmentType.AWS_CLOUD, + api_key='test-api-key', + account_id='account-123' + ) + provider = AuthProvider(config) + + token = provider.get_token() + + assert token == 'aws-token-456' + mock_token_manager.get_token.assert_called_once() + + +class TestAuthProviderGovCloud: + """Tests for AuthProvider with GOV_CLOUD environment.""" + + @patch('wxdi.common.auth.auth_provider.GovCloudAuthenticator') + def test_gov_cloud_authenticator_creation(self, mock_gov_cloud_auth): + """Test that CustomAuthenticator is created for GOV_CLOUD.""" + config = AuthConfig( + environment_type=EnvironmentType.GOV_CLOUD, + api_key='test-api-key' + ) + + provider = AuthProvider(config) + + mock_gov_cloud_auth.assert_called_once_with( + apikey='test-api-key', + url='https://dai.ibmforusgov.com/api/rest/mcsp/apikeys/token', + disable_ssl_verification=True + ) + + @patch('wxdi.common.auth.auth_provider.GovCloudAuthenticator') + def test_gov_cloud_get_token(self, mock_gov_cloud_auth): + """Test getting token from GOV_CLOUD authenticator.""" + mock_token_manager = Mock() + mock_token_manager.get_token.return_value = 'gov-token-789' + mock_authenticator = Mock() + mock_authenticator.token_manager = mock_token_manager + mock_gov_cloud_auth.return_value = mock_authenticator + + config = AuthConfig( + environment_type=EnvironmentType.GOV_CLOUD, + api_key='test-api-key' + ) + provider = AuthProvider(config) + + token = provider.get_token() + + assert token == 'gov-token-789' + mock_token_manager.get_token.assert_called_once() + + +class TestAuthProviderOnPrem: + """Tests for AuthProvider with ON_PREM environment.""" + + @patch('wxdi.common.auth.auth_provider.CloudPakForDataAuthenticator') + def test_on_prem_with_api_key(self, mock_cp4d_auth): + """Test CloudPakForDataAuthenticator creation with API key.""" + config = AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com', + username='admin', + api_key='test-key' + ) + + provider = AuthProvider(config) + + mock_cp4d_auth.assert_called_once_with( + username='admin', + url='https://cpd.example.com/icp4d-api/v1/authorize', + apikey='test-key', + disable_ssl_verification=True + ) + + @patch('wxdi.common.auth.auth_provider.CloudPakForDataAuthenticator') + def test_on_prem_with_password(self, mock_cp4d_auth): + """Test CloudPakForDataAuthenticator creation with password.""" + config = AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com', + username='admin', + password='secret' + ) + + provider = AuthProvider(config) + + mock_cp4d_auth.assert_called_once_with( + username='admin', + password='secret', + url='https://cpd.example.com/icp4d-api/v1/authorize', + disable_ssl_verification=True + ) + + @patch('wxdi.common.auth.auth_provider.CloudPakForDataAuthenticator') + def test_on_prem_get_token(self, mock_cp4d_auth): + """Test getting token from ON_PREM authenticator.""" + mock_token_manager = Mock() + mock_token_manager.get_token.return_value = 'on-prem-token-abc' + mock_authenticator = Mock() + mock_authenticator.token_manager = mock_token_manager + mock_cp4d_auth.return_value = mock_authenticator + + config = AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com', + username='admin', + api_key='test-key' + ) + provider = AuthProvider(config) + + token = provider.get_token() + + assert token == 'on-prem-token-abc' + mock_token_manager.get_token.assert_called_once() + + +class TestAuthProviderEdgeCases: + """Tests for edge cases and error handling.""" + + def test_invalid_config_type(self): + """Test that invalid config type raises AttributeError.""" + with pytest.raises(AttributeError): + AuthProvider("invalid-config") # type: ignore + + @patch('wxdi.common.auth.auth_provider.IAMAuthenticator') + def test_token_manager_none(self, mock_iam_auth): + """Test handling when token_manager is None.""" + mock_authenticator = Mock() + mock_authenticator.token_manager = None + mock_iam_auth.return_value = mock_authenticator + + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-api-key' + ) + provider = AuthProvider(config) + + with pytest.raises(Exception, match="does not have token_manager"): + provider.get_token() + + @patch('wxdi.common.auth.auth_provider.IAMAuthenticator') + def test_multiple_token_requests(self, mock_iam_auth): + """Test multiple token requests use the same authenticator.""" + mock_token_manager = Mock() + mock_token_manager.get_token.side_effect = ['token1', 'token2', 'token3'] + mock_authenticator = Mock() + mock_authenticator.token_manager = mock_token_manager + mock_iam_auth.return_value = mock_authenticator + + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-api-key' + ) + provider = AuthProvider(config) + + token1 = provider.get_token() + token2 = provider.get_token() + token3 = provider.get_token() + + assert token1 == 'token1' + assert token2 == 'token2' + assert token3 == 'token3' + assert mock_token_manager.get_token.call_count == 3 + # Authenticator should only be created once + assert mock_iam_auth.call_count == 1 + + +class TestAuthProviderIntegration: + """Integration tests for AuthProvider.""" + + @patch('wxdi.common.auth.auth_provider.IAMAuthenticator') + @patch('wxdi.common.auth.auth_provider.GovCloudAuthenticator') + @patch('wxdi.common.auth.auth_provider.MCSPV2Authenticator') + @patch('wxdi.common.auth.auth_provider.CloudPakForDataAuthenticator') + def test_different_environments_use_different_authenticators( + self, mock_cp4d, mock_mcsp, mock_custom, mock_iam + ): + """Test that different environments create different authenticators.""" + # IBM Cloud + config1 = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='key1' + ) + AuthProvider(config1) + assert mock_iam.call_count == 1 + + # AWS Cloud - uses MCSPV2Authenticator + config2 = AuthConfig( + environment_type=EnvironmentType.AWS_CLOUD, + api_key='key2', + account_id='account-123' + ) + AuthProvider(config2) + assert mock_mcsp.call_count == 1 + + # Gov Cloud - uses CustomAuthenticator + config3 = AuthConfig( + environment_type=EnvironmentType.GOV_CLOUD, + api_key='key3' + ) + AuthProvider(config3) + assert mock_custom.call_count == 1 + + # On-Prem + config4 = AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com', + username='admin', + api_key='key4' + ) + AuthProvider(config4) + assert mock_cp4d.call_count == 1 \ No newline at end of file diff --git a/tests/src/auth/test_gov_cloud_authenticator.py b/tests/src/auth/test_gov_cloud_authenticator.py new file mode 100644 index 0000000..fe3bc15 --- /dev/null +++ b/tests/src/auth/test_gov_cloud_authenticator.py @@ -0,0 +1,259 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for GovCloudAuthenticator module. + +Tests the GovCloudAuthenticator class and its integration with GovCloudTokenManager. +""" + +import pytest +from unittest.mock import Mock, patch, MagicMock +from wxdi.common.auth.gov_cloud_authenticator import GovCloudAuthenticator + + +class TestGovCloudAuthenticatorInitialization: + """Tests for GovCloudAuthenticator initialization.""" + + def test_initialization_with_required_params(self): + """Test initialization with required parameters.""" + authenticator = GovCloudAuthenticator( + apikey='test-api-key', + url='https://example.com/token' + ) + + assert authenticator.token_manager is not None + assert authenticator.token_manager.apikey == 'test-api-key' + assert authenticator.token_manager.url == 'https://example.com/token' + + def test_initialization_with_ssl_verification_disabled(self): + """Test initialization with SSL verification disabled.""" + authenticator = GovCloudAuthenticator( + apikey='test-api-key', + url='https://example.com/token', + disable_ssl_verification=True + ) + + assert authenticator.token_manager.disable_ssl_verification is True + + def test_initialization_with_ssl_verification_enabled(self): + """Test initialization with SSL verification enabled (default).""" + authenticator = GovCloudAuthenticator( + apikey='test-api-key', + url='https://example.com/token' + ) + + assert authenticator.token_manager.disable_ssl_verification is False + + +class TestGovCloudAuthenticatorValidation: + """Tests for GovCloudAuthenticator validation.""" + + def test_validate_with_valid_config(self): + """Test validation with valid configuration.""" + authenticator = GovCloudAuthenticator( + apikey='test-api-key', + url='https://example.com/token' + ) + + # Should not raise any exception + authenticator.validate() + + def test_validate_with_missing_apikey(self): + """Test validation fails with missing API key.""" + with pytest.raises(ValueError, match="apikey"): + GovCloudAuthenticator( + apikey='', + url='https://example.com/token' + ) + + def test_validate_with_missing_url(self): + """Test validation fails with missing URL.""" + with pytest.raises(ValueError, match="url"): + GovCloudAuthenticator( + apikey='test-api-key', + url='' + ) + + +class TestGovCloudAuthenticatorTokenManagement: + """Tests for GovCloudAuthenticator token management.""" + + @patch('wxdi.common.auth.gov_cloud_authenticator.GovCloudTokenManager') + def test_token_manager_creation(self, mock_token_manager_class): + """Test that GovCloudTokenManager is created correctly.""" + mock_token_manager = Mock() + mock_token_manager_class.return_value = mock_token_manager + + authenticator = GovCloudAuthenticator( + apikey='test-api-key', + url='https://example.com/token' + ) + + mock_token_manager_class.assert_called_once_with( + apikey='test-api-key', + url='https://example.com/token', + disable_ssl_verification=False, + headers=None, + proxies=None + ) + assert authenticator.token_manager == mock_token_manager + + @patch('wxdi.common.auth.gov_cloud_authenticator.GovCloudTokenManager') + def test_get_token(self, mock_token_manager_class): + """Test getting token through token manager.""" + mock_token_manager = Mock() + mock_token_manager.get_token.return_value = 'test-token-abc123' + mock_token_manager_class.return_value = mock_token_manager + + authenticator = GovCloudAuthenticator( + apikey='test-api-key', + url='https://example.com/token' + ) + + token = authenticator.token_manager.get_token() + + assert token == 'test-token-abc123' + mock_token_manager.get_token.assert_called_once() + + +class TestGovCloudAuthenticatorInheritance: + """Tests for GovCloudAuthenticator inheritance from IAMRequestBasedAuthenticator.""" + + def test_inherits_from_iam_request_based_authenticator(self): + """Test that GovCloudAuthenticator inherits from IAMRequestBasedAuthenticator.""" + from ibm_cloud_sdk_core.authenticators.iam_request_based_authenticator import IAMRequestBasedAuthenticator + + authenticator = GovCloudAuthenticator( + apikey='test-api-key', + url='https://example.com/token' + ) + + assert isinstance(authenticator, IAMRequestBasedAuthenticator) + + def test_has_authentication_type(self): + """Test that authenticator has authentication_type attribute.""" + authenticator = GovCloudAuthenticator( + apikey='test-api-key', + url='https://example.com/token' + ) + + # IAMRequestBasedAuthenticator should have authentication_type + assert hasattr(authenticator, 'authentication_type') + + +class TestGovCloudAuthenticatorEdgeCases: + """Tests for edge cases and error handling.""" + + def test_none_apikey(self): + """Test that None API key raises error.""" + with pytest.raises((ValueError, TypeError)): + GovCloudAuthenticator( + apikey=None, # type: ignore + url='https://example.com/token' + ) + + def test_none_url(self): + """Test that None URL raises error.""" + with pytest.raises((ValueError, TypeError)): + GovCloudAuthenticator( + apikey='test-api-key', + url=None # type: ignore + ) + + def test_empty_string_apikey(self): + """Test that empty string API key raises error.""" + with pytest.raises(ValueError): + GovCloudAuthenticator( + apikey='', + url='https://example.com/token' + ) + + def test_empty_string_url(self): + """Test that empty string URL raises error.""" + with pytest.raises(ValueError): + GovCloudAuthenticator( + apikey='test-api-key', + url='' + ) + + def test_whitespace_apikey(self): + """Test that whitespace-only API key is accepted but may fail on validation.""" + # GovCloudAuthenticator doesn't strip whitespace, so it accepts it + authenticator = GovCloudAuthenticator( + apikey=' ', + url='https://example.com/token' + ) + # The whitespace apikey is stored as-is + assert authenticator.apikey == ' ' + + def test_whitespace_url(self): + """Test that whitespace-only URL is accepted but may fail on validation.""" + # GovCloudAuthenticator doesn't strip whitespace, so it accepts it + authenticator = GovCloudAuthenticator( + apikey='test-api-key', + url=' ' + ) + # The whitespace url is stored as-is + assert authenticator.url == ' ' + + +class TestGovCloudAuthenticatorIntegration: + """Integration tests for GovCloudAuthenticator.""" + + @patch('wxdi.common.auth.gov_cloud_authenticator.GovCloudTokenManager') + def test_full_authentication_flow(self, mock_token_manager_class): + """Test complete authentication flow.""" + mock_token_manager = Mock() + mock_token_manager.get_token.return_value = 'integration-token-xyz' + mock_token_manager_class.return_value = mock_token_manager + + # Create authenticator + authenticator = GovCloudAuthenticator( + apikey='integration-key', + url='https://integration.example.com/token' + ) + + # Validate + authenticator.validate() + + # Get token + token = authenticator.token_manager.get_token() + + assert token == 'integration-token-xyz' + mock_token_manager.get_token.assert_called_once() + + @patch('wxdi.common.auth.gov_cloud_authenticator.GovCloudTokenManager') + def test_multiple_token_requests(self, mock_token_manager_class): + """Test multiple token requests use the same token manager.""" + mock_token_manager = Mock() + mock_token_manager.get_token.side_effect = ['token1', 'token2', 'token3'] + mock_token_manager_class.return_value = mock_token_manager + + authenticator = GovCloudAuthenticator( + apikey='test-key', + url='https://example.com/token' + ) + + token1 = authenticator.token_manager.get_token() + token2 = authenticator.token_manager.get_token() + token3 = authenticator.token_manager.get_token() + + assert token1 == 'token1' + assert token2 == 'token2' + assert token3 == 'token3' + assert mock_token_manager.get_token.call_count == 3 + # Token manager should only be created once + assert mock_token_manager_class.call_count == 1 \ No newline at end of file diff --git a/tests/src/auth/test_gov_cloud_token_manager.py b/tests/src/auth/test_gov_cloud_token_manager.py new file mode 100644 index 0000000..30fa736 --- /dev/null +++ b/tests/src/auth/test_gov_cloud_token_manager.py @@ -0,0 +1,403 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for GovCloudTokenManager module. + +Tests the GovCloudTokenManager class and its token management functionality. +""" + +import pytest +import time +from unittest.mock import Mock, patch, MagicMock +from wxdi.common.auth.gov_cloud_token_manager import GovCloudTokenManager + +# Valid JWT token for testing (header.payload.signature format) +VALID_JWT_TOKEN = 'eyJhbGciOiAiUlMyNTYiLCAidHlwIjogIkpXVCJ9.eyJleHAiOiA5OTk5OTk5OTk5LCAiaWF0IjogMTIzNDU2Nzg5MH0.ZmFrZS1zaWduYXR1cmU' + + +class TestGovCloudTokenManagerInitialization: + """Tests for GovCloudTokenManager initialization.""" + + def test_initialization_with_required_params(self): + """Test initialization with required parameters.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + assert token_manager.apikey == 'test-api-key' + assert token_manager.url == 'https://example.com/token' + assert token_manager.disable_ssl_verification is False + assert token_manager.headers == {} + assert token_manager.proxies == {} + + def test_initialization_with_ssl_verification_disabled(self): + """Test initialization with SSL verification disabled.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token', + disable_ssl_verification=True + ) + + assert token_manager.disable_ssl_verification is True + + def test_initialization_with_custom_headers(self): + """Test initialization with custom headers.""" + custom_headers = {'X-Custom-Header': 'value'} + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token', + headers=custom_headers + ) + + assert token_manager.headers == custom_headers + + def test_initialization_with_proxies(self): + """Test initialization with proxy configuration.""" + proxies = {'http': 'http://proxy.example.com:8080'} + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token', + proxies=proxies + ) + + assert token_manager.proxies == proxies + + +class TestGovCloudTokenManagerInheritance: + """Tests for GovCloudTokenManager inheritance from JWTTokenManager.""" + + def test_inherits_from_jwt_token_manager(self): + """Test that GovCloudTokenManager inherits from JWTTokenManager.""" + from ibm_cloud_sdk_core.token_managers.jwt_token_manager import JWTTokenManager + + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + assert isinstance(token_manager, JWTTokenManager) + + def test_has_get_token_method(self): + """Test that token manager has get_token method from parent.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + assert hasattr(token_manager, 'get_token') + assert callable(token_manager.get_token) + + +class TestGovCloudTokenManagerRequestToken: + """Tests for request_token method.""" + + @patch.object(GovCloudTokenManager, '_request') + def test_request_token_makes_post_request(self, mock_request): + """Test that request_token makes a POST request.""" + mock_request.return_value = { + 'token': 'test-token-123', + 'expires_in': 3600 + } + + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + response = token_manager.request_token() + + mock_request.assert_called_once() + call_args = mock_request.call_args + assert call_args[1]['method'] == 'POST' + assert call_args[1]['url'] == 'https://example.com/token' + + @patch.object(GovCloudTokenManager, '_request') + def test_request_token_uses_form_urlencoded(self, mock_request): + """Test that request_token uses application/x-www-form-urlencoded.""" + mock_request.return_value = { + 'token': 'test-token-123', + 'expires_in': 3600 + } + + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + token_manager.request_token() + + call_args = mock_request.call_args + headers = call_args[1]['headers'] + assert headers['Content-Type'] == 'application/x-www-form-urlencoded' + assert headers['Accept'] == 'application/json' + + @patch.object(GovCloudTokenManager, '_request') + def test_request_token_sends_apikey(self, mock_request): + """Test that request_token sends the API key in the request.""" + mock_request.return_value = { + 'token': 'test-token-123', + 'expires_in': 3600 + } + + token_manager = GovCloudTokenManager( + apikey='my-secret-key', + url='https://example.com/token' + ) + + token_manager.request_token() + + call_args = mock_request.call_args + data = call_args[1]['data'] + assert data['apikey'] == 'my-secret-key' + + @patch.object(GovCloudTokenManager, '_request') + def test_request_token_includes_custom_headers(self, mock_request): + """Test that custom headers are included in the request.""" + mock_request.return_value = { + 'token': 'test-token-123', + 'expires_in': 3600 + } + + custom_headers = {'X-Custom-Header': 'custom-value'} + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token', + headers=custom_headers + ) + + token_manager.request_token() + + call_args = mock_request.call_args + headers = call_args[1]['headers'] + assert headers['X-Custom-Header'] == 'custom-value' + + @patch.object(GovCloudTokenManager, '_request') + def test_request_token_returns_dict(self, mock_request): + """Test that request_token returns a dictionary.""" + expected_response = { + 'token': 'test-token-456', + 'expires_in': 7200 + } + mock_request.return_value = expected_response + + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + response = token_manager.request_token() + + assert response == expected_response + assert isinstance(response, dict) + + +class TestGovCloudTokenManagerSaveTokenInfo: + """Tests for _save_token_info method.""" + + def test_save_token_info_with_token_field(self): + """Test saving token info when response has 'token' field.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + token_response = { + 'token': VALID_JWT_TOKEN, + 'expires_in': 3600 + } + + token_manager._save_token_info(token_response) + + assert token_manager.access_token == VALID_JWT_TOKEN + + def test_save_token_info_with_access_token_field(self): + """Test saving token info when response has 'access_token' field.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + # Note: JWTTokenManager expects 'token' field, not 'access_token' + # The token manager will look for 'token' field first + token_response = { + 'token': VALID_JWT_TOKEN, + 'expires_in': 3600 + } + + token_manager._save_token_info(token_response) + + assert token_manager.access_token == VALID_JWT_TOKEN + + def test_save_token_info_prefers_token_over_access_token(self): + """Test that 'token' field is preferred over 'access_token'.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + token_response = { + 'token': VALID_JWT_TOKEN, + 'access_token': VALID_JWT_TOKEN, + 'expires_in': 3600 + } + + token_manager._save_token_info(token_response) + + assert token_manager.access_token == VALID_JWT_TOKEN + + def test_save_token_info_raises_error_without_token(self): + """Test that error is raised when no token is in response.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + token_response = { + 'expires_in': 3600 + } + + # JWTTokenManager will raise an error when token field is missing + with pytest.raises(Exception): # Could be ValueError or KeyError + token_manager._save_token_info(token_response) + + def test_save_token_info_with_expires_in(self): + """Test saving token info with expires_in field.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + token_response = { + 'token': VALID_JWT_TOKEN, + 'expires_in': 3600 + } + + token_manager._save_token_info(token_response) + + # JWTTokenManager extracts expiration from JWT token payload (exp field) + # Our JWT token has exp=9999999999 + assert token_manager.expire_time == 9999999999 + assert token_manager.access_token == VALID_JWT_TOKEN + + def test_save_token_info_with_expiration(self): + """Test saving token info with expiration timestamp.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + token_response = { + 'token': VALID_JWT_TOKEN, + 'expiration': 1234567890 + } + + token_manager._save_token_info(token_response) + + # JWTTokenManager extracts expiration from JWT token payload (exp field) + # Our JWT token has exp=9999999999, not from the 'expiration' field + assert token_manager.expire_time == 9999999999 + assert token_manager.access_token == VALID_JWT_TOKEN + + def test_save_token_info_default_expiration(self): + """Test that expiration is extracted from JWT token.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + token_response = { + 'token': VALID_JWT_TOKEN + } + + token_manager._save_token_info(token_response) + + # JWTTokenManager extracts expiration from JWT token payload (exp field) + # Our JWT token has exp=9999999999 + assert token_manager.expire_time == 9999999999 + assert token_manager.access_token == VALID_JWT_TOKEN + + +class TestGovCloudTokenManagerEdgeCases: + """Tests for edge cases and error handling.""" + + def test_empty_apikey(self): + """Test that empty API key is handled.""" + token_manager = GovCloudTokenManager( + apikey='', + url='https://example.com/token' + ) + + # Should initialize but may fail on actual request + assert token_manager.apikey == '' + + def test_empty_url(self): + """Test that empty URL is handled.""" + token_manager = GovCloudTokenManager( + apikey='test-key', + url='' + ) + + # Should initialize but may fail on actual request + assert token_manager.url == '' + + def test_none_headers_defaults_to_empty_dict(self): + """Test that None headers defaults to empty dict.""" + token_manager = GovCloudTokenManager( + apikey='test-key', + url='https://example.com/token', + headers=None + ) + + assert token_manager.headers == {} + + def test_none_proxies_defaults_to_empty_dict(self): + """Test that None proxies defaults to empty dict.""" + token_manager = GovCloudTokenManager( + apikey='test-key', + url='https://example.com/token', + proxies=None + ) + + assert token_manager.proxies == {} + + +class TestGovCloudTokenManagerIntegration: + """Integration tests for GovCloudTokenManager.""" + + @patch.object(GovCloudTokenManager, '_request') + def test_full_token_lifecycle(self, mock_request): + """Test complete token request and save lifecycle.""" + mock_request.return_value = { + 'token': VALID_JWT_TOKEN, + 'expires_in': 7200, + 'refresh_token': 'refresh-lifecycle' + } + + token_manager = GovCloudTokenManager( + apikey='lifecycle-key', + url='https://example.com/token' + ) + + # Request token + response = token_manager.request_token() + + # Save token info + token_manager._save_token_info(response) + + # Verify token was saved + assert token_manager.access_token == VALID_JWT_TOKEN + assert hasattr(token_manager, 'expire_time') \ No newline at end of file diff --git a/tests/src/data_product_recommender/__init__.py b/tests/src/data_product_recommender/__init__.py new file mode 100644 index 0000000..2d49fb5 --- /dev/null +++ b/tests/src/data_product_recommender/__init__.py @@ -0,0 +1,19 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Data Product Recommender test package""" + +# Made with Bob diff --git a/tests/src/data_product_recommender/test_data_product_recommender.py b/tests/src/data_product_recommender/test_data_product_recommender.py new file mode 100644 index 0000000..df31423 --- /dev/null +++ b/tests/src/data_product_recommender/test_data_product_recommender.py @@ -0,0 +1,613 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Unit tests for DataProductRecommender +""" + +import pytest +import pandas as pd +import json +import tempfile +import os +from datetime import datetime, timedelta +from wxdi.data_product_recommender.recommender import ( + DataProductRecommender, + normalize_query_pattern, + FIVE_STARS, FOUR_STARS, THREE_STARS, TWO_STARS, ONE_STAR, + EXCELLENT_CANDIDATE, GOOD_CANDIDATE, FAIR_CANDIDATE, WEAK_CANDIDATE, POOR_CANDIDATE +) +from wxdi.data_product_recommender.platforms import SnowflakeQueryParser + + +class TestNormalizeQueryPattern: + """Tests for normalize_query_pattern function""" + + def test_normalize_simple_query(self): + """Test normalizing a simple query""" + query = "SELECT * FROM SALES.CUSTOMERS WHERE id = 123" + pattern = normalize_query_pattern(query) + assert pattern == "SELECT * FROM SALES.CUSTOMERS WHERE ID = ?" + + def test_normalize_query_with_strings(self): + """Test normalizing query with string literals""" + query = "SELECT * FROM SALES.CUSTOMERS WHERE name = 'John Doe'" + pattern = normalize_query_pattern(query) + assert pattern == "SELECT * FROM SALES.CUSTOMERS WHERE NAME = ?" + + def test_normalize_query_with_double_quotes(self): + """Test normalizing query with double-quoted strings""" + query = 'SELECT * FROM SALES.CUSTOMERS WHERE name = "John Doe"' + pattern = normalize_query_pattern(query) + assert pattern == "SELECT * FROM SALES.CUSTOMERS WHERE NAME = ?" + + def test_normalize_query_with_decimals(self): + """Test normalizing query with decimal numbers""" + query = "SELECT * FROM SALES.ORDERS WHERE amount > 123.45" + pattern = normalize_query_pattern(query) + assert pattern == "SELECT * FROM SALES.ORDERS WHERE AMOUNT > ?" + + def test_normalize_query_with_dates(self): + """Test normalizing query with date literals""" + query = "SELECT * FROM SALES.ORDERS WHERE date = '2024-01-01'" + pattern = normalize_query_pattern(query) + assert "?" in pattern + + def test_normalize_empty_query(self): + """Test normalizing empty query""" + assert normalize_query_pattern("") == "" + assert normalize_query_pattern(None) == "" + + def test_normalize_query_whitespace(self): + """Test normalizing query with multiple whitespaces""" + query = "SELECT * FROM SALES.CUSTOMERS" + pattern = normalize_query_pattern(query) + assert pattern == "SELECT * FROM SALES.CUSTOMERS" + + def test_normalize_query_with_escaped_quotes(self): + """Test normalizing query with escaped quotes""" + query = "SELECT * FROM SALES.CUSTOMERS WHERE name = 'O\\'Brien'" + pattern = normalize_query_pattern(query) + assert "?" in pattern + + +class TestDataProductRecommender: + """Tests for DataProductRecommender class""" + + def setup_method(self): + """Setup test fixtures""" + self.parser = SnowflakeQueryParser() + self.recommender = DataProductRecommender(self.parser) + + def test_initialization(self): + """Test recommender initialization""" + assert self.recommender.parser is not None + assert self.recommender.query_logs is None + assert self.recommender.table_metrics is None + assert self.recommender.query_patterns is None + + def test_load_query_logs_from_json_file(self): + """Test loading query logs from JSON file""" + # Create temporary JSON file + test_data = [ + { + 'query_text': 'SELECT * FROM SALES.CUSTOMERS', + 'user_name': 'user1', + 'start_time': '2025-01-01' + }, + { + 'query_text': 'SELECT * FROM SALES.ORDERS', + 'user_name': 'user2', + 'start_time': '2025-01-02' + } + ] + + with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: + json.dump(test_data, f) + temp_file = f.name + + try: + # Load the data + df = self.recommender.load_query_logs_from_json_file(temp_file) + + # Verify + assert df is not None + assert len(df) == 2 + assert 'query_text' in df.columns + assert 'user' in df.columns + assert 'tables' in df.columns + finally: + os.unlink(temp_file) + + def test_load_query_logs_from_csv_file(self): + """Test loading query logs from CSV file""" + # Create temporary CSV file + test_data = pd.DataFrame({ + 'query_text': ['SELECT * FROM SALES.CUSTOMERS', 'SELECT * FROM SALES.ORDERS'], + 'user_name': ['user1', 'user2'], + 'start_time': ['2025-01-01', '2025-01-02'] + }) + + with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f: + test_data.to_csv(f.name, index=False) + temp_file = f.name + + try: + # Load the data + df = self.recommender.load_query_logs_from_csv_file(temp_file) + + # Verify + assert df is not None + assert len(df) == 2 + assert 'query_text' in df.columns + assert 'user' in df.columns + assert 'tables' in df.columns + finally: + os.unlink(temp_file) + + def test_calculate_metrics(self): + """Test calculating table metrics""" + # Setup test data + test_data = pd.DataFrame({ + 'query_text': [ + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.ORDERS', + 'SELECT * FROM SALES.CUSTOMERS JOIN SALES.ORDERS ON c.id = o.customer_id' + ], + 'user': ['user1', 'user2', 'user1', 'user3'], + 'start_time': ['2025-01-01', '2025-01-02', '2025-01-03', '2025-01-04'] + }) + + # Manually add tables column + test_data['tables'] = [ + ['SALES.CUSTOMERS'], + ['SALES.CUSTOMERS'], + ['SALES.ORDERS'], + ['SALES.CUSTOMERS', 'SALES.ORDERS'] + ] + + self.recommender.query_logs = test_data + + # Calculate metrics + metrics = self.recommender.calculate_metrics() + + # Verify + assert metrics is not None + assert len(metrics) == 2 # Two unique tables + assert 'table' in metrics.columns + assert 'query_count' in metrics.columns + assert 'unique_users' in metrics.columns + assert 'related_tables' in metrics.columns + + # Check specific values + customers_row = metrics[metrics['table'] == 'SALES.CUSTOMERS'].iloc[0] + assert customers_row['query_count'] == 3 + assert customers_row['unique_users'] == 3 + + def test_score_tables(self): + """Test scoring tables""" + # Setup test data with metrics including temporal fields + self.recommender.table_metrics = pd.DataFrame({ + 'table': ['SALES.CUSTOMERS', 'SALES.ORDERS'], + 'query_count': [10, 5], + 'unique_users': [5, 3], + 'related_tables': [['SALES.ORDERS'], ['SALES.CUSTOMERS']], + 'related_table_count': [1, 1], + 'recency_score': [0.9, 0.5], + 'consistency_score': [0.8, 0.6] + }) + + # Score tables + scored = self.recommender.score_tables() + + # Verify + assert scored is not None + assert len(scored) == 2 + assert 'recommendation_score' in scored.columns + assert scored.iloc[0]['recommendation_score'] > scored.iloc[1]['recommendation_score'] + + def test_identify_table_groups(self): + """Test identifying table groups""" + # Setup test data + test_data = pd.DataFrame({ + 'query_text': ['query1', 'query2', 'query3'], + 'user': ['user1', 'user2', 'user3'], + 'start_time': ['2025-01-01', '2025-01-02', '2025-01-03'], + 'tables': [ + ['SALES.CUSTOMERS', 'SALES.ORDERS'], + ['SALES.CUSTOMERS', 'SALES.ORDERS'], + ['SALES.ORDERS', 'PRODUCT.CATALOG'] + ] + }) + + self.recommender.query_logs = test_data + + # Identify groups + groups = self.recommender.identify_table_groups(min_cooccurrence=2) + + # Verify + assert groups is not None + assert len(groups) >= 1 + # The group (SALES.CUSTOMERS, SALES.ORDERS) should appear with count 2 + assert any(count == 2 for _, count in groups) + + def test_recommend_data_products(self): + """Test generating recommendations""" + # Setup complete test data + test_data = pd.DataFrame({ + 'query_text': ['query1', 'query2', 'query3'], + 'user': ['user1', 'user2', 'user3'], + 'start_time': ['2025-01-01', '2025-01-02', '2025-01-03'], + 'tables': [ + ['SALES.CUSTOMERS'], + ['SALES.CUSTOMERS'], + ['SALES.ORDERS'] + ] + }) + + self.recommender.query_logs = test_data + self.recommender.calculate_metrics() + + # Generate recommendations + recommendations = self.recommender.recommend_data_products( + num_recommendations=2 + ) + + # Verify structure + assert recommendations is not None + assert 'individual_tables' in recommendations + assert 'table_groups' in recommendations + assert 'metadata' in recommendations + assert len(recommendations['individual_tables']) <= 2 + + # Verify metadata + metadata = recommendations['metadata'] + assert 'total_tables' in metadata + assert 'recommended_tables' in metadata + assert 'highest_score' in metadata + assert 'min_score_threshold' in metadata + assert metadata['total_tables'] >= metadata['recommended_tables'] + assert metadata['min_score_threshold'] is None # No threshold in this test + + def test_export_recommendations_markdown(self): + """Test exporting recommendations to markdown""" + # Setup test data + self.recommender.query_logs = pd.DataFrame({ + 'query_text': ['query1'], + 'user': ['user1'], + 'start_time': ['2025-01-01'], + 'tables': [['SALES.CUSTOMERS']] + }) + + self.recommender.table_metrics = pd.DataFrame({ + 'table': ['SALES.CUSTOMERS'], + 'query_count': [10], + 'unique_users': [5], + 'related_tables': [[]], + 'related_table_count': [0] + }) + + recommendations = { + 'individual_tables': [ + { + 'table': 'SALES.CUSTOMERS', + 'query_count': 10, + 'unique_users': 5, + 'recommendation_score': 85.5, + 'related_tables': [] + } + ] + } + + # Export to temporary file + with tempfile.NamedTemporaryFile(mode='w', suffix='.md', delete=False) as f: + temp_file = f.name + + try: + self.recommender.export_recommendations_markdown(recommendations, temp_file) + + # Verify file was created and has content + assert os.path.exists(temp_file) + with open(temp_file, 'r') as f: + content = f.read() + assert 'Data Product Recommendations' in content + assert 'SALES.CUSTOMERS' in content + finally: + os.unlink(temp_file) + + def test_export_recommendations_json(self): + """Test exporting recommendations to JSON""" + # Setup test data + self.recommender.query_logs = pd.DataFrame({ + 'query_text': ['query1'], + 'user': ['user1'], + 'start_time': ['2025-01-01'], + 'tables': [['SALES.CUSTOMERS']] + }) + + recommendations = { + 'individual_tables': [ + { + 'table': 'SALES.CUSTOMERS', + 'query_count': 10, + 'unique_users': 5, + 'recommendation_score': 85.5, + 'recency_score': 0.9, + 'consistency_score': 0.8 + } + ], + 'metadata': { + 'total_tables': 1, + 'recommended_tables': 1 + } + } + + # Export to temporary file + with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: + temp_file = f.name + + try: + self.recommender.export_recommendations_json(recommendations, temp_file) + + # Verify file was created and has valid JSON + assert os.path.exists(temp_file) + with open(temp_file, 'r') as f: + data = json.load(f) + # JSON export uses 'recommendations' key, not 'individual_tables' + assert 'recommendations' in data + assert 'metadata' in data + assert len(data['recommendations']) == 1 + # Verify structure + assert data['recommendations'][0]['type'] == 'standalone' + assert 'score' in data['recommendations'][0] + assert 'rating' in data['recommendations'][0] + finally: + os.unlink(temp_file) + + def test_calculate_temporal_metrics(self): + """Test calculating temporal metrics (recency and consistency)""" + # Setup test data with specific dates + base_date = datetime(2025, 1, 1) + test_data = pd.DataFrame({ + 'query_text': [ + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.CUSTOMERS' + ], + 'user': ['user1', 'user1', 'user1'], + 'start_time': [ + base_date, + base_date + timedelta(days=1), + base_date + timedelta(days=2) + ], + 'tables': [ + ['SALES.CUSTOMERS'], + ['SALES.CUSTOMERS'], + ['SALES.CUSTOMERS'] + ] + }) + + self.recommender.query_logs = test_data + metrics = self.recommender.calculate_metrics() + + # Verify temporal metrics are calculated + assert 'recency_score' in metrics.columns + assert 'consistency_score' in metrics.columns + assert 'days_since_last_query' in metrics.columns + assert metrics.iloc[0]['recency_score'] > 0 + + def test_recommend_with_table_groups(self): + """Test recommendations include table groups""" + # Setup test data with co-occurring tables + test_data = pd.DataFrame({ + 'query_text': ['query1', 'query2', 'query3'], + 'user': ['user1', 'user2', 'user3'], + 'start_time': ['2025-01-01', '2025-01-02', '2025-01-03'], + 'tables': [ + ['SALES.CUSTOMERS', 'SALES.ORDERS'], + ['SALES.CUSTOMERS', 'SALES.ORDERS'], + ['SALES.CUSTOMERS', 'SALES.ORDERS'] + ] + }) + + self.recommender.query_logs = test_data + self.recommender.calculate_metrics() + + recommendations = self.recommender.recommend_data_products( + num_recommendations=10 + ) + + # Verify table groups are identified + assert 'table_groups' in recommendations + + def test_score_tables_with_custom_weights(self): + """Test scoring tables with custom weights""" + # Setup test data + self.recommender.table_metrics = pd.DataFrame({ + 'table': ['SALES.CUSTOMERS'], + 'query_count': [10], + 'unique_users': [5], + 'related_tables': [[]], + 'related_table_count': [0], + 'recency_score': [0.9], + 'consistency_score': [0.8] + }) + + # Score with custom weights + scored = self.recommender.score_tables( + query_weight=0.5, + user_weight=0.3, + recency_weight=0.1, + consistency_weight=0.1 + ) + + assert 'recommendation_score' in scored.columns + assert len(scored) == 1 + + def test_get_rating_label(self): + """Test rating label assignment""" + # _get_rating_label returns just a string label, not a tuple + assert self.recommender._get_rating_label(95) == 'excellent' + assert self.recommender._get_rating_label(75) == 'good' + assert self.recommender._get_rating_label(55) == 'fair' + assert self.recommender._get_rating_label(35) == 'weak' + assert self.recommender._get_rating_label(15) == 'poor' + + def test_get_star_rating(self): + """Test star rating assignment""" + # _get_star_rating returns a tuple of (stars, label) + stars, label = self.recommender._get_star_rating(95) + assert stars == FIVE_STARS + assert label == EXCELLENT_CANDIDATE + + stars, label = self.recommender._get_star_rating(75) + assert stars == FOUR_STARS + assert label == GOOD_CANDIDATE + + stars, label = self.recommender._get_star_rating(55) + assert stars == THREE_STARS + assert label == FAIR_CANDIDATE + + stars, label = self.recommender._get_star_rating(35) + assert stars == TWO_STARS + assert label == WEAK_CANDIDATE + + stars, label = self.recommender._get_star_rating(15) + assert stars == ONE_STAR + assert label == POOR_CANDIDATE + + def test_load_query_logs_filters_empty_tables(self): + """Test that queries with no table references are filtered out""" + # Create test data with some queries having no tables + test_data = [ + { + 'query_text': 'SELECT * FROM SALES.CUSTOMERS', + 'user_name': 'user1', + 'start_time': '2025-01-01' + }, + { + 'query_text': 'SHOW TABLES', # No table reference + 'user_name': 'user2', + 'start_time': '2025-01-02' + } + ] + + with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: + json.dump(test_data, f) + temp_file = f.name + + try: + df = self.recommender.load_query_logs_from_json_file(temp_file) + # Should only have 1 query (the one with table reference) + assert len(df) >= 0 # May vary based on parser + finally: + os.unlink(temp_file) + + def test_calculate_metrics_with_single_timestamp(self): + """Test metrics calculation with single timestamp per table""" + test_data = pd.DataFrame({ + 'query_text': ['SELECT * FROM SALES.CUSTOMERS'], + 'user': ['user1'], + 'start_time': [datetime(2025, 1, 1)], + 'tables': [['SALES.CUSTOMERS']] + }) + + self.recommender.query_logs = test_data + metrics = self.recommender.calculate_metrics() + + # Should handle single timestamp gracefully + assert len(metrics) == 1 + assert metrics.iloc[0]['consistency_score'] == 0.0 # Single timestamp = 0 consistency + + def test_recommend_with_min_score_filters_tables(self): + """Test that min_score filters out low-scoring tables""" + # Setup test data with varying query counts + test_data = pd.DataFrame({ + 'query_text': [ + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.ORDERS' + ], + 'user': ['user1', 'user2', 'user3', 'user1'], + 'start_time': ['2025-01-01', '2025-01-02', '2025-01-03', '2025-01-04'], + 'tables': [ + ['SALES.CUSTOMERS'], + ['SALES.CUSTOMERS'], + ['SALES.CUSTOMERS'], + ['SALES.ORDERS'] + ] + }) + + self.recommender.query_logs = test_data + self.recommender.calculate_metrics() + + # Get recommendations with high min_score + recommendations = self.recommender.recommend_data_products( + num_recommendations=10, + min_score=70.0 + ) + + # Should filter out low-scoring tables + assert recommendations['metadata']['min_score_threshold'] == 70.0 + for table in recommendations['individual_tables']: + assert table['recommendation_score'] >= 70.0 + + +class TestRecommenderEdgeCases: + """Test edge cases for recommender""" + + def setup_method(self): + """Setup test fixtures""" + self.parser = SnowflakeQueryParser() + self.recommender = DataProductRecommender(self.parser) + + def test_calculate_metrics_without_loaded_data(self): + """Test calculating metrics without loading data first""" + with pytest.raises(ValueError, match="Query logs not loaded"): + self.recommender.calculate_metrics() + + def test_score_tables_without_metrics(self): + """Test scoring tables without calculating metrics first""" + with pytest.raises(ValueError, match="Metrics not calculated"): + self.recommender.score_tables() + + def test_identify_table_groups_without_loaded_data(self): + """Test identifying groups without loading data first""" + with pytest.raises(ValueError, match="Query logs not loaded"): + self.recommender.identify_table_groups() + + def test_load_csv_with_missing_columns(self): + """Test loading CSV with missing required columns""" + # Create CSV with missing columns + test_data = pd.DataFrame({ + 'query_text': ['SELECT * FROM table1'], + 'start_time': ['2025-01-01'] + # Missing 'user' column + }) + + with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f: + test_data.to_csv(f.name, index=False) + temp_file = f.name + + try: + with pytest.raises(ValueError, match="missing required columns"): + self.recommender.load_query_logs_from_csv_file(temp_file) + finally: + os.unlink(temp_file) + +# Made with Bob diff --git a/tests/src/data_product_recommender/test_data_product_recommender_cli.py b/tests/src/data_product_recommender/test_data_product_recommender_cli.py new file mode 100644 index 0000000..63d5db6 --- /dev/null +++ b/tests/src/data_product_recommender/test_data_product_recommender_cli.py @@ -0,0 +1,306 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Unit tests for Data Product Recommender CLI +""" + +import pytest +import json +import tempfile +import os +from pathlib import Path +from unittest.mock import patch, MagicMock +import pandas as pd + +from wxdi.data_product_recommender.cli import main +from wxdi.data_product_recommender.platforms import SnowflakeQueryParser + + +class TestCLI: + """Tests for CLI functionality""" + + def test_cli_with_csv_snowflake(self): + """Test CLI with CSV file and Snowflake platform""" + # Create temporary CSV file + test_data = pd.DataFrame({ + 'query_text': ['SELECT * FROM SALES.CUSTOMERS', 'SELECT * FROM SALES.ORDERS'], + 'user_name': ['user1', 'user2'], + 'start_time': ['2025-01-01', '2025-01-02'] + }) + + with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f: + test_data.to_csv(f.name, index=False) + csv_file = f.name + + with tempfile.TemporaryDirectory() as output_dir: + try: + test_args = [ + 'cli.py', + '--platform', 'snowflake', + '--input-file', csv_file, + '--output', output_dir, + '--num-recommendations', '5' + ] + + with patch('sys.argv', test_args): + main() + + # Verify output file was created + output_files = list(Path(output_dir).glob('recommendations_*.md')) + assert len(output_files) == 1 + + finally: + os.unlink(csv_file) + + def test_cli_with_json_databricks(self): + """Test CLI with JSON file and Databricks platform""" + # Create temporary JSON file + test_data = [ + { + 'statement_text': 'SELECT * FROM CLINICAL.PATIENTS', + 'executed_by': 'user1', + 'start_time': '2025-01-01' + }, + { + 'statement_text': 'SELECT * FROM CLINICAL.VISITS', + 'executed_by': 'user2', + 'start_time': '2025-01-02' + } + ] + + with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: + json.dump(test_data, f) + json_file = f.name + + with tempfile.TemporaryDirectory() as output_dir: + try: + test_args = [ + 'cli.py', + '--platform', 'databricks', + '--input-file', json_file, + '--output', output_dir, + '--output-format', 'json', + '--num-recommendations', '10' + ] + + with patch('sys.argv', test_args): + main() + + # Verify JSON output file was created + output_files = list(Path(output_dir).glob('recommendations_*.json')) + assert len(output_files) == 1 + + # Verify JSON is valid + with open(output_files[0], 'r') as f: + recommendations = json.load(f) + # JSON export uses 'recommendations' key, not 'individual_tables' + assert 'recommendations' in recommendations + assert 'metadata' in recommendations + + finally: + os.unlink(json_file) + + def test_cli_with_bigquery(self): + """Test CLI with BigQuery platform""" + # Create temporary CSV file + test_data = pd.DataFrame({ + 'query': ['SELECT * FROM PRODUCT.CATALOG', 'SELECT * FROM PRODUCT.INVENTORY'], + 'user_email': ['user1@example.com', 'user2@example.com'], + 'start_time': ['2025-01-01', '2025-01-02'] + }) + + with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f: + test_data.to_csv(f.name, index=False) + csv_file = f.name + + with tempfile.TemporaryDirectory() as output_dir: + try: + test_args = [ + 'cli.py', + '--platform', 'bigquery', + '--input-file', csv_file, + '--output', output_dir + ] + + with patch('sys.argv', test_args): + main() + + # Verify output was created + output_files = list(Path(output_dir).glob('recommendations_*.md')) + assert len(output_files) == 1 + + finally: + os.unlink(csv_file) + + def test_cli_with_watsonxdata(self): + """Test CLI with watsonx.data platform""" + # Create temporary CSV file + test_data = pd.DataFrame({ + 'query': ['SELECT * FROM NETWORK.SUBSCRIBERS', 'SELECT * FROM NETWORK.USAGE'], + 'user': ['user1', 'user2'], + 'created': ['2025-01-01', '2025-01-02'] + }) + + with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f: + test_data.to_csv(f.name, index=False) + csv_file = f.name + + with tempfile.TemporaryDirectory() as output_dir: + try: + test_args = [ + 'cli.py', + '--platform', 'watsonxdata', + '--input-file', csv_file, + '--output', output_dir + ] + + with patch('sys.argv', test_args): + main() + + # Verify output was created + output_files = list(Path(output_dir).glob('recommendations_*.md')) + assert len(output_files) == 1 + + finally: + os.unlink(csv_file) + + def test_cli_with_min_score_threshold(self): + """Test CLI with minimum score threshold""" + # Create temporary CSV file with multiple queries + test_data = pd.DataFrame({ + 'query_text': [ + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.ORDERS' + ], + 'user_name': ['user1', 'user2', 'user3', 'user1'], + 'start_time': ['2025-01-01', '2025-01-02', '2025-01-03', '2025-01-04'] + }) + + with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f: + test_data.to_csv(f.name, index=False) + csv_file = f.name + + with tempfile.TemporaryDirectory() as output_dir: + try: + test_args = [ + 'cli.py', + '--platform', 'snowflake', + '--input-file', csv_file, + '--output', output_dir, + '--min-score', '50.0' + ] + + with patch('sys.argv', test_args): + main() + + # Verify output was created + output_files = list(Path(output_dir).glob('recommendations_*.md')) + assert len(output_files) == 1 + + finally: + os.unlink(csv_file) + + def test_cli_invalid_file_type(self): + """Test CLI with invalid file type""" + with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False) as f: + f.write("invalid file") + txt_file = f.name + + try: + test_args = [ + 'cli.py', + '--platform', 'snowflake', + '--input-file', txt_file, + '--output', 'output' + ] + + with patch('sys.argv', test_args): + with pytest.raises(ValueError, match="Invalid --input-file type"): + main() + finally: + os.unlink(txt_file) + + def test_cli_creates_output_directory(self): + """Test that CLI creates output directory if it doesn't exist""" + # Create temporary CSV file + test_data = pd.DataFrame({ + 'query_text': ['SELECT * FROM SALES.CUSTOMERS'], + 'user_name': ['user1'], + 'start_time': ['2025-01-01'] + }) + + with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f: + test_data.to_csv(f.name, index=False) + csv_file = f.name + + with tempfile.TemporaryDirectory() as temp_dir: + output_dir = Path(temp_dir) / 'nested' / 'output' / 'dir' + + try: + test_args = [ + 'cli.py', + '--platform', 'snowflake', + '--input-file', csv_file, + '--output', str(output_dir) + ] + + with patch('sys.argv', test_args): + main() + + # Verify directory was created + assert output_dir.exists() + assert output_dir.is_dir() + + finally: + os.unlink(csv_file) + + def test_cli_markdown_output_format(self): + """Test CLI with explicit markdown output format""" + test_data = pd.DataFrame({ + 'query_text': ['SELECT * FROM SALES.CUSTOMERS'], + 'user_name': ['user1'], + 'start_time': ['2025-01-01'] + }) + + with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f: + test_data.to_csv(f.name, index=False) + csv_file = f.name + + with tempfile.TemporaryDirectory() as output_dir: + try: + test_args = [ + 'cli.py', + '--platform', 'snowflake', + '--input-file', csv_file, + '--output', output_dir, + '--output-format', 'markdown' + ] + + with patch('sys.argv', test_args): + main() + + # Verify markdown file was created + output_files = list(Path(output_dir).glob('recommendations_*.md')) + assert len(output_files) == 1 + + finally: + os.unlink(csv_file) + + +# Made with Bob \ No newline at end of file diff --git a/tests/src/data_product_recommender/test_data_product_recommender_conftest.py b/tests/src/data_product_recommender/test_data_product_recommender_conftest.py new file mode 100644 index 0000000..2d662bf --- /dev/null +++ b/tests/src/data_product_recommender/test_data_product_recommender_conftest.py @@ -0,0 +1,144 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Pytest configuration and shared fixtures +""" + +import pytest +import pandas as pd +import json +import tempfile +import os + + +@pytest.fixture +def sample_query_logs(): + """Sample query logs for testing""" + return pd.DataFrame({ + 'query_text': [ + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.CUSTOMERS WHERE active = true', + 'SELECT * FROM SALES.ORDERS', + 'SELECT * FROM SALES.CUSTOMERS c JOIN SALES.ORDERS o ON c.id = o.customer_id', + 'SELECT * FROM PRODUCT.CATALOG' + ], + 'user': ['user1', 'user2', 'user1', 'user3', 'user2'], + 'start_time': [ + '2025-01-01 10:00:00', + '2025-01-01 11:00:00', + '2025-01-01 12:00:00', + '2025-01-01 13:00:00', + '2025-01-01 14:00:00' + ] + }) + + +@pytest.fixture +def sample_query_logs_with_tables(): + """Sample query logs with tables already extracted""" + return pd.DataFrame({ + 'query_text': [ + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.CUSTOMERS WHERE active = true', + 'SELECT * FROM SALES.ORDERS', + 'SELECT * FROM SALES.CUSTOMERS c JOIN SALES.ORDERS o ON c.id = o.customer_id', + 'SELECT * FROM PRODUCT.CATALOG' + ], + 'user': ['user1', 'user2', 'user1', 'user3', 'user2'], + 'start_time': [ + '2025-01-01 10:00:00', + '2025-01-01 11:00:00', + '2025-01-01 12:00:00', + '2025-01-01 13:00:00', + '2025-01-01 14:00:00' + ], + 'tables': [ + ['SALES.CUSTOMERS'], + ['SALES.CUSTOMERS'], + ['SALES.ORDERS'], + ['SALES.CUSTOMERS', 'SALES.ORDERS'], + ['PRODUCT.CATALOG'] + ] + }) + + +@pytest.fixture +def sample_table_metrics(): + """Sample table metrics for testing""" + return pd.DataFrame({ + 'table': ['SALES.CUSTOMERS', 'SALES.ORDERS', 'PRODUCT.CATALOG'], + 'query_count': [10, 5, 2], + 'unique_users': [5, 3, 1], + 'related_tables': [ + ['SALES.ORDERS', 'PRODUCT.CATALOG'], + ['SALES.CUSTOMERS'], + [] + ], + 'related_table_count': [2, 1, 0] + }) + + +@pytest.fixture +def temp_json_file(sample_query_logs): + """Create a temporary JSON file with sample data""" + data = sample_query_logs.to_dict('records') + + # Rename columns to match raw format + for record in data: + record['user_name'] = record.pop('user') + + with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: + json.dump(data, f) + temp_file = f.name + + yield temp_file + + # Cleanup + if os.path.exists(temp_file): + os.unlink(temp_file) + + +@pytest.fixture +def temp_csv_file(sample_query_logs): + """Create a temporary CSV file with sample data""" + # Rename columns to match raw format + df = sample_query_logs.copy() + df = df.rename(columns={'user': 'user_name'}) + + with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f: + df.to_csv(f.name, index=False) + temp_file = f.name + + yield temp_file + + # Cleanup + if os.path.exists(temp_file): + os.unlink(temp_file) + + +@pytest.fixture +def temp_output_dir(): + """Create a temporary output directory""" + temp_dir = tempfile.mkdtemp() + yield temp_dir + + # Cleanup + import shutil + if os.path.exists(temp_dir): + shutil.rmtree(temp_dir) + +# Made with Bob diff --git a/tests/src/data_product_recommender/test_data_product_recommender_parsers.py b/tests/src/data_product_recommender/test_data_product_recommender_parsers.py new file mode 100644 index 0000000..03142cb --- /dev/null +++ b/tests/src/data_product_recommender/test_data_product_recommender_parsers.py @@ -0,0 +1,236 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Unit tests for query log parsers +""" + +import pytest +import pandas as pd +from wxdi.data_product_recommender.platforms import ( + SnowflakeQueryParser, + DatabricksQueryParser, + BigQueryQueryParser, + WatsonxDataQueryParser +) + + +class TestSnowflakeQueryParser: + """Tests for Snowflake query parser""" + + def setup_method(self): + """Setup test fixtures""" + self.parser = SnowflakeQueryParser() + + def test_normalize_columns(self): + """Test column normalization""" + df = pd.DataFrame({ + 'query_text': ['SELECT * FROM table1'], + 'user_name': ['user1'], + 'start_time': ['2025-01-01'] + }) + + result = self.parser.normalize_columns(df) + + assert 'query_text' in result.columns + assert 'user' in result.columns + assert 'start_time' in result.columns + + def test_extract_tables_simple(self): + """Test extracting tables from simple query""" + query = "SELECT * FROM SALES.CUSTOMERS" + tables = self.parser.extract_tables(query) + + assert len(tables) == 1 + assert 'SALES.CUSTOMERS' in tables + + def test_extract_tables_with_join(self): + """Test extracting tables from query with JOIN""" + query = "SELECT * FROM SALES.ORDERS o JOIN SALES.CUSTOMERS c ON o.customer_id = c.id" + tables = self.parser.extract_tables(query) + + assert len(tables) == 2 + assert 'SALES.ORDERS' in tables + assert 'SALES.CUSTOMERS' in tables + + def test_extract_tables_multiple_joins(self): + """Test extracting tables from query with multiple JOINs""" + query = """ + SELECT * FROM SALES.ORDERS o + LEFT JOIN SALES.CUSTOMERS c ON o.customer_id = c.id + INNER JOIN PRODUCT.CATALOG p ON o.product_id = p.id + """ + tables = self.parser.extract_tables(query) + + assert len(tables) == 3 + assert 'SALES.ORDERS' in tables + assert 'SALES.CUSTOMERS' in tables + assert 'PRODUCT.CATALOG' in tables + + def test_extract_tables_empty_query(self): + """Test extracting tables from empty query""" + tables = self.parser.extract_tables("") + assert len(tables) == 0 + + def test_extract_tables_none_query(self): + """Test extracting tables from None query""" + tables = self.parser.extract_tables(None) + assert len(tables) == 0 + + +class TestDatabricksQueryParser: + """Tests for Databricks query parser""" + + def setup_method(self): + """Setup test fixtures""" + self.parser = DatabricksQueryParser() + + def test_normalize_columns(self): + """Test column normalization""" + df = pd.DataFrame({ + 'statement_text': ['SELECT * FROM table1'], + 'executed_by': ['user1'], + 'start_time': ['2025-01-01'] + }) + + result = self.parser.normalize_columns(df) + + assert 'query_text' in result.columns + assert 'user' in result.columns + assert 'start_time' in result.columns + + def test_extract_tables_simple(self): + """Test extracting tables from simple query""" + query = "SELECT * FROM CLINICAL.PATIENTS" + tables = self.parser.extract_tables(query) + + assert len(tables) == 1 + assert 'CLINICAL.PATIENTS' in tables + + +class TestBigQueryQueryParser: + """Tests for BigQuery query parser""" + + def setup_method(self): + """Setup test fixtures""" + self.parser = BigQueryQueryParser() + + def test_normalize_columns(self): + """Test column normalization""" + df = pd.DataFrame({ + 'query': ['SELECT * FROM table1'], + 'user_email': ['user1@example.com'], + 'start_time': ['2025-01-01'] + }) + + result = self.parser.normalize_columns(df) + + assert 'query_text' in result.columns + assert 'user' in result.columns + assert 'start_time' in result.columns + + def test_extract_tables_with_backticks(self): + """Test extracting tables from BigQuery query with backticks""" + query = "SELECT * FROM `project-id.PRODUCT.CATALOG`" + tables = self.parser.extract_tables(query) + + assert len(tables) == 1 + assert 'PRODUCT.CATALOG' in tables + + def test_extract_tables_multiple_with_backticks(self): + """Test extracting multiple tables with project IDs""" + query = """ + SELECT * FROM `project-id.SALES.ORDERS` o + JOIN `project-id.CUSTOMER.PROFILES` c ON o.customer_id = c.id + """ + tables = self.parser.extract_tables(query) + + assert len(tables) == 2 + assert 'SALES.ORDERS' in tables + assert 'CUSTOMER.PROFILES' in tables + + +class TestWatsonxDataQueryParser: + """Tests for watsonx.data query parser""" + + def setup_method(self): + """Setup test fixtures""" + self.parser = WatsonxDataQueryParser() + + def test_normalize_columns(self): + """Test column normalization""" + df = pd.DataFrame({ + 'query': ['SELECT * FROM table1'], + 'user': ['user1'], + 'created': ['2025-01-01'] + }) + + result = self.parser.normalize_columns(df) + + assert 'query_text' in result.columns + assert 'user' in result.columns + assert 'start_time' in result.columns + + def test_extract_tables_simple(self): + """Test extracting tables from simple query""" + query = "SELECT * FROM NETWORK.SUBSCRIBERS" + tables = self.parser.extract_tables(query) + + assert len(tables) == 1 + assert 'NETWORK.SUBSCRIBERS' in tables + + +class TestParserEdgeCases: + """Test edge cases across all parsers""" + + @pytest.mark.parametrize("parser_class", [ + SnowflakeQueryParser, + DatabricksQueryParser, + BigQueryQueryParser, + WatsonxDataQueryParser + ]) + def test_extract_tables_with_subquery(self, parser_class): + """Test extracting tables from query with subquery""" + parser = parser_class() + query = """ + SELECT * FROM SALES.ORDERS + WHERE customer_id IN (SELECT id FROM SALES.CUSTOMERS WHERE active = true) + """ + tables = parser.extract_tables(query) + + # Should extract both tables + assert len(tables) >= 2 + + @pytest.mark.parametrize("parser_class", [ + SnowflakeQueryParser, + DatabricksQueryParser, + BigQueryQueryParser, + WatsonxDataQueryParser + ]) + def test_extract_tables_case_insensitive(self, parser_class): + """Test that table extraction is case-insensitive""" + parser = parser_class() + query_upper = "SELECT * FROM SALES.ORDERS" + query_lower = "select * from sales.orders" + + tables_upper = parser.extract_tables(query_upper) + tables_lower = parser.extract_tables(query_lower) + + # Both should extract the same table (normalized to uppercase) + assert len(tables_upper) == 1 + assert len(tables_lower) == 1 + +# Made with Bob diff --git a/tests/src/dph_services/__init__.py b/tests/src/dph_services/__init__.py new file mode 100644 index 0000000..db6be80 --- /dev/null +++ b/tests/src/dph_services/__init__.py @@ -0,0 +1,19 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""DPH Services test package""" + +# Made with Bob diff --git a/tests/src/dph_services/test_common.py b/tests/src/dph_services/test_common.py new file mode 100644 index 0000000..5a0e5f7 --- /dev/null +++ b/tests/src/dph_services/test_common.py @@ -0,0 +1,50 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Test methods in the common module +""" + +import unittest +from wxdi.dph_services import common + + +class TestCommon(unittest.TestCase): + """ + Test methods in the common module + """ + + def test_get_sdk_headers(self): + """ + Test the get_sdk_headers method + """ + headers = common.get_sdk_headers( + service_name='dph_services', service_version='V1', operation_id='operation1' + ) + self.assertIsNotNone(headers) + self.assertIsNotNone(headers.get('User-Agent')) + self.assertIn('data-product-python-sdk', headers.get('User-Agent')) + + def test_get_system_info(self): + """ + Test the get_system_info method + """ + system_info = common.get_system_info() + self.assertIsNotNone(system_info) + self.assertIn('lang=', system_info) + self.assertIn('arch=', system_info) + self.assertIn('os=', system_info) + self.assertIn('python.version=', system_info) diff --git a/tests/src/dph_services/test_dph_v1.py b/tests/src/dph_services/test_dph_v1.py new file mode 100644 index 0000000..d465a30 --- /dev/null +++ b/tests/src/dph_services/test_dph_v1.py @@ -0,0 +1,14205 @@ +# -*- coding: utf-8 -*- +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Unit Tests for DphV1 +""" + +from datetime import datetime, timezone +from ibm_cloud_sdk_core.authenticators.no_auth_authenticator import NoAuthAuthenticator +from ibm_cloud_sdk_core.utils import datetime_to_string, string_to_datetime +import inspect +import io +import json +import os +import pytest +import re +import requests +import responses +import tempfile +import urllib +from wxdi.dph_services.dph_v1 import * + + +_service = DphV1( + authenticator=NoAuthAuthenticator() +) + +_base_url = 'https://fake' +_service.set_service_url(_base_url) + + +def preprocess_url(operation_path: str): + """ + Returns the request url associated with the specified operation path. + This will be base_url concatenated with a quoted version of operation_path. + The returned request URL is used to register the mock response so it needs + to match the request URL that is formed by the requests library. + """ + + # Form the request URL from the base URL and operation path. + request_url = _base_url + operation_path + + # If the request url does NOT end with a /, then just return it as-is. + # Otherwise, return a regular expression that matches one or more trailing /. + if not request_url.endswith('/'): + return request_url + return re.compile(request_url.rstrip('/') + '/+') + + +############################################################################## +# Start of Service: Configuration +############################################################################## +# region + + +class TestNewInstance: + """ + Test Class for new_instance + """ + + def test_new_instance(self): + """ + new_instance() + """ + os.environ['TEST_SERVICE_AUTH_TYPE'] = 'noAuth' + + service = DphV1.new_instance( + service_name='TEST_SERVICE', + ) + + assert service is not None + assert isinstance(service, DphV1) + + def test_new_instance_without_authenticator(self): + """ + new_instance_without_authenticator() + """ + with pytest.raises(ValueError, match='authenticator must be provided'): + service = DphV1.new_instance( + service_name='TEST_SERVICE_NOT_FOUND', + ) + + +class TestGetInitializeStatus: + """ + Test Class for get_initialize_status + """ + + @responses.activate + def test_get_initialize_status_all_params(self): + """ + get_initialize_status() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/configuration/initialize/status') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "href": "https://api.example.com/configuration/initialize/status?catalog_id=d29c42eb-7100-4b7a-8257-c196dbcca1cd", "status": "not_started", "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "last_started_at": "2023-08-21T15:24:06.021Z", "last_finished_at": "2023-08-21T20:24:34.450Z", "initialized_options": [{"name": "name", "version": 1}], "workflows": {"data_access": {"definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}, "request_new_product": {"definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + container_id = 'testString' + + # Invoke method + response = _service.get_initialize_status( + container_id=container_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + + def test_get_initialize_status_all_params_with_retries(self): + # Enable retries and run test_get_initialize_status_all_params. + _service.enable_retries() + self.test_get_initialize_status_all_params() + + # Disable retries and run test_get_initialize_status_all_params. + _service.disable_retries() + self.test_get_initialize_status_all_params() + + @responses.activate + def test_get_initialize_status_required_params(self): + """ + test_get_initialize_status_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/configuration/initialize/status') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "href": "https://api.example.com/configuration/initialize/status?catalog_id=d29c42eb-7100-4b7a-8257-c196dbcca1cd", "status": "not_started", "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "last_started_at": "2023-08-21T15:24:06.021Z", "last_finished_at": "2023-08-21T20:24:34.450Z", "initialized_options": [{"name": "name", "version": 1}], "workflows": {"data_access": {"definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}, "request_new_product": {"definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Invoke method + response = _service.get_initialize_status() + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_initialize_status_required_params_with_retries(self): + # Enable retries and run test_get_initialize_status_required_params. + _service.enable_retries() + self.test_get_initialize_status_required_params() + + # Disable retries and run test_get_initialize_status_required_params. + _service.disable_retries() + self.test_get_initialize_status_required_params() + + +class TestGetServiceIdCredentials: + """ + Test Class for get_service_id_credentials + """ + + @responses.activate + def test_get_service_id_credentials_all_params(self): + """ + get_service_id_credentials() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/configuration/credentials') + mock_response = '{"name": "data-product-admin-service-id-API-key", "created_at": "2019-01-01T12:00:00.000Z"}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Invoke method + response = _service.get_service_id_credentials() + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_service_id_credentials_all_params_with_retries(self): + # Enable retries and run test_get_service_id_credentials_all_params. + _service.enable_retries() + self.test_get_service_id_credentials_all_params() + + # Disable retries and run test_get_service_id_credentials_all_params. + _service.disable_retries() + self.test_get_service_id_credentials_all_params() + + +class TestInitialize: + """ + Test Class for initialize + """ + + @responses.activate + def test_initialize_all_params(self): + """ + initialize() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/configuration/initialize') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "href": "https://api.example.com/configuration/initialize/status?catalog_id=d29c42eb-7100-4b7a-8257-c196dbcca1cd", "status": "not_started", "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "last_started_at": "2023-08-21T15:24:06.021Z", "last_finished_at": "2023-08-21T20:24:34.450Z", "initialized_options": [{"name": "name", "version": 1}], "workflows": {"data_access": {"definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}, "request_new_product": {"definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=202, + ) + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Set up parameter values + container = container_reference_model + include = ['delivery_methods', 'domains_multi_industry', 'data_product_samples', 'workflows', 'project', 'catalog_configurations'] + + # Invoke method + response = _service.initialize( + container=container, + include=include, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 202 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['container'] == container_reference_model + assert req_body['include'] == ['delivery_methods', 'domains_multi_industry', 'data_product_samples', 'workflows', 'project', 'catalog_configurations'] + + def test_initialize_all_params_with_retries(self): + # Enable retries and run test_initialize_all_params. + _service.enable_retries() + self.test_initialize_all_params() + + # Disable retries and run test_initialize_all_params. + _service.disable_retries() + self.test_initialize_all_params() + + +class TestManageApiKeys: + """ + Test Class for manage_api_keys + """ + + @responses.activate + def test_manage_api_keys_all_params(self): + """ + manage_api_keys() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/configuration/rotate_credentials') + responses.add( + responses.POST, + url, + status=204, + ) + + # Invoke method + response = _service.manage_api_keys() + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 204 + + def test_manage_api_keys_all_params_with_retries(self): + # Enable retries and run test_manage_api_keys_all_params. + _service.enable_retries() + self.test_manage_api_keys_all_params() + + # Disable retries and run test_manage_api_keys_all_params. + _service.disable_retries() + self.test_manage_api_keys_all_params() + + +# endregion +############################################################################## +# End of Service: Configuration +############################################################################## + +############################################################################## +# Start of Service: DataAssetVisualization +############################################################################## +# region + + +class TestNewInstance: + """ + Test Class for new_instance + """ + + def test_new_instance(self): + """ + new_instance() + """ + os.environ['TEST_SERVICE_AUTH_TYPE'] = 'noAuth' + + service = DphV1.new_instance( + service_name='TEST_SERVICE', + ) + + assert service is not None + assert isinstance(service, DphV1) + + def test_new_instance_without_authenticator(self): + """ + new_instance_without_authenticator() + """ + with pytest.raises(ValueError, match='authenticator must be provided'): + service = DphV1.new_instance( + service_name='TEST_SERVICE_NOT_FOUND', + ) + + +class TestCreateDataAssetVisualization: + """ + Test Class for create_data_asset_visualization + """ + + @responses.activate + def test_create_data_asset_visualization_all_params(self): + """ + create_data_asset_visualization() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_asset/visualization') + mock_response = '{"results": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a Visualization model + visualization_model = {} + visualization_model['id'] = 'testString' + visualization_model['name'] = 'testString' + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = '2be8f727-c5d2-4cb0-9216-f9888f428048' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = 'caeee3f3-756e-47d5-846d-da4600809e22' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ErrorMessage model + error_message_model = {} + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + # Construct a dict representation of a DataAssetRelationship model + data_asset_relationship_model = {} + data_asset_relationship_model['visualization'] = visualization_model + data_asset_relationship_model['asset'] = asset_reference_model + data_asset_relationship_model['related_asset'] = asset_reference_model + data_asset_relationship_model['error'] = error_message_model + + # Set up parameter values + assets = [data_asset_relationship_model] + + # Invoke method + response = _service.create_data_asset_visualization( + assets=assets, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['assets'] == [data_asset_relationship_model] + + def test_create_data_asset_visualization_all_params_with_retries(self): + # Enable retries and run test_create_data_asset_visualization_all_params. + _service.enable_retries() + self.test_create_data_asset_visualization_all_params() + + # Disable retries and run test_create_data_asset_visualization_all_params. + _service.disable_retries() + self.test_create_data_asset_visualization_all_params() + + +class TestReinitiateDataAssetVisualization: + """ + Test Class for reinitiate_data_asset_visualization + """ + + @responses.activate + def test_reinitiate_data_asset_visualization_all_params(self): + """ + reinitiate_data_asset_visualization() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_asset/visualization/reinitiate') + mock_response = '{"results": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a Visualization model + visualization_model = {} + visualization_model['id'] = 'testString' + visualization_model['name'] = 'testString' + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = '2be8f727-c5d2-4cb0-9216-f9888f428048' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = 'caeee3f3-756e-47d5-846d-da4600809e22' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ErrorMessage model + error_message_model = {} + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + # Construct a dict representation of a DataAssetRelationship model + data_asset_relationship_model = {} + data_asset_relationship_model['visualization'] = visualization_model + data_asset_relationship_model['asset'] = asset_reference_model + data_asset_relationship_model['related_asset'] = asset_reference_model + data_asset_relationship_model['error'] = error_message_model + + # Set up parameter values + assets = [data_asset_relationship_model] + + # Invoke method + response = _service.reinitiate_data_asset_visualization( + assets=assets, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['assets'] == [data_asset_relationship_model] + + def test_reinitiate_data_asset_visualization_all_params_with_retries(self): + # Enable retries and run test_reinitiate_data_asset_visualization_all_params. + _service.enable_retries() + self.test_reinitiate_data_asset_visualization_all_params() + + # Disable retries and run test_reinitiate_data_asset_visualization_all_params. + _service.disable_retries() + self.test_reinitiate_data_asset_visualization_all_params() + + +# endregion +############################################################################## +# End of Service: DataAssetVisualization +############################################################################## + +############################################################################## +# Start of Service: DataProducts +############################################################################## +# region + + +class TestNewInstance: + """ + Test Class for new_instance + """ + + def test_new_instance(self): + """ + new_instance() + """ + os.environ['TEST_SERVICE_AUTH_TYPE'] = 'noAuth' + + service = DphV1.new_instance( + service_name='TEST_SERVICE', + ) + + assert service is not None + assert isinstance(service, DphV1) + + def test_new_instance_without_authenticator(self): + """ + new_instance_without_authenticator() + """ + with pytest.raises(ValueError, match='authenticator must be provided'): + service = DphV1.new_instance( + service_name='TEST_SERVICE_NOT_FOUND', + ) + + +class TestListDataProducts: + """ + Test Class for list_data_products + """ + + @responses.activate + def test_list_data_products_all_params(self): + """ + list_data_products() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "data_products": [{"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "name": "name"}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + limit = 200 + start = 'testString' + + # Invoke method + response = _service.list_data_products( + limit=limit, + start=start, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'limit={}'.format(limit) in query_string + assert 'start={}'.format(start) in query_string + + def test_list_data_products_all_params_with_retries(self): + # Enable retries and run test_list_data_products_all_params. + _service.enable_retries() + self.test_list_data_products_all_params() + + # Disable retries and run test_list_data_products_all_params. + _service.disable_retries() + self.test_list_data_products_all_params() + + @responses.activate + def test_list_data_products_required_params(self): + """ + test_list_data_products_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "data_products": [{"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "name": "name"}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Invoke method + response = _service.list_data_products() + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_list_data_products_required_params_with_retries(self): + # Enable retries and run test_list_data_products_required_params. + _service.enable_retries() + self.test_list_data_products_required_params() + + # Disable retries and run test_list_data_products_required_params. + _service.disable_retries() + self.test_list_data_products_required_params() + + @responses.activate + def test_list_data_products_with_pager_get_next(self): + """ + test_list_data_products_with_pager_get_next() + """ + # Set up a two-page mock response + url = preprocess_url('/data_product_exchange/v1/data_products') + mock_response1 = '{"next":{"start":"1"},"total_count":2,"limit":1,"data_products":[{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"name":"name"}]}' + mock_response2 = '{"total_count":2,"limit":1,"data_products":[{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"name":"name"}]}' + responses.add( + responses.GET, + url, + body=mock_response1, + content_type='application/json', + status=200, + ) + responses.add( + responses.GET, + url, + body=mock_response2, + content_type='application/json', + status=200, + ) + + # Exercise the pager class for this operation + all_results = [] + pager = DataProductsPager( + client=_service, + limit=10, + ) + while pager.has_next(): + next_page = pager.get_next() + assert next_page is not None + all_results.extend(next_page) + assert len(all_results) == 2 + + @responses.activate + def test_list_data_products_with_pager_get_all(self): + """ + test_list_data_products_with_pager_get_all() + """ + # Set up a two-page mock response + url = preprocess_url('/data_product_exchange/v1/data_products') + mock_response1 = '{"next":{"start":"1"},"total_count":2,"limit":1,"data_products":[{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"name":"name"}]}' + mock_response2 = '{"total_count":2,"limit":1,"data_products":[{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"name":"name"}]}' + responses.add( + responses.GET, + url, + body=mock_response1, + content_type='application/json', + status=200, + ) + responses.add( + responses.GET, + url, + body=mock_response2, + content_type='application/json', + status=200, + ) + + # Exercise the pager class for this operation + pager = DataProductsPager( + client=_service, + limit=10, + ) + all_results = pager.get_all() + assert all_results is not None + assert len(all_results) == 2 + + +class TestCreateDataProduct: + """ + Test Class for create_data_product + """ + + @responses.activate + def test_create_data_product_all_params(self): + """ + create_data_product() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products') + mock_response = '{"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "name": "name", "latest_release": {"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}, "drafts": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a DataProductDraftVersionRelease model + data_product_draft_version_release_model = {} + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a dict representation of a DataProductIdentity model + data_product_identity_model = {} + data_product_identity_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_identity_model['release'] = data_product_draft_version_release_model + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a UseCase model + use_case_model = {} + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'owner' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractTerms model + contract_terms_model = {} + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + # Construct a dict representation of a AssetPartReference model + asset_part_reference_model = {} + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + # Construct a dict representation of a EngineDetailsModel model + engine_details_model_model = {} + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + # Construct a dict representation of a ProducerInputModel model + producer_input_model_model = {} + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + # Construct a dict representation of a DeliveryMethodPropertiesModel model + delivery_method_properties_model_model = {} + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + # Construct a dict representation of a DeliveryMethod model + delivery_method_model = {} + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + # Construct a dict representation of a DataProductPart model + data_product_part_model = {} + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + # Construct a dict representation of a DataProductCustomWorkflowDefinition model + data_product_custom_workflow_definition_model = {} + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a dict representation of a DataProductOrderAccessRequest model + data_product_order_access_request_model = {} + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + # Construct a dict representation of a DataProductWorkflows model + data_product_workflows_model = {} + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + # Construct a dict representation of a AssetListAccessControl model + asset_list_access_control_model = {} + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + # Construct a dict representation of a ContainerIdentity model + container_identity_model = {} + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a dict representation of a AssetPrototype model + asset_prototype_model = {} + asset_prototype_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_prototype_model['container'] = container_identity_model + + # Construct a dict representation of a DataProductDraftPrototype model + data_product_draft_prototype_model = {} + data_product_draft_prototype_model['version'] = '1.0.0' + data_product_draft_prototype_model['state'] = 'draft' + data_product_draft_prototype_model['data_product'] = data_product_identity_model + data_product_draft_prototype_model['name'] = 'My New Data Product' + data_product_draft_prototype_model['description'] = 'This is a description of My Data Product.' + data_product_draft_prototype_model['tags'] = ['testString'] + data_product_draft_prototype_model['use_cases'] = [use_case_model] + data_product_draft_prototype_model['types'] = ['data'] + data_product_draft_prototype_model['contract_terms'] = [contract_terms_model] + data_product_draft_prototype_model['domain'] = domain_model + data_product_draft_prototype_model['parts_out'] = [data_product_part_model] + data_product_draft_prototype_model['workflows'] = data_product_workflows_model + data_product_draft_prototype_model['dataview_enabled'] = True + data_product_draft_prototype_model['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_draft_prototype_model['access_control'] = asset_list_access_control_model + data_product_draft_prototype_model['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_draft_prototype_model['sub_container'] = container_identity_model + data_product_draft_prototype_model['is_restricted'] = True + data_product_draft_prototype_model['asset'] = asset_prototype_model + + # Set up parameter values + drafts = [data_product_draft_prototype_model] + limit = 200 + start = 'testString' + + # Invoke method + response = _service.create_data_product( + drafts, + limit=limit, + start=start, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'limit={}'.format(limit) in query_string + assert 'start={}'.format(start) in query_string + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['drafts'] == [data_product_draft_prototype_model] + + def test_create_data_product_all_params_with_retries(self): + # Enable retries and run test_create_data_product_all_params. + _service.enable_retries() + self.test_create_data_product_all_params() + + # Disable retries and run test_create_data_product_all_params. + _service.disable_retries() + self.test_create_data_product_all_params() + + @responses.activate + def test_create_data_product_required_params(self): + """ + test_create_data_product_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products') + mock_response = '{"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "name": "name", "latest_release": {"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}, "drafts": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a DataProductDraftVersionRelease model + data_product_draft_version_release_model = {} + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a dict representation of a DataProductIdentity model + data_product_identity_model = {} + data_product_identity_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_identity_model['release'] = data_product_draft_version_release_model + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a UseCase model + use_case_model = {} + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'owner' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractTerms model + contract_terms_model = {} + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + # Construct a dict representation of a AssetPartReference model + asset_part_reference_model = {} + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + # Construct a dict representation of a EngineDetailsModel model + engine_details_model_model = {} + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + # Construct a dict representation of a ProducerInputModel model + producer_input_model_model = {} + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + # Construct a dict representation of a DeliveryMethodPropertiesModel model + delivery_method_properties_model_model = {} + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + # Construct a dict representation of a DeliveryMethod model + delivery_method_model = {} + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + # Construct a dict representation of a DataProductPart model + data_product_part_model = {} + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + # Construct a dict representation of a DataProductCustomWorkflowDefinition model + data_product_custom_workflow_definition_model = {} + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a dict representation of a DataProductOrderAccessRequest model + data_product_order_access_request_model = {} + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + # Construct a dict representation of a DataProductWorkflows model + data_product_workflows_model = {} + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + # Construct a dict representation of a AssetListAccessControl model + asset_list_access_control_model = {} + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + # Construct a dict representation of a ContainerIdentity model + container_identity_model = {} + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a dict representation of a AssetPrototype model + asset_prototype_model = {} + asset_prototype_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_prototype_model['container'] = container_identity_model + + # Construct a dict representation of a DataProductDraftPrototype model + data_product_draft_prototype_model = {} + data_product_draft_prototype_model['version'] = '1.0.0' + data_product_draft_prototype_model['state'] = 'draft' + data_product_draft_prototype_model['data_product'] = data_product_identity_model + data_product_draft_prototype_model['name'] = 'My New Data Product' + data_product_draft_prototype_model['description'] = 'This is a description of My Data Product.' + data_product_draft_prototype_model['tags'] = ['testString'] + data_product_draft_prototype_model['use_cases'] = [use_case_model] + data_product_draft_prototype_model['types'] = ['data'] + data_product_draft_prototype_model['contract_terms'] = [contract_terms_model] + data_product_draft_prototype_model['domain'] = domain_model + data_product_draft_prototype_model['parts_out'] = [data_product_part_model] + data_product_draft_prototype_model['workflows'] = data_product_workflows_model + data_product_draft_prototype_model['dataview_enabled'] = True + data_product_draft_prototype_model['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_draft_prototype_model['access_control'] = asset_list_access_control_model + data_product_draft_prototype_model['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_draft_prototype_model['sub_container'] = container_identity_model + data_product_draft_prototype_model['is_restricted'] = True + data_product_draft_prototype_model['asset'] = asset_prototype_model + + # Set up parameter values + drafts = [data_product_draft_prototype_model] + + # Invoke method + response = _service.create_data_product( + drafts, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['drafts'] == [data_product_draft_prototype_model] + + def test_create_data_product_required_params_with_retries(self): + # Enable retries and run test_create_data_product_required_params. + _service.enable_retries() + self.test_create_data_product_required_params() + + # Disable retries and run test_create_data_product_required_params. + _service.disable_retries() + self.test_create_data_product_required_params() + + @responses.activate + def test_create_data_product_value_error(self): + """ + test_create_data_product_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products') + mock_response = '{"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "name": "name", "latest_release": {"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}, "drafts": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a DataProductDraftVersionRelease model + data_product_draft_version_release_model = {} + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a dict representation of a DataProductIdentity model + data_product_identity_model = {} + data_product_identity_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_identity_model['release'] = data_product_draft_version_release_model + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a UseCase model + use_case_model = {} + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'owner' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractTerms model + contract_terms_model = {} + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + # Construct a dict representation of a AssetPartReference model + asset_part_reference_model = {} + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + # Construct a dict representation of a EngineDetailsModel model + engine_details_model_model = {} + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + # Construct a dict representation of a ProducerInputModel model + producer_input_model_model = {} + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + # Construct a dict representation of a DeliveryMethodPropertiesModel model + delivery_method_properties_model_model = {} + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + # Construct a dict representation of a DeliveryMethod model + delivery_method_model = {} + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + # Construct a dict representation of a DataProductPart model + data_product_part_model = {} + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + # Construct a dict representation of a DataProductCustomWorkflowDefinition model + data_product_custom_workflow_definition_model = {} + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a dict representation of a DataProductOrderAccessRequest model + data_product_order_access_request_model = {} + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + # Construct a dict representation of a DataProductWorkflows model + data_product_workflows_model = {} + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + # Construct a dict representation of a AssetListAccessControl model + asset_list_access_control_model = {} + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + # Construct a dict representation of a ContainerIdentity model + container_identity_model = {} + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a dict representation of a AssetPrototype model + asset_prototype_model = {} + asset_prototype_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_prototype_model['container'] = container_identity_model + + # Construct a dict representation of a DataProductDraftPrototype model + data_product_draft_prototype_model = {} + data_product_draft_prototype_model['version'] = '1.0.0' + data_product_draft_prototype_model['state'] = 'draft' + data_product_draft_prototype_model['data_product'] = data_product_identity_model + data_product_draft_prototype_model['name'] = 'My New Data Product' + data_product_draft_prototype_model['description'] = 'This is a description of My Data Product.' + data_product_draft_prototype_model['tags'] = ['testString'] + data_product_draft_prototype_model['use_cases'] = [use_case_model] + data_product_draft_prototype_model['types'] = ['data'] + data_product_draft_prototype_model['contract_terms'] = [contract_terms_model] + data_product_draft_prototype_model['domain'] = domain_model + data_product_draft_prototype_model['parts_out'] = [data_product_part_model] + data_product_draft_prototype_model['workflows'] = data_product_workflows_model + data_product_draft_prototype_model['dataview_enabled'] = True + data_product_draft_prototype_model['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_draft_prototype_model['access_control'] = asset_list_access_control_model + data_product_draft_prototype_model['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_draft_prototype_model['sub_container'] = container_identity_model + data_product_draft_prototype_model['is_restricted'] = True + data_product_draft_prototype_model['asset'] = asset_prototype_model + + # Set up parameter values + drafts = [data_product_draft_prototype_model] + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "drafts": drafts, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.create_data_product(**req_copy) + + def test_create_data_product_value_error_with_retries(self): + # Enable retries and run test_create_data_product_value_error. + _service.enable_retries() + self.test_create_data_product_value_error() + + # Disable retries and run test_create_data_product_value_error. + _service.disable_retries() + self.test_create_data_product_value_error() + + +class TestGetDataProduct: + """ + Test Class for get_data_product + """ + + @responses.activate + def test_get_data_product_all_params(self): + """ + get_data_product() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString') + mock_response = '{"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "name": "name", "latest_release": {"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}, "drafts": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + + # Invoke method + response = _service.get_data_product( + data_product_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_data_product_all_params_with_retries(self): + # Enable retries and run test_get_data_product_all_params. + _service.enable_retries() + self.test_get_data_product_all_params() + + # Disable retries and run test_get_data_product_all_params. + _service.disable_retries() + self.test_get_data_product_all_params() + + @responses.activate + def test_get_data_product_value_error(self): + """ + test_get_data_product_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString') + mock_response = '{"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "name": "name", "latest_release": {"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}, "drafts": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_data_product(**req_copy) + + def test_get_data_product_value_error_with_retries(self): + # Enable retries and run test_get_data_product_value_error. + _service.enable_retries() + self.test_get_data_product_value_error() + + # Disable retries and run test_get_data_product_value_error. + _service.disable_retries() + self.test_get_data_product_value_error() + + +# endregion +############################################################################## +# End of Service: DataProducts +############################################################################## + +############################################################################## +# Start of Service: DataProductDrafts +############################################################################## +# region + + +class TestNewInstance: + """ + Test Class for new_instance + """ + + def test_new_instance(self): + """ + new_instance() + """ + os.environ['TEST_SERVICE_AUTH_TYPE'] = 'noAuth' + + service = DphV1.new_instance( + service_name='TEST_SERVICE', + ) + + assert service is not None + assert isinstance(service, DphV1) + + def test_new_instance_without_authenticator(self): + """ + new_instance_without_authenticator() + """ + with pytest.raises(ValueError, match='authenticator must be provided'): + service = DphV1.new_instance( + service_name='TEST_SERVICE_NOT_FOUND', + ) + + +class TestCompleteDraftContractTermsDocument: + """ + Test Class for complete_draft_contract_terms_document + """ + + @responses.activate + def test_complete_draft_contract_terms_document_all_params(self): + """ + complete_draft_contract_terms_document() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents/testString/complete') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + + # Invoke method + response = _service.complete_draft_contract_terms_document( + data_product_id, + draft_id, + contract_terms_id, + document_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_complete_draft_contract_terms_document_all_params_with_retries(self): + # Enable retries and run test_complete_draft_contract_terms_document_all_params. + _service.enable_retries() + self.test_complete_draft_contract_terms_document_all_params() + + # Disable retries and run test_complete_draft_contract_terms_document_all_params. + _service.disable_retries() + self.test_complete_draft_contract_terms_document_all_params() + + @responses.activate + def test_complete_draft_contract_terms_document_value_error(self): + """ + test_complete_draft_contract_terms_document_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents/testString/complete') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "contract_terms_id": contract_terms_id, + "document_id": document_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.complete_draft_contract_terms_document(**req_copy) + + def test_complete_draft_contract_terms_document_value_error_with_retries(self): + # Enable retries and run test_complete_draft_contract_terms_document_value_error. + _service.enable_retries() + self.test_complete_draft_contract_terms_document_value_error() + + # Disable retries and run test_complete_draft_contract_terms_document_value_error. + _service.disable_retries() + self.test_complete_draft_contract_terms_document_value_error() + + +class TestListDataProductDrafts: + """ + Test Class for list_data_product_drafts + """ + + @responses.activate + def test_list_data_product_drafts_all_params(self): + """ + list_data_product_drafts() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "drafts": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + asset_container_id = 'testString' + version = 'testString' + limit = 200 + start = 'testString' + + # Invoke method + response = _service.list_data_product_drafts( + data_product_id, + asset_container_id=asset_container_id, + version=version, + limit=limit, + start=start, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'asset.container.id={}'.format(asset_container_id) in query_string + assert 'version={}'.format(version) in query_string + assert 'limit={}'.format(limit) in query_string + assert 'start={}'.format(start) in query_string + + def test_list_data_product_drafts_all_params_with_retries(self): + # Enable retries and run test_list_data_product_drafts_all_params. + _service.enable_retries() + self.test_list_data_product_drafts_all_params() + + # Disable retries and run test_list_data_product_drafts_all_params. + _service.disable_retries() + self.test_list_data_product_drafts_all_params() + + @responses.activate + def test_list_data_product_drafts_required_params(self): + """ + test_list_data_product_drafts_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "drafts": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + + # Invoke method + response = _service.list_data_product_drafts( + data_product_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_list_data_product_drafts_required_params_with_retries(self): + # Enable retries and run test_list_data_product_drafts_required_params. + _service.enable_retries() + self.test_list_data_product_drafts_required_params() + + # Disable retries and run test_list_data_product_drafts_required_params. + _service.disable_retries() + self.test_list_data_product_drafts_required_params() + + @responses.activate + def test_list_data_product_drafts_value_error(self): + """ + test_list_data_product_drafts_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "drafts": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.list_data_product_drafts(**req_copy) + + def test_list_data_product_drafts_value_error_with_retries(self): + # Enable retries and run test_list_data_product_drafts_value_error. + _service.enable_retries() + self.test_list_data_product_drafts_value_error() + + # Disable retries and run test_list_data_product_drafts_value_error. + _service.disable_retries() + self.test_list_data_product_drafts_value_error() + + @responses.activate + def test_list_data_product_drafts_with_pager_get_next(self): + """ + test_list_data_product_drafts_with_pager_get_next() + """ + # Set up a two-page mock response + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts') + mock_response1 = '{"next":{"start":"1"},"total_count":2,"limit":1,"drafts":[{"version":"1.0.0","state":"draft","data_product":{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"name":"My Data Product","description":"This is a description of My Data Product.","tags":["tags"],"use_cases":[{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}],"types":["data"],"contract_terms":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"id":"id","documents":[{"url":"url","type":"terms_and_conditions","name":"name","id":"2b0bf220-079c-11ee-be56-0242ac120002","attachment":{"id":"id"},"upload_url":"upload_url"}],"error_msg":"error_msg","overview":{"api_version":"v3.0.1","kind":"DataContract","name":"Sample Data Contract","version":"0.0.0","domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"more_info":"List of links to sources that provide more details on the data contract."},"description":{"purpose":"Used for customer behavior analysis.","limitations":"Data cannot be used for marketing.","usage":"Data should be used only for analytics.","more_info":[{"type":"privacy-statement","url":"https://moreinfo.example.com"}],"custom_properties":"{\\"property1\\":\\"value1\\"}"},"organization":[{"user_id":"IBMid-691000IN4G","role":"owner"}],"roles":[{"role":"owner"}],"price":{"amount":"100.0","currency":"USD","unit":"megabyte"},"sla":[{"default_element":"Standard SLA Policy","properties":[{"property":"Uptime Guarantee","value":"99.9"}]}],"support_and_communication":[{"channel":"Email Support","url":"https://support.example.com"}],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}],"contract_test":{"status":"pass","last_tested_time":"last_tested_time","message":"message"},"servers":[{"server":"server","asset":{"id":"id","name":"name"},"connection_id":"connection_id","type":"type","description":"description","environment":"environment","account":"account","catalog":"catalog","database":"database","dataset":"dataset","delimiter":"delimiter","endpoint_url":"endpoint_url","format":"format","host":"host","location":"location","path":"path","port":"port","project":"project","region":"region","region_name":"region_name","schema":"schema","service_name":"service_name","staging_dir":"staging_dir","stream":"stream","warehouse":"warehouse","roles":["roles"],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}]}],"schema":[{"asset_id":"2b0bf220-079c-11ee-be56-0242ac120002","connection_id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","description":"description","connection_path":"connection_path","physical_type":"physical_type","properties":[{"name":"name","type":{"type":"type","length":"length","scale":"scale","nullable":"nullable","signed":"signed","native_type":"native_type"},"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}],"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}]}],"domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"parts_out":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"type":"data_asset"},"delivery_methods":[{"id":"09cf5fcc-cb9d-4995-a8e4-16517b25229f","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"getproperties":{"producer_input":{"engine_details":{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]},"engines":[{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]}]}}}]}],"workflows":{"order_access_request":{"task_assignee_users":["task_assignee_users"],"pre_approved_users":["pre_approved_users"],"custom_workflow_definition":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"}}},"dataview_enabled":true,"comments":"Comments by a producer that are provided either at the time of data product version creation or retiring","access_control":{"owner":"IBMid-696000KYV9"},"last_updated_at":"2019-01-01T12:00:00.000Z","sub_container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd"},"is_restricted":false,"id":"2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd","asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}}]}' + mock_response2 = '{"total_count":2,"limit":1,"drafts":[{"version":"1.0.0","state":"draft","data_product":{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"name":"My Data Product","description":"This is a description of My Data Product.","tags":["tags"],"use_cases":[{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}],"types":["data"],"contract_terms":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"id":"id","documents":[{"url":"url","type":"terms_and_conditions","name":"name","id":"2b0bf220-079c-11ee-be56-0242ac120002","attachment":{"id":"id"},"upload_url":"upload_url"}],"error_msg":"error_msg","overview":{"api_version":"v3.0.1","kind":"DataContract","name":"Sample Data Contract","version":"0.0.0","domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"more_info":"List of links to sources that provide more details on the data contract."},"description":{"purpose":"Used for customer behavior analysis.","limitations":"Data cannot be used for marketing.","usage":"Data should be used only for analytics.","more_info":[{"type":"privacy-statement","url":"https://moreinfo.example.com"}],"custom_properties":"{\\"property1\\":\\"value1\\"}"},"organization":[{"user_id":"IBMid-691000IN4G","role":"owner"}],"roles":[{"role":"owner"}],"price":{"amount":"100.0","currency":"USD","unit":"megabyte"},"sla":[{"default_element":"Standard SLA Policy","properties":[{"property":"Uptime Guarantee","value":"99.9"}]}],"support_and_communication":[{"channel":"Email Support","url":"https://support.example.com"}],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}],"contract_test":{"status":"pass","last_tested_time":"last_tested_time","message":"message"},"servers":[{"server":"server","asset":{"id":"id","name":"name"},"connection_id":"connection_id","type":"type","description":"description","environment":"environment","account":"account","catalog":"catalog","database":"database","dataset":"dataset","delimiter":"delimiter","endpoint_url":"endpoint_url","format":"format","host":"host","location":"location","path":"path","port":"port","project":"project","region":"region","region_name":"region_name","schema":"schema","service_name":"service_name","staging_dir":"staging_dir","stream":"stream","warehouse":"warehouse","roles":["roles"],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}]}],"schema":[{"asset_id":"2b0bf220-079c-11ee-be56-0242ac120002","connection_id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","description":"description","connection_path":"connection_path","physical_type":"physical_type","properties":[{"name":"name","type":{"type":"type","length":"length","scale":"scale","nullable":"nullable","signed":"signed","native_type":"native_type"},"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}],"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}]}],"domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"parts_out":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"type":"data_asset"},"delivery_methods":[{"id":"09cf5fcc-cb9d-4995-a8e4-16517b25229f","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"getproperties":{"producer_input":{"engine_details":{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]},"engines":[{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]}]}}}]}],"workflows":{"order_access_request":{"task_assignee_users":["task_assignee_users"],"pre_approved_users":["pre_approved_users"],"custom_workflow_definition":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"}}},"dataview_enabled":true,"comments":"Comments by a producer that are provided either at the time of data product version creation or retiring","access_control":{"owner":"IBMid-696000KYV9"},"last_updated_at":"2019-01-01T12:00:00.000Z","sub_container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd"},"is_restricted":false,"id":"2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd","asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response1, + content_type='application/json', + status=200, + ) + responses.add( + responses.GET, + url, + body=mock_response2, + content_type='application/json', + status=200, + ) + + # Exercise the pager class for this operation + all_results = [] + pager = DataProductDraftsPager( + client=_service, + data_product_id='testString', + asset_container_id='testString', + version='testString', + limit=10, + ) + while pager.has_next(): + next_page = pager.get_next() + assert next_page is not None + all_results.extend(next_page) + assert len(all_results) == 2 + + @responses.activate + def test_list_data_product_drafts_with_pager_get_all(self): + """ + test_list_data_product_drafts_with_pager_get_all() + """ + # Set up a two-page mock response + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts') + mock_response1 = '{"next":{"start":"1"},"total_count":2,"limit":1,"drafts":[{"version":"1.0.0","state":"draft","data_product":{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"name":"My Data Product","description":"This is a description of My Data Product.","tags":["tags"],"use_cases":[{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}],"types":["data"],"contract_terms":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"id":"id","documents":[{"url":"url","type":"terms_and_conditions","name":"name","id":"2b0bf220-079c-11ee-be56-0242ac120002","attachment":{"id":"id"},"upload_url":"upload_url"}],"error_msg":"error_msg","overview":{"api_version":"v3.0.1","kind":"DataContract","name":"Sample Data Contract","version":"0.0.0","domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"more_info":"List of links to sources that provide more details on the data contract."},"description":{"purpose":"Used for customer behavior analysis.","limitations":"Data cannot be used for marketing.","usage":"Data should be used only for analytics.","more_info":[{"type":"privacy-statement","url":"https://moreinfo.example.com"}],"custom_properties":"{\\"property1\\":\\"value1\\"}"},"organization":[{"user_id":"IBMid-691000IN4G","role":"owner"}],"roles":[{"role":"owner"}],"price":{"amount":"100.0","currency":"USD","unit":"megabyte"},"sla":[{"default_element":"Standard SLA Policy","properties":[{"property":"Uptime Guarantee","value":"99.9"}]}],"support_and_communication":[{"channel":"Email Support","url":"https://support.example.com"}],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}],"contract_test":{"status":"pass","last_tested_time":"last_tested_time","message":"message"},"servers":[{"server":"server","asset":{"id":"id","name":"name"},"connection_id":"connection_id","type":"type","description":"description","environment":"environment","account":"account","catalog":"catalog","database":"database","dataset":"dataset","delimiter":"delimiter","endpoint_url":"endpoint_url","format":"format","host":"host","location":"location","path":"path","port":"port","project":"project","region":"region","region_name":"region_name","schema":"schema","service_name":"service_name","staging_dir":"staging_dir","stream":"stream","warehouse":"warehouse","roles":["roles"],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}]}],"schema":[{"asset_id":"2b0bf220-079c-11ee-be56-0242ac120002","connection_id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","description":"description","connection_path":"connection_path","physical_type":"physical_type","properties":[{"name":"name","type":{"type":"type","length":"length","scale":"scale","nullable":"nullable","signed":"signed","native_type":"native_type"},"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}],"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}]}],"domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"parts_out":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"type":"data_asset"},"delivery_methods":[{"id":"09cf5fcc-cb9d-4995-a8e4-16517b25229f","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"getproperties":{"producer_input":{"engine_details":{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]},"engines":[{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]}]}}}]}],"workflows":{"order_access_request":{"task_assignee_users":["task_assignee_users"],"pre_approved_users":["pre_approved_users"],"custom_workflow_definition":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"}}},"dataview_enabled":true,"comments":"Comments by a producer that are provided either at the time of data product version creation or retiring","access_control":{"owner":"IBMid-696000KYV9"},"last_updated_at":"2019-01-01T12:00:00.000Z","sub_container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd"},"is_restricted":false,"id":"2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd","asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}}]}' + mock_response2 = '{"total_count":2,"limit":1,"drafts":[{"version":"1.0.0","state":"draft","data_product":{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"name":"My Data Product","description":"This is a description of My Data Product.","tags":["tags"],"use_cases":[{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}],"types":["data"],"contract_terms":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"id":"id","documents":[{"url":"url","type":"terms_and_conditions","name":"name","id":"2b0bf220-079c-11ee-be56-0242ac120002","attachment":{"id":"id"},"upload_url":"upload_url"}],"error_msg":"error_msg","overview":{"api_version":"v3.0.1","kind":"DataContract","name":"Sample Data Contract","version":"0.0.0","domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"more_info":"List of links to sources that provide more details on the data contract."},"description":{"purpose":"Used for customer behavior analysis.","limitations":"Data cannot be used for marketing.","usage":"Data should be used only for analytics.","more_info":[{"type":"privacy-statement","url":"https://moreinfo.example.com"}],"custom_properties":"{\\"property1\\":\\"value1\\"}"},"organization":[{"user_id":"IBMid-691000IN4G","role":"owner"}],"roles":[{"role":"owner"}],"price":{"amount":"100.0","currency":"USD","unit":"megabyte"},"sla":[{"default_element":"Standard SLA Policy","properties":[{"property":"Uptime Guarantee","value":"99.9"}]}],"support_and_communication":[{"channel":"Email Support","url":"https://support.example.com"}],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}],"contract_test":{"status":"pass","last_tested_time":"last_tested_time","message":"message"},"servers":[{"server":"server","asset":{"id":"id","name":"name"},"connection_id":"connection_id","type":"type","description":"description","environment":"environment","account":"account","catalog":"catalog","database":"database","dataset":"dataset","delimiter":"delimiter","endpoint_url":"endpoint_url","format":"format","host":"host","location":"location","path":"path","port":"port","project":"project","region":"region","region_name":"region_name","schema":"schema","service_name":"service_name","staging_dir":"staging_dir","stream":"stream","warehouse":"warehouse","roles":["roles"],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}]}],"schema":[{"asset_id":"2b0bf220-079c-11ee-be56-0242ac120002","connection_id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","description":"description","connection_path":"connection_path","physical_type":"physical_type","properties":[{"name":"name","type":{"type":"type","length":"length","scale":"scale","nullable":"nullable","signed":"signed","native_type":"native_type"},"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}],"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}]}],"domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"parts_out":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"type":"data_asset"},"delivery_methods":[{"id":"09cf5fcc-cb9d-4995-a8e4-16517b25229f","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"getproperties":{"producer_input":{"engine_details":{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]},"engines":[{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]}]}}}]}],"workflows":{"order_access_request":{"task_assignee_users":["task_assignee_users"],"pre_approved_users":["pre_approved_users"],"custom_workflow_definition":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"}}},"dataview_enabled":true,"comments":"Comments by a producer that are provided either at the time of data product version creation or retiring","access_control":{"owner":"IBMid-696000KYV9"},"last_updated_at":"2019-01-01T12:00:00.000Z","sub_container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd"},"is_restricted":false,"id":"2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd","asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response1, + content_type='application/json', + status=200, + ) + responses.add( + responses.GET, + url, + body=mock_response2, + content_type='application/json', + status=200, + ) + + # Exercise the pager class for this operation + pager = DataProductDraftsPager( + client=_service, + data_product_id='testString', + asset_container_id='testString', + version='testString', + limit=10, + ) + all_results = pager.get_all() + assert all_results is not None + assert len(all_results) == 2 + + +class TestCreateDataProductDraft: + """ + Test Class for create_data_product_draft + """ + + @responses.activate + def test_create_data_product_draft_all_params(self): + """ + create_data_product_draft() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a ContainerIdentity model + container_identity_model = {} + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a dict representation of a AssetPrototype model + asset_prototype_model = {} + asset_prototype_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_prototype_model['container'] = container_identity_model + + # Construct a dict representation of a DataProductDraftVersionRelease model + data_product_draft_version_release_model = {} + data_product_draft_version_release_model['id'] = '8bf83660-11fe-4427-a72a-8d8359af24e3' + + # Construct a dict representation of a DataProductIdentity model + data_product_identity_model = {} + data_product_identity_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_identity_model['release'] = data_product_draft_version_release_model + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a UseCase model + use_case_model = {} + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'owner' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractTerms model + contract_terms_model = {} + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + # Construct a dict representation of a AssetPartReference model + asset_part_reference_model = {} + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + # Construct a dict representation of a EngineDetailsModel model + engine_details_model_model = {} + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + # Construct a dict representation of a ProducerInputModel model + producer_input_model_model = {} + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + # Construct a dict representation of a DeliveryMethodPropertiesModel model + delivery_method_properties_model_model = {} + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + # Construct a dict representation of a DeliveryMethod model + delivery_method_model = {} + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + # Construct a dict representation of a DataProductPart model + data_product_part_model = {} + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + # Construct a dict representation of a DataProductCustomWorkflowDefinition model + data_product_custom_workflow_definition_model = {} + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a dict representation of a DataProductOrderAccessRequest model + data_product_order_access_request_model = {} + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + # Construct a dict representation of a DataProductWorkflows model + data_product_workflows_model = {} + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + # Construct a dict representation of a AssetListAccessControl model + asset_list_access_control_model = {} + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + # Set up parameter values + data_product_id = 'testString' + asset = asset_prototype_model + version = '1.2.0' + state = 'draft' + data_product = data_product_identity_model + name = 'testString' + description = 'testString' + tags = ['testString'] + use_cases = [use_case_model] + types = ['data'] + contract_terms = [contract_terms_model] + domain = domain_model + parts_out = [data_product_part_model] + workflows = data_product_workflows_model + dataview_enabled = True + comments = 'testString' + access_control = asset_list_access_control_model + last_updated_at = string_to_datetime('2019-01-01T12:00:00.000Z') + sub_container = container_identity_model + is_restricted = True + + # Invoke method + response = _service.create_data_product_draft( + data_product_id, + asset, + version=version, + state=state, + data_product=data_product, + name=name, + description=description, + tags=tags, + use_cases=use_cases, + types=types, + contract_terms=contract_terms, + domain=domain, + parts_out=parts_out, + workflows=workflows, + dataview_enabled=dataview_enabled, + comments=comments, + access_control=access_control, + last_updated_at=last_updated_at, + sub_container=sub_container, + is_restricted=is_restricted, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['asset'] == asset_prototype_model + assert req_body['version'] == '1.2.0' + assert req_body['state'] == 'draft' + assert req_body['data_product'] == data_product_identity_model + assert req_body['name'] == 'testString' + assert req_body['description'] == 'testString' + assert req_body['tags'] == ['testString'] + assert req_body['use_cases'] == [use_case_model] + assert req_body['types'] == ['data'] + assert req_body['contract_terms'] == [contract_terms_model] + assert req_body['domain'] == domain_model + assert req_body['parts_out'] == [data_product_part_model] + assert req_body['workflows'] == data_product_workflows_model + assert req_body['dataview_enabled'] == True + assert req_body['comments'] == 'testString' + assert req_body['access_control'] == asset_list_access_control_model + assert req_body['last_updated_at'] == '2019-01-01T12:00:00Z' + assert req_body['sub_container'] == container_identity_model + assert req_body['is_restricted'] == True + + def test_create_data_product_draft_all_params_with_retries(self): + # Enable retries and run test_create_data_product_draft_all_params. + _service.enable_retries() + self.test_create_data_product_draft_all_params() + + # Disable retries and run test_create_data_product_draft_all_params. + _service.disable_retries() + self.test_create_data_product_draft_all_params() + + @responses.activate + def test_create_data_product_draft_value_error(self): + """ + test_create_data_product_draft_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a ContainerIdentity model + container_identity_model = {} + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a dict representation of a AssetPrototype model + asset_prototype_model = {} + asset_prototype_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_prototype_model['container'] = container_identity_model + + # Construct a dict representation of a DataProductDraftVersionRelease model + data_product_draft_version_release_model = {} + data_product_draft_version_release_model['id'] = '8bf83660-11fe-4427-a72a-8d8359af24e3' + + # Construct a dict representation of a DataProductIdentity model + data_product_identity_model = {} + data_product_identity_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_identity_model['release'] = data_product_draft_version_release_model + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a UseCase model + use_case_model = {} + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'owner' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractTerms model + contract_terms_model = {} + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + # Construct a dict representation of a AssetPartReference model + asset_part_reference_model = {} + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + # Construct a dict representation of a EngineDetailsModel model + engine_details_model_model = {} + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + # Construct a dict representation of a ProducerInputModel model + producer_input_model_model = {} + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + # Construct a dict representation of a DeliveryMethodPropertiesModel model + delivery_method_properties_model_model = {} + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + # Construct a dict representation of a DeliveryMethod model + delivery_method_model = {} + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + # Construct a dict representation of a DataProductPart model + data_product_part_model = {} + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + # Construct a dict representation of a DataProductCustomWorkflowDefinition model + data_product_custom_workflow_definition_model = {} + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a dict representation of a DataProductOrderAccessRequest model + data_product_order_access_request_model = {} + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + # Construct a dict representation of a DataProductWorkflows model + data_product_workflows_model = {} + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + # Construct a dict representation of a AssetListAccessControl model + asset_list_access_control_model = {} + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + # Set up parameter values + data_product_id = 'testString' + asset = asset_prototype_model + version = '1.2.0' + state = 'draft' + data_product = data_product_identity_model + name = 'testString' + description = 'testString' + tags = ['testString'] + use_cases = [use_case_model] + types = ['data'] + contract_terms = [contract_terms_model] + domain = domain_model + parts_out = [data_product_part_model] + workflows = data_product_workflows_model + dataview_enabled = True + comments = 'testString' + access_control = asset_list_access_control_model + last_updated_at = string_to_datetime('2019-01-01T12:00:00.000Z') + sub_container = container_identity_model + is_restricted = True + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "asset": asset, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.create_data_product_draft(**req_copy) + + def test_create_data_product_draft_value_error_with_retries(self): + # Enable retries and run test_create_data_product_draft_value_error. + _service.enable_retries() + self.test_create_data_product_draft_value_error() + + # Disable retries and run test_create_data_product_draft_value_error. + _service.disable_retries() + self.test_create_data_product_draft_value_error() + + +class TestCreateDraftContractTermsDocument: + """ + Test Class for create_draft_contract_terms_document + """ + + @responses.activate + def test_create_draft_contract_terms_document_all_params(self): + """ + create_draft_contract_terms_document() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + type = 'terms_and_conditions' + name = 'Terms and conditions document' + url = 'testString' + + # Invoke method + response = _service.create_draft_contract_terms_document( + data_product_id, + draft_id, + contract_terms_id, + type, + name, + url=url, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['type'] == 'terms_and_conditions' + assert req_body['name'] == 'Terms and conditions document' + assert req_body['url'] == 'testString' + + def test_create_draft_contract_terms_document_all_params_with_retries(self): + # Enable retries and run test_create_draft_contract_terms_document_all_params. + _service.enable_retries() + self.test_create_draft_contract_terms_document_all_params() + + # Disable retries and run test_create_draft_contract_terms_document_all_params. + _service.disable_retries() + self.test_create_draft_contract_terms_document_all_params() + + @responses.activate + def test_create_draft_contract_terms_document_value_error(self): + """ + test_create_draft_contract_terms_document_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + type = 'terms_and_conditions' + name = 'Terms and conditions document' + url = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "contract_terms_id": contract_terms_id, + "type": type, + "name": name, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.create_draft_contract_terms_document(**req_copy) + + def test_create_draft_contract_terms_document_value_error_with_retries(self): + # Enable retries and run test_create_draft_contract_terms_document_value_error. + _service.enable_retries() + self.test_create_draft_contract_terms_document_value_error() + + # Disable retries and run test_create_draft_contract_terms_document_value_error. + _service.disable_retries() + self.test_create_draft_contract_terms_document_value_error() + + +class TestGetDataProductDraft: + """ + Test Class for get_data_product_draft + """ + + @responses.activate + def test_get_data_product_draft_all_params(self): + """ + get_data_product_draft() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + + # Invoke method + response = _service.get_data_product_draft( + data_product_id, + draft_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_data_product_draft_all_params_with_retries(self): + # Enable retries and run test_get_data_product_draft_all_params. + _service.enable_retries() + self.test_get_data_product_draft_all_params() + + # Disable retries and run test_get_data_product_draft_all_params. + _service.disable_retries() + self.test_get_data_product_draft_all_params() + + @responses.activate + def test_get_data_product_draft_value_error(self): + """ + test_get_data_product_draft_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_data_product_draft(**req_copy) + + def test_get_data_product_draft_value_error_with_retries(self): + # Enable retries and run test_get_data_product_draft_value_error. + _service.enable_retries() + self.test_get_data_product_draft_value_error() + + # Disable retries and run test_get_data_product_draft_value_error. + _service.disable_retries() + self.test_get_data_product_draft_value_error() + + +class TestDeleteDataProductDraft: + """ + Test Class for delete_data_product_draft + """ + + @responses.activate + def test_delete_data_product_draft_all_params(self): + """ + delete_data_product_draft() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString') + responses.add( + responses.DELETE, + url, + status=204, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + + # Invoke method + response = _service.delete_data_product_draft( + data_product_id, + draft_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 204 + + def test_delete_data_product_draft_all_params_with_retries(self): + # Enable retries and run test_delete_data_product_draft_all_params. + _service.enable_retries() + self.test_delete_data_product_draft_all_params() + + # Disable retries and run test_delete_data_product_draft_all_params. + _service.disable_retries() + self.test_delete_data_product_draft_all_params() + + @responses.activate + def test_delete_data_product_draft_value_error(self): + """ + test_delete_data_product_draft_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString') + responses.add( + responses.DELETE, + url, + status=204, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.delete_data_product_draft(**req_copy) + + def test_delete_data_product_draft_value_error_with_retries(self): + # Enable retries and run test_delete_data_product_draft_value_error. + _service.enable_retries() + self.test_delete_data_product_draft_value_error() + + # Disable retries and run test_delete_data_product_draft_value_error. + _service.disable_retries() + self.test_delete_data_product_draft_value_error() + + +class TestUpdateDataProductDraft: + """ + Test Class for update_data_product_draft + """ + + @responses.activate + def test_update_data_product_draft_all_params(self): + """ + update_data_product_draft() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Invoke method + response = _service.update_data_product_draft( + data_product_id, + draft_id, + json_patch_instructions, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body == json_patch_instructions + + def test_update_data_product_draft_all_params_with_retries(self): + # Enable retries and run test_update_data_product_draft_all_params. + _service.enable_retries() + self.test_update_data_product_draft_all_params() + + # Disable retries and run test_update_data_product_draft_all_params. + _service.disable_retries() + self.test_update_data_product_draft_all_params() + + @responses.activate + def test_update_data_product_draft_value_error(self): + """ + test_update_data_product_draft_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "json_patch_instructions": json_patch_instructions, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.update_data_product_draft(**req_copy) + + def test_update_data_product_draft_value_error_with_retries(self): + # Enable retries and run test_update_data_product_draft_value_error. + _service.enable_retries() + self.test_update_data_product_draft_value_error() + + # Disable retries and run test_update_data_product_draft_value_error. + _service.disable_retries() + self.test_update_data_product_draft_value_error() + + +class TestGetDraftContractTermsDocument: + """ + Test Class for get_draft_contract_terms_document + """ + + @responses.activate + def test_get_draft_contract_terms_document_all_params(self): + """ + get_draft_contract_terms_document() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents/testString') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + + # Invoke method + response = _service.get_draft_contract_terms_document( + data_product_id, + draft_id, + contract_terms_id, + document_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_draft_contract_terms_document_all_params_with_retries(self): + # Enable retries and run test_get_draft_contract_terms_document_all_params. + _service.enable_retries() + self.test_get_draft_contract_terms_document_all_params() + + # Disable retries and run test_get_draft_contract_terms_document_all_params. + _service.disable_retries() + self.test_get_draft_contract_terms_document_all_params() + + @responses.activate + def test_get_draft_contract_terms_document_value_error(self): + """ + test_get_draft_contract_terms_document_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents/testString') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "contract_terms_id": contract_terms_id, + "document_id": document_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_draft_contract_terms_document(**req_copy) + + def test_get_draft_contract_terms_document_value_error_with_retries(self): + # Enable retries and run test_get_draft_contract_terms_document_value_error. + _service.enable_retries() + self.test_get_draft_contract_terms_document_value_error() + + # Disable retries and run test_get_draft_contract_terms_document_value_error. + _service.disable_retries() + self.test_get_draft_contract_terms_document_value_error() + + +class TestDeleteDraftContractTermsDocument: + """ + Test Class for delete_draft_contract_terms_document + """ + + @responses.activate + def test_delete_draft_contract_terms_document_all_params(self): + """ + delete_draft_contract_terms_document() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents/testString') + responses.add( + responses.DELETE, + url, + status=204, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + + # Invoke method + response = _service.delete_draft_contract_terms_document( + data_product_id, + draft_id, + contract_terms_id, + document_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 204 + + def test_delete_draft_contract_terms_document_all_params_with_retries(self): + # Enable retries and run test_delete_draft_contract_terms_document_all_params. + _service.enable_retries() + self.test_delete_draft_contract_terms_document_all_params() + + # Disable retries and run test_delete_draft_contract_terms_document_all_params. + _service.disable_retries() + self.test_delete_draft_contract_terms_document_all_params() + + @responses.activate + def test_delete_draft_contract_terms_document_value_error(self): + """ + test_delete_draft_contract_terms_document_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents/testString') + responses.add( + responses.DELETE, + url, + status=204, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "contract_terms_id": contract_terms_id, + "document_id": document_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.delete_draft_contract_terms_document(**req_copy) + + def test_delete_draft_contract_terms_document_value_error_with_retries(self): + # Enable retries and run test_delete_draft_contract_terms_document_value_error. + _service.enable_retries() + self.test_delete_draft_contract_terms_document_value_error() + + # Disable retries and run test_delete_draft_contract_terms_document_value_error. + _service.disable_retries() + self.test_delete_draft_contract_terms_document_value_error() + + +class TestUpdateDraftContractTermsDocument: + """ + Test Class for update_draft_contract_terms_document + """ + + @responses.activate + def test_update_draft_contract_terms_document_all_params(self): + """ + update_draft_contract_terms_document() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents/testString') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Invoke method + response = _service.update_draft_contract_terms_document( + data_product_id, + draft_id, + contract_terms_id, + document_id, + json_patch_instructions, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body == json_patch_instructions + + def test_update_draft_contract_terms_document_all_params_with_retries(self): + # Enable retries and run test_update_draft_contract_terms_document_all_params. + _service.enable_retries() + self.test_update_draft_contract_terms_document_all_params() + + # Disable retries and run test_update_draft_contract_terms_document_all_params. + _service.disable_retries() + self.test_update_draft_contract_terms_document_all_params() + + @responses.activate + def test_update_draft_contract_terms_document_value_error(self): + """ + test_update_draft_contract_terms_document_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents/testString') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "contract_terms_id": contract_terms_id, + "document_id": document_id, + "json_patch_instructions": json_patch_instructions, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.update_draft_contract_terms_document(**req_copy) + + def test_update_draft_contract_terms_document_value_error_with_retries(self): + # Enable retries and run test_update_draft_contract_terms_document_value_error. + _service.enable_retries() + self.test_update_draft_contract_terms_document_value_error() + + # Disable retries and run test_update_draft_contract_terms_document_value_error. + _service.disable_retries() + self.test_update_draft_contract_terms_document_value_error() + + +class TestGetDataProductDraftContractTerms: + """ + Test Class for get_data_product_draft_contract_terms + """ + + @responses.activate + def test_get_data_product_draft_contract_terms_all_params(self): + """ + get_data_product_draft_contract_terms() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString') + mock_response = '{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + accept = 'application/json' + include_contract_documents = True + autopopulate_server_information = False + server_asset_id = 'testString' + + # Invoke method + response = _service.get_data_product_draft_contract_terms( + data_product_id, + draft_id, + contract_terms_id, + accept=accept, + include_contract_documents=include_contract_documents, + autopopulate_server_information=autopopulate_server_information, + server_asset_id=server_asset_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'include_contract_documents={}'.format('true' if include_contract_documents else 'false') in query_string + assert 'autopopulate_server_information={}'.format('true' if autopopulate_server_information else 'false') in query_string + assert 'server_asset_id={}'.format(server_asset_id) in query_string + + def test_get_data_product_draft_contract_terms_all_params_with_retries(self): + # Enable retries and run test_get_data_product_draft_contract_terms_all_params. + _service.enable_retries() + self.test_get_data_product_draft_contract_terms_all_params() + + # Disable retries and run test_get_data_product_draft_contract_terms_all_params. + _service.disable_retries() + self.test_get_data_product_draft_contract_terms_all_params() + + @responses.activate + def test_get_data_product_draft_contract_terms_required_params(self): + """ + test_get_data_product_draft_contract_terms_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString') + mock_response = '{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + + # Invoke method + response = _service.get_data_product_draft_contract_terms( + data_product_id, + draft_id, + contract_terms_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_data_product_draft_contract_terms_required_params_with_retries(self): + # Enable retries and run test_get_data_product_draft_contract_terms_required_params. + _service.enable_retries() + self.test_get_data_product_draft_contract_terms_required_params() + + # Disable retries and run test_get_data_product_draft_contract_terms_required_params. + _service.disable_retries() + self.test_get_data_product_draft_contract_terms_required_params() + + @responses.activate + def test_get_data_product_draft_contract_terms_value_error(self): + """ + test_get_data_product_draft_contract_terms_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString') + mock_response = '{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "contract_terms_id": contract_terms_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_data_product_draft_contract_terms(**req_copy) + + def test_get_data_product_draft_contract_terms_value_error_with_retries(self): + # Enable retries and run test_get_data_product_draft_contract_terms_value_error. + _service.enable_retries() + self.test_get_data_product_draft_contract_terms_value_error() + + # Disable retries and run test_get_data_product_draft_contract_terms_value_error. + _service.disable_retries() + self.test_get_data_product_draft_contract_terms_value_error() + + +class TestReplaceDataProductDraftContractTerms: + """ + Test Class for replace_data_product_draft_contract_terms + """ + + @responses.activate + def test_replace_data_product_draft_contract_terms_all_params(self): + """ + replace_data_product_draft_contract_terms() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString') + mock_response = '{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}' + responses.add( + responses.PUT, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'https://ibm.com/document' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'Terms and Conditions' + contract_terms_document_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + domain_model['name'] = 'domain_name' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = 'v0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://www.moreinfo.example.coms' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Intended purpose for the provided data.' + description_model['limitations'] = 'Technical, compliance, and legal limitations for data use.' + description_model['usage'] = 'Recommended usage of the data.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = 'Custom properties that are not part of the standard.' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'IAM Role' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = 'Amount' + pricing_model['currency'] = 'Currency' + pricing_model['unit'] = 'Unit' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'slaproperty' + contract_template_sla_property_model['value'] = 'slavalue' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'sladefaultelement' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'channel' + contract_template_support_and_communication_model['url'] = 'https://www.example.coms' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'The name of the key.' + contract_template_custom_property_model['value'] = 'The value of the key.' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = '684d6aa0-9f93-4564-8a20-e354bc469857' + contract_asset_model['name'] = 'PAYMENT_TRANSACTIONS1' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'snowflake-server-01' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = '8d7701be-709a-49c0-ae4e-a7daeaae6def' + contract_server_model['type'] = 'snowflake' + contract_server_model['description'] = 'Snowflake analytics server' + contract_server_model['environment'] = 'dev' + contract_server_model['account'] = 'acc-456' + contract_server_model['catalog'] = 'analytics_cat' + contract_server_model['database'] = 'analytics_db' + contract_server_model['dataset'] = 'customer_data' + contract_server_model['delimiter'] = ',' + contract_server_model['endpoint_url'] = 'https://xy12345.snowflakecomputing.com' + contract_server_model['format'] = 'parquet' + contract_server_model['host'] = 'xy12345.snowflakecomputing.com' + contract_server_model['location'] = 'Mumbai' + contract_server_model['path'] = '/analytics/data' + contract_server_model['port'] = '443' + contract_server_model['project'] = 'projectY' + contract_server_model['region'] = 'ap-south-1' + contract_server_model['region_name'] = 'Asia South 1' + contract_server_model['schema'] = 'PAYMENT_TRANSACTIONS1' + contract_server_model['service_name'] = 'snowflake' + contract_server_model['staging_dir'] = '/snowflake/staging' + contract_server_model['stream'] = 'stream_analytics' + contract_server_model['warehouse'] = 'wh_xlarge' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'varchar' + contract_schema_property_type_model['length'] = '1024' + contract_schema_property_type_model['scale'] = '0' + contract_schema_property_type_model['nullable'] = 'true' + contract_schema_property_type_model['signed'] = 'false' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'product_brand_code' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '09ca6b40-7c89-412a-8951-ad820da709d1' + contract_schema_model['connection_id'] = '6cc57d4d-2229-438f-91a0-2c455556422b' + contract_schema_model['name'] = '000000_0-2025-06-20-20-28-52.csv' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = '/dpx-test-bucket/000000_0-2025-06-20-20-28-52.csv' + contract_schema_model['physical_type'] = 'text/csv' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + asset = asset_reference_model + id = 'testString' + documents = [contract_terms_document_model] + error_msg = 'testString' + overview = overview_model + description = description_model + organization = [contract_template_organization_model] + roles = [roles_model] + price = pricing_model + sla = [contract_template_sla_model] + support_and_communication = [contract_template_support_and_communication_model] + custom_properties = [contract_template_custom_property_model] + contract_test = contract_test_model + servers = [contract_server_model] + schema = [contract_schema_model] + + # Invoke method + response = _service.replace_data_product_draft_contract_terms( + data_product_id, + draft_id, + contract_terms_id, + asset=asset, + id=id, + documents=documents, + error_msg=error_msg, + overview=overview, + description=description, + organization=organization, + roles=roles, + price=price, + sla=sla, + support_and_communication=support_and_communication, + custom_properties=custom_properties, + contract_test=contract_test, + servers=servers, + schema=schema, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['asset'] == asset_reference_model + assert req_body['id'] == 'testString' + assert req_body['documents'] == [contract_terms_document_model] + assert req_body['error_msg'] == 'testString' + assert req_body['overview'] == overview_model + assert req_body['description'] == description_model + assert req_body['organization'] == [contract_template_organization_model] + assert req_body['roles'] == [roles_model] + assert req_body['price'] == pricing_model + assert req_body['sla'] == [contract_template_sla_model] + assert req_body['support_and_communication'] == [contract_template_support_and_communication_model] + assert req_body['custom_properties'] == [contract_template_custom_property_model] + assert req_body['contract_test'] == contract_test_model + assert req_body['servers'] == [contract_server_model] + assert req_body['schema'] == [contract_schema_model] + + def test_replace_data_product_draft_contract_terms_all_params_with_retries(self): + # Enable retries and run test_replace_data_product_draft_contract_terms_all_params. + _service.enable_retries() + self.test_replace_data_product_draft_contract_terms_all_params() + + # Disable retries and run test_replace_data_product_draft_contract_terms_all_params. + _service.disable_retries() + self.test_replace_data_product_draft_contract_terms_all_params() + + @responses.activate + def test_replace_data_product_draft_contract_terms_value_error(self): + """ + test_replace_data_product_draft_contract_terms_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString') + mock_response = '{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}' + responses.add( + responses.PUT, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'https://ibm.com/document' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'Terms and Conditions' + contract_terms_document_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + domain_model['name'] = 'domain_name' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = 'v0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://www.moreinfo.example.coms' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Intended purpose for the provided data.' + description_model['limitations'] = 'Technical, compliance, and legal limitations for data use.' + description_model['usage'] = 'Recommended usage of the data.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = 'Custom properties that are not part of the standard.' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'IAM Role' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = 'Amount' + pricing_model['currency'] = 'Currency' + pricing_model['unit'] = 'Unit' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'slaproperty' + contract_template_sla_property_model['value'] = 'slavalue' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'sladefaultelement' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'channel' + contract_template_support_and_communication_model['url'] = 'https://www.example.coms' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'The name of the key.' + contract_template_custom_property_model['value'] = 'The value of the key.' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = '684d6aa0-9f93-4564-8a20-e354bc469857' + contract_asset_model['name'] = 'PAYMENT_TRANSACTIONS1' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'snowflake-server-01' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = '8d7701be-709a-49c0-ae4e-a7daeaae6def' + contract_server_model['type'] = 'snowflake' + contract_server_model['description'] = 'Snowflake analytics server' + contract_server_model['environment'] = 'dev' + contract_server_model['account'] = 'acc-456' + contract_server_model['catalog'] = 'analytics_cat' + contract_server_model['database'] = 'analytics_db' + contract_server_model['dataset'] = 'customer_data' + contract_server_model['delimiter'] = ',' + contract_server_model['endpoint_url'] = 'https://xy12345.snowflakecomputing.com' + contract_server_model['format'] = 'parquet' + contract_server_model['host'] = 'xy12345.snowflakecomputing.com' + contract_server_model['location'] = 'Mumbai' + contract_server_model['path'] = '/analytics/data' + contract_server_model['port'] = '443' + contract_server_model['project'] = 'projectY' + contract_server_model['region'] = 'ap-south-1' + contract_server_model['region_name'] = 'Asia South 1' + contract_server_model['schema'] = 'PAYMENT_TRANSACTIONS1' + contract_server_model['service_name'] = 'snowflake' + contract_server_model['staging_dir'] = '/snowflake/staging' + contract_server_model['stream'] = 'stream_analytics' + contract_server_model['warehouse'] = 'wh_xlarge' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'varchar' + contract_schema_property_type_model['length'] = '1024' + contract_schema_property_type_model['scale'] = '0' + contract_schema_property_type_model['nullable'] = 'true' + contract_schema_property_type_model['signed'] = 'false' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'product_brand_code' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '09ca6b40-7c89-412a-8951-ad820da709d1' + contract_schema_model['connection_id'] = '6cc57d4d-2229-438f-91a0-2c455556422b' + contract_schema_model['name'] = '000000_0-2025-06-20-20-28-52.csv' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = '/dpx-test-bucket/000000_0-2025-06-20-20-28-52.csv' + contract_schema_model['physical_type'] = 'text/csv' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + asset = asset_reference_model + id = 'testString' + documents = [contract_terms_document_model] + error_msg = 'testString' + overview = overview_model + description = description_model + organization = [contract_template_organization_model] + roles = [roles_model] + price = pricing_model + sla = [contract_template_sla_model] + support_and_communication = [contract_template_support_and_communication_model] + custom_properties = [contract_template_custom_property_model] + contract_test = contract_test_model + servers = [contract_server_model] + schema = [contract_schema_model] + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "contract_terms_id": contract_terms_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.replace_data_product_draft_contract_terms(**req_copy) + + def test_replace_data_product_draft_contract_terms_value_error_with_retries(self): + # Enable retries and run test_replace_data_product_draft_contract_terms_value_error. + _service.enable_retries() + self.test_replace_data_product_draft_contract_terms_value_error() + + # Disable retries and run test_replace_data_product_draft_contract_terms_value_error. + _service.disable_retries() + self.test_replace_data_product_draft_contract_terms_value_error() + + +class TestUpdateDataProductDraftContractTerms: + """ + Test Class for update_data_product_draft_contract_terms + """ + + @responses.activate + def test_update_data_product_draft_contract_terms_all_params(self): + """ + update_data_product_draft_contract_terms() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString') + mock_response = '{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Invoke method + response = _service.update_data_product_draft_contract_terms( + data_product_id, + draft_id, + contract_terms_id, + json_patch_instructions, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body == json_patch_instructions + + def test_update_data_product_draft_contract_terms_all_params_with_retries(self): + # Enable retries and run test_update_data_product_draft_contract_terms_all_params. + _service.enable_retries() + self.test_update_data_product_draft_contract_terms_all_params() + + # Disable retries and run test_update_data_product_draft_contract_terms_all_params. + _service.disable_retries() + self.test_update_data_product_draft_contract_terms_all_params() + + @responses.activate + def test_update_data_product_draft_contract_terms_value_error(self): + """ + test_update_data_product_draft_contract_terms_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString') + mock_response = '{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "contract_terms_id": contract_terms_id, + "json_patch_instructions": json_patch_instructions, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.update_data_product_draft_contract_terms(**req_copy) + + def test_update_data_product_draft_contract_terms_value_error_with_retries(self): + # Enable retries and run test_update_data_product_draft_contract_terms_value_error. + _service.enable_retries() + self.test_update_data_product_draft_contract_terms_value_error() + + # Disable retries and run test_update_data_product_draft_contract_terms_value_error. + _service.disable_retries() + self.test_update_data_product_draft_contract_terms_value_error() + + +class TestGetContractTermsInSpecifiedFormat: + """ + Test Class for get_contract_terms_in_specified_format + """ + + @responses.activate + def test_get_contract_terms_in_specified_format_all_params(self): + """ + get_contract_terms_in_specified_format() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/format') + mock_response = 'This is a mock binary response.' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/odcs+yaml', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + format = 'testString' + format_version = 'testString' + accept = 'application/odcs+yaml' + + # Invoke method + response = _service.get_contract_terms_in_specified_format( + data_product_id, + draft_id, + contract_terms_id, + format, + format_version, + accept=accept, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'format={}'.format(format) in query_string + assert 'format_version={}'.format(format_version) in query_string + + def test_get_contract_terms_in_specified_format_all_params_with_retries(self): + # Enable retries and run test_get_contract_terms_in_specified_format_all_params. + _service.enable_retries() + self.test_get_contract_terms_in_specified_format_all_params() + + # Disable retries and run test_get_contract_terms_in_specified_format_all_params. + _service.disable_retries() + self.test_get_contract_terms_in_specified_format_all_params() + + @responses.activate + def test_get_contract_terms_in_specified_format_required_params(self): + """ + test_get_contract_terms_in_specified_format_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/format') + mock_response = 'This is a mock binary response.' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/odcs+yaml', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + format = 'testString' + format_version = 'testString' + + # Invoke method + response = _service.get_contract_terms_in_specified_format( + data_product_id, + draft_id, + contract_terms_id, + format, + format_version, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'format={}'.format(format) in query_string + assert 'format_version={}'.format(format_version) in query_string + + def test_get_contract_terms_in_specified_format_required_params_with_retries(self): + # Enable retries and run test_get_contract_terms_in_specified_format_required_params. + _service.enable_retries() + self.test_get_contract_terms_in_specified_format_required_params() + + # Disable retries and run test_get_contract_terms_in_specified_format_required_params. + _service.disable_retries() + self.test_get_contract_terms_in_specified_format_required_params() + + @responses.activate + def test_get_contract_terms_in_specified_format_value_error(self): + """ + test_get_contract_terms_in_specified_format_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/format') + mock_response = 'This is a mock binary response.' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/odcs+yaml', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + format = 'testString' + format_version = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "contract_terms_id": contract_terms_id, + "format": format, + "format_version": format_version, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_contract_terms_in_specified_format(**req_copy) + + def test_get_contract_terms_in_specified_format_value_error_with_retries(self): + # Enable retries and run test_get_contract_terms_in_specified_format_value_error. + _service.enable_retries() + self.test_get_contract_terms_in_specified_format_value_error() + + # Disable retries and run test_get_contract_terms_in_specified_format_value_error. + _service.disable_retries() + self.test_get_contract_terms_in_specified_format_value_error() + + +class TestPublishDataProductDraft: + """ + Test Class for publish_data_product_draft + """ + + @responses.activate + def test_publish_data_product_draft_all_params(self): + """ + publish_data_product_draft() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/publish') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + + # Invoke method + response = _service.publish_data_product_draft( + data_product_id, + draft_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_publish_data_product_draft_all_params_with_retries(self): + # Enable retries and run test_publish_data_product_draft_all_params. + _service.enable_retries() + self.test_publish_data_product_draft_all_params() + + # Disable retries and run test_publish_data_product_draft_all_params. + _service.disable_retries() + self.test_publish_data_product_draft_all_params() + + @responses.activate + def test_publish_data_product_draft_value_error(self): + """ + test_publish_data_product_draft_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/publish') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.publish_data_product_draft(**req_copy) + + def test_publish_data_product_draft_value_error_with_retries(self): + # Enable retries and run test_publish_data_product_draft_value_error. + _service.enable_retries() + self.test_publish_data_product_draft_value_error() + + # Disable retries and run test_publish_data_product_draft_value_error. + _service.disable_retries() + self.test_publish_data_product_draft_value_error() + + +# endregion +############################################################################## +# End of Service: DataProductDrafts +############################################################################## + +############################################################################## +# Start of Service: DataProductReleases +############################################################################## +# region + + +class TestNewInstance: + """ + Test Class for new_instance + """ + + def test_new_instance(self): + """ + new_instance() + """ + os.environ['TEST_SERVICE_AUTH_TYPE'] = 'noAuth' + + service = DphV1.new_instance( + service_name='TEST_SERVICE', + ) + + assert service is not None + assert isinstance(service, DphV1) + + def test_new_instance_without_authenticator(self): + """ + new_instance_without_authenticator() + """ + with pytest.raises(ValueError, match='authenticator must be provided'): + service = DphV1.new_instance( + service_name='TEST_SERVICE_NOT_FOUND', + ) + + +class TestGetDataProductRelease: + """ + Test Class for get_data_product_release + """ + + @responses.activate + def test_get_data_product_release_all_params(self): + """ + get_data_product_release() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + check_caller_approval = False + + # Invoke method + response = _service.get_data_product_release( + data_product_id, + release_id, + check_caller_approval=check_caller_approval, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'check_caller_approval={}'.format('true' if check_caller_approval else 'false') in query_string + + def test_get_data_product_release_all_params_with_retries(self): + # Enable retries and run test_get_data_product_release_all_params. + _service.enable_retries() + self.test_get_data_product_release_all_params() + + # Disable retries and run test_get_data_product_release_all_params. + _service.disable_retries() + self.test_get_data_product_release_all_params() + + @responses.activate + def test_get_data_product_release_required_params(self): + """ + test_get_data_product_release_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + + # Invoke method + response = _service.get_data_product_release( + data_product_id, + release_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_data_product_release_required_params_with_retries(self): + # Enable retries and run test_get_data_product_release_required_params. + _service.enable_retries() + self.test_get_data_product_release_required_params() + + # Disable retries and run test_get_data_product_release_required_params. + _service.disable_retries() + self.test_get_data_product_release_required_params() + + @responses.activate + def test_get_data_product_release_value_error(self): + """ + test_get_data_product_release_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "release_id": release_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_data_product_release(**req_copy) + + def test_get_data_product_release_value_error_with_retries(self): + # Enable retries and run test_get_data_product_release_value_error. + _service.enable_retries() + self.test_get_data_product_release_value_error() + + # Disable retries and run test_get_data_product_release_value_error. + _service.disable_retries() + self.test_get_data_product_release_value_error() + + +class TestUpdateDataProductRelease: + """ + Test Class for update_data_product_release + """ + + @responses.activate + def test_update_data_product_release_all_params(self): + """ + update_data_product_release() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Invoke method + response = _service.update_data_product_release( + data_product_id, + release_id, + json_patch_instructions, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body == json_patch_instructions + + def test_update_data_product_release_all_params_with_retries(self): + # Enable retries and run test_update_data_product_release_all_params. + _service.enable_retries() + self.test_update_data_product_release_all_params() + + # Disable retries and run test_update_data_product_release_all_params. + _service.disable_retries() + self.test_update_data_product_release_all_params() + + @responses.activate + def test_update_data_product_release_value_error(self): + """ + test_update_data_product_release_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "release_id": release_id, + "json_patch_instructions": json_patch_instructions, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.update_data_product_release(**req_copy) + + def test_update_data_product_release_value_error_with_retries(self): + # Enable retries and run test_update_data_product_release_value_error. + _service.enable_retries() + self.test_update_data_product_release_value_error() + + # Disable retries and run test_update_data_product_release_value_error. + _service.disable_retries() + self.test_update_data_product_release_value_error() + + +class TestGetReleaseContractTermsDocument: + """ + Test Class for get_release_contract_terms_document + """ + + @responses.activate + def test_get_release_contract_terms_document_all_params(self): + """ + get_release_contract_terms_document() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/contract_terms/testString/documents/testString') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + + # Invoke method + response = _service.get_release_contract_terms_document( + data_product_id, + release_id, + contract_terms_id, + document_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_release_contract_terms_document_all_params_with_retries(self): + # Enable retries and run test_get_release_contract_terms_document_all_params. + _service.enable_retries() + self.test_get_release_contract_terms_document_all_params() + + # Disable retries and run test_get_release_contract_terms_document_all_params. + _service.disable_retries() + self.test_get_release_contract_terms_document_all_params() + + @responses.activate + def test_get_release_contract_terms_document_value_error(self): + """ + test_get_release_contract_terms_document_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/contract_terms/testString/documents/testString') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "release_id": release_id, + "contract_terms_id": contract_terms_id, + "document_id": document_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_release_contract_terms_document(**req_copy) + + def test_get_release_contract_terms_document_value_error_with_retries(self): + # Enable retries and run test_get_release_contract_terms_document_value_error. + _service.enable_retries() + self.test_get_release_contract_terms_document_value_error() + + # Disable retries and run test_get_release_contract_terms_document_value_error. + _service.disable_retries() + self.test_get_release_contract_terms_document_value_error() + + +class TestGetPublishedDataProductDraftContractTerms: + """ + Test Class for get_published_data_product_draft_contract_terms + """ + + @responses.activate + def test_get_published_data_product_draft_contract_terms_all_params(self): + """ + get_published_data_product_draft_contract_terms() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/contract_terms/testString') + mock_response = 'This is a mock binary response.' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/odcs+yaml', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + contract_terms_id = 'testString' + accept = 'application/odcs+yaml' + include_contract_documents = True + + # Invoke method + response = _service.get_published_data_product_draft_contract_terms( + data_product_id, + release_id, + contract_terms_id, + accept=accept, + include_contract_documents=include_contract_documents, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'include_contract_documents={}'.format('true' if include_contract_documents else 'false') in query_string + + def test_get_published_data_product_draft_contract_terms_all_params_with_retries(self): + # Enable retries and run test_get_published_data_product_draft_contract_terms_all_params. + _service.enable_retries() + self.test_get_published_data_product_draft_contract_terms_all_params() + + # Disable retries and run test_get_published_data_product_draft_contract_terms_all_params. + _service.disable_retries() + self.test_get_published_data_product_draft_contract_terms_all_params() + + @responses.activate + def test_get_published_data_product_draft_contract_terms_required_params(self): + """ + test_get_published_data_product_draft_contract_terms_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/contract_terms/testString') + mock_response = 'This is a mock binary response.' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/odcs+yaml', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + contract_terms_id = 'testString' + + # Invoke method + response = _service.get_published_data_product_draft_contract_terms( + data_product_id, + release_id, + contract_terms_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_published_data_product_draft_contract_terms_required_params_with_retries(self): + # Enable retries and run test_get_published_data_product_draft_contract_terms_required_params. + _service.enable_retries() + self.test_get_published_data_product_draft_contract_terms_required_params() + + # Disable retries and run test_get_published_data_product_draft_contract_terms_required_params. + _service.disable_retries() + self.test_get_published_data_product_draft_contract_terms_required_params() + + @responses.activate + def test_get_published_data_product_draft_contract_terms_value_error(self): + """ + test_get_published_data_product_draft_contract_terms_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/contract_terms/testString') + mock_response = 'This is a mock binary response.' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/odcs+yaml', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + contract_terms_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "release_id": release_id, + "contract_terms_id": contract_terms_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_published_data_product_draft_contract_terms(**req_copy) + + def test_get_published_data_product_draft_contract_terms_value_error_with_retries(self): + # Enable retries and run test_get_published_data_product_draft_contract_terms_value_error. + _service.enable_retries() + self.test_get_published_data_product_draft_contract_terms_value_error() + + # Disable retries and run test_get_published_data_product_draft_contract_terms_value_error. + _service.disable_retries() + self.test_get_published_data_product_draft_contract_terms_value_error() + + +class TestListDataProductReleases: + """ + Test Class for list_data_product_releases + """ + + @responses.activate + def test_list_data_product_releases_all_params(self): + """ + list_data_product_releases() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "releases": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + asset_container_id = 'testString' + state = ['available'] + version = 'testString' + limit = 200 + start = 'testString' + + # Invoke method + response = _service.list_data_product_releases( + data_product_id, + asset_container_id=asset_container_id, + state=state, + version=version, + limit=limit, + start=start, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'asset.container.id={}'.format(asset_container_id) in query_string + assert 'state={}'.format(','.join(state)) in query_string + assert 'version={}'.format(version) in query_string + assert 'limit={}'.format(limit) in query_string + assert 'start={}'.format(start) in query_string + + def test_list_data_product_releases_all_params_with_retries(self): + # Enable retries and run test_list_data_product_releases_all_params. + _service.enable_retries() + self.test_list_data_product_releases_all_params() + + # Disable retries and run test_list_data_product_releases_all_params. + _service.disable_retries() + self.test_list_data_product_releases_all_params() + + @responses.activate + def test_list_data_product_releases_required_params(self): + """ + test_list_data_product_releases_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "releases": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + + # Invoke method + response = _service.list_data_product_releases( + data_product_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_list_data_product_releases_required_params_with_retries(self): + # Enable retries and run test_list_data_product_releases_required_params. + _service.enable_retries() + self.test_list_data_product_releases_required_params() + + # Disable retries and run test_list_data_product_releases_required_params. + _service.disable_retries() + self.test_list_data_product_releases_required_params() + + @responses.activate + def test_list_data_product_releases_value_error(self): + """ + test_list_data_product_releases_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "releases": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.list_data_product_releases(**req_copy) + + def test_list_data_product_releases_value_error_with_retries(self): + # Enable retries and run test_list_data_product_releases_value_error. + _service.enable_retries() + self.test_list_data_product_releases_value_error() + + # Disable retries and run test_list_data_product_releases_value_error. + _service.disable_retries() + self.test_list_data_product_releases_value_error() + + @responses.activate + def test_list_data_product_releases_with_pager_get_next(self): + """ + test_list_data_product_releases_with_pager_get_next() + """ + # Set up a two-page mock response + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases') + mock_response1 = '{"next":{"start":"1"},"total_count":2,"limit":1,"releases":[{"version":"1.0.0","state":"draft","data_product":{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"name":"My Data Product","description":"This is a description of My Data Product.","tags":["tags"],"use_cases":[{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}],"types":["data"],"contract_terms":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"id":"id","documents":[{"url":"url","type":"terms_and_conditions","name":"name","id":"2b0bf220-079c-11ee-be56-0242ac120002","attachment":{"id":"id"},"upload_url":"upload_url"}],"error_msg":"error_msg","overview":{"api_version":"v3.0.1","kind":"DataContract","name":"Sample Data Contract","version":"0.0.0","domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"more_info":"List of links to sources that provide more details on the data contract."},"description":{"purpose":"Used for customer behavior analysis.","limitations":"Data cannot be used for marketing.","usage":"Data should be used only for analytics.","more_info":[{"type":"privacy-statement","url":"https://moreinfo.example.com"}],"custom_properties":"{\\"property1\\":\\"value1\\"}"},"organization":[{"user_id":"IBMid-691000IN4G","role":"owner"}],"roles":[{"role":"owner"}],"price":{"amount":"100.0","currency":"USD","unit":"megabyte"},"sla":[{"default_element":"Standard SLA Policy","properties":[{"property":"Uptime Guarantee","value":"99.9"}]}],"support_and_communication":[{"channel":"Email Support","url":"https://support.example.com"}],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}],"contract_test":{"status":"pass","last_tested_time":"last_tested_time","message":"message"},"servers":[{"server":"server","asset":{"id":"id","name":"name"},"connection_id":"connection_id","type":"type","description":"description","environment":"environment","account":"account","catalog":"catalog","database":"database","dataset":"dataset","delimiter":"delimiter","endpoint_url":"endpoint_url","format":"format","host":"host","location":"location","path":"path","port":"port","project":"project","region":"region","region_name":"region_name","schema":"schema","service_name":"service_name","staging_dir":"staging_dir","stream":"stream","warehouse":"warehouse","roles":["roles"],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}]}],"schema":[{"asset_id":"2b0bf220-079c-11ee-be56-0242ac120002","connection_id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","description":"description","connection_path":"connection_path","physical_type":"physical_type","properties":[{"name":"name","type":{"type":"type","length":"length","scale":"scale","nullable":"nullable","signed":"signed","native_type":"native_type"},"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}],"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}]}],"domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"parts_out":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"type":"data_asset"},"delivery_methods":[{"id":"09cf5fcc-cb9d-4995-a8e4-16517b25229f","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"getproperties":{"producer_input":{"engine_details":{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]},"engines":[{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]}]}}}]}],"workflows":{"order_access_request":{"task_assignee_users":["task_assignee_users"],"pre_approved_users":["pre_approved_users"],"custom_workflow_definition":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"}}},"dataview_enabled":true,"comments":"Comments by a producer that are provided either at the time of data product version creation or retiring","access_control":{"owner":"IBMid-696000KYV9"},"last_updated_at":"2019-01-01T12:00:00.000Z","sub_container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd"},"is_restricted":false,"id":"2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd","asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}}]}' + mock_response2 = '{"total_count":2,"limit":1,"releases":[{"version":"1.0.0","state":"draft","data_product":{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"name":"My Data Product","description":"This is a description of My Data Product.","tags":["tags"],"use_cases":[{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}],"types":["data"],"contract_terms":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"id":"id","documents":[{"url":"url","type":"terms_and_conditions","name":"name","id":"2b0bf220-079c-11ee-be56-0242ac120002","attachment":{"id":"id"},"upload_url":"upload_url"}],"error_msg":"error_msg","overview":{"api_version":"v3.0.1","kind":"DataContract","name":"Sample Data Contract","version":"0.0.0","domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"more_info":"List of links to sources that provide more details on the data contract."},"description":{"purpose":"Used for customer behavior analysis.","limitations":"Data cannot be used for marketing.","usage":"Data should be used only for analytics.","more_info":[{"type":"privacy-statement","url":"https://moreinfo.example.com"}],"custom_properties":"{\\"property1\\":\\"value1\\"}"},"organization":[{"user_id":"IBMid-691000IN4G","role":"owner"}],"roles":[{"role":"owner"}],"price":{"amount":"100.0","currency":"USD","unit":"megabyte"},"sla":[{"default_element":"Standard SLA Policy","properties":[{"property":"Uptime Guarantee","value":"99.9"}]}],"support_and_communication":[{"channel":"Email Support","url":"https://support.example.com"}],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}],"contract_test":{"status":"pass","last_tested_time":"last_tested_time","message":"message"},"servers":[{"server":"server","asset":{"id":"id","name":"name"},"connection_id":"connection_id","type":"type","description":"description","environment":"environment","account":"account","catalog":"catalog","database":"database","dataset":"dataset","delimiter":"delimiter","endpoint_url":"endpoint_url","format":"format","host":"host","location":"location","path":"path","port":"port","project":"project","region":"region","region_name":"region_name","schema":"schema","service_name":"service_name","staging_dir":"staging_dir","stream":"stream","warehouse":"warehouse","roles":["roles"],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}]}],"schema":[{"asset_id":"2b0bf220-079c-11ee-be56-0242ac120002","connection_id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","description":"description","connection_path":"connection_path","physical_type":"physical_type","properties":[{"name":"name","type":{"type":"type","length":"length","scale":"scale","nullable":"nullable","signed":"signed","native_type":"native_type"},"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}],"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}]}],"domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"parts_out":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"type":"data_asset"},"delivery_methods":[{"id":"09cf5fcc-cb9d-4995-a8e4-16517b25229f","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"getproperties":{"producer_input":{"engine_details":{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]},"engines":[{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]}]}}}]}],"workflows":{"order_access_request":{"task_assignee_users":["task_assignee_users"],"pre_approved_users":["pre_approved_users"],"custom_workflow_definition":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"}}},"dataview_enabled":true,"comments":"Comments by a producer that are provided either at the time of data product version creation or retiring","access_control":{"owner":"IBMid-696000KYV9"},"last_updated_at":"2019-01-01T12:00:00.000Z","sub_container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd"},"is_restricted":false,"id":"2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd","asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response1, + content_type='application/json', + status=200, + ) + responses.add( + responses.GET, + url, + body=mock_response2, + content_type='application/json', + status=200, + ) + + # Exercise the pager class for this operation + all_results = [] + pager = DataProductReleasesPager( + client=_service, + data_product_id='testString', + asset_container_id='testString', + state=['available'], + version='testString', + limit=10, + ) + while pager.has_next(): + next_page = pager.get_next() + assert next_page is not None + all_results.extend(next_page) + assert len(all_results) == 2 + + @responses.activate + def test_list_data_product_releases_with_pager_get_all(self): + """ + test_list_data_product_releases_with_pager_get_all() + """ + # Set up a two-page mock response + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases') + mock_response1 = '{"next":{"start":"1"},"total_count":2,"limit":1,"releases":[{"version":"1.0.0","state":"draft","data_product":{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"name":"My Data Product","description":"This is a description of My Data Product.","tags":["tags"],"use_cases":[{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}],"types":["data"],"contract_terms":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"id":"id","documents":[{"url":"url","type":"terms_and_conditions","name":"name","id":"2b0bf220-079c-11ee-be56-0242ac120002","attachment":{"id":"id"},"upload_url":"upload_url"}],"error_msg":"error_msg","overview":{"api_version":"v3.0.1","kind":"DataContract","name":"Sample Data Contract","version":"0.0.0","domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"more_info":"List of links to sources that provide more details on the data contract."},"description":{"purpose":"Used for customer behavior analysis.","limitations":"Data cannot be used for marketing.","usage":"Data should be used only for analytics.","more_info":[{"type":"privacy-statement","url":"https://moreinfo.example.com"}],"custom_properties":"{\\"property1\\":\\"value1\\"}"},"organization":[{"user_id":"IBMid-691000IN4G","role":"owner"}],"roles":[{"role":"owner"}],"price":{"amount":"100.0","currency":"USD","unit":"megabyte"},"sla":[{"default_element":"Standard SLA Policy","properties":[{"property":"Uptime Guarantee","value":"99.9"}]}],"support_and_communication":[{"channel":"Email Support","url":"https://support.example.com"}],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}],"contract_test":{"status":"pass","last_tested_time":"last_tested_time","message":"message"},"servers":[{"server":"server","asset":{"id":"id","name":"name"},"connection_id":"connection_id","type":"type","description":"description","environment":"environment","account":"account","catalog":"catalog","database":"database","dataset":"dataset","delimiter":"delimiter","endpoint_url":"endpoint_url","format":"format","host":"host","location":"location","path":"path","port":"port","project":"project","region":"region","region_name":"region_name","schema":"schema","service_name":"service_name","staging_dir":"staging_dir","stream":"stream","warehouse":"warehouse","roles":["roles"],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}]}],"schema":[{"asset_id":"2b0bf220-079c-11ee-be56-0242ac120002","connection_id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","description":"description","connection_path":"connection_path","physical_type":"physical_type","properties":[{"name":"name","type":{"type":"type","length":"length","scale":"scale","nullable":"nullable","signed":"signed","native_type":"native_type"},"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}],"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}]}],"domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"parts_out":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"type":"data_asset"},"delivery_methods":[{"id":"09cf5fcc-cb9d-4995-a8e4-16517b25229f","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"getproperties":{"producer_input":{"engine_details":{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]},"engines":[{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]}]}}}]}],"workflows":{"order_access_request":{"task_assignee_users":["task_assignee_users"],"pre_approved_users":["pre_approved_users"],"custom_workflow_definition":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"}}},"dataview_enabled":true,"comments":"Comments by a producer that are provided either at the time of data product version creation or retiring","access_control":{"owner":"IBMid-696000KYV9"},"last_updated_at":"2019-01-01T12:00:00.000Z","sub_container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd"},"is_restricted":false,"id":"2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd","asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}}]}' + mock_response2 = '{"total_count":2,"limit":1,"releases":[{"version":"1.0.0","state":"draft","data_product":{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"name":"My Data Product","description":"This is a description of My Data Product.","tags":["tags"],"use_cases":[{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}],"types":["data"],"contract_terms":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"id":"id","documents":[{"url":"url","type":"terms_and_conditions","name":"name","id":"2b0bf220-079c-11ee-be56-0242ac120002","attachment":{"id":"id"},"upload_url":"upload_url"}],"error_msg":"error_msg","overview":{"api_version":"v3.0.1","kind":"DataContract","name":"Sample Data Contract","version":"0.0.0","domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"more_info":"List of links to sources that provide more details on the data contract."},"description":{"purpose":"Used for customer behavior analysis.","limitations":"Data cannot be used for marketing.","usage":"Data should be used only for analytics.","more_info":[{"type":"privacy-statement","url":"https://moreinfo.example.com"}],"custom_properties":"{\\"property1\\":\\"value1\\"}"},"organization":[{"user_id":"IBMid-691000IN4G","role":"owner"}],"roles":[{"role":"owner"}],"price":{"amount":"100.0","currency":"USD","unit":"megabyte"},"sla":[{"default_element":"Standard SLA Policy","properties":[{"property":"Uptime Guarantee","value":"99.9"}]}],"support_and_communication":[{"channel":"Email Support","url":"https://support.example.com"}],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}],"contract_test":{"status":"pass","last_tested_time":"last_tested_time","message":"message"},"servers":[{"server":"server","asset":{"id":"id","name":"name"},"connection_id":"connection_id","type":"type","description":"description","environment":"environment","account":"account","catalog":"catalog","database":"database","dataset":"dataset","delimiter":"delimiter","endpoint_url":"endpoint_url","format":"format","host":"host","location":"location","path":"path","port":"port","project":"project","region":"region","region_name":"region_name","schema":"schema","service_name":"service_name","staging_dir":"staging_dir","stream":"stream","warehouse":"warehouse","roles":["roles"],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}]}],"schema":[{"asset_id":"2b0bf220-079c-11ee-be56-0242ac120002","connection_id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","description":"description","connection_path":"connection_path","physical_type":"physical_type","properties":[{"name":"name","type":{"type":"type","length":"length","scale":"scale","nullable":"nullable","signed":"signed","native_type":"native_type"},"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}],"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}]}],"domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"parts_out":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"type":"data_asset"},"delivery_methods":[{"id":"09cf5fcc-cb9d-4995-a8e4-16517b25229f","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"getproperties":{"producer_input":{"engine_details":{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]},"engines":[{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]}]}}}]}],"workflows":{"order_access_request":{"task_assignee_users":["task_assignee_users"],"pre_approved_users":["pre_approved_users"],"custom_workflow_definition":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"}}},"dataview_enabled":true,"comments":"Comments by a producer that are provided either at the time of data product version creation or retiring","access_control":{"owner":"IBMid-696000KYV9"},"last_updated_at":"2019-01-01T12:00:00.000Z","sub_container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd"},"is_restricted":false,"id":"2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd","asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response1, + content_type='application/json', + status=200, + ) + responses.add( + responses.GET, + url, + body=mock_response2, + content_type='application/json', + status=200, + ) + + # Exercise the pager class for this operation + pager = DataProductReleasesPager( + client=_service, + data_product_id='testString', + asset_container_id='testString', + state=['available'], + version='testString', + limit=10, + ) + all_results = pager.get_all() + assert all_results is not None + assert len(all_results) == 2 + + +class TestRetireDataProductRelease: + """ + Test Class for retire_data_product_release + """ + + @responses.activate + def test_retire_data_product_release_all_params(self): + """ + retire_data_product_release() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/retire') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + revoke_access = False + start_at = 'testString' + + # Invoke method + response = _service.retire_data_product_release( + data_product_id, + release_id, + revoke_access=revoke_access, + start_at=start_at, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'revoke_access={}'.format('true' if revoke_access else 'false') in query_string + assert 'start_at={}'.format(start_at) in query_string + + def test_retire_data_product_release_all_params_with_retries(self): + # Enable retries and run test_retire_data_product_release_all_params. + _service.enable_retries() + self.test_retire_data_product_release_all_params() + + # Disable retries and run test_retire_data_product_release_all_params. + _service.disable_retries() + self.test_retire_data_product_release_all_params() + + @responses.activate + def test_retire_data_product_release_required_params(self): + """ + test_retire_data_product_release_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/retire') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + + # Invoke method + response = _service.retire_data_product_release( + data_product_id, + release_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_retire_data_product_release_required_params_with_retries(self): + # Enable retries and run test_retire_data_product_release_required_params. + _service.enable_retries() + self.test_retire_data_product_release_required_params() + + # Disable retries and run test_retire_data_product_release_required_params. + _service.disable_retries() + self.test_retire_data_product_release_required_params() + + @responses.activate + def test_retire_data_product_release_value_error(self): + """ + test_retire_data_product_release_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/retire') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "release_id": release_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.retire_data_product_release(**req_copy) + + def test_retire_data_product_release_value_error_with_retries(self): + # Enable retries and run test_retire_data_product_release_value_error. + _service.enable_retries() + self.test_retire_data_product_release_value_error() + + # Disable retries and run test_retire_data_product_release_value_error. + _service.disable_retries() + self.test_retire_data_product_release_value_error() + + +class TestCreateRevokeAccessProcess: + """ + Test Class for create_revoke_access_process + """ + + @responses.activate + def test_create_revoke_access_process_all_params(self): + """ + create_revoke_access_process() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/revoke_access') + mock_response = '{"message": "message"}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=202, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + body = io.BytesIO(b'This is a mock file.').getvalue() + content_type = 'testString' + + # Invoke method + response = _service.create_revoke_access_process( + data_product_id, + release_id, + body=body, + content_type=content_type, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 202 + # Validate body params + + def test_create_revoke_access_process_all_params_with_retries(self): + # Enable retries and run test_create_revoke_access_process_all_params. + _service.enable_retries() + self.test_create_revoke_access_process_all_params() + + # Disable retries and run test_create_revoke_access_process_all_params. + _service.disable_retries() + self.test_create_revoke_access_process_all_params() + + @responses.activate + def test_create_revoke_access_process_required_params(self): + """ + test_create_revoke_access_process_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/revoke_access') + mock_response = '{"message": "message"}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=202, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + + # Invoke method + response = _service.create_revoke_access_process( + data_product_id, + release_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 202 + + def test_create_revoke_access_process_required_params_with_retries(self): + # Enable retries and run test_create_revoke_access_process_required_params. + _service.enable_retries() + self.test_create_revoke_access_process_required_params() + + # Disable retries and run test_create_revoke_access_process_required_params. + _service.disable_retries() + self.test_create_revoke_access_process_required_params() + + @responses.activate + def test_create_revoke_access_process_value_error(self): + """ + test_create_revoke_access_process_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/revoke_access') + mock_response = '{"message": "message"}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=202, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "release_id": release_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.create_revoke_access_process(**req_copy) + + def test_create_revoke_access_process_value_error_with_retries(self): + # Enable retries and run test_create_revoke_access_process_value_error. + _service.enable_retries() + self.test_create_revoke_access_process_value_error() + + # Disable retries and run test_create_revoke_access_process_value_error. + _service.disable_retries() + self.test_create_revoke_access_process_value_error() + + +# endregion +############################################################################## +# End of Service: DataProductReleases +############################################################################## + +############################################################################## +# Start of Service: DataProductContractTemplates +############################################################################## +# region + + +class TestNewInstance: + """ + Test Class for new_instance + """ + + def test_new_instance(self): + """ + new_instance() + """ + os.environ['TEST_SERVICE_AUTH_TYPE'] = 'noAuth' + + service = DphV1.new_instance( + service_name='TEST_SERVICE', + ) + + assert service is not None + assert isinstance(service, DphV1) + + def test_new_instance_without_authenticator(self): + """ + new_instance_without_authenticator() + """ + with pytest.raises(ValueError, match='authenticator must be provided'): + service = DphV1.new_instance( + service_name='TEST_SERVICE_NOT_FOUND', + ) + + +class TestListDataProductContractTemplate: + """ + Test Class for list_data_product_contract_template + """ + + @responses.activate + def test_list_data_product_contract_template_all_params(self): + """ + list_data_product_contract_template() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates') + mock_response = '{"contract_templates": [{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "id": "20aa7c97-cfcc-4d16-ae76-2ca1847ce733", "creator_id": "IBMid-123456ABC", "created_at": "2025-06-26T12:30:20.000Z", "name": "Sample Data Contract Template", "error": {"code": "code", "message": "message"}, "contract_terms": {"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + container_id = 'testString' + contract_template_name = 'testString' + domain_ids = 'testString' + + # Invoke method + response = _service.list_data_product_contract_template( + container_id=container_id, + contract_template_name=contract_template_name, + domain_ids=domain_ids, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + assert 'contract_template.name={}'.format(contract_template_name) in query_string + assert 'domain.ids={}'.format(domain_ids) in query_string + + def test_list_data_product_contract_template_all_params_with_retries(self): + # Enable retries and run test_list_data_product_contract_template_all_params. + _service.enable_retries() + self.test_list_data_product_contract_template_all_params() + + # Disable retries and run test_list_data_product_contract_template_all_params. + _service.disable_retries() + self.test_list_data_product_contract_template_all_params() + + @responses.activate + def test_list_data_product_contract_template_required_params(self): + """ + test_list_data_product_contract_template_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates') + mock_response = '{"contract_templates": [{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "id": "20aa7c97-cfcc-4d16-ae76-2ca1847ce733", "creator_id": "IBMid-123456ABC", "created_at": "2025-06-26T12:30:20.000Z", "name": "Sample Data Contract Template", "error": {"code": "code", "message": "message"}, "contract_terms": {"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Invoke method + response = _service.list_data_product_contract_template() + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_list_data_product_contract_template_required_params_with_retries(self): + # Enable retries and run test_list_data_product_contract_template_required_params. + _service.enable_retries() + self.test_list_data_product_contract_template_required_params() + + # Disable retries and run test_list_data_product_contract_template_required_params. + _service.disable_retries() + self.test_list_data_product_contract_template_required_params() + + +class TestCreateContractTemplate: + """ + Test Class for create_contract_template + """ + + @responses.activate + def test_create_contract_template_all_params(self): + """ + create_contract_template() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "id": "20aa7c97-cfcc-4d16-ae76-2ca1847ce733", "creator_id": "IBMid-123456ABC", "created_at": "2025-06-26T12:30:20.000Z", "name": "Sample Data Contract Template", "error": {"code": "code", "message": "message"}, "contract_terms": {"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = '531f74a-01c8-4e91-8e29-b018db683c86' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a ErrorMessage model + error_message_model = {} + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = '0094ebe9-abc3-473b-80ea-c777ede095ea' + domain_model['name'] = 'Test Domain New' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://www.moreinfo.example.coms' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Intended purpose for the provided data.' + description_model['limitations'] = 'Technical, compliance, and legal limitations for data use.' + description_model['usage'] = 'Recommended usage of the data.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = 'Custom properties that are not part of the standard.' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'IAM Role' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = '100.00' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'slaproperty' + contract_template_sla_property_model['value'] = 'slavalue' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'sladefaultelement' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'channel' + contract_template_support_and_communication_model['url'] = 'https://www.example.coms' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'propertykey' + contract_template_custom_property_model['value'] = 'propertyvalue' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractTerms model + contract_terms_model = {} + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + # Set up parameter values + container = container_reference_model + id = 'testString' + creator_id = 'testString' + created_at = 'testString' + name = 'Sample Data Contract Template' + error = error_message_model + contract_terms = contract_terms_model + container_id = 'testString' + contract_template_name = 'testString' + domain_ids = 'testString' + + # Invoke method + response = _service.create_contract_template( + container, + id=id, + creator_id=creator_id, + created_at=created_at, + name=name, + error=error, + contract_terms=contract_terms, + container_id=container_id, + contract_template_name=contract_template_name, + domain_ids=domain_ids, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + assert 'contract_template.name={}'.format(contract_template_name) in query_string + assert 'domain.ids={}'.format(domain_ids) in query_string + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['container'] == container_reference_model + assert req_body['id'] == 'testString' + assert req_body['creator_id'] == 'testString' + assert req_body['created_at'] == 'testString' + assert req_body['name'] == 'Sample Data Contract Template' + assert req_body['error'] == error_message_model + assert req_body['contract_terms'] == contract_terms_model + + def test_create_contract_template_all_params_with_retries(self): + # Enable retries and run test_create_contract_template_all_params. + _service.enable_retries() + self.test_create_contract_template_all_params() + + # Disable retries and run test_create_contract_template_all_params. + _service.disable_retries() + self.test_create_contract_template_all_params() + + @responses.activate + def test_create_contract_template_required_params(self): + """ + test_create_contract_template_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "id": "20aa7c97-cfcc-4d16-ae76-2ca1847ce733", "creator_id": "IBMid-123456ABC", "created_at": "2025-06-26T12:30:20.000Z", "name": "Sample Data Contract Template", "error": {"code": "code", "message": "message"}, "contract_terms": {"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = '531f74a-01c8-4e91-8e29-b018db683c86' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a ErrorMessage model + error_message_model = {} + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = '0094ebe9-abc3-473b-80ea-c777ede095ea' + domain_model['name'] = 'Test Domain New' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://www.moreinfo.example.coms' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Intended purpose for the provided data.' + description_model['limitations'] = 'Technical, compliance, and legal limitations for data use.' + description_model['usage'] = 'Recommended usage of the data.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = 'Custom properties that are not part of the standard.' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'IAM Role' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = '100.00' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'slaproperty' + contract_template_sla_property_model['value'] = 'slavalue' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'sladefaultelement' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'channel' + contract_template_support_and_communication_model['url'] = 'https://www.example.coms' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'propertykey' + contract_template_custom_property_model['value'] = 'propertyvalue' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractTerms model + contract_terms_model = {} + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + # Set up parameter values + container = container_reference_model + id = 'testString' + creator_id = 'testString' + created_at = 'testString' + name = 'Sample Data Contract Template' + error = error_message_model + contract_terms = contract_terms_model + + # Invoke method + response = _service.create_contract_template( + container, + id=id, + creator_id=creator_id, + created_at=created_at, + name=name, + error=error, + contract_terms=contract_terms, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['container'] == container_reference_model + assert req_body['id'] == 'testString' + assert req_body['creator_id'] == 'testString' + assert req_body['created_at'] == 'testString' + assert req_body['name'] == 'Sample Data Contract Template' + assert req_body['error'] == error_message_model + assert req_body['contract_terms'] == contract_terms_model + + def test_create_contract_template_required_params_with_retries(self): + # Enable retries and run test_create_contract_template_required_params. + _service.enable_retries() + self.test_create_contract_template_required_params() + + # Disable retries and run test_create_contract_template_required_params. + _service.disable_retries() + self.test_create_contract_template_required_params() + + @responses.activate + def test_create_contract_template_value_error(self): + """ + test_create_contract_template_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "id": "20aa7c97-cfcc-4d16-ae76-2ca1847ce733", "creator_id": "IBMid-123456ABC", "created_at": "2025-06-26T12:30:20.000Z", "name": "Sample Data Contract Template", "error": {"code": "code", "message": "message"}, "contract_terms": {"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = '531f74a-01c8-4e91-8e29-b018db683c86' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a ErrorMessage model + error_message_model = {} + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = '0094ebe9-abc3-473b-80ea-c777ede095ea' + domain_model['name'] = 'Test Domain New' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://www.moreinfo.example.coms' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Intended purpose for the provided data.' + description_model['limitations'] = 'Technical, compliance, and legal limitations for data use.' + description_model['usage'] = 'Recommended usage of the data.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = 'Custom properties that are not part of the standard.' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'IAM Role' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = '100.00' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'slaproperty' + contract_template_sla_property_model['value'] = 'slavalue' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'sladefaultelement' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'channel' + contract_template_support_and_communication_model['url'] = 'https://www.example.coms' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'propertykey' + contract_template_custom_property_model['value'] = 'propertyvalue' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractTerms model + contract_terms_model = {} + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + # Set up parameter values + container = container_reference_model + id = 'testString' + creator_id = 'testString' + created_at = 'testString' + name = 'Sample Data Contract Template' + error = error_message_model + contract_terms = contract_terms_model + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "container": container, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.create_contract_template(**req_copy) + + def test_create_contract_template_value_error_with_retries(self): + # Enable retries and run test_create_contract_template_value_error. + _service.enable_retries() + self.test_create_contract_template_value_error() + + # Disable retries and run test_create_contract_template_value_error. + _service.disable_retries() + self.test_create_contract_template_value_error() + + +class TestGetContractTemplate: + """ + Test Class for get_contract_template + """ + + @responses.activate + def test_get_contract_template_all_params(self): + """ + get_contract_template() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates/testString') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "id": "20aa7c97-cfcc-4d16-ae76-2ca1847ce733", "creator_id": "IBMid-123456ABC", "created_at": "2025-06-26T12:30:20.000Z", "name": "Sample Data Contract Template", "error": {"code": "code", "message": "message"}, "contract_terms": {"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + contract_template_id = 'testString' + container_id = 'testString' + + # Invoke method + response = _service.get_contract_template( + contract_template_id, + container_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + + def test_get_contract_template_all_params_with_retries(self): + # Enable retries and run test_get_contract_template_all_params. + _service.enable_retries() + self.test_get_contract_template_all_params() + + # Disable retries and run test_get_contract_template_all_params. + _service.disable_retries() + self.test_get_contract_template_all_params() + + @responses.activate + def test_get_contract_template_value_error(self): + """ + test_get_contract_template_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates/testString') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "id": "20aa7c97-cfcc-4d16-ae76-2ca1847ce733", "creator_id": "IBMid-123456ABC", "created_at": "2025-06-26T12:30:20.000Z", "name": "Sample Data Contract Template", "error": {"code": "code", "message": "message"}, "contract_terms": {"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + contract_template_id = 'testString' + container_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "contract_template_id": contract_template_id, + "container_id": container_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_contract_template(**req_copy) + + def test_get_contract_template_value_error_with_retries(self): + # Enable retries and run test_get_contract_template_value_error. + _service.enable_retries() + self.test_get_contract_template_value_error() + + # Disable retries and run test_get_contract_template_value_error. + _service.disable_retries() + self.test_get_contract_template_value_error() + + +class TestDeleteDataProductContractTemplate: + """ + Test Class for delete_data_product_contract_template + """ + + @responses.activate + def test_delete_data_product_contract_template_all_params(self): + """ + delete_data_product_contract_template() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates/testString') + responses.add( + responses.DELETE, + url, + status=204, + ) + + # Set up parameter values + contract_template_id = 'testString' + container_id = 'testString' + + # Invoke method + response = _service.delete_data_product_contract_template( + contract_template_id, + container_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 204 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + + def test_delete_data_product_contract_template_all_params_with_retries(self): + # Enable retries and run test_delete_data_product_contract_template_all_params. + _service.enable_retries() + self.test_delete_data_product_contract_template_all_params() + + # Disable retries and run test_delete_data_product_contract_template_all_params. + _service.disable_retries() + self.test_delete_data_product_contract_template_all_params() + + @responses.activate + def test_delete_data_product_contract_template_value_error(self): + """ + test_delete_data_product_contract_template_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates/testString') + responses.add( + responses.DELETE, + url, + status=204, + ) + + # Set up parameter values + contract_template_id = 'testString' + container_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "contract_template_id": contract_template_id, + "container_id": container_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.delete_data_product_contract_template(**req_copy) + + def test_delete_data_product_contract_template_value_error_with_retries(self): + # Enable retries and run test_delete_data_product_contract_template_value_error. + _service.enable_retries() + self.test_delete_data_product_contract_template_value_error() + + # Disable retries and run test_delete_data_product_contract_template_value_error. + _service.disable_retries() + self.test_delete_data_product_contract_template_value_error() + + +class TestUpdateDataProductContractTemplate: + """ + Test Class for update_data_product_contract_template + """ + + @responses.activate + def test_update_data_product_contract_template_all_params(self): + """ + update_data_product_contract_template() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates/testString') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "id": "20aa7c97-cfcc-4d16-ae76-2ca1847ce733", "creator_id": "IBMid-123456ABC", "created_at": "2025-06-26T12:30:20.000Z", "name": "Sample Data Contract Template", "error": {"code": "code", "message": "message"}, "contract_terms": {"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + contract_template_id = 'testString' + container_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Invoke method + response = _service.update_data_product_contract_template( + contract_template_id, + container_id, + json_patch_instructions, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body == json_patch_instructions + + def test_update_data_product_contract_template_all_params_with_retries(self): + # Enable retries and run test_update_data_product_contract_template_all_params. + _service.enable_retries() + self.test_update_data_product_contract_template_all_params() + + # Disable retries and run test_update_data_product_contract_template_all_params. + _service.disable_retries() + self.test_update_data_product_contract_template_all_params() + + @responses.activate + def test_update_data_product_contract_template_value_error(self): + """ + test_update_data_product_contract_template_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates/testString') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "id": "20aa7c97-cfcc-4d16-ae76-2ca1847ce733", "creator_id": "IBMid-123456ABC", "created_at": "2025-06-26T12:30:20.000Z", "name": "Sample Data Contract Template", "error": {"code": "code", "message": "message"}, "contract_terms": {"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + contract_template_id = 'testString' + container_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "contract_template_id": contract_template_id, + "container_id": container_id, + "json_patch_instructions": json_patch_instructions, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.update_data_product_contract_template(**req_copy) + + def test_update_data_product_contract_template_value_error_with_retries(self): + # Enable retries and run test_update_data_product_contract_template_value_error. + _service.enable_retries() + self.test_update_data_product_contract_template_value_error() + + # Disable retries and run test_update_data_product_contract_template_value_error. + _service.disable_retries() + self.test_update_data_product_contract_template_value_error() + + +# endregion +############################################################################## +# End of Service: DataProductContractTemplates +############################################################################## + +############################################################################## +# Start of Service: DataProductDomains +############################################################################## +# region + + +class TestNewInstance: + """ + Test Class for new_instance + """ + + def test_new_instance(self): + """ + new_instance() + """ + os.environ['TEST_SERVICE_AUTH_TYPE'] = 'noAuth' + + service = DphV1.new_instance( + service_name='TEST_SERVICE', + ) + + assert service is not None + assert isinstance(service, DphV1) + + def test_new_instance_without_authenticator(self): + """ + new_instance_without_authenticator() + """ + with pytest.raises(ValueError, match='authenticator must be provided'): + service = DphV1.new_instance( + service_name='TEST_SERVICE_NOT_FOUND', + ) + + +class TestListDataProductDomains: + """ + Test Class for list_data_product_domains + """ + + @responses.activate + def test_list_data_product_domains_all_params(self): + """ + list_data_product_domains() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains') + mock_response = '{"domains": [{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "name": "Operations", "description": "This is a description of the data product domain.", "id": "id", "created_by": "created_by", "member_roles": {"user_iam_id": "user_iam_id", "roles": ["roles"]}, "properties": {"value": "value"}, "sub_domains": [{"name": "Operations", "id": "id", "description": "description"}], "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + container_id = 'testString' + include_subdomains = True + + # Invoke method + response = _service.list_data_product_domains( + container_id=container_id, + include_subdomains=include_subdomains, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + assert 'include_subdomains={}'.format('true' if include_subdomains else 'false') in query_string + + def test_list_data_product_domains_all_params_with_retries(self): + # Enable retries and run test_list_data_product_domains_all_params. + _service.enable_retries() + self.test_list_data_product_domains_all_params() + + # Disable retries and run test_list_data_product_domains_all_params. + _service.disable_retries() + self.test_list_data_product_domains_all_params() + + @responses.activate + def test_list_data_product_domains_required_params(self): + """ + test_list_data_product_domains_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains') + mock_response = '{"domains": [{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "name": "Operations", "description": "This is a description of the data product domain.", "id": "id", "created_by": "created_by", "member_roles": {"user_iam_id": "user_iam_id", "roles": ["roles"]}, "properties": {"value": "value"}, "sub_domains": [{"name": "Operations", "id": "id", "description": "description"}], "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Invoke method + response = _service.list_data_product_domains() + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_list_data_product_domains_required_params_with_retries(self): + # Enable retries and run test_list_data_product_domains_required_params. + _service.enable_retries() + self.test_list_data_product_domains_required_params() + + # Disable retries and run test_list_data_product_domains_required_params. + _service.disable_retries() + self.test_list_data_product_domains_required_params() + + +class TestCreateDataProductDomain: + """ + Test Class for create_data_product_domain + """ + + @responses.activate + def test_create_data_product_domain_all_params(self): + """ + create_data_product_domain() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "name": "Operations", "description": "This is a description of the data product domain.", "id": "id", "created_by": "created_by", "member_roles": {"user_iam_id": "user_iam_id", "roles": ["roles"]}, "properties": {"value": "value"}, "sub_domains": [{"name": "Operations", "id": "id", "description": "description"}], "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'ed580171-a6e4-4b93-973f-ae2f2f62991b' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a ErrorExtraResource model + error_extra_resource_model = {} + error_extra_resource_model['id'] = 'testString' + error_extra_resource_model['timestamp'] = '2019-01-01T12:00:00Z' + error_extra_resource_model['environment_name'] = 'testString' + error_extra_resource_model['http_status'] = 0 + error_extra_resource_model['source_cluster'] = 0 + error_extra_resource_model['source_component'] = 0 + error_extra_resource_model['transaction_id'] = 0 + + # Construct a dict representation of a ErrorModelResource model + error_model_resource_model = {} + error_model_resource_model['code'] = 'request_body_error' + error_model_resource_model['message'] = 'testString' + error_model_resource_model['extra'] = error_extra_resource_model + error_model_resource_model['more_info'] = 'testString' + + # Construct a dict representation of a MemberRolesSchema model + member_roles_schema_model = {} + member_roles_schema_model['user_iam_id'] = 'testString' + member_roles_schema_model['roles'] = ['testString'] + + # Construct a dict representation of a PropertiesSchema model + properties_schema_model = {} + properties_schema_model['value'] = 'testString' + + # Construct a dict representation of a InitializeSubDomain model + initialize_sub_domain_model = {} + initialize_sub_domain_model['name'] = 'Sub domain 1' + initialize_sub_domain_model['id'] = 'testString' + initialize_sub_domain_model['description'] = 'New sub domain 1' + + # Construct a dict representation of a ContainerIdentity model + container_identity_model = {} + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Set up parameter values + container = container_reference_model + trace = 'testString' + errors = [error_model_resource_model] + name = 'Test domain' + description = 'The sample description for new domain' + id = 'testString' + created_by = 'testString' + member_roles = member_roles_schema_model + properties = properties_schema_model + sub_domains = [initialize_sub_domain_model] + sub_container = container_identity_model + link_to_subcontainers = False + + # Invoke method + response = _service.create_data_product_domain( + container, + trace=trace, + errors=errors, + name=name, + description=description, + id=id, + created_by=created_by, + member_roles=member_roles, + properties=properties, + sub_domains=sub_domains, + sub_container=sub_container, + link_to_subcontainers=link_to_subcontainers, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'link_to_subcontainers={}'.format('true' if link_to_subcontainers else 'false') in query_string + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['container'] == container_reference_model + assert req_body['trace'] == 'testString' + assert req_body['errors'] == [error_model_resource_model] + assert req_body['name'] == 'Test domain' + assert req_body['description'] == 'The sample description for new domain' + assert req_body['id'] == 'testString' + assert req_body['created_by'] == 'testString' + assert req_body['member_roles'] == member_roles_schema_model + assert req_body['properties'] == properties_schema_model + assert req_body['sub_domains'] == [initialize_sub_domain_model] + assert req_body['sub_container'] == container_identity_model + + def test_create_data_product_domain_all_params_with_retries(self): + # Enable retries and run test_create_data_product_domain_all_params. + _service.enable_retries() + self.test_create_data_product_domain_all_params() + + # Disable retries and run test_create_data_product_domain_all_params. + _service.disable_retries() + self.test_create_data_product_domain_all_params() + + @responses.activate + def test_create_data_product_domain_required_params(self): + """ + test_create_data_product_domain_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "name": "Operations", "description": "This is a description of the data product domain.", "id": "id", "created_by": "created_by", "member_roles": {"user_iam_id": "user_iam_id", "roles": ["roles"]}, "properties": {"value": "value"}, "sub_domains": [{"name": "Operations", "id": "id", "description": "description"}], "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'ed580171-a6e4-4b93-973f-ae2f2f62991b' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a ErrorExtraResource model + error_extra_resource_model = {} + error_extra_resource_model['id'] = 'testString' + error_extra_resource_model['timestamp'] = '2019-01-01T12:00:00Z' + error_extra_resource_model['environment_name'] = 'testString' + error_extra_resource_model['http_status'] = 0 + error_extra_resource_model['source_cluster'] = 0 + error_extra_resource_model['source_component'] = 0 + error_extra_resource_model['transaction_id'] = 0 + + # Construct a dict representation of a ErrorModelResource model + error_model_resource_model = {} + error_model_resource_model['code'] = 'request_body_error' + error_model_resource_model['message'] = 'testString' + error_model_resource_model['extra'] = error_extra_resource_model + error_model_resource_model['more_info'] = 'testString' + + # Construct a dict representation of a MemberRolesSchema model + member_roles_schema_model = {} + member_roles_schema_model['user_iam_id'] = 'testString' + member_roles_schema_model['roles'] = ['testString'] + + # Construct a dict representation of a PropertiesSchema model + properties_schema_model = {} + properties_schema_model['value'] = 'testString' + + # Construct a dict representation of a InitializeSubDomain model + initialize_sub_domain_model = {} + initialize_sub_domain_model['name'] = 'Sub domain 1' + initialize_sub_domain_model['id'] = 'testString' + initialize_sub_domain_model['description'] = 'New sub domain 1' + + # Construct a dict representation of a ContainerIdentity model + container_identity_model = {} + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Set up parameter values + container = container_reference_model + trace = 'testString' + errors = [error_model_resource_model] + name = 'Test domain' + description = 'The sample description for new domain' + id = 'testString' + created_by = 'testString' + member_roles = member_roles_schema_model + properties = properties_schema_model + sub_domains = [initialize_sub_domain_model] + sub_container = container_identity_model + + # Invoke method + response = _service.create_data_product_domain( + container, + trace=trace, + errors=errors, + name=name, + description=description, + id=id, + created_by=created_by, + member_roles=member_roles, + properties=properties, + sub_domains=sub_domains, + sub_container=sub_container, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['container'] == container_reference_model + assert req_body['trace'] == 'testString' + assert req_body['errors'] == [error_model_resource_model] + assert req_body['name'] == 'Test domain' + assert req_body['description'] == 'The sample description for new domain' + assert req_body['id'] == 'testString' + assert req_body['created_by'] == 'testString' + assert req_body['member_roles'] == member_roles_schema_model + assert req_body['properties'] == properties_schema_model + assert req_body['sub_domains'] == [initialize_sub_domain_model] + assert req_body['sub_container'] == container_identity_model + + def test_create_data_product_domain_required_params_with_retries(self): + # Enable retries and run test_create_data_product_domain_required_params. + _service.enable_retries() + self.test_create_data_product_domain_required_params() + + # Disable retries and run test_create_data_product_domain_required_params. + _service.disable_retries() + self.test_create_data_product_domain_required_params() + + @responses.activate + def test_create_data_product_domain_value_error(self): + """ + test_create_data_product_domain_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "name": "Operations", "description": "This is a description of the data product domain.", "id": "id", "created_by": "created_by", "member_roles": {"user_iam_id": "user_iam_id", "roles": ["roles"]}, "properties": {"value": "value"}, "sub_domains": [{"name": "Operations", "id": "id", "description": "description"}], "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'ed580171-a6e4-4b93-973f-ae2f2f62991b' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a ErrorExtraResource model + error_extra_resource_model = {} + error_extra_resource_model['id'] = 'testString' + error_extra_resource_model['timestamp'] = '2019-01-01T12:00:00Z' + error_extra_resource_model['environment_name'] = 'testString' + error_extra_resource_model['http_status'] = 0 + error_extra_resource_model['source_cluster'] = 0 + error_extra_resource_model['source_component'] = 0 + error_extra_resource_model['transaction_id'] = 0 + + # Construct a dict representation of a ErrorModelResource model + error_model_resource_model = {} + error_model_resource_model['code'] = 'request_body_error' + error_model_resource_model['message'] = 'testString' + error_model_resource_model['extra'] = error_extra_resource_model + error_model_resource_model['more_info'] = 'testString' + + # Construct a dict representation of a MemberRolesSchema model + member_roles_schema_model = {} + member_roles_schema_model['user_iam_id'] = 'testString' + member_roles_schema_model['roles'] = ['testString'] + + # Construct a dict representation of a PropertiesSchema model + properties_schema_model = {} + properties_schema_model['value'] = 'testString' + + # Construct a dict representation of a InitializeSubDomain model + initialize_sub_domain_model = {} + initialize_sub_domain_model['name'] = 'Sub domain 1' + initialize_sub_domain_model['id'] = 'testString' + initialize_sub_domain_model['description'] = 'New sub domain 1' + + # Construct a dict representation of a ContainerIdentity model + container_identity_model = {} + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Set up parameter values + container = container_reference_model + trace = 'testString' + errors = [error_model_resource_model] + name = 'Test domain' + description = 'The sample description for new domain' + id = 'testString' + created_by = 'testString' + member_roles = member_roles_schema_model + properties = properties_schema_model + sub_domains = [initialize_sub_domain_model] + sub_container = container_identity_model + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "container": container, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.create_data_product_domain(**req_copy) + + def test_create_data_product_domain_value_error_with_retries(self): + # Enable retries and run test_create_data_product_domain_value_error. + _service.enable_retries() + self.test_create_data_product_domain_value_error() + + # Disable retries and run test_create_data_product_domain_value_error. + _service.disable_retries() + self.test_create_data_product_domain_value_error() + + +class TestCreateDataProductSubdomain: + """ + Test Class for create_data_product_subdomain + """ + + @responses.activate + def test_create_data_product_subdomain_all_params(self): + """ + create_data_product_subdomain() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString/subdomains') + mock_response = '{"name": "Operations", "id": "id", "description": "description"}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Set up parameter values + domain_id = 'testString' + container_id = 'testString' + name = 'Sub domain 1' + id = 'testString' + description = 'New sub domain 1' + + # Invoke method + response = _service.create_data_product_subdomain( + domain_id, + container_id, + name=name, + id=id, + description=description, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['name'] == 'Sub domain 1' + assert req_body['id'] == 'testString' + assert req_body['description'] == 'New sub domain 1' + + def test_create_data_product_subdomain_all_params_with_retries(self): + # Enable retries and run test_create_data_product_subdomain_all_params. + _service.enable_retries() + self.test_create_data_product_subdomain_all_params() + + # Disable retries and run test_create_data_product_subdomain_all_params. + _service.disable_retries() + self.test_create_data_product_subdomain_all_params() + + @responses.activate + def test_create_data_product_subdomain_value_error(self): + """ + test_create_data_product_subdomain_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString/subdomains') + mock_response = '{"name": "Operations", "id": "id", "description": "description"}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Set up parameter values + domain_id = 'testString' + container_id = 'testString' + name = 'Sub domain 1' + id = 'testString' + description = 'New sub domain 1' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "domain_id": domain_id, + "container_id": container_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.create_data_product_subdomain(**req_copy) + + def test_create_data_product_subdomain_value_error_with_retries(self): + # Enable retries and run test_create_data_product_subdomain_value_error. + _service.enable_retries() + self.test_create_data_product_subdomain_value_error() + + # Disable retries and run test_create_data_product_subdomain_value_error. + _service.disable_retries() + self.test_create_data_product_subdomain_value_error() + + +class TestGetDomain: + """ + Test Class for get_domain + """ + + @responses.activate + def test_get_domain_all_params(self): + """ + get_domain() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "name": "Operations", "description": "This is a description of the data product domain.", "id": "id", "created_by": "created_by", "member_roles": {"user_iam_id": "user_iam_id", "roles": ["roles"]}, "properties": {"value": "value"}, "sub_domains": [{"name": "Operations", "id": "id", "description": "description"}], "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + domain_id = 'testString' + + # Invoke method + response = _service.get_domain( + domain_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_domain_all_params_with_retries(self): + # Enable retries and run test_get_domain_all_params. + _service.enable_retries() + self.test_get_domain_all_params() + + # Disable retries and run test_get_domain_all_params. + _service.disable_retries() + self.test_get_domain_all_params() + + @responses.activate + def test_get_domain_value_error(self): + """ + test_get_domain_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "name": "Operations", "description": "This is a description of the data product domain.", "id": "id", "created_by": "created_by", "member_roles": {"user_iam_id": "user_iam_id", "roles": ["roles"]}, "properties": {"value": "value"}, "sub_domains": [{"name": "Operations", "id": "id", "description": "description"}], "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + domain_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "domain_id": domain_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_domain(**req_copy) + + def test_get_domain_value_error_with_retries(self): + # Enable retries and run test_get_domain_value_error. + _service.enable_retries() + self.test_get_domain_value_error() + + # Disable retries and run test_get_domain_value_error. + _service.disable_retries() + self.test_get_domain_value_error() + + +class TestDeleteDomain: + """ + Test Class for delete_domain + """ + + @responses.activate + def test_delete_domain_all_params(self): + """ + delete_domain() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString') + responses.add( + responses.DELETE, + url, + status=204, + ) + + # Set up parameter values + domain_id = 'testString' + + # Invoke method + response = _service.delete_domain( + domain_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 204 + + def test_delete_domain_all_params_with_retries(self): + # Enable retries and run test_delete_domain_all_params. + _service.enable_retries() + self.test_delete_domain_all_params() + + # Disable retries and run test_delete_domain_all_params. + _service.disable_retries() + self.test_delete_domain_all_params() + + @responses.activate + def test_delete_domain_value_error(self): + """ + test_delete_domain_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString') + responses.add( + responses.DELETE, + url, + status=204, + ) + + # Set up parameter values + domain_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "domain_id": domain_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.delete_domain(**req_copy) + + def test_delete_domain_value_error_with_retries(self): + # Enable retries and run test_delete_domain_value_error. + _service.enable_retries() + self.test_delete_domain_value_error() + + # Disable retries and run test_delete_domain_value_error. + _service.disable_retries() + self.test_delete_domain_value_error() + + +class TestUpdateDataProductDomain: + """ + Test Class for update_data_product_domain + """ + + @responses.activate + def test_update_data_product_domain_all_params(self): + """ + update_data_product_domain() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "name": "Operations", "description": "This is a description of the data product domain.", "id": "id", "created_by": "created_by", "member_roles": {"user_iam_id": "user_iam_id", "roles": ["roles"]}, "properties": {"value": "value"}, "sub_domains": [{"name": "Operations", "id": "id", "description": "description"}], "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + domain_id = 'testString' + container_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Invoke method + response = _service.update_data_product_domain( + domain_id, + container_id, + json_patch_instructions, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body == json_patch_instructions + + def test_update_data_product_domain_all_params_with_retries(self): + # Enable retries and run test_update_data_product_domain_all_params. + _service.enable_retries() + self.test_update_data_product_domain_all_params() + + # Disable retries and run test_update_data_product_domain_all_params. + _service.disable_retries() + self.test_update_data_product_domain_all_params() + + @responses.activate + def test_update_data_product_domain_value_error(self): + """ + test_update_data_product_domain_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "name": "Operations", "description": "This is a description of the data product domain.", "id": "id", "created_by": "created_by", "member_roles": {"user_iam_id": "user_iam_id", "roles": ["roles"]}, "properties": {"value": "value"}, "sub_domains": [{"name": "Operations", "id": "id", "description": "description"}], "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + domain_id = 'testString' + container_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "domain_id": domain_id, + "container_id": container_id, + "json_patch_instructions": json_patch_instructions, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.update_data_product_domain(**req_copy) + + def test_update_data_product_domain_value_error_with_retries(self): + # Enable retries and run test_update_data_product_domain_value_error. + _service.enable_retries() + self.test_update_data_product_domain_value_error() + + # Disable retries and run test_update_data_product_domain_value_error. + _service.disable_retries() + self.test_update_data_product_domain_value_error() + + +class TestGetDataProductByDomain: + """ + Test Class for get_data_product_by_domain + """ + + @responses.activate + def test_get_data_product_by_domain_all_params(self): + """ + get_data_product_by_domain() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString/data_products') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "data_product_versions": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + domain_id = 'testString' + container_id = 'testString' + + # Invoke method + response = _service.get_data_product_by_domain( + domain_id, + container_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + + def test_get_data_product_by_domain_all_params_with_retries(self): + # Enable retries and run test_get_data_product_by_domain_all_params. + _service.enable_retries() + self.test_get_data_product_by_domain_all_params() + + # Disable retries and run test_get_data_product_by_domain_all_params. + _service.disable_retries() + self.test_get_data_product_by_domain_all_params() + + @responses.activate + def test_get_data_product_by_domain_value_error(self): + """ + test_get_data_product_by_domain_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString/data_products') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "data_product_versions": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + domain_id = 'testString' + container_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "domain_id": domain_id, + "container_id": container_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_data_product_by_domain(**req_copy) + + def test_get_data_product_by_domain_value_error_with_retries(self): + # Enable retries and run test_get_data_product_by_domain_value_error. + _service.enable_retries() + self.test_get_data_product_by_domain_value_error() + + # Disable retries and run test_get_data_product_by_domain_value_error. + _service.disable_retries() + self.test_get_data_product_by_domain_value_error() + + +# endregion +############################################################################## +# End of Service: DataProductDomains +############################################################################## + +############################################################################## +# Start of Service: BucketServices +############################################################################## +# region + + +class TestNewInstance: + """ + Test Class for new_instance + """ + + def test_new_instance(self): + """ + new_instance() + """ + os.environ['TEST_SERVICE_AUTH_TYPE'] = 'noAuth' + + service = DphV1.new_instance( + service_name='TEST_SERVICE', + ) + + assert service is not None + assert isinstance(service, DphV1) + + def test_new_instance_without_authenticator(self): + """ + new_instance_without_authenticator() + """ + with pytest.raises(ValueError, match='authenticator must be provided'): + service = DphV1.new_instance( + service_name='TEST_SERVICE_NOT_FOUND', + ) + + +class TestCreateS3Bucket: + """ + Test Class for create_s3_bucket + """ + + @responses.activate + def test_create_s3_bucket_all_params(self): + """ + create_s3_bucket() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/bucket') + mock_response = '{"bucket_name": "bucket_name", "bucket_location": "bucket_location", "role_arn": "role_arn", "bucket_type": "bucket_type", "shared": true}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Set up parameter values + is_shared = True + + # Invoke method + response = _service.create_s3_bucket( + is_shared, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'is_shared={}'.format('true' if is_shared else 'false') in query_string + + def test_create_s3_bucket_all_params_with_retries(self): + # Enable retries and run test_create_s3_bucket_all_params. + _service.enable_retries() + self.test_create_s3_bucket_all_params() + + # Disable retries and run test_create_s3_bucket_all_params. + _service.disable_retries() + self.test_create_s3_bucket_all_params() + + @responses.activate + def test_create_s3_bucket_value_error(self): + """ + test_create_s3_bucket_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/bucket') + mock_response = '{"bucket_name": "bucket_name", "bucket_location": "bucket_location", "role_arn": "role_arn", "bucket_type": "bucket_type", "shared": true}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Set up parameter values + is_shared = True + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "is_shared": is_shared, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.create_s3_bucket(**req_copy) + + def test_create_s3_bucket_value_error_with_retries(self): + # Enable retries and run test_create_s3_bucket_value_error. + _service.enable_retries() + self.test_create_s3_bucket_value_error() + + # Disable retries and run test_create_s3_bucket_value_error. + _service.disable_retries() + self.test_create_s3_bucket_value_error() + + +class TestGetS3BucketValidation: + """ + Test Class for get_s3_bucket_validation + """ + + @responses.activate + def test_get_s3_bucket_validation_all_params(self): + """ + get_s3_bucket_validation() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/bucket/validate/testString') + mock_response = '{"bucket_exists": false}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + bucket_name = 'testString' + + # Invoke method + response = _service.get_s3_bucket_validation( + bucket_name, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_s3_bucket_validation_all_params_with_retries(self): + # Enable retries and run test_get_s3_bucket_validation_all_params. + _service.enable_retries() + self.test_get_s3_bucket_validation_all_params() + + # Disable retries and run test_get_s3_bucket_validation_all_params. + _service.disable_retries() + self.test_get_s3_bucket_validation_all_params() + + @responses.activate + def test_get_s3_bucket_validation_value_error(self): + """ + test_get_s3_bucket_validation_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/bucket/validate/testString') + mock_response = '{"bucket_exists": false}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + bucket_name = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "bucket_name": bucket_name, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_s3_bucket_validation(**req_copy) + + def test_get_s3_bucket_validation_value_error_with_retries(self): + # Enable retries and run test_get_s3_bucket_validation_value_error. + _service.enable_retries() + self.test_get_s3_bucket_validation_value_error() + + # Disable retries and run test_get_s3_bucket_validation_value_error. + _service.disable_retries() + self.test_get_s3_bucket_validation_value_error() + + +# endregion +############################################################################## +# End of Service: BucketServices +############################################################################## + +############################################################################## +# Start of Service: DataProductRevokeAccessJobRuns +############################################################################## +# region + + +class TestNewInstance: + """ + Test Class for new_instance + """ + + def test_new_instance(self): + """ + new_instance() + """ + os.environ['TEST_SERVICE_AUTH_TYPE'] = 'noAuth' + + service = DphV1.new_instance( + service_name='TEST_SERVICE', + ) + + assert service is not None + assert isinstance(service, DphV1) + + def test_new_instance_without_authenticator(self): + """ + new_instance_without_authenticator() + """ + with pytest.raises(ValueError, match='authenticator must be provided'): + service = DphV1.new_instance( + service_name='TEST_SERVICE_NOT_FOUND', + ) + + +class TestGetRevokeAccessProcessState: + """ + Test Class for get_revoke_access_process_state + """ + + @responses.activate + def test_get_revoke_access_process_state_all_params(self): + """ + get_revoke_access_process_state() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_product_revoke_access/job_runs') + mock_response = '{"results": [{"metadata": {"anyKey": "anyValue"}, "entity": {"anyKey": "anyValue"}}], "total_count": 42, "next": {"query": "ibm_data_product_revoke_access.state:(SCHEDULED OR FAILED)", "limit": 1, "bookmark": "MQ==", "include": "entity", "skip": 0}}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + release_id = 'testString' + limit = 200 + start = 'testString' + + # Invoke method + response = _service.get_revoke_access_process_state( + release_id, + limit=limit, + start=start, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'release_id={}'.format(release_id) in query_string + assert 'limit={}'.format(limit) in query_string + assert 'start={}'.format(start) in query_string + + def test_get_revoke_access_process_state_all_params_with_retries(self): + # Enable retries and run test_get_revoke_access_process_state_all_params. + _service.enable_retries() + self.test_get_revoke_access_process_state_all_params() + + # Disable retries and run test_get_revoke_access_process_state_all_params. + _service.disable_retries() + self.test_get_revoke_access_process_state_all_params() + + @responses.activate + def test_get_revoke_access_process_state_required_params(self): + """ + test_get_revoke_access_process_state_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_product_revoke_access/job_runs') + mock_response = '{"results": [{"metadata": {"anyKey": "anyValue"}, "entity": {"anyKey": "anyValue"}}], "total_count": 42, "next": {"query": "ibm_data_product_revoke_access.state:(SCHEDULED OR FAILED)", "limit": 1, "bookmark": "MQ==", "include": "entity", "skip": 0}}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + release_id = 'testString' + + # Invoke method + response = _service.get_revoke_access_process_state( + release_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'release_id={}'.format(release_id) in query_string + + def test_get_revoke_access_process_state_required_params_with_retries(self): + # Enable retries and run test_get_revoke_access_process_state_required_params. + _service.enable_retries() + self.test_get_revoke_access_process_state_required_params() + + # Disable retries and run test_get_revoke_access_process_state_required_params. + _service.disable_retries() + self.test_get_revoke_access_process_state_required_params() + + @responses.activate + def test_get_revoke_access_process_state_value_error(self): + """ + test_get_revoke_access_process_state_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_product_revoke_access/job_runs') + mock_response = '{"results": [{"metadata": {"anyKey": "anyValue"}, "entity": {"anyKey": "anyValue"}}], "total_count": 42, "next": {"query": "ibm_data_product_revoke_access.state:(SCHEDULED OR FAILED)", "limit": 1, "bookmark": "MQ==", "include": "entity", "skip": 0}}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + release_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "release_id": release_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_revoke_access_process_state(**req_copy) + + def test_get_revoke_access_process_state_value_error_with_retries(self): + # Enable retries and run test_get_revoke_access_process_state_value_error. + _service.enable_retries() + self.test_get_revoke_access_process_state_value_error() + + # Disable retries and run test_get_revoke_access_process_state_value_error. + _service.disable_retries() + self.test_get_revoke_access_process_state_value_error() + + +# endregion +############################################################################## +# End of Service: DataProductRevokeAccessJobRuns +############################################################################## + + +############################################################################## +# Start of Model Tests +############################################################################## +# region + + +class TestModel_Asset: + """ + Test Class for Asset + """ + + def test_asset_serialization(self): + """ + Test serialization/deserialization for Asset + """ + + # Construct a json representation of a Asset model + asset_model_json = {} + asset_model_json['metadata'] = {'anyKey': 'anyValue'} + asset_model_json['entity'] = {'anyKey': 'anyValue'} + + # Construct a model instance of Asset by calling from_dict on the json representation + asset_model = Asset.from_dict(asset_model_json) + assert asset_model != False + + # Construct a model instance of Asset by calling from_dict on the json representation + asset_model_dict = Asset.from_dict(asset_model_json).__dict__ + asset_model2 = Asset(**asset_model_dict) + + # Verify the model instances are equivalent + assert asset_model == asset_model2 + + # Convert model instance back to dict and verify no loss of data + asset_model_json2 = asset_model.to_dict() + assert asset_model_json2 == asset_model_json + + +class TestModel_AssetListAccessControl: + """ + Test Class for AssetListAccessControl + """ + + def test_asset_list_access_control_serialization(self): + """ + Test serialization/deserialization for AssetListAccessControl + """ + + # Construct a json representation of a AssetListAccessControl model + asset_list_access_control_model_json = {} + asset_list_access_control_model_json['owner'] = 'IBMid-696000KYV9' + + # Construct a model instance of AssetListAccessControl by calling from_dict on the json representation + asset_list_access_control_model = AssetListAccessControl.from_dict(asset_list_access_control_model_json) + assert asset_list_access_control_model != False + + # Construct a model instance of AssetListAccessControl by calling from_dict on the json representation + asset_list_access_control_model_dict = AssetListAccessControl.from_dict(asset_list_access_control_model_json).__dict__ + asset_list_access_control_model2 = AssetListAccessControl(**asset_list_access_control_model_dict) + + # Verify the model instances are equivalent + assert asset_list_access_control_model == asset_list_access_control_model2 + + # Convert model instance back to dict and verify no loss of data + asset_list_access_control_model_json2 = asset_list_access_control_model.to_dict() + assert asset_list_access_control_model_json2 == asset_list_access_control_model_json + + +class TestModel_AssetPartReference: + """ + Test Class for AssetPartReference + """ + + def test_asset_part_reference_serialization(self): + """ + Test serialization/deserialization for AssetPartReference + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a AssetPartReference model + asset_part_reference_model_json = {} + asset_part_reference_model_json['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model_json['name'] = 'testString' + asset_part_reference_model_json['container'] = container_reference_model + asset_part_reference_model_json['type'] = 'data_asset' + + # Construct a model instance of AssetPartReference by calling from_dict on the json representation + asset_part_reference_model = AssetPartReference.from_dict(asset_part_reference_model_json) + assert asset_part_reference_model != False + + # Construct a model instance of AssetPartReference by calling from_dict on the json representation + asset_part_reference_model_dict = AssetPartReference.from_dict(asset_part_reference_model_json).__dict__ + asset_part_reference_model2 = AssetPartReference(**asset_part_reference_model_dict) + + # Verify the model instances are equivalent + assert asset_part_reference_model == asset_part_reference_model2 + + # Convert model instance back to dict and verify no loss of data + asset_part_reference_model_json2 = asset_part_reference_model.to_dict() + assert asset_part_reference_model_json2 == asset_part_reference_model_json + + +class TestModel_AssetPrototype: + """ + Test Class for AssetPrototype + """ + + def test_asset_prototype_serialization(self): + """ + Test serialization/deserialization for AssetPrototype + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a json representation of a AssetPrototype model + asset_prototype_model_json = {} + asset_prototype_model_json['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_prototype_model_json['container'] = container_identity_model + + # Construct a model instance of AssetPrototype by calling from_dict on the json representation + asset_prototype_model = AssetPrototype.from_dict(asset_prototype_model_json) + assert asset_prototype_model != False + + # Construct a model instance of AssetPrototype by calling from_dict on the json representation + asset_prototype_model_dict = AssetPrototype.from_dict(asset_prototype_model_json).__dict__ + asset_prototype_model2 = AssetPrototype(**asset_prototype_model_dict) + + # Verify the model instances are equivalent + assert asset_prototype_model == asset_prototype_model2 + + # Convert model instance back to dict and verify no loss of data + asset_prototype_model_json2 = asset_prototype_model.to_dict() + assert asset_prototype_model_json2 == asset_prototype_model_json + + +class TestModel_AssetReference: + """ + Test Class for AssetReference + """ + + def test_asset_reference_serialization(self): + """ + Test serialization/deserialization for AssetReference + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a AssetReference model + asset_reference_model_json = {} + asset_reference_model_json['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model_json['name'] = 'testString' + asset_reference_model_json['container'] = container_reference_model + + # Construct a model instance of AssetReference by calling from_dict on the json representation + asset_reference_model = AssetReference.from_dict(asset_reference_model_json) + assert asset_reference_model != False + + # Construct a model instance of AssetReference by calling from_dict on the json representation + asset_reference_model_dict = AssetReference.from_dict(asset_reference_model_json).__dict__ + asset_reference_model2 = AssetReference(**asset_reference_model_dict) + + # Verify the model instances are equivalent + assert asset_reference_model == asset_reference_model2 + + # Convert model instance back to dict and verify no loss of data + asset_reference_model_json2 = asset_reference_model.to_dict() + assert asset_reference_model_json2 == asset_reference_model_json + + +class TestModel_BucketResponse: + """ + Test Class for BucketResponse + """ + + def test_bucket_response_serialization(self): + """ + Test serialization/deserialization for BucketResponse + """ + + # Construct a json representation of a BucketResponse model + bucket_response_model_json = {} + bucket_response_model_json['bucket_name'] = 'testString' + bucket_response_model_json['bucket_location'] = 'testString' + bucket_response_model_json['role_arn'] = 'testString' + bucket_response_model_json['bucket_type'] = 'testString' + bucket_response_model_json['shared'] = True + + # Construct a model instance of BucketResponse by calling from_dict on the json representation + bucket_response_model = BucketResponse.from_dict(bucket_response_model_json) + assert bucket_response_model != False + + # Construct a model instance of BucketResponse by calling from_dict on the json representation + bucket_response_model_dict = BucketResponse.from_dict(bucket_response_model_json).__dict__ + bucket_response_model2 = BucketResponse(**bucket_response_model_dict) + + # Verify the model instances are equivalent + assert bucket_response_model == bucket_response_model2 + + # Convert model instance back to dict and verify no loss of data + bucket_response_model_json2 = bucket_response_model.to_dict() + assert bucket_response_model_json2 == bucket_response_model_json + + +class TestModel_BucketValidationResponse: + """ + Test Class for BucketValidationResponse + """ + + def test_bucket_validation_response_serialization(self): + """ + Test serialization/deserialization for BucketValidationResponse + """ + + # Construct a json representation of a BucketValidationResponse model + bucket_validation_response_model_json = {} + bucket_validation_response_model_json['bucket_exists'] = True + + # Construct a model instance of BucketValidationResponse by calling from_dict on the json representation + bucket_validation_response_model = BucketValidationResponse.from_dict(bucket_validation_response_model_json) + assert bucket_validation_response_model != False + + # Construct a model instance of BucketValidationResponse by calling from_dict on the json representation + bucket_validation_response_model_dict = BucketValidationResponse.from_dict(bucket_validation_response_model_json).__dict__ + bucket_validation_response_model2 = BucketValidationResponse(**bucket_validation_response_model_dict) + + # Verify the model instances are equivalent + assert bucket_validation_response_model == bucket_validation_response_model2 + + # Convert model instance back to dict and verify no loss of data + bucket_validation_response_model_json2 = bucket_validation_response_model.to_dict() + assert bucket_validation_response_model_json2 == bucket_validation_response_model_json + + +class TestModel_ContainerIdentity: + """ + Test Class for ContainerIdentity + """ + + def test_container_identity_serialization(self): + """ + Test serialization/deserialization for ContainerIdentity + """ + + # Construct a json representation of a ContainerIdentity model + container_identity_model_json = {} + container_identity_model_json['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a model instance of ContainerIdentity by calling from_dict on the json representation + container_identity_model = ContainerIdentity.from_dict(container_identity_model_json) + assert container_identity_model != False + + # Construct a model instance of ContainerIdentity by calling from_dict on the json representation + container_identity_model_dict = ContainerIdentity.from_dict(container_identity_model_json).__dict__ + container_identity_model2 = ContainerIdentity(**container_identity_model_dict) + + # Verify the model instances are equivalent + assert container_identity_model == container_identity_model2 + + # Convert model instance back to dict and verify no loss of data + container_identity_model_json2 = container_identity_model.to_dict() + assert container_identity_model_json2 == container_identity_model_json + + +class TestModel_ContainerReference: + """ + Test Class for ContainerReference + """ + + def test_container_reference_serialization(self): + """ + Test serialization/deserialization for ContainerReference + """ + + # Construct a json representation of a ContainerReference model + container_reference_model_json = {} + container_reference_model_json['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model_json['type'] = 'catalog' + + # Construct a model instance of ContainerReference by calling from_dict on the json representation + container_reference_model = ContainerReference.from_dict(container_reference_model_json) + assert container_reference_model != False + + # Construct a model instance of ContainerReference by calling from_dict on the json representation + container_reference_model_dict = ContainerReference.from_dict(container_reference_model_json).__dict__ + container_reference_model2 = ContainerReference(**container_reference_model_dict) + + # Verify the model instances are equivalent + assert container_reference_model == container_reference_model2 + + # Convert model instance back to dict and verify no loss of data + container_reference_model_json2 = container_reference_model.to_dict() + assert container_reference_model_json2 == container_reference_model_json + + +class TestModel_ContractAsset: + """ + Test Class for ContractAsset + """ + + def test_contract_asset_serialization(self): + """ + Test serialization/deserialization for ContractAsset + """ + + # Construct a json representation of a ContractAsset model + contract_asset_model_json = {} + contract_asset_model_json['id'] = 'testString' + contract_asset_model_json['name'] = 'testString' + + # Construct a model instance of ContractAsset by calling from_dict on the json representation + contract_asset_model = ContractAsset.from_dict(contract_asset_model_json) + assert contract_asset_model != False + + # Construct a model instance of ContractAsset by calling from_dict on the json representation + contract_asset_model_dict = ContractAsset.from_dict(contract_asset_model_json).__dict__ + contract_asset_model2 = ContractAsset(**contract_asset_model_dict) + + # Verify the model instances are equivalent + assert contract_asset_model == contract_asset_model2 + + # Convert model instance back to dict and verify no loss of data + contract_asset_model_json2 = contract_asset_model.to_dict() + assert contract_asset_model_json2 == contract_asset_model_json + + +class TestModel_ContractQualityRule: + """ + Test Class for ContractQualityRule + """ + + def test_contract_quality_rule_serialization(self): + """ + Test serialization/deserialization for ContractQualityRule + """ + + # Construct a json representation of a ContractQualityRule model + contract_quality_rule_model_json = {} + contract_quality_rule_model_json['type'] = 'sql' + contract_quality_rule_model_json['description'] = 'testString' + contract_quality_rule_model_json['rule'] = 'testString' + contract_quality_rule_model_json['implementation'] = 'testString' + contract_quality_rule_model_json['engine'] = 'testString' + contract_quality_rule_model_json['must_be_less_than'] = 'testString' + contract_quality_rule_model_json['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model_json['must_be_greater_than'] = 'testString' + contract_quality_rule_model_json['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model_json['must_be_between'] = ['testString'] + contract_quality_rule_model_json['must_not_be_between'] = ['testString'] + contract_quality_rule_model_json['must_be'] = 'testString' + contract_quality_rule_model_json['must_not_be'] = 'testString' + contract_quality_rule_model_json['name'] = 'testString' + contract_quality_rule_model_json['unit'] = 'testString' + contract_quality_rule_model_json['query'] = 'testString' + + # Construct a model instance of ContractQualityRule by calling from_dict on the json representation + contract_quality_rule_model = ContractQualityRule.from_dict(contract_quality_rule_model_json) + assert contract_quality_rule_model != False + + # Construct a model instance of ContractQualityRule by calling from_dict on the json representation + contract_quality_rule_model_dict = ContractQualityRule.from_dict(contract_quality_rule_model_json).__dict__ + contract_quality_rule_model2 = ContractQualityRule(**contract_quality_rule_model_dict) + + # Verify the model instances are equivalent + assert contract_quality_rule_model == contract_quality_rule_model2 + + # Convert model instance back to dict and verify no loss of data + contract_quality_rule_model_json2 = contract_quality_rule_model.to_dict() + assert contract_quality_rule_model_json2 == contract_quality_rule_model_json + + +class TestModel_ContractSchema: + """ + Test Class for ContractSchema + """ + + def test_contract_schema_serialization(self): + """ + Test serialization/deserialization for ContractSchema + """ + + # Construct dict forms of any model objects needed in order to build this model. + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a json representation of a ContractSchema model + contract_schema_model_json = {} + contract_schema_model_json['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model_json['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model_json['name'] = 'testString' + contract_schema_model_json['description'] = 'testString' + contract_schema_model_json['connection_path'] = 'testString' + contract_schema_model_json['physical_type'] = 'testString' + contract_schema_model_json['properties'] = [contract_schema_property_model] + contract_schema_model_json['quality'] = [contract_quality_rule_model] + + # Construct a model instance of ContractSchema by calling from_dict on the json representation + contract_schema_model = ContractSchema.from_dict(contract_schema_model_json) + assert contract_schema_model != False + + # Construct a model instance of ContractSchema by calling from_dict on the json representation + contract_schema_model_dict = ContractSchema.from_dict(contract_schema_model_json).__dict__ + contract_schema_model2 = ContractSchema(**contract_schema_model_dict) + + # Verify the model instances are equivalent + assert contract_schema_model == contract_schema_model2 + + # Convert model instance back to dict and verify no loss of data + contract_schema_model_json2 = contract_schema_model.to_dict() + assert contract_schema_model_json2 == contract_schema_model_json + + +class TestModel_ContractSchemaProperty: + """ + Test Class for ContractSchemaProperty + """ + + def test_contract_schema_property_serialization(self): + """ + Test serialization/deserialization for ContractSchemaProperty + """ + + # Construct dict forms of any model objects needed in order to build this model. + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a json representation of a ContractSchemaProperty model + contract_schema_property_model_json = {} + contract_schema_property_model_json['name'] = 'testString' + contract_schema_property_model_json['type'] = contract_schema_property_type_model + contract_schema_property_model_json['quality'] = [contract_quality_rule_model] + + # Construct a model instance of ContractSchemaProperty by calling from_dict on the json representation + contract_schema_property_model = ContractSchemaProperty.from_dict(contract_schema_property_model_json) + assert contract_schema_property_model != False + + # Construct a model instance of ContractSchemaProperty by calling from_dict on the json representation + contract_schema_property_model_dict = ContractSchemaProperty.from_dict(contract_schema_property_model_json).__dict__ + contract_schema_property_model2 = ContractSchemaProperty(**contract_schema_property_model_dict) + + # Verify the model instances are equivalent + assert contract_schema_property_model == contract_schema_property_model2 + + # Convert model instance back to dict and verify no loss of data + contract_schema_property_model_json2 = contract_schema_property_model.to_dict() + assert contract_schema_property_model_json2 == contract_schema_property_model_json + + +class TestModel_ContractSchemaPropertyType: + """ + Test Class for ContractSchemaPropertyType + """ + + def test_contract_schema_property_type_serialization(self): + """ + Test serialization/deserialization for ContractSchemaPropertyType + """ + + # Construct a json representation of a ContractSchemaPropertyType model + contract_schema_property_type_model_json = {} + contract_schema_property_type_model_json['type'] = 'testString' + contract_schema_property_type_model_json['length'] = 'testString' + contract_schema_property_type_model_json['scale'] = 'testString' + contract_schema_property_type_model_json['nullable'] = 'testString' + contract_schema_property_type_model_json['signed'] = 'testString' + contract_schema_property_type_model_json['native_type'] = 'testString' + + # Construct a model instance of ContractSchemaPropertyType by calling from_dict on the json representation + contract_schema_property_type_model = ContractSchemaPropertyType.from_dict(contract_schema_property_type_model_json) + assert contract_schema_property_type_model != False + + # Construct a model instance of ContractSchemaPropertyType by calling from_dict on the json representation + contract_schema_property_type_model_dict = ContractSchemaPropertyType.from_dict(contract_schema_property_type_model_json).__dict__ + contract_schema_property_type_model2 = ContractSchemaPropertyType(**contract_schema_property_type_model_dict) + + # Verify the model instances are equivalent + assert contract_schema_property_type_model == contract_schema_property_type_model2 + + # Convert model instance back to dict and verify no loss of data + contract_schema_property_type_model_json2 = contract_schema_property_type_model.to_dict() + assert contract_schema_property_type_model_json2 == contract_schema_property_type_model_json + + +class TestModel_ContractServer: + """ + Test Class for ContractServer + """ + + def test_contract_server_serialization(self): + """ + Test serialization/deserialization for ContractServer + """ + + # Construct dict forms of any model objects needed in order to build this model. + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + # Construct a json representation of a ContractServer model + contract_server_model_json = {} + contract_server_model_json['server'] = 'testString' + contract_server_model_json['asset'] = contract_asset_model + contract_server_model_json['connection_id'] = 'testString' + contract_server_model_json['type'] = 'testString' + contract_server_model_json['description'] = 'testString' + contract_server_model_json['environment'] = 'testString' + contract_server_model_json['account'] = 'testString' + contract_server_model_json['catalog'] = 'testString' + contract_server_model_json['database'] = 'testString' + contract_server_model_json['dataset'] = 'testString' + contract_server_model_json['delimiter'] = 'testString' + contract_server_model_json['endpoint_url'] = 'testString' + contract_server_model_json['format'] = 'testString' + contract_server_model_json['host'] = 'testString' + contract_server_model_json['location'] = 'testString' + contract_server_model_json['path'] = 'testString' + contract_server_model_json['port'] = 'testString' + contract_server_model_json['project'] = 'testString' + contract_server_model_json['region'] = 'testString' + contract_server_model_json['region_name'] = 'testString' + contract_server_model_json['schema'] = 'testString' + contract_server_model_json['service_name'] = 'testString' + contract_server_model_json['staging_dir'] = 'testString' + contract_server_model_json['stream'] = 'testString' + contract_server_model_json['warehouse'] = 'testString' + contract_server_model_json['roles'] = ['testString'] + contract_server_model_json['custom_properties'] = [contract_template_custom_property_model] + + # Construct a model instance of ContractServer by calling from_dict on the json representation + contract_server_model = ContractServer.from_dict(contract_server_model_json) + assert contract_server_model != False + + # Construct a model instance of ContractServer by calling from_dict on the json representation + contract_server_model_dict = ContractServer.from_dict(contract_server_model_json).__dict__ + contract_server_model2 = ContractServer(**contract_server_model_dict) + + # Verify the model instances are equivalent + assert contract_server_model == contract_server_model2 + + # Convert model instance back to dict and verify no loss of data + contract_server_model_json2 = contract_server_model.to_dict() + assert contract_server_model_json2 == contract_server_model_json + + +class TestModel_ContractTemplateCustomProperty: + """ + Test Class for ContractTemplateCustomProperty + """ + + def test_contract_template_custom_property_serialization(self): + """ + Test serialization/deserialization for ContractTemplateCustomProperty + """ + + # Construct a json representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model_json = {} + contract_template_custom_property_model_json['key'] = 'customPropertyKey' + contract_template_custom_property_model_json['value'] = 'customPropertyValue' + + # Construct a model instance of ContractTemplateCustomProperty by calling from_dict on the json representation + contract_template_custom_property_model = ContractTemplateCustomProperty.from_dict(contract_template_custom_property_model_json) + assert contract_template_custom_property_model != False + + # Construct a model instance of ContractTemplateCustomProperty by calling from_dict on the json representation + contract_template_custom_property_model_dict = ContractTemplateCustomProperty.from_dict(contract_template_custom_property_model_json).__dict__ + contract_template_custom_property_model2 = ContractTemplateCustomProperty(**contract_template_custom_property_model_dict) + + # Verify the model instances are equivalent + assert contract_template_custom_property_model == contract_template_custom_property_model2 + + # Convert model instance back to dict and verify no loss of data + contract_template_custom_property_model_json2 = contract_template_custom_property_model.to_dict() + assert contract_template_custom_property_model_json2 == contract_template_custom_property_model_json + + +class TestModel_ContractTemplateOrganization: + """ + Test Class for ContractTemplateOrganization + """ + + def test_contract_template_organization_serialization(self): + """ + Test serialization/deserialization for ContractTemplateOrganization + """ + + # Construct a json representation of a ContractTemplateOrganization model + contract_template_organization_model_json = {} + contract_template_organization_model_json['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model_json['role'] = 'owner' + + # Construct a model instance of ContractTemplateOrganization by calling from_dict on the json representation + contract_template_organization_model = ContractTemplateOrganization.from_dict(contract_template_organization_model_json) + assert contract_template_organization_model != False + + # Construct a model instance of ContractTemplateOrganization by calling from_dict on the json representation + contract_template_organization_model_dict = ContractTemplateOrganization.from_dict(contract_template_organization_model_json).__dict__ + contract_template_organization_model2 = ContractTemplateOrganization(**contract_template_organization_model_dict) + + # Verify the model instances are equivalent + assert contract_template_organization_model == contract_template_organization_model2 + + # Convert model instance back to dict and verify no loss of data + contract_template_organization_model_json2 = contract_template_organization_model.to_dict() + assert contract_template_organization_model_json2 == contract_template_organization_model_json + + +class TestModel_ContractTemplateSLA: + """ + Test Class for ContractTemplateSLA + """ + + def test_contract_template_sla_serialization(self): + """ + Test serialization/deserialization for ContractTemplateSLA + """ + + # Construct dict forms of any model objects needed in order to build this model. + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + # Construct a json representation of a ContractTemplateSLA model + contract_template_sla_model_json = {} + contract_template_sla_model_json['default_element'] = 'Standard SLA Policy' + contract_template_sla_model_json['properties'] = [contract_template_sla_property_model] + + # Construct a model instance of ContractTemplateSLA by calling from_dict on the json representation + contract_template_sla_model = ContractTemplateSLA.from_dict(contract_template_sla_model_json) + assert contract_template_sla_model != False + + # Construct a model instance of ContractTemplateSLA by calling from_dict on the json representation + contract_template_sla_model_dict = ContractTemplateSLA.from_dict(contract_template_sla_model_json).__dict__ + contract_template_sla_model2 = ContractTemplateSLA(**contract_template_sla_model_dict) + + # Verify the model instances are equivalent + assert contract_template_sla_model == contract_template_sla_model2 + + # Convert model instance back to dict and verify no loss of data + contract_template_sla_model_json2 = contract_template_sla_model.to_dict() + assert contract_template_sla_model_json2 == contract_template_sla_model_json + + +class TestModel_ContractTemplateSLAProperty: + """ + Test Class for ContractTemplateSLAProperty + """ + + def test_contract_template_sla_property_serialization(self): + """ + Test serialization/deserialization for ContractTemplateSLAProperty + """ + + # Construct a json representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model_json = {} + contract_template_sla_property_model_json['property'] = 'Uptime Guarantee' + contract_template_sla_property_model_json['value'] = '99.9' + + # Construct a model instance of ContractTemplateSLAProperty by calling from_dict on the json representation + contract_template_sla_property_model = ContractTemplateSLAProperty.from_dict(contract_template_sla_property_model_json) + assert contract_template_sla_property_model != False + + # Construct a model instance of ContractTemplateSLAProperty by calling from_dict on the json representation + contract_template_sla_property_model_dict = ContractTemplateSLAProperty.from_dict(contract_template_sla_property_model_json).__dict__ + contract_template_sla_property_model2 = ContractTemplateSLAProperty(**contract_template_sla_property_model_dict) + + # Verify the model instances are equivalent + assert contract_template_sla_property_model == contract_template_sla_property_model2 + + # Convert model instance back to dict and verify no loss of data + contract_template_sla_property_model_json2 = contract_template_sla_property_model.to_dict() + assert contract_template_sla_property_model_json2 == contract_template_sla_property_model_json + + +class TestModel_ContractTemplateSupportAndCommunication: + """ + Test Class for ContractTemplateSupportAndCommunication + """ + + def test_contract_template_support_and_communication_serialization(self): + """ + Test serialization/deserialization for ContractTemplateSupportAndCommunication + """ + + # Construct a json representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model_json = {} + contract_template_support_and_communication_model_json['channel'] = 'Email Support' + contract_template_support_and_communication_model_json['url'] = 'https://support.example.com' + + # Construct a model instance of ContractTemplateSupportAndCommunication by calling from_dict on the json representation + contract_template_support_and_communication_model = ContractTemplateSupportAndCommunication.from_dict(contract_template_support_and_communication_model_json) + assert contract_template_support_and_communication_model != False + + # Construct a model instance of ContractTemplateSupportAndCommunication by calling from_dict on the json representation + contract_template_support_and_communication_model_dict = ContractTemplateSupportAndCommunication.from_dict(contract_template_support_and_communication_model_json).__dict__ + contract_template_support_and_communication_model2 = ContractTemplateSupportAndCommunication(**contract_template_support_and_communication_model_dict) + + # Verify the model instances are equivalent + assert contract_template_support_and_communication_model == contract_template_support_and_communication_model2 + + # Convert model instance back to dict and verify no loss of data + contract_template_support_and_communication_model_json2 = contract_template_support_and_communication_model.to_dict() + assert contract_template_support_and_communication_model_json2 == contract_template_support_and_communication_model_json + + +class TestModel_ContractTerms: + """ + Test Class for ContractTerms + """ + + def test_contract_terms_serialization(self): + """ + Test serialization/deserialization for ContractTerms + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Construct a json representation of a ContractTerms model + contract_terms_model_json = {} + contract_terms_model_json['asset'] = asset_reference_model + contract_terms_model_json['id'] = 'testString' + contract_terms_model_json['documents'] = [contract_terms_document_model] + contract_terms_model_json['error_msg'] = 'testString' + contract_terms_model_json['overview'] = overview_model + contract_terms_model_json['description'] = description_model + contract_terms_model_json['organization'] = [contract_template_organization_model] + contract_terms_model_json['roles'] = [roles_model] + contract_terms_model_json['price'] = pricing_model + contract_terms_model_json['sla'] = [contract_template_sla_model] + contract_terms_model_json['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model_json['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model_json['contract_test'] = contract_test_model + contract_terms_model_json['servers'] = [contract_server_model] + contract_terms_model_json['schema'] = [contract_schema_model] + + # Construct a model instance of ContractTerms by calling from_dict on the json representation + contract_terms_model = ContractTerms.from_dict(contract_terms_model_json) + assert contract_terms_model != False + + # Construct a model instance of ContractTerms by calling from_dict on the json representation + contract_terms_model_dict = ContractTerms.from_dict(contract_terms_model_json).__dict__ + contract_terms_model2 = ContractTerms(**contract_terms_model_dict) + + # Verify the model instances are equivalent + assert contract_terms_model == contract_terms_model2 + + # Convert model instance back to dict and verify no loss of data + contract_terms_model_json2 = contract_terms_model.to_dict() + assert contract_terms_model_json2 == contract_terms_model_json + + +class TestModel_ContractTermsDocument: + """ + Test Class for ContractTermsDocument + """ + + def test_contract_terms_document_serialization(self): + """ + Test serialization/deserialization for ContractTermsDocument + """ + + # Construct dict forms of any model objects needed in order to build this model. + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a json representation of a ContractTermsDocument model + contract_terms_document_model_json = {} + contract_terms_document_model_json['url'] = 'testString' + contract_terms_document_model_json['type'] = 'terms_and_conditions' + contract_terms_document_model_json['name'] = 'testString' + contract_terms_document_model_json['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model_json['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model_json['upload_url'] = 'testString' + + # Construct a model instance of ContractTermsDocument by calling from_dict on the json representation + contract_terms_document_model = ContractTermsDocument.from_dict(contract_terms_document_model_json) + assert contract_terms_document_model != False + + # Construct a model instance of ContractTermsDocument by calling from_dict on the json representation + contract_terms_document_model_dict = ContractTermsDocument.from_dict(contract_terms_document_model_json).__dict__ + contract_terms_document_model2 = ContractTermsDocument(**contract_terms_document_model_dict) + + # Verify the model instances are equivalent + assert contract_terms_document_model == contract_terms_document_model2 + + # Convert model instance back to dict and verify no loss of data + contract_terms_document_model_json2 = contract_terms_document_model.to_dict() + assert contract_terms_document_model_json2 == contract_terms_document_model_json + + +class TestModel_ContractTermsDocumentAttachment: + """ + Test Class for ContractTermsDocumentAttachment + """ + + def test_contract_terms_document_attachment_serialization(self): + """ + Test serialization/deserialization for ContractTermsDocumentAttachment + """ + + # Construct a json representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model_json = {} + contract_terms_document_attachment_model_json['id'] = 'testString' + + # Construct a model instance of ContractTermsDocumentAttachment by calling from_dict on the json representation + contract_terms_document_attachment_model = ContractTermsDocumentAttachment.from_dict(contract_terms_document_attachment_model_json) + assert contract_terms_document_attachment_model != False + + # Construct a model instance of ContractTermsDocumentAttachment by calling from_dict on the json representation + contract_terms_document_attachment_model_dict = ContractTermsDocumentAttachment.from_dict(contract_terms_document_attachment_model_json).__dict__ + contract_terms_document_attachment_model2 = ContractTermsDocumentAttachment(**contract_terms_document_attachment_model_dict) + + # Verify the model instances are equivalent + assert contract_terms_document_attachment_model == contract_terms_document_attachment_model2 + + # Convert model instance back to dict and verify no loss of data + contract_terms_document_attachment_model_json2 = contract_terms_document_attachment_model.to_dict() + assert contract_terms_document_attachment_model_json2 == contract_terms_document_attachment_model_json + + +class TestModel_ContractTermsMoreInfo: + """ + Test Class for ContractTermsMoreInfo + """ + + def test_contract_terms_more_info_serialization(self): + """ + Test serialization/deserialization for ContractTermsMoreInfo + """ + + # Construct a json representation of a ContractTermsMoreInfo model + contract_terms_more_info_model_json = {} + contract_terms_more_info_model_json['type'] = 'privacy-statement' + contract_terms_more_info_model_json['url'] = 'https://moreinfo.example.com' + + # Construct a model instance of ContractTermsMoreInfo by calling from_dict on the json representation + contract_terms_more_info_model = ContractTermsMoreInfo.from_dict(contract_terms_more_info_model_json) + assert contract_terms_more_info_model != False + + # Construct a model instance of ContractTermsMoreInfo by calling from_dict on the json representation + contract_terms_more_info_model_dict = ContractTermsMoreInfo.from_dict(contract_terms_more_info_model_json).__dict__ + contract_terms_more_info_model2 = ContractTermsMoreInfo(**contract_terms_more_info_model_dict) + + # Verify the model instances are equivalent + assert contract_terms_more_info_model == contract_terms_more_info_model2 + + # Convert model instance back to dict and verify no loss of data + contract_terms_more_info_model_json2 = contract_terms_more_info_model.to_dict() + assert contract_terms_more_info_model_json2 == contract_terms_more_info_model_json + + +class TestModel_ContractTest: + """ + Test Class for ContractTest + """ + + def test_contract_test_serialization(self): + """ + Test serialization/deserialization for ContractTest + """ + + # Construct a json representation of a ContractTest model + contract_test_model_json = {} + contract_test_model_json['status'] = 'pass' + contract_test_model_json['last_tested_time'] = 'testString' + contract_test_model_json['message'] = 'testString' + + # Construct a model instance of ContractTest by calling from_dict on the json representation + contract_test_model = ContractTest.from_dict(contract_test_model_json) + assert contract_test_model != False + + # Construct a model instance of ContractTest by calling from_dict on the json representation + contract_test_model_dict = ContractTest.from_dict(contract_test_model_json).__dict__ + contract_test_model2 = ContractTest(**contract_test_model_dict) + + # Verify the model instances are equivalent + assert contract_test_model == contract_test_model2 + + # Convert model instance back to dict and verify no loss of data + contract_test_model_json2 = contract_test_model.to_dict() + assert contract_test_model_json2 == contract_test_model_json + + +class TestModel_DataAssetRelationship: + """ + Test Class for DataAssetRelationship + """ + + def test_data_asset_relationship_serialization(self): + """ + Test serialization/deserialization for DataAssetRelationship + """ + + # Construct dict forms of any model objects needed in order to build this model. + + visualization_model = {} # Visualization + visualization_model['id'] = 'testString' + visualization_model['name'] = 'testString' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + error_message_model = {} # ErrorMessage + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + # Construct a json representation of a DataAssetRelationship model + data_asset_relationship_model_json = {} + data_asset_relationship_model_json['visualization'] = visualization_model + data_asset_relationship_model_json['asset'] = asset_reference_model + data_asset_relationship_model_json['related_asset'] = asset_reference_model + data_asset_relationship_model_json['error'] = error_message_model + + # Construct a model instance of DataAssetRelationship by calling from_dict on the json representation + data_asset_relationship_model = DataAssetRelationship.from_dict(data_asset_relationship_model_json) + assert data_asset_relationship_model != False + + # Construct a model instance of DataAssetRelationship by calling from_dict on the json representation + data_asset_relationship_model_dict = DataAssetRelationship.from_dict(data_asset_relationship_model_json).__dict__ + data_asset_relationship_model2 = DataAssetRelationship(**data_asset_relationship_model_dict) + + # Verify the model instances are equivalent + assert data_asset_relationship_model == data_asset_relationship_model2 + + # Convert model instance back to dict and verify no loss of data + data_asset_relationship_model_json2 = data_asset_relationship_model.to_dict() + assert data_asset_relationship_model_json2 == data_asset_relationship_model_json + + +class TestModel_DataAssetVisualizationRes: + """ + Test Class for DataAssetVisualizationRes + """ + + def test_data_asset_visualization_res_serialization(self): + """ + Test serialization/deserialization for DataAssetVisualizationRes + """ + + # Construct dict forms of any model objects needed in order to build this model. + + visualization_model = {} # Visualization + visualization_model['id'] = 'testString' + visualization_model['name'] = 'testString' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + error_message_model = {} # ErrorMessage + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + data_asset_relationship_model = {} # DataAssetRelationship + data_asset_relationship_model['visualization'] = visualization_model + data_asset_relationship_model['asset'] = asset_reference_model + data_asset_relationship_model['related_asset'] = asset_reference_model + data_asset_relationship_model['error'] = error_message_model + + # Construct a json representation of a DataAssetVisualizationRes model + data_asset_visualization_res_model_json = {} + data_asset_visualization_res_model_json['results'] = [data_asset_relationship_model] + + # Construct a model instance of DataAssetVisualizationRes by calling from_dict on the json representation + data_asset_visualization_res_model = DataAssetVisualizationRes.from_dict(data_asset_visualization_res_model_json) + assert data_asset_visualization_res_model != False + + # Construct a model instance of DataAssetVisualizationRes by calling from_dict on the json representation + data_asset_visualization_res_model_dict = DataAssetVisualizationRes.from_dict(data_asset_visualization_res_model_json).__dict__ + data_asset_visualization_res_model2 = DataAssetVisualizationRes(**data_asset_visualization_res_model_dict) + + # Verify the model instances are equivalent + assert data_asset_visualization_res_model == data_asset_visualization_res_model2 + + # Convert model instance back to dict and verify no loss of data + data_asset_visualization_res_model_json2 = data_asset_visualization_res_model.to_dict() + assert data_asset_visualization_res_model_json2 == data_asset_visualization_res_model_json + + +class TestModel_DataProduct: + """ + Test Class for DataProduct + """ + + def test_data_product_serialization(self): + """ + Test serialization/deserialization for DataProduct + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_version_summary_data_product_model = {} # DataProductVersionSummaryDataProduct + data_product_version_summary_data_product_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_version_summary_data_product_model['release'] = data_product_draft_version_release_model + data_product_version_summary_data_product_model['container'] = container_reference_model + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + data_product_version_summary_model = {} # DataProductVersionSummary + data_product_version_summary_model['version'] = '1.0.0' + data_product_version_summary_model['state'] = 'draft' + data_product_version_summary_model['data_product'] = data_product_version_summary_data_product_model + data_product_version_summary_model['name'] = 'My Data Product' + data_product_version_summary_model['description'] = 'This is a description of My Data Product.' + data_product_version_summary_model['tags'] = ['testString'] + data_product_version_summary_model['use_cases'] = [use_case_model] + data_product_version_summary_model['types'] = ['data'] + data_product_version_summary_model['contract_terms'] = [contract_terms_model] + data_product_version_summary_model['domain'] = domain_model + data_product_version_summary_model['parts_out'] = [data_product_part_model] + data_product_version_summary_model['workflows'] = data_product_workflows_model + data_product_version_summary_model['dataview_enabled'] = True + data_product_version_summary_model['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_version_summary_model['access_control'] = asset_list_access_control_model + data_product_version_summary_model['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_version_summary_model['sub_container'] = container_identity_model + data_product_version_summary_model['is_restricted'] = True + data_product_version_summary_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd' + data_product_version_summary_model['asset'] = asset_reference_model + + # Construct a json representation of a DataProduct model + data_product_model_json = {} + data_product_model_json['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_model_json['release'] = data_product_draft_version_release_model + data_product_model_json['container'] = container_reference_model + data_product_model_json['name'] = 'testString' + data_product_model_json['latest_release'] = data_product_version_summary_model + data_product_model_json['drafts'] = [data_product_version_summary_model] + + # Construct a model instance of DataProduct by calling from_dict on the json representation + data_product_model = DataProduct.from_dict(data_product_model_json) + assert data_product_model != False + + # Construct a model instance of DataProduct by calling from_dict on the json representation + data_product_model_dict = DataProduct.from_dict(data_product_model_json).__dict__ + data_product_model2 = DataProduct(**data_product_model_dict) + + # Verify the model instances are equivalent + assert data_product_model == data_product_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_model_json2 = data_product_model.to_dict() + assert data_product_model_json2 == data_product_model_json + + +class TestModel_DataProductCollection: + """ + Test Class for DataProductCollection + """ + + def test_data_product_collection_serialization(self): + """ + Test serialization/deserialization for DataProductCollection + """ + + # Construct dict forms of any model objects needed in order to build this model. + + first_page_model = {} # FirstPage + first_page_model['href'] = 'https://api.example.com/collection' + + next_page_model = {} # NextPage + next_page_model['href'] = 'https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + next_page_model['start'] = 'eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_summary_model = {} # DataProductSummary + data_product_summary_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_summary_model['release'] = data_product_draft_version_release_model + data_product_summary_model['container'] = container_reference_model + data_product_summary_model['name'] = 'testString' + + # Construct a json representation of a DataProductCollection model + data_product_collection_model_json = {} + data_product_collection_model_json['limit'] = 200 + data_product_collection_model_json['first'] = first_page_model + data_product_collection_model_json['next'] = next_page_model + data_product_collection_model_json['total_results'] = 200 + data_product_collection_model_json['data_products'] = [data_product_summary_model] + + # Construct a model instance of DataProductCollection by calling from_dict on the json representation + data_product_collection_model = DataProductCollection.from_dict(data_product_collection_model_json) + assert data_product_collection_model != False + + # Construct a model instance of DataProductCollection by calling from_dict on the json representation + data_product_collection_model_dict = DataProductCollection.from_dict(data_product_collection_model_json).__dict__ + data_product_collection_model2 = DataProductCollection(**data_product_collection_model_dict) + + # Verify the model instances are equivalent + assert data_product_collection_model == data_product_collection_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_collection_model_json2 = data_product_collection_model.to_dict() + assert data_product_collection_model_json2 == data_product_collection_model_json + + +class TestModel_DataProductContractTemplate: + """ + Test Class for DataProductContractTemplate + """ + + def test_data_product_contract_template_serialization(self): + """ + Test serialization/deserialization for DataProductContractTemplate + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + error_message_model = {} # ErrorMessage + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + # Construct a json representation of a DataProductContractTemplate model + data_product_contract_template_model_json = {} + data_product_contract_template_model_json['container'] = container_reference_model + data_product_contract_template_model_json['id'] = '20aa7c97-cfcc-4d16-ae76-2ca1847ce733' + data_product_contract_template_model_json['creator_id'] = 'IBMid-123456ABC' + data_product_contract_template_model_json['created_at'] = '2025-06-26T12:30:20.000Z' + data_product_contract_template_model_json['name'] = 'Sample Data Contract Template' + data_product_contract_template_model_json['error'] = error_message_model + data_product_contract_template_model_json['contract_terms'] = contract_terms_model + + # Construct a model instance of DataProductContractTemplate by calling from_dict on the json representation + data_product_contract_template_model = DataProductContractTemplate.from_dict(data_product_contract_template_model_json) + assert data_product_contract_template_model != False + + # Construct a model instance of DataProductContractTemplate by calling from_dict on the json representation + data_product_contract_template_model_dict = DataProductContractTemplate.from_dict(data_product_contract_template_model_json).__dict__ + data_product_contract_template_model2 = DataProductContractTemplate(**data_product_contract_template_model_dict) + + # Verify the model instances are equivalent + assert data_product_contract_template_model == data_product_contract_template_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_contract_template_model_json2 = data_product_contract_template_model.to_dict() + assert data_product_contract_template_model_json2 == data_product_contract_template_model_json + + +class TestModel_DataProductContractTemplateCollection: + """ + Test Class for DataProductContractTemplateCollection + """ + + def test_data_product_contract_template_collection_serialization(self): + """ + Test serialization/deserialization for DataProductContractTemplateCollection + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + error_message_model = {} # ErrorMessage + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + data_product_contract_template_model = {} # DataProductContractTemplate + data_product_contract_template_model['container'] = container_reference_model + data_product_contract_template_model['id'] = '20aa7c97-cfcc-4d16-ae76-2ca1847ce733' + data_product_contract_template_model['creator_id'] = 'IBMid-123456ABC' + data_product_contract_template_model['created_at'] = '2025-06-26T12:30:20.000Z' + data_product_contract_template_model['name'] = 'Sample Data Contract Template' + data_product_contract_template_model['error'] = error_message_model + data_product_contract_template_model['contract_terms'] = contract_terms_model + + # Construct a json representation of a DataProductContractTemplateCollection model + data_product_contract_template_collection_model_json = {} + data_product_contract_template_collection_model_json['contract_templates'] = [data_product_contract_template_model] + + # Construct a model instance of DataProductContractTemplateCollection by calling from_dict on the json representation + data_product_contract_template_collection_model = DataProductContractTemplateCollection.from_dict(data_product_contract_template_collection_model_json) + assert data_product_contract_template_collection_model != False + + # Construct a model instance of DataProductContractTemplateCollection by calling from_dict on the json representation + data_product_contract_template_collection_model_dict = DataProductContractTemplateCollection.from_dict(data_product_contract_template_collection_model_json).__dict__ + data_product_contract_template_collection_model2 = DataProductContractTemplateCollection(**data_product_contract_template_collection_model_dict) + + # Verify the model instances are equivalent + assert data_product_contract_template_collection_model == data_product_contract_template_collection_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_contract_template_collection_model_json2 = data_product_contract_template_collection_model.to_dict() + assert data_product_contract_template_collection_model_json2 == data_product_contract_template_collection_model_json + + +class TestModel_DataProductCustomWorkflowDefinition: + """ + Test Class for DataProductCustomWorkflowDefinition + """ + + def test_data_product_custom_workflow_definition_serialization(self): + """ + Test serialization/deserialization for DataProductCustomWorkflowDefinition + """ + + # Construct a json representation of a DataProductCustomWorkflowDefinition model + data_product_custom_workflow_definition_model_json = {} + data_product_custom_workflow_definition_model_json['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a model instance of DataProductCustomWorkflowDefinition by calling from_dict on the json representation + data_product_custom_workflow_definition_model = DataProductCustomWorkflowDefinition.from_dict(data_product_custom_workflow_definition_model_json) + assert data_product_custom_workflow_definition_model != False + + # Construct a model instance of DataProductCustomWorkflowDefinition by calling from_dict on the json representation + data_product_custom_workflow_definition_model_dict = DataProductCustomWorkflowDefinition.from_dict(data_product_custom_workflow_definition_model_json).__dict__ + data_product_custom_workflow_definition_model2 = DataProductCustomWorkflowDefinition(**data_product_custom_workflow_definition_model_dict) + + # Verify the model instances are equivalent + assert data_product_custom_workflow_definition_model == data_product_custom_workflow_definition_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_custom_workflow_definition_model_json2 = data_product_custom_workflow_definition_model.to_dict() + assert data_product_custom_workflow_definition_model_json2 == data_product_custom_workflow_definition_model_json + + +class TestModel_DataProductDomain: + """ + Test Class for DataProductDomain + """ + + def test_data_product_domain_serialization(self): + """ + Test serialization/deserialization for DataProductDomain + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + error_extra_resource_model = {} # ErrorExtraResource + error_extra_resource_model['id'] = 'testString' + error_extra_resource_model['timestamp'] = '2019-01-01T12:00:00Z' + error_extra_resource_model['environment_name'] = 'testString' + error_extra_resource_model['http_status'] = 0 + error_extra_resource_model['source_cluster'] = 0 + error_extra_resource_model['source_component'] = 0 + error_extra_resource_model['transaction_id'] = 0 + + error_model_resource_model = {} # ErrorModelResource + error_model_resource_model['code'] = 'request_body_error' + error_model_resource_model['message'] = 'testString' + error_model_resource_model['extra'] = error_extra_resource_model + error_model_resource_model['more_info'] = 'testString' + + member_roles_schema_model = {} # MemberRolesSchema + member_roles_schema_model['user_iam_id'] = 'testString' + member_roles_schema_model['roles'] = ['testString'] + + properties_schema_model = {} # PropertiesSchema + properties_schema_model['value'] = 'testString' + + initialize_sub_domain_model = {} # InitializeSubDomain + initialize_sub_domain_model['name'] = 'Operations' + initialize_sub_domain_model['id'] = 'testString' + initialize_sub_domain_model['description'] = 'testString' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a json representation of a DataProductDomain model + data_product_domain_model_json = {} + data_product_domain_model_json['container'] = container_reference_model + data_product_domain_model_json['trace'] = 'testString' + data_product_domain_model_json['errors'] = [error_model_resource_model] + data_product_domain_model_json['name'] = 'Operations' + data_product_domain_model_json['description'] = 'This is a description of the data product domain.' + data_product_domain_model_json['id'] = 'testString' + data_product_domain_model_json['created_by'] = 'testString' + data_product_domain_model_json['member_roles'] = member_roles_schema_model + data_product_domain_model_json['properties'] = properties_schema_model + data_product_domain_model_json['sub_domains'] = [initialize_sub_domain_model] + data_product_domain_model_json['sub_container'] = container_identity_model + + # Construct a model instance of DataProductDomain by calling from_dict on the json representation + data_product_domain_model = DataProductDomain.from_dict(data_product_domain_model_json) + assert data_product_domain_model != False + + # Construct a model instance of DataProductDomain by calling from_dict on the json representation + data_product_domain_model_dict = DataProductDomain.from_dict(data_product_domain_model_json).__dict__ + data_product_domain_model2 = DataProductDomain(**data_product_domain_model_dict) + + # Verify the model instances are equivalent + assert data_product_domain_model == data_product_domain_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_domain_model_json2 = data_product_domain_model.to_dict() + assert data_product_domain_model_json2 == data_product_domain_model_json + + +class TestModel_DataProductDomainCollection: + """ + Test Class for DataProductDomainCollection + """ + + def test_data_product_domain_collection_serialization(self): + """ + Test serialization/deserialization for DataProductDomainCollection + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + error_extra_resource_model = {} # ErrorExtraResource + error_extra_resource_model['id'] = 'testString' + error_extra_resource_model['timestamp'] = '2019-01-01T12:00:00Z' + error_extra_resource_model['environment_name'] = 'testString' + error_extra_resource_model['http_status'] = 0 + error_extra_resource_model['source_cluster'] = 0 + error_extra_resource_model['source_component'] = 0 + error_extra_resource_model['transaction_id'] = 0 + + error_model_resource_model = {} # ErrorModelResource + error_model_resource_model['code'] = 'request_body_error' + error_model_resource_model['message'] = 'testString' + error_model_resource_model['extra'] = error_extra_resource_model + error_model_resource_model['more_info'] = 'testString' + + member_roles_schema_model = {} # MemberRolesSchema + member_roles_schema_model['user_iam_id'] = 'testString' + member_roles_schema_model['roles'] = ['testString'] + + properties_schema_model = {} # PropertiesSchema + properties_schema_model['value'] = 'testString' + + initialize_sub_domain_model = {} # InitializeSubDomain + initialize_sub_domain_model['name'] = 'Operations' + initialize_sub_domain_model['id'] = 'testString' + initialize_sub_domain_model['description'] = 'testString' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + data_product_domain_model = {} # DataProductDomain + data_product_domain_model['container'] = container_reference_model + data_product_domain_model['trace'] = 'testString' + data_product_domain_model['errors'] = [error_model_resource_model] + data_product_domain_model['name'] = 'Operations' + data_product_domain_model['description'] = 'This is a description of the data product domain.' + data_product_domain_model['id'] = 'testString' + data_product_domain_model['created_by'] = 'testString' + data_product_domain_model['member_roles'] = member_roles_schema_model + data_product_domain_model['properties'] = properties_schema_model + data_product_domain_model['sub_domains'] = [initialize_sub_domain_model] + data_product_domain_model['sub_container'] = container_identity_model + + # Construct a json representation of a DataProductDomainCollection model + data_product_domain_collection_model_json = {} + data_product_domain_collection_model_json['domains'] = [data_product_domain_model] + + # Construct a model instance of DataProductDomainCollection by calling from_dict on the json representation + data_product_domain_collection_model = DataProductDomainCollection.from_dict(data_product_domain_collection_model_json) + assert data_product_domain_collection_model != False + + # Construct a model instance of DataProductDomainCollection by calling from_dict on the json representation + data_product_domain_collection_model_dict = DataProductDomainCollection.from_dict(data_product_domain_collection_model_json).__dict__ + data_product_domain_collection_model2 = DataProductDomainCollection(**data_product_domain_collection_model_dict) + + # Verify the model instances are equivalent + assert data_product_domain_collection_model == data_product_domain_collection_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_domain_collection_model_json2 = data_product_domain_collection_model.to_dict() + assert data_product_domain_collection_model_json2 == data_product_domain_collection_model_json + + +class TestModel_DataProductDraft: + """ + Test Class for DataProductDraft + """ + + def test_data_product_draft_serialization(self): + """ + Test serialization/deserialization for DataProductDraft + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_draft_data_product_model = {} # DataProductDraftDataProduct + data_product_draft_data_product_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_draft_data_product_model['release'] = data_product_draft_version_release_model + data_product_draft_data_product_model['container'] = container_reference_model + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + visualization_model = {} # Visualization + visualization_model['id'] = 'testString' + visualization_model['name'] = 'testString' + + error_message_model = {} # ErrorMessage + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + data_asset_relationship_model = {} # DataAssetRelationship + data_asset_relationship_model['visualization'] = visualization_model + data_asset_relationship_model['asset'] = asset_reference_model + data_asset_relationship_model['related_asset'] = asset_reference_model + data_asset_relationship_model['error'] = error_message_model + + # Construct a json representation of a DataProductDraft model + data_product_draft_model_json = {} + data_product_draft_model_json['version'] = '1.0.0' + data_product_draft_model_json['state'] = 'draft' + data_product_draft_model_json['data_product'] = data_product_draft_data_product_model + data_product_draft_model_json['name'] = 'My Data Product' + data_product_draft_model_json['description'] = 'This is a description of My Data Product.' + data_product_draft_model_json['tags'] = ['testString'] + data_product_draft_model_json['use_cases'] = [use_case_model] + data_product_draft_model_json['types'] = ['data'] + data_product_draft_model_json['contract_terms'] = [contract_terms_model] + data_product_draft_model_json['domain'] = domain_model + data_product_draft_model_json['parts_out'] = [data_product_part_model] + data_product_draft_model_json['workflows'] = data_product_workflows_model + data_product_draft_model_json['dataview_enabled'] = True + data_product_draft_model_json['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_draft_model_json['access_control'] = asset_list_access_control_model + data_product_draft_model_json['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_draft_model_json['sub_container'] = container_identity_model + data_product_draft_model_json['is_restricted'] = True + data_product_draft_model_json['id'] = '2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd' + data_product_draft_model_json['asset'] = asset_reference_model + data_product_draft_model_json['published_by'] = 'testString' + data_product_draft_model_json['published_at'] = '2019-01-01T12:00:00Z' + data_product_draft_model_json['created_by'] = 'testString' + data_product_draft_model_json['created_at'] = '2019-01-01T12:00:00Z' + data_product_draft_model_json['properties'] = {'anyKey': 'anyValue'} + data_product_draft_model_json['visualization_errors'] = [data_asset_relationship_model] + + # Construct a model instance of DataProductDraft by calling from_dict on the json representation + data_product_draft_model = DataProductDraft.from_dict(data_product_draft_model_json) + assert data_product_draft_model != False + + # Construct a model instance of DataProductDraft by calling from_dict on the json representation + data_product_draft_model_dict = DataProductDraft.from_dict(data_product_draft_model_json).__dict__ + data_product_draft_model2 = DataProductDraft(**data_product_draft_model_dict) + + # Verify the model instances are equivalent + assert data_product_draft_model == data_product_draft_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_draft_model_json2 = data_product_draft_model.to_dict() + assert data_product_draft_model_json2 == data_product_draft_model_json + + +class TestModel_DataProductDraftCollection: + """ + Test Class for DataProductDraftCollection + """ + + def test_data_product_draft_collection_serialization(self): + """ + Test serialization/deserialization for DataProductDraftCollection + """ + + # Construct dict forms of any model objects needed in order to build this model. + + first_page_model = {} # FirstPage + first_page_model['href'] = 'https://api.example.com/collection' + + next_page_model = {} # NextPage + next_page_model['href'] = 'https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + next_page_model['start'] = 'eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_draft_summary_data_product_model = {} # DataProductDraftSummaryDataProduct + data_product_draft_summary_data_product_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_draft_summary_data_product_model['release'] = data_product_draft_version_release_model + data_product_draft_summary_data_product_model['container'] = container_reference_model + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + data_product_draft_summary_model = {} # DataProductDraftSummary + data_product_draft_summary_model['version'] = '1.0.0' + data_product_draft_summary_model['state'] = 'draft' + data_product_draft_summary_model['data_product'] = data_product_draft_summary_data_product_model + data_product_draft_summary_model['name'] = 'My Data Product' + data_product_draft_summary_model['description'] = 'This is a description of My Data Product.' + data_product_draft_summary_model['tags'] = ['testString'] + data_product_draft_summary_model['use_cases'] = [use_case_model] + data_product_draft_summary_model['types'] = ['data'] + data_product_draft_summary_model['contract_terms'] = [contract_terms_model] + data_product_draft_summary_model['domain'] = domain_model + data_product_draft_summary_model['parts_out'] = [data_product_part_model] + data_product_draft_summary_model['workflows'] = data_product_workflows_model + data_product_draft_summary_model['dataview_enabled'] = True + data_product_draft_summary_model['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_draft_summary_model['access_control'] = asset_list_access_control_model + data_product_draft_summary_model['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_draft_summary_model['sub_container'] = container_identity_model + data_product_draft_summary_model['is_restricted'] = True + data_product_draft_summary_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd' + data_product_draft_summary_model['asset'] = asset_reference_model + + # Construct a json representation of a DataProductDraftCollection model + data_product_draft_collection_model_json = {} + data_product_draft_collection_model_json['limit'] = 200 + data_product_draft_collection_model_json['first'] = first_page_model + data_product_draft_collection_model_json['next'] = next_page_model + data_product_draft_collection_model_json['total_results'] = 200 + data_product_draft_collection_model_json['drafts'] = [data_product_draft_summary_model] + + # Construct a model instance of DataProductDraftCollection by calling from_dict on the json representation + data_product_draft_collection_model = DataProductDraftCollection.from_dict(data_product_draft_collection_model_json) + assert data_product_draft_collection_model != False + + # Construct a model instance of DataProductDraftCollection by calling from_dict on the json representation + data_product_draft_collection_model_dict = DataProductDraftCollection.from_dict(data_product_draft_collection_model_json).__dict__ + data_product_draft_collection_model2 = DataProductDraftCollection(**data_product_draft_collection_model_dict) + + # Verify the model instances are equivalent + assert data_product_draft_collection_model == data_product_draft_collection_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_draft_collection_model_json2 = data_product_draft_collection_model.to_dict() + assert data_product_draft_collection_model_json2 == data_product_draft_collection_model_json + + +class TestModel_DataProductDraftDataProduct: + """ + Test Class for DataProductDraftDataProduct + """ + + def test_data_product_draft_data_product_serialization(self): + """ + Test serialization/deserialization for DataProductDraftDataProduct + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a DataProductDraftDataProduct model + data_product_draft_data_product_model_json = {} + data_product_draft_data_product_model_json['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_draft_data_product_model_json['release'] = data_product_draft_version_release_model + data_product_draft_data_product_model_json['container'] = container_reference_model + + # Construct a model instance of DataProductDraftDataProduct by calling from_dict on the json representation + data_product_draft_data_product_model = DataProductDraftDataProduct.from_dict(data_product_draft_data_product_model_json) + assert data_product_draft_data_product_model != False + + # Construct a model instance of DataProductDraftDataProduct by calling from_dict on the json representation + data_product_draft_data_product_model_dict = DataProductDraftDataProduct.from_dict(data_product_draft_data_product_model_json).__dict__ + data_product_draft_data_product_model2 = DataProductDraftDataProduct(**data_product_draft_data_product_model_dict) + + # Verify the model instances are equivalent + assert data_product_draft_data_product_model == data_product_draft_data_product_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_draft_data_product_model_json2 = data_product_draft_data_product_model.to_dict() + assert data_product_draft_data_product_model_json2 == data_product_draft_data_product_model_json + + +class TestModel_DataProductDraftPrototype: + """ + Test Class for DataProductDraftPrototype + """ + + def test_data_product_draft_prototype_serialization(self): + """ + Test serialization/deserialization for DataProductDraftPrototype + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_identity_model = {} # DataProductIdentity + data_product_identity_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_identity_model['release'] = data_product_draft_version_release_model + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + asset_prototype_model = {} # AssetPrototype + asset_prototype_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_prototype_model['container'] = container_identity_model + + # Construct a json representation of a DataProductDraftPrototype model + data_product_draft_prototype_model_json = {} + data_product_draft_prototype_model_json['version'] = '1.0.0' + data_product_draft_prototype_model_json['state'] = 'draft' + data_product_draft_prototype_model_json['data_product'] = data_product_identity_model + data_product_draft_prototype_model_json['name'] = 'My Data Product' + data_product_draft_prototype_model_json['description'] = 'This is a description of My Data Product.' + data_product_draft_prototype_model_json['tags'] = ['testString'] + data_product_draft_prototype_model_json['use_cases'] = [use_case_model] + data_product_draft_prototype_model_json['types'] = ['data'] + data_product_draft_prototype_model_json['contract_terms'] = [contract_terms_model] + data_product_draft_prototype_model_json['domain'] = domain_model + data_product_draft_prototype_model_json['parts_out'] = [data_product_part_model] + data_product_draft_prototype_model_json['workflows'] = data_product_workflows_model + data_product_draft_prototype_model_json['dataview_enabled'] = True + data_product_draft_prototype_model_json['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_draft_prototype_model_json['access_control'] = asset_list_access_control_model + data_product_draft_prototype_model_json['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_draft_prototype_model_json['sub_container'] = container_identity_model + data_product_draft_prototype_model_json['is_restricted'] = True + data_product_draft_prototype_model_json['asset'] = asset_prototype_model + + # Construct a model instance of DataProductDraftPrototype by calling from_dict on the json representation + data_product_draft_prototype_model = DataProductDraftPrototype.from_dict(data_product_draft_prototype_model_json) + assert data_product_draft_prototype_model != False + + # Construct a model instance of DataProductDraftPrototype by calling from_dict on the json representation + data_product_draft_prototype_model_dict = DataProductDraftPrototype.from_dict(data_product_draft_prototype_model_json).__dict__ + data_product_draft_prototype_model2 = DataProductDraftPrototype(**data_product_draft_prototype_model_dict) + + # Verify the model instances are equivalent + assert data_product_draft_prototype_model == data_product_draft_prototype_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_draft_prototype_model_json2 = data_product_draft_prototype_model.to_dict() + assert data_product_draft_prototype_model_json2 == data_product_draft_prototype_model_json + + +class TestModel_DataProductDraftSummary: + """ + Test Class for DataProductDraftSummary + """ + + def test_data_product_draft_summary_serialization(self): + """ + Test serialization/deserialization for DataProductDraftSummary + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_draft_summary_data_product_model = {} # DataProductDraftSummaryDataProduct + data_product_draft_summary_data_product_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_draft_summary_data_product_model['release'] = data_product_draft_version_release_model + data_product_draft_summary_data_product_model['container'] = container_reference_model + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a json representation of a DataProductDraftSummary model + data_product_draft_summary_model_json = {} + data_product_draft_summary_model_json['version'] = '1.0.0' + data_product_draft_summary_model_json['state'] = 'draft' + data_product_draft_summary_model_json['data_product'] = data_product_draft_summary_data_product_model + data_product_draft_summary_model_json['name'] = 'My Data Product' + data_product_draft_summary_model_json['description'] = 'This is a description of My Data Product.' + data_product_draft_summary_model_json['tags'] = ['testString'] + data_product_draft_summary_model_json['use_cases'] = [use_case_model] + data_product_draft_summary_model_json['types'] = ['data'] + data_product_draft_summary_model_json['contract_terms'] = [contract_terms_model] + data_product_draft_summary_model_json['domain'] = domain_model + data_product_draft_summary_model_json['parts_out'] = [data_product_part_model] + data_product_draft_summary_model_json['workflows'] = data_product_workflows_model + data_product_draft_summary_model_json['dataview_enabled'] = True + data_product_draft_summary_model_json['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_draft_summary_model_json['access_control'] = asset_list_access_control_model + data_product_draft_summary_model_json['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_draft_summary_model_json['sub_container'] = container_identity_model + data_product_draft_summary_model_json['is_restricted'] = True + data_product_draft_summary_model_json['id'] = '2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd' + data_product_draft_summary_model_json['asset'] = asset_reference_model + + # Construct a model instance of DataProductDraftSummary by calling from_dict on the json representation + data_product_draft_summary_model = DataProductDraftSummary.from_dict(data_product_draft_summary_model_json) + assert data_product_draft_summary_model != False + + # Construct a model instance of DataProductDraftSummary by calling from_dict on the json representation + data_product_draft_summary_model_dict = DataProductDraftSummary.from_dict(data_product_draft_summary_model_json).__dict__ + data_product_draft_summary_model2 = DataProductDraftSummary(**data_product_draft_summary_model_dict) + + # Verify the model instances are equivalent + assert data_product_draft_summary_model == data_product_draft_summary_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_draft_summary_model_json2 = data_product_draft_summary_model.to_dict() + assert data_product_draft_summary_model_json2 == data_product_draft_summary_model_json + + +class TestModel_DataProductDraftSummaryDataProduct: + """ + Test Class for DataProductDraftSummaryDataProduct + """ + + def test_data_product_draft_summary_data_product_serialization(self): + """ + Test serialization/deserialization for DataProductDraftSummaryDataProduct + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a DataProductDraftSummaryDataProduct model + data_product_draft_summary_data_product_model_json = {} + data_product_draft_summary_data_product_model_json['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_draft_summary_data_product_model_json['release'] = data_product_draft_version_release_model + data_product_draft_summary_data_product_model_json['container'] = container_reference_model + + # Construct a model instance of DataProductDraftSummaryDataProduct by calling from_dict on the json representation + data_product_draft_summary_data_product_model = DataProductDraftSummaryDataProduct.from_dict(data_product_draft_summary_data_product_model_json) + assert data_product_draft_summary_data_product_model != False + + # Construct a model instance of DataProductDraftSummaryDataProduct by calling from_dict on the json representation + data_product_draft_summary_data_product_model_dict = DataProductDraftSummaryDataProduct.from_dict(data_product_draft_summary_data_product_model_json).__dict__ + data_product_draft_summary_data_product_model2 = DataProductDraftSummaryDataProduct(**data_product_draft_summary_data_product_model_dict) + + # Verify the model instances are equivalent + assert data_product_draft_summary_data_product_model == data_product_draft_summary_data_product_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_draft_summary_data_product_model_json2 = data_product_draft_summary_data_product_model.to_dict() + assert data_product_draft_summary_data_product_model_json2 == data_product_draft_summary_data_product_model_json + + +class TestModel_DataProductDraftVersionRelease: + """ + Test Class for DataProductDraftVersionRelease + """ + + def test_data_product_draft_version_release_serialization(self): + """ + Test serialization/deserialization for DataProductDraftVersionRelease + """ + + # Construct a json representation of a DataProductDraftVersionRelease model + data_product_draft_version_release_model_json = {} + data_product_draft_version_release_model_json['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a model instance of DataProductDraftVersionRelease by calling from_dict on the json representation + data_product_draft_version_release_model = DataProductDraftVersionRelease.from_dict(data_product_draft_version_release_model_json) + assert data_product_draft_version_release_model != False + + # Construct a model instance of DataProductDraftVersionRelease by calling from_dict on the json representation + data_product_draft_version_release_model_dict = DataProductDraftVersionRelease.from_dict(data_product_draft_version_release_model_json).__dict__ + data_product_draft_version_release_model2 = DataProductDraftVersionRelease(**data_product_draft_version_release_model_dict) + + # Verify the model instances are equivalent + assert data_product_draft_version_release_model == data_product_draft_version_release_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_draft_version_release_model_json2 = data_product_draft_version_release_model.to_dict() + assert data_product_draft_version_release_model_json2 == data_product_draft_version_release_model_json + + +class TestModel_DataProductIdentity: + """ + Test Class for DataProductIdentity + """ + + def test_data_product_identity_serialization(self): + """ + Test serialization/deserialization for DataProductIdentity + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a json representation of a DataProductIdentity model + data_product_identity_model_json = {} + data_product_identity_model_json['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_identity_model_json['release'] = data_product_draft_version_release_model + + # Construct a model instance of DataProductIdentity by calling from_dict on the json representation + data_product_identity_model = DataProductIdentity.from_dict(data_product_identity_model_json) + assert data_product_identity_model != False + + # Construct a model instance of DataProductIdentity by calling from_dict on the json representation + data_product_identity_model_dict = DataProductIdentity.from_dict(data_product_identity_model_json).__dict__ + data_product_identity_model2 = DataProductIdentity(**data_product_identity_model_dict) + + # Verify the model instances are equivalent + assert data_product_identity_model == data_product_identity_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_identity_model_json2 = data_product_identity_model.to_dict() + assert data_product_identity_model_json2 == data_product_identity_model_json + + +class TestModel_DataProductOrderAccessRequest: + """ + Test Class for DataProductOrderAccessRequest + """ + + def test_data_product_order_access_request_serialization(self): + """ + Test serialization/deserialization for DataProductOrderAccessRequest + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a json representation of a DataProductOrderAccessRequest model + data_product_order_access_request_model_json = {} + data_product_order_access_request_model_json['task_assignee_users'] = ['testString'] + data_product_order_access_request_model_json['pre_approved_users'] = ['testString'] + data_product_order_access_request_model_json['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + # Construct a model instance of DataProductOrderAccessRequest by calling from_dict on the json representation + data_product_order_access_request_model = DataProductOrderAccessRequest.from_dict(data_product_order_access_request_model_json) + assert data_product_order_access_request_model != False + + # Construct a model instance of DataProductOrderAccessRequest by calling from_dict on the json representation + data_product_order_access_request_model_dict = DataProductOrderAccessRequest.from_dict(data_product_order_access_request_model_json).__dict__ + data_product_order_access_request_model2 = DataProductOrderAccessRequest(**data_product_order_access_request_model_dict) + + # Verify the model instances are equivalent + assert data_product_order_access_request_model == data_product_order_access_request_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_order_access_request_model_json2 = data_product_order_access_request_model.to_dict() + assert data_product_order_access_request_model_json2 == data_product_order_access_request_model_json + + +class TestModel_DataProductPart: + """ + Test Class for DataProductPart + """ + + def test_data_product_part_serialization(self): + """ + Test serialization/deserialization for DataProductPart + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + # Construct a json representation of a DataProductPart model + data_product_part_model_json = {} + data_product_part_model_json['asset'] = asset_part_reference_model + data_product_part_model_json['delivery_methods'] = [delivery_method_model] + + # Construct a model instance of DataProductPart by calling from_dict on the json representation + data_product_part_model = DataProductPart.from_dict(data_product_part_model_json) + assert data_product_part_model != False + + # Construct a model instance of DataProductPart by calling from_dict on the json representation + data_product_part_model_dict = DataProductPart.from_dict(data_product_part_model_json).__dict__ + data_product_part_model2 = DataProductPart(**data_product_part_model_dict) + + # Verify the model instances are equivalent + assert data_product_part_model == data_product_part_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_part_model_json2 = data_product_part_model.to_dict() + assert data_product_part_model_json2 == data_product_part_model_json + + +class TestModel_DataProductRelease: + """ + Test Class for DataProductRelease + """ + + def test_data_product_release_serialization(self): + """ + Test serialization/deserialization for DataProductRelease + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_release_data_product_model = {} # DataProductReleaseDataProduct + data_product_release_data_product_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_release_data_product_model['release'] = data_product_draft_version_release_model + data_product_release_data_product_model['container'] = container_reference_model + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + visualization_model = {} # Visualization + visualization_model['id'] = 'testString' + visualization_model['name'] = 'testString' + + error_message_model = {} # ErrorMessage + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + data_asset_relationship_model = {} # DataAssetRelationship + data_asset_relationship_model['visualization'] = visualization_model + data_asset_relationship_model['asset'] = asset_reference_model + data_asset_relationship_model['related_asset'] = asset_reference_model + data_asset_relationship_model['error'] = error_message_model + + # Construct a json representation of a DataProductRelease model + data_product_release_model_json = {} + data_product_release_model_json['version'] = '1.0.0' + data_product_release_model_json['state'] = 'draft' + data_product_release_model_json['data_product'] = data_product_release_data_product_model + data_product_release_model_json['name'] = 'My Data Product' + data_product_release_model_json['description'] = 'This is a description of My Data Product.' + data_product_release_model_json['tags'] = ['testString'] + data_product_release_model_json['use_cases'] = [use_case_model] + data_product_release_model_json['types'] = ['data'] + data_product_release_model_json['contract_terms'] = [contract_terms_model] + data_product_release_model_json['domain'] = domain_model + data_product_release_model_json['parts_out'] = [data_product_part_model] + data_product_release_model_json['workflows'] = data_product_workflows_model + data_product_release_model_json['dataview_enabled'] = True + data_product_release_model_json['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_release_model_json['access_control'] = asset_list_access_control_model + data_product_release_model_json['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_release_model_json['sub_container'] = container_identity_model + data_product_release_model_json['is_restricted'] = True + data_product_release_model_json['id'] = '2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd' + data_product_release_model_json['asset'] = asset_reference_model + data_product_release_model_json['published_by'] = 'testString' + data_product_release_model_json['published_at'] = '2019-01-01T12:00:00Z' + data_product_release_model_json['created_by'] = 'testString' + data_product_release_model_json['created_at'] = '2019-01-01T12:00:00Z' + data_product_release_model_json['properties'] = {'anyKey': 'anyValue'} + data_product_release_model_json['visualization_errors'] = [data_asset_relationship_model] + + # Construct a model instance of DataProductRelease by calling from_dict on the json representation + data_product_release_model = DataProductRelease.from_dict(data_product_release_model_json) + assert data_product_release_model != False + + # Construct a model instance of DataProductRelease by calling from_dict on the json representation + data_product_release_model_dict = DataProductRelease.from_dict(data_product_release_model_json).__dict__ + data_product_release_model2 = DataProductRelease(**data_product_release_model_dict) + + # Verify the model instances are equivalent + assert data_product_release_model == data_product_release_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_release_model_json2 = data_product_release_model.to_dict() + assert data_product_release_model_json2 == data_product_release_model_json + + +class TestModel_DataProductReleaseCollection: + """ + Test Class for DataProductReleaseCollection + """ + + def test_data_product_release_collection_serialization(self): + """ + Test serialization/deserialization for DataProductReleaseCollection + """ + + # Construct dict forms of any model objects needed in order to build this model. + + first_page_model = {} # FirstPage + first_page_model['href'] = 'https://api.example.com/collection' + + next_page_model = {} # NextPage + next_page_model['href'] = 'https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + next_page_model['start'] = 'eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_release_summary_data_product_model = {} # DataProductReleaseSummaryDataProduct + data_product_release_summary_data_product_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_release_summary_data_product_model['release'] = data_product_draft_version_release_model + data_product_release_summary_data_product_model['container'] = container_reference_model + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + data_product_release_summary_model = {} # DataProductReleaseSummary + data_product_release_summary_model['version'] = '1.0.0' + data_product_release_summary_model['state'] = 'draft' + data_product_release_summary_model['data_product'] = data_product_release_summary_data_product_model + data_product_release_summary_model['name'] = 'My Data Product' + data_product_release_summary_model['description'] = 'This is a description of My Data Product.' + data_product_release_summary_model['tags'] = ['testString'] + data_product_release_summary_model['use_cases'] = [use_case_model] + data_product_release_summary_model['types'] = ['data'] + data_product_release_summary_model['contract_terms'] = [contract_terms_model] + data_product_release_summary_model['domain'] = domain_model + data_product_release_summary_model['parts_out'] = [data_product_part_model] + data_product_release_summary_model['workflows'] = data_product_workflows_model + data_product_release_summary_model['dataview_enabled'] = True + data_product_release_summary_model['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_release_summary_model['access_control'] = asset_list_access_control_model + data_product_release_summary_model['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_release_summary_model['sub_container'] = container_identity_model + data_product_release_summary_model['is_restricted'] = True + data_product_release_summary_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd' + data_product_release_summary_model['asset'] = asset_reference_model + + # Construct a json representation of a DataProductReleaseCollection model + data_product_release_collection_model_json = {} + data_product_release_collection_model_json['limit'] = 200 + data_product_release_collection_model_json['first'] = first_page_model + data_product_release_collection_model_json['next'] = next_page_model + data_product_release_collection_model_json['total_results'] = 200 + data_product_release_collection_model_json['releases'] = [data_product_release_summary_model] + + # Construct a model instance of DataProductReleaseCollection by calling from_dict on the json representation + data_product_release_collection_model = DataProductReleaseCollection.from_dict(data_product_release_collection_model_json) + assert data_product_release_collection_model != False + + # Construct a model instance of DataProductReleaseCollection by calling from_dict on the json representation + data_product_release_collection_model_dict = DataProductReleaseCollection.from_dict(data_product_release_collection_model_json).__dict__ + data_product_release_collection_model2 = DataProductReleaseCollection(**data_product_release_collection_model_dict) + + # Verify the model instances are equivalent + assert data_product_release_collection_model == data_product_release_collection_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_release_collection_model_json2 = data_product_release_collection_model.to_dict() + assert data_product_release_collection_model_json2 == data_product_release_collection_model_json + + +class TestModel_DataProductReleaseDataProduct: + """ + Test Class for DataProductReleaseDataProduct + """ + + def test_data_product_release_data_product_serialization(self): + """ + Test serialization/deserialization for DataProductReleaseDataProduct + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a DataProductReleaseDataProduct model + data_product_release_data_product_model_json = {} + data_product_release_data_product_model_json['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_release_data_product_model_json['release'] = data_product_draft_version_release_model + data_product_release_data_product_model_json['container'] = container_reference_model + + # Construct a model instance of DataProductReleaseDataProduct by calling from_dict on the json representation + data_product_release_data_product_model = DataProductReleaseDataProduct.from_dict(data_product_release_data_product_model_json) + assert data_product_release_data_product_model != False + + # Construct a model instance of DataProductReleaseDataProduct by calling from_dict on the json representation + data_product_release_data_product_model_dict = DataProductReleaseDataProduct.from_dict(data_product_release_data_product_model_json).__dict__ + data_product_release_data_product_model2 = DataProductReleaseDataProduct(**data_product_release_data_product_model_dict) + + # Verify the model instances are equivalent + assert data_product_release_data_product_model == data_product_release_data_product_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_release_data_product_model_json2 = data_product_release_data_product_model.to_dict() + assert data_product_release_data_product_model_json2 == data_product_release_data_product_model_json + + +class TestModel_DataProductReleaseSummary: + """ + Test Class for DataProductReleaseSummary + """ + + def test_data_product_release_summary_serialization(self): + """ + Test serialization/deserialization for DataProductReleaseSummary + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_release_summary_data_product_model = {} # DataProductReleaseSummaryDataProduct + data_product_release_summary_data_product_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_release_summary_data_product_model['release'] = data_product_draft_version_release_model + data_product_release_summary_data_product_model['container'] = container_reference_model + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a json representation of a DataProductReleaseSummary model + data_product_release_summary_model_json = {} + data_product_release_summary_model_json['version'] = '1.0.0' + data_product_release_summary_model_json['state'] = 'draft' + data_product_release_summary_model_json['data_product'] = data_product_release_summary_data_product_model + data_product_release_summary_model_json['name'] = 'My Data Product' + data_product_release_summary_model_json['description'] = 'This is a description of My Data Product.' + data_product_release_summary_model_json['tags'] = ['testString'] + data_product_release_summary_model_json['use_cases'] = [use_case_model] + data_product_release_summary_model_json['types'] = ['data'] + data_product_release_summary_model_json['contract_terms'] = [contract_terms_model] + data_product_release_summary_model_json['domain'] = domain_model + data_product_release_summary_model_json['parts_out'] = [data_product_part_model] + data_product_release_summary_model_json['workflows'] = data_product_workflows_model + data_product_release_summary_model_json['dataview_enabled'] = True + data_product_release_summary_model_json['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_release_summary_model_json['access_control'] = asset_list_access_control_model + data_product_release_summary_model_json['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_release_summary_model_json['sub_container'] = container_identity_model + data_product_release_summary_model_json['is_restricted'] = True + data_product_release_summary_model_json['id'] = '2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd' + data_product_release_summary_model_json['asset'] = asset_reference_model + + # Construct a model instance of DataProductReleaseSummary by calling from_dict on the json representation + data_product_release_summary_model = DataProductReleaseSummary.from_dict(data_product_release_summary_model_json) + assert data_product_release_summary_model != False + + # Construct a model instance of DataProductReleaseSummary by calling from_dict on the json representation + data_product_release_summary_model_dict = DataProductReleaseSummary.from_dict(data_product_release_summary_model_json).__dict__ + data_product_release_summary_model2 = DataProductReleaseSummary(**data_product_release_summary_model_dict) + + # Verify the model instances are equivalent + assert data_product_release_summary_model == data_product_release_summary_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_release_summary_model_json2 = data_product_release_summary_model.to_dict() + assert data_product_release_summary_model_json2 == data_product_release_summary_model_json + + +class TestModel_DataProductReleaseSummaryDataProduct: + """ + Test Class for DataProductReleaseSummaryDataProduct + """ + + def test_data_product_release_summary_data_product_serialization(self): + """ + Test serialization/deserialization for DataProductReleaseSummaryDataProduct + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a DataProductReleaseSummaryDataProduct model + data_product_release_summary_data_product_model_json = {} + data_product_release_summary_data_product_model_json['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_release_summary_data_product_model_json['release'] = data_product_draft_version_release_model + data_product_release_summary_data_product_model_json['container'] = container_reference_model + + # Construct a model instance of DataProductReleaseSummaryDataProduct by calling from_dict on the json representation + data_product_release_summary_data_product_model = DataProductReleaseSummaryDataProduct.from_dict(data_product_release_summary_data_product_model_json) + assert data_product_release_summary_data_product_model != False + + # Construct a model instance of DataProductReleaseSummaryDataProduct by calling from_dict on the json representation + data_product_release_summary_data_product_model_dict = DataProductReleaseSummaryDataProduct.from_dict(data_product_release_summary_data_product_model_json).__dict__ + data_product_release_summary_data_product_model2 = DataProductReleaseSummaryDataProduct(**data_product_release_summary_data_product_model_dict) + + # Verify the model instances are equivalent + assert data_product_release_summary_data_product_model == data_product_release_summary_data_product_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_release_summary_data_product_model_json2 = data_product_release_summary_data_product_model.to_dict() + assert data_product_release_summary_data_product_model_json2 == data_product_release_summary_data_product_model_json + + +class TestModel_DataProductSummary: + """ + Test Class for DataProductSummary + """ + + def test_data_product_summary_serialization(self): + """ + Test serialization/deserialization for DataProductSummary + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a DataProductSummary model + data_product_summary_model_json = {} + data_product_summary_model_json['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_summary_model_json['release'] = data_product_draft_version_release_model + data_product_summary_model_json['container'] = container_reference_model + data_product_summary_model_json['name'] = 'testString' + + # Construct a model instance of DataProductSummary by calling from_dict on the json representation + data_product_summary_model = DataProductSummary.from_dict(data_product_summary_model_json) + assert data_product_summary_model != False + + # Construct a model instance of DataProductSummary by calling from_dict on the json representation + data_product_summary_model_dict = DataProductSummary.from_dict(data_product_summary_model_json).__dict__ + data_product_summary_model2 = DataProductSummary(**data_product_summary_model_dict) + + # Verify the model instances are equivalent + assert data_product_summary_model == data_product_summary_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_summary_model_json2 = data_product_summary_model.to_dict() + assert data_product_summary_model_json2 == data_product_summary_model_json + + +class TestModel_DataProductVersionCollection: + """ + Test Class for DataProductVersionCollection + """ + + def test_data_product_version_collection_serialization(self): + """ + Test serialization/deserialization for DataProductVersionCollection + """ + + # Construct dict forms of any model objects needed in order to build this model. + + first_page_model = {} # FirstPage + first_page_model['href'] = 'https://api.example.com/collection' + + next_page_model = {} # NextPage + next_page_model['href'] = 'https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + next_page_model['start'] = 'eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_version_summary_data_product_model = {} # DataProductVersionSummaryDataProduct + data_product_version_summary_data_product_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_version_summary_data_product_model['release'] = data_product_draft_version_release_model + data_product_version_summary_data_product_model['container'] = container_reference_model + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + data_product_version_summary_model = {} # DataProductVersionSummary + data_product_version_summary_model['version'] = '1.0.0' + data_product_version_summary_model['state'] = 'draft' + data_product_version_summary_model['data_product'] = data_product_version_summary_data_product_model + data_product_version_summary_model['name'] = 'My Data Product' + data_product_version_summary_model['description'] = 'This is a description of My Data Product.' + data_product_version_summary_model['tags'] = ['testString'] + data_product_version_summary_model['use_cases'] = [use_case_model] + data_product_version_summary_model['types'] = ['data'] + data_product_version_summary_model['contract_terms'] = [contract_terms_model] + data_product_version_summary_model['domain'] = domain_model + data_product_version_summary_model['parts_out'] = [data_product_part_model] + data_product_version_summary_model['workflows'] = data_product_workflows_model + data_product_version_summary_model['dataview_enabled'] = True + data_product_version_summary_model['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_version_summary_model['access_control'] = asset_list_access_control_model + data_product_version_summary_model['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_version_summary_model['sub_container'] = container_identity_model + data_product_version_summary_model['is_restricted'] = True + data_product_version_summary_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd' + data_product_version_summary_model['asset'] = asset_reference_model + + # Construct a json representation of a DataProductVersionCollection model + data_product_version_collection_model_json = {} + data_product_version_collection_model_json['limit'] = 200 + data_product_version_collection_model_json['first'] = first_page_model + data_product_version_collection_model_json['next'] = next_page_model + data_product_version_collection_model_json['total_results'] = 200 + data_product_version_collection_model_json['data_product_versions'] = [data_product_version_summary_model] + + # Construct a model instance of DataProductVersionCollection by calling from_dict on the json representation + data_product_version_collection_model = DataProductVersionCollection.from_dict(data_product_version_collection_model_json) + assert data_product_version_collection_model != False + + # Construct a model instance of DataProductVersionCollection by calling from_dict on the json representation + data_product_version_collection_model_dict = DataProductVersionCollection.from_dict(data_product_version_collection_model_json).__dict__ + data_product_version_collection_model2 = DataProductVersionCollection(**data_product_version_collection_model_dict) + + # Verify the model instances are equivalent + assert data_product_version_collection_model == data_product_version_collection_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_version_collection_model_json2 = data_product_version_collection_model.to_dict() + assert data_product_version_collection_model_json2 == data_product_version_collection_model_json + + +class TestModel_DataProductVersionSummary: + """ + Test Class for DataProductVersionSummary + """ + + def test_data_product_version_summary_serialization(self): + """ + Test serialization/deserialization for DataProductVersionSummary + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_version_summary_data_product_model = {} # DataProductVersionSummaryDataProduct + data_product_version_summary_data_product_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_version_summary_data_product_model['release'] = data_product_draft_version_release_model + data_product_version_summary_data_product_model['container'] = container_reference_model + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a json representation of a DataProductVersionSummary model + data_product_version_summary_model_json = {} + data_product_version_summary_model_json['version'] = '1.0.0' + data_product_version_summary_model_json['state'] = 'draft' + data_product_version_summary_model_json['data_product'] = data_product_version_summary_data_product_model + data_product_version_summary_model_json['name'] = 'My Data Product' + data_product_version_summary_model_json['description'] = 'This is a description of My Data Product.' + data_product_version_summary_model_json['tags'] = ['testString'] + data_product_version_summary_model_json['use_cases'] = [use_case_model] + data_product_version_summary_model_json['types'] = ['data'] + data_product_version_summary_model_json['contract_terms'] = [contract_terms_model] + data_product_version_summary_model_json['domain'] = domain_model + data_product_version_summary_model_json['parts_out'] = [data_product_part_model] + data_product_version_summary_model_json['workflows'] = data_product_workflows_model + data_product_version_summary_model_json['dataview_enabled'] = True + data_product_version_summary_model_json['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_version_summary_model_json['access_control'] = asset_list_access_control_model + data_product_version_summary_model_json['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_version_summary_model_json['sub_container'] = container_identity_model + data_product_version_summary_model_json['is_restricted'] = True + data_product_version_summary_model_json['id'] = '2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd' + data_product_version_summary_model_json['asset'] = asset_reference_model + + # Construct a model instance of DataProductVersionSummary by calling from_dict on the json representation + data_product_version_summary_model = DataProductVersionSummary.from_dict(data_product_version_summary_model_json) + assert data_product_version_summary_model != False + + # Construct a model instance of DataProductVersionSummary by calling from_dict on the json representation + data_product_version_summary_model_dict = DataProductVersionSummary.from_dict(data_product_version_summary_model_json).__dict__ + data_product_version_summary_model2 = DataProductVersionSummary(**data_product_version_summary_model_dict) + + # Verify the model instances are equivalent + assert data_product_version_summary_model == data_product_version_summary_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_version_summary_model_json2 = data_product_version_summary_model.to_dict() + assert data_product_version_summary_model_json2 == data_product_version_summary_model_json + + +class TestModel_DataProductVersionSummaryDataProduct: + """ + Test Class for DataProductVersionSummaryDataProduct + """ + + def test_data_product_version_summary_data_product_serialization(self): + """ + Test serialization/deserialization for DataProductVersionSummaryDataProduct + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a DataProductVersionSummaryDataProduct model + data_product_version_summary_data_product_model_json = {} + data_product_version_summary_data_product_model_json['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_version_summary_data_product_model_json['release'] = data_product_draft_version_release_model + data_product_version_summary_data_product_model_json['container'] = container_reference_model + + # Construct a model instance of DataProductVersionSummaryDataProduct by calling from_dict on the json representation + data_product_version_summary_data_product_model = DataProductVersionSummaryDataProduct.from_dict(data_product_version_summary_data_product_model_json) + assert data_product_version_summary_data_product_model != False + + # Construct a model instance of DataProductVersionSummaryDataProduct by calling from_dict on the json representation + data_product_version_summary_data_product_model_dict = DataProductVersionSummaryDataProduct.from_dict(data_product_version_summary_data_product_model_json).__dict__ + data_product_version_summary_data_product_model2 = DataProductVersionSummaryDataProduct(**data_product_version_summary_data_product_model_dict) + + # Verify the model instances are equivalent + assert data_product_version_summary_data_product_model == data_product_version_summary_data_product_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_version_summary_data_product_model_json2 = data_product_version_summary_data_product_model.to_dict() + assert data_product_version_summary_data_product_model_json2 == data_product_version_summary_data_product_model_json + + +class TestModel_DataProductWorkflows: + """ + Test Class for DataProductWorkflows + """ + + def test_data_product_workflows_serialization(self): + """ + Test serialization/deserialization for DataProductWorkflows + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + # Construct a json representation of a DataProductWorkflows model + data_product_workflows_model_json = {} + data_product_workflows_model_json['order_access_request'] = data_product_order_access_request_model + + # Construct a model instance of DataProductWorkflows by calling from_dict on the json representation + data_product_workflows_model = DataProductWorkflows.from_dict(data_product_workflows_model_json) + assert data_product_workflows_model != False + + # Construct a model instance of DataProductWorkflows by calling from_dict on the json representation + data_product_workflows_model_dict = DataProductWorkflows.from_dict(data_product_workflows_model_json).__dict__ + data_product_workflows_model2 = DataProductWorkflows(**data_product_workflows_model_dict) + + # Verify the model instances are equivalent + assert data_product_workflows_model == data_product_workflows_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_workflows_model_json2 = data_product_workflows_model.to_dict() + assert data_product_workflows_model_json2 == data_product_workflows_model_json + + +class TestModel_DeliveryMethod: + """ + Test Class for DeliveryMethod + """ + + def test_delivery_method_serialization(self): + """ + Test serialization/deserialization for DeliveryMethod + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + # Construct a json representation of a DeliveryMethod model + delivery_method_model_json = {} + delivery_method_model_json['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model_json['container'] = container_reference_model + delivery_method_model_json['getproperties'] = delivery_method_properties_model_model + + # Construct a model instance of DeliveryMethod by calling from_dict on the json representation + delivery_method_model = DeliveryMethod.from_dict(delivery_method_model_json) + assert delivery_method_model != False + + # Construct a model instance of DeliveryMethod by calling from_dict on the json representation + delivery_method_model_dict = DeliveryMethod.from_dict(delivery_method_model_json).__dict__ + delivery_method_model2 = DeliveryMethod(**delivery_method_model_dict) + + # Verify the model instances are equivalent + assert delivery_method_model == delivery_method_model2 + + # Convert model instance back to dict and verify no loss of data + delivery_method_model_json2 = delivery_method_model.to_dict() + assert delivery_method_model_json2 == delivery_method_model_json + + +class TestModel_DeliveryMethodPropertiesModel: + """ + Test Class for DeliveryMethodPropertiesModel + """ + + def test_delivery_method_properties_model_serialization(self): + """ + Test serialization/deserialization for DeliveryMethodPropertiesModel + """ + + # Construct dict forms of any model objects needed in order to build this model. + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + # Construct a json representation of a DeliveryMethodPropertiesModel model + delivery_method_properties_model_model_json = {} + delivery_method_properties_model_model_json['producer_input'] = producer_input_model_model + + # Construct a model instance of DeliveryMethodPropertiesModel by calling from_dict on the json representation + delivery_method_properties_model_model = DeliveryMethodPropertiesModel.from_dict(delivery_method_properties_model_model_json) + assert delivery_method_properties_model_model != False + + # Construct a model instance of DeliveryMethodPropertiesModel by calling from_dict on the json representation + delivery_method_properties_model_model_dict = DeliveryMethodPropertiesModel.from_dict(delivery_method_properties_model_model_json).__dict__ + delivery_method_properties_model_model2 = DeliveryMethodPropertiesModel(**delivery_method_properties_model_model_dict) + + # Verify the model instances are equivalent + assert delivery_method_properties_model_model == delivery_method_properties_model_model2 + + # Convert model instance back to dict and verify no loss of data + delivery_method_properties_model_model_json2 = delivery_method_properties_model_model.to_dict() + assert delivery_method_properties_model_model_json2 == delivery_method_properties_model_model_json + + +class TestModel_Description: + """ + Test Class for Description + """ + + def test_description_serialization(self): + """ + Test serialization/deserialization for Description + """ + + # Construct dict forms of any model objects needed in order to build this model. + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + # Construct a json representation of a Description model + description_model_json = {} + description_model_json['purpose'] = 'Used for customer behavior analysis.' + description_model_json['limitations'] = 'Data cannot be used for marketing.' + description_model_json['usage'] = 'Data should be used only for analytics.' + description_model_json['more_info'] = [contract_terms_more_info_model] + description_model_json['custom_properties'] = '{"property1":"value1"}' + + # Construct a model instance of Description by calling from_dict on the json representation + description_model = Description.from_dict(description_model_json) + assert description_model != False + + # Construct a model instance of Description by calling from_dict on the json representation + description_model_dict = Description.from_dict(description_model_json).__dict__ + description_model2 = Description(**description_model_dict) + + # Verify the model instances are equivalent + assert description_model == description_model2 + + # Convert model instance back to dict and verify no loss of data + description_model_json2 = description_model.to_dict() + assert description_model_json2 == description_model_json + + +class TestModel_Domain: + """ + Test Class for Domain + """ + + def test_domain_serialization(self): + """ + Test serialization/deserialization for Domain + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a Domain model + domain_model_json = {} + domain_model_json['id'] = 'testString' + domain_model_json['name'] = 'testString' + domain_model_json['container'] = container_reference_model + + # Construct a model instance of Domain by calling from_dict on the json representation + domain_model = Domain.from_dict(domain_model_json) + assert domain_model != False + + # Construct a model instance of Domain by calling from_dict on the json representation + domain_model_dict = Domain.from_dict(domain_model_json).__dict__ + domain_model2 = Domain(**domain_model_dict) + + # Verify the model instances are equivalent + assert domain_model == domain_model2 + + # Convert model instance back to dict and verify no loss of data + domain_model_json2 = domain_model.to_dict() + assert domain_model_json2 == domain_model_json + + +class TestModel_EngineDetailsModel: + """ + Test Class for EngineDetailsModel + """ + + def test_engine_details_model_serialization(self): + """ + Test serialization/deserialization for EngineDetailsModel + """ + + # Construct a json representation of a EngineDetailsModel model + engine_details_model_model_json = {} + engine_details_model_model_json['display_name'] = 'Iceberg Engine' + engine_details_model_model_json['engine_id'] = 'presto767' + engine_details_model_model_json['engine_port'] = '34567' + engine_details_model_model_json['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model_json['engine_type'] = 'spark' + engine_details_model_model_json['associated_catalogs'] = ['testString'] + + # Construct a model instance of EngineDetailsModel by calling from_dict on the json representation + engine_details_model_model = EngineDetailsModel.from_dict(engine_details_model_model_json) + assert engine_details_model_model != False + + # Construct a model instance of EngineDetailsModel by calling from_dict on the json representation + engine_details_model_model_dict = EngineDetailsModel.from_dict(engine_details_model_model_json).__dict__ + engine_details_model_model2 = EngineDetailsModel(**engine_details_model_model_dict) + + # Verify the model instances are equivalent + assert engine_details_model_model == engine_details_model_model2 + + # Convert model instance back to dict and verify no loss of data + engine_details_model_model_json2 = engine_details_model_model.to_dict() + assert engine_details_model_model_json2 == engine_details_model_model_json + + +class TestModel_ErrorExtraResource: + """ + Test Class for ErrorExtraResource + """ + + def test_error_extra_resource_serialization(self): + """ + Test serialization/deserialization for ErrorExtraResource + """ + + # Construct a json representation of a ErrorExtraResource model + error_extra_resource_model_json = {} + error_extra_resource_model_json['id'] = 'testString' + error_extra_resource_model_json['timestamp'] = '2019-01-01T12:00:00Z' + error_extra_resource_model_json['environment_name'] = 'testString' + error_extra_resource_model_json['http_status'] = 0 + error_extra_resource_model_json['source_cluster'] = 0 + error_extra_resource_model_json['source_component'] = 0 + error_extra_resource_model_json['transaction_id'] = 0 + + # Construct a model instance of ErrorExtraResource by calling from_dict on the json representation + error_extra_resource_model = ErrorExtraResource.from_dict(error_extra_resource_model_json) + assert error_extra_resource_model != False + + # Construct a model instance of ErrorExtraResource by calling from_dict on the json representation + error_extra_resource_model_dict = ErrorExtraResource.from_dict(error_extra_resource_model_json).__dict__ + error_extra_resource_model2 = ErrorExtraResource(**error_extra_resource_model_dict) + + # Verify the model instances are equivalent + assert error_extra_resource_model == error_extra_resource_model2 + + # Convert model instance back to dict and verify no loss of data + error_extra_resource_model_json2 = error_extra_resource_model.to_dict() + assert error_extra_resource_model_json2 == error_extra_resource_model_json + + +class TestModel_ErrorMessage: + """ + Test Class for ErrorMessage + """ + + def test_error_message_serialization(self): + """ + Test serialization/deserialization for ErrorMessage + """ + + # Construct a json representation of a ErrorMessage model + error_message_model_json = {} + error_message_model_json['code'] = 'testString' + error_message_model_json['message'] = 'testString' + + # Construct a model instance of ErrorMessage by calling from_dict on the json representation + error_message_model = ErrorMessage.from_dict(error_message_model_json) + assert error_message_model != False + + # Construct a model instance of ErrorMessage by calling from_dict on the json representation + error_message_model_dict = ErrorMessage.from_dict(error_message_model_json).__dict__ + error_message_model2 = ErrorMessage(**error_message_model_dict) + + # Verify the model instances are equivalent + assert error_message_model == error_message_model2 + + # Convert model instance back to dict and verify no loss of data + error_message_model_json2 = error_message_model.to_dict() + assert error_message_model_json2 == error_message_model_json + + +class TestModel_ErrorModelResource: + """ + Test Class for ErrorModelResource + """ + + def test_error_model_resource_serialization(self): + """ + Test serialization/deserialization for ErrorModelResource + """ + + # Construct dict forms of any model objects needed in order to build this model. + + error_extra_resource_model = {} # ErrorExtraResource + error_extra_resource_model['id'] = 'testString' + error_extra_resource_model['timestamp'] = '2019-01-01T12:00:00Z' + error_extra_resource_model['environment_name'] = 'testString' + error_extra_resource_model['http_status'] = 0 + error_extra_resource_model['source_cluster'] = 0 + error_extra_resource_model['source_component'] = 0 + error_extra_resource_model['transaction_id'] = 0 + + # Construct a json representation of a ErrorModelResource model + error_model_resource_model_json = {} + error_model_resource_model_json['code'] = 'request_body_error' + error_model_resource_model_json['message'] = 'testString' + error_model_resource_model_json['extra'] = error_extra_resource_model + error_model_resource_model_json['more_info'] = 'testString' + + # Construct a model instance of ErrorModelResource by calling from_dict on the json representation + error_model_resource_model = ErrorModelResource.from_dict(error_model_resource_model_json) + assert error_model_resource_model != False + + # Construct a model instance of ErrorModelResource by calling from_dict on the json representation + error_model_resource_model_dict = ErrorModelResource.from_dict(error_model_resource_model_json).__dict__ + error_model_resource_model2 = ErrorModelResource(**error_model_resource_model_dict) + + # Verify the model instances are equivalent + assert error_model_resource_model == error_model_resource_model2 + + # Convert model instance back to dict and verify no loss of data + error_model_resource_model_json2 = error_model_resource_model.to_dict() + assert error_model_resource_model_json2 == error_model_resource_model_json + + +class TestModel_FirstPage: + """ + Test Class for FirstPage + """ + + def test_first_page_serialization(self): + """ + Test serialization/deserialization for FirstPage + """ + + # Construct a json representation of a FirstPage model + first_page_model_json = {} + first_page_model_json['href'] = 'https://api.example.com/collection' + + # Construct a model instance of FirstPage by calling from_dict on the json representation + first_page_model = FirstPage.from_dict(first_page_model_json) + assert first_page_model != False + + # Construct a model instance of FirstPage by calling from_dict on the json representation + first_page_model_dict = FirstPage.from_dict(first_page_model_json).__dict__ + first_page_model2 = FirstPage(**first_page_model_dict) + + # Verify the model instances are equivalent + assert first_page_model == first_page_model2 + + # Convert model instance back to dict and verify no loss of data + first_page_model_json2 = first_page_model.to_dict() + assert first_page_model_json2 == first_page_model_json + + +class TestModel_InitializeResource: + """ + Test Class for InitializeResource + """ + + def test_initialize_resource_serialization(self): + """ + Test serialization/deserialization for InitializeResource + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + error_extra_resource_model = {} # ErrorExtraResource + error_extra_resource_model['id'] = 'testString' + error_extra_resource_model['timestamp'] = '2019-01-01T12:00:00Z' + error_extra_resource_model['environment_name'] = 'testString' + error_extra_resource_model['http_status'] = 0 + error_extra_resource_model['source_cluster'] = 0 + error_extra_resource_model['source_component'] = 0 + error_extra_resource_model['transaction_id'] = 0 + + error_model_resource_model = {} # ErrorModelResource + error_model_resource_model['code'] = 'request_body_error' + error_model_resource_model['message'] = 'testString' + error_model_resource_model['extra'] = error_extra_resource_model + error_model_resource_model['more_info'] = 'testString' + + initialized_option_model = {} # InitializedOption + initialized_option_model['name'] = 'testString' + initialized_option_model['version'] = 1 + + workflow_definition_reference_model = {} # WorkflowDefinitionReference + workflow_definition_reference_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + provided_workflow_resource_model = {} # ProvidedWorkflowResource + provided_workflow_resource_model['definition'] = workflow_definition_reference_model + + provided_catalog_workflows_model = {} # ProvidedCatalogWorkflows + provided_catalog_workflows_model['data_access'] = provided_workflow_resource_model + provided_catalog_workflows_model['request_new_product'] = provided_workflow_resource_model + + # Construct a json representation of a InitializeResource model + initialize_resource_model_json = {} + initialize_resource_model_json['container'] = container_reference_model + initialize_resource_model_json['href'] = 'https://api.example.com/configuration/initialize/status?catalog_id=d29c42eb-7100-4b7a-8257-c196dbcca1cd' + initialize_resource_model_json['status'] = 'not_started' + initialize_resource_model_json['trace'] = 'testString' + initialize_resource_model_json['errors'] = [error_model_resource_model] + initialize_resource_model_json['last_started_at'] = '2023-08-21T15:24:06.021000Z' + initialize_resource_model_json['last_finished_at'] = '2023-08-21T20:24:34.450000Z' + initialize_resource_model_json['initialized_options'] = [initialized_option_model] + initialize_resource_model_json['workflows'] = provided_catalog_workflows_model + + # Construct a model instance of InitializeResource by calling from_dict on the json representation + initialize_resource_model = InitializeResource.from_dict(initialize_resource_model_json) + assert initialize_resource_model != False + + # Construct a model instance of InitializeResource by calling from_dict on the json representation + initialize_resource_model_dict = InitializeResource.from_dict(initialize_resource_model_json).__dict__ + initialize_resource_model2 = InitializeResource(**initialize_resource_model_dict) + + # Verify the model instances are equivalent + assert initialize_resource_model == initialize_resource_model2 + + # Convert model instance back to dict and verify no loss of data + initialize_resource_model_json2 = initialize_resource_model.to_dict() + assert initialize_resource_model_json2 == initialize_resource_model_json + + +class TestModel_InitializeSubDomain: + """ + Test Class for InitializeSubDomain + """ + + def test_initialize_sub_domain_serialization(self): + """ + Test serialization/deserialization for InitializeSubDomain + """ + + # Construct a json representation of a InitializeSubDomain model + initialize_sub_domain_model_json = {} + initialize_sub_domain_model_json['name'] = 'Operations' + initialize_sub_domain_model_json['id'] = 'testString' + initialize_sub_domain_model_json['description'] = 'testString' + + # Construct a model instance of InitializeSubDomain by calling from_dict on the json representation + initialize_sub_domain_model = InitializeSubDomain.from_dict(initialize_sub_domain_model_json) + assert initialize_sub_domain_model != False + + # Construct a model instance of InitializeSubDomain by calling from_dict on the json representation + initialize_sub_domain_model_dict = InitializeSubDomain.from_dict(initialize_sub_domain_model_json).__dict__ + initialize_sub_domain_model2 = InitializeSubDomain(**initialize_sub_domain_model_dict) + + # Verify the model instances are equivalent + assert initialize_sub_domain_model == initialize_sub_domain_model2 + + # Convert model instance back to dict and verify no loss of data + initialize_sub_domain_model_json2 = initialize_sub_domain_model.to_dict() + assert initialize_sub_domain_model_json2 == initialize_sub_domain_model_json + + +class TestModel_InitializedOption: + """ + Test Class for InitializedOption + """ + + def test_initialized_option_serialization(self): + """ + Test serialization/deserialization for InitializedOption + """ + + # Construct a json representation of a InitializedOption model + initialized_option_model_json = {} + initialized_option_model_json['name'] = 'testString' + initialized_option_model_json['version'] = 1 + + # Construct a model instance of InitializedOption by calling from_dict on the json representation + initialized_option_model = InitializedOption.from_dict(initialized_option_model_json) + assert initialized_option_model != False + + # Construct a model instance of InitializedOption by calling from_dict on the json representation + initialized_option_model_dict = InitializedOption.from_dict(initialized_option_model_json).__dict__ + initialized_option_model2 = InitializedOption(**initialized_option_model_dict) + + # Verify the model instances are equivalent + assert initialized_option_model == initialized_option_model2 + + # Convert model instance back to dict and verify no loss of data + initialized_option_model_json2 = initialized_option_model.to_dict() + assert initialized_option_model_json2 == initialized_option_model_json + + +class TestModel_JsonPatchOperation: + """ + Test Class for JsonPatchOperation + """ + + def test_json_patch_operation_serialization(self): + """ + Test serialization/deserialization for JsonPatchOperation + """ + + # Construct a json representation of a JsonPatchOperation model + json_patch_operation_model_json = {} + json_patch_operation_model_json['op'] = 'add' + json_patch_operation_model_json['path'] = 'testString' + json_patch_operation_model_json['from'] = 'testString' + json_patch_operation_model_json['value'] = 'testString' + + # Construct a model instance of JsonPatchOperation by calling from_dict on the json representation + json_patch_operation_model = JsonPatchOperation.from_dict(json_patch_operation_model_json) + assert json_patch_operation_model != False + + # Construct a model instance of JsonPatchOperation by calling from_dict on the json representation + json_patch_operation_model_dict = JsonPatchOperation.from_dict(json_patch_operation_model_json).__dict__ + json_patch_operation_model2 = JsonPatchOperation(**json_patch_operation_model_dict) + + # Verify the model instances are equivalent + assert json_patch_operation_model == json_patch_operation_model2 + + # Convert model instance back to dict and verify no loss of data + json_patch_operation_model_json2 = json_patch_operation_model.to_dict() + assert json_patch_operation_model_json2 == json_patch_operation_model_json + + +class TestModel_MemberRolesSchema: + """ + Test Class for MemberRolesSchema + """ + + def test_member_roles_schema_serialization(self): + """ + Test serialization/deserialization for MemberRolesSchema + """ + + # Construct a json representation of a MemberRolesSchema model + member_roles_schema_model_json = {} + member_roles_schema_model_json['user_iam_id'] = 'testString' + member_roles_schema_model_json['roles'] = ['testString'] + + # Construct a model instance of MemberRolesSchema by calling from_dict on the json representation + member_roles_schema_model = MemberRolesSchema.from_dict(member_roles_schema_model_json) + assert member_roles_schema_model != False + + # Construct a model instance of MemberRolesSchema by calling from_dict on the json representation + member_roles_schema_model_dict = MemberRolesSchema.from_dict(member_roles_schema_model_json).__dict__ + member_roles_schema_model2 = MemberRolesSchema(**member_roles_schema_model_dict) + + # Verify the model instances are equivalent + assert member_roles_schema_model == member_roles_schema_model2 + + # Convert model instance back to dict and verify no loss of data + member_roles_schema_model_json2 = member_roles_schema_model.to_dict() + assert member_roles_schema_model_json2 == member_roles_schema_model_json + + +class TestModel_NextPage: + """ + Test Class for NextPage + """ + + def test_next_page_serialization(self): + """ + Test serialization/deserialization for NextPage + """ + + # Construct a json representation of a NextPage model + next_page_model_json = {} + next_page_model_json['href'] = 'https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + next_page_model_json['start'] = 'eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + + # Construct a model instance of NextPage by calling from_dict on the json representation + next_page_model = NextPage.from_dict(next_page_model_json) + assert next_page_model != False + + # Construct a model instance of NextPage by calling from_dict on the json representation + next_page_model_dict = NextPage.from_dict(next_page_model_json).__dict__ + next_page_model2 = NextPage(**next_page_model_dict) + + # Verify the model instances are equivalent + assert next_page_model == next_page_model2 + + # Convert model instance back to dict and verify no loss of data + next_page_model_json2 = next_page_model.to_dict() + assert next_page_model_json2 == next_page_model_json + + +class TestModel_Overview: + """ + Test Class for Overview + """ + + def test_overview_serialization(self): + """ + Test serialization/deserialization for Overview + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + # Construct a json representation of a Overview model + overview_model_json = {} + overview_model_json['api_version'] = 'v3.0.1' + overview_model_json['kind'] = 'DataContract' + overview_model_json['name'] = 'Sample Data Contract' + overview_model_json['version'] = '0.0.0' + overview_model_json['domain'] = domain_model + overview_model_json['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a model instance of Overview by calling from_dict on the json representation + overview_model = Overview.from_dict(overview_model_json) + assert overview_model != False + + # Construct a model instance of Overview by calling from_dict on the json representation + overview_model_dict = Overview.from_dict(overview_model_json).__dict__ + overview_model2 = Overview(**overview_model_dict) + + # Verify the model instances are equivalent + assert overview_model == overview_model2 + + # Convert model instance back to dict and verify no loss of data + overview_model_json2 = overview_model.to_dict() + assert overview_model_json2 == overview_model_json + + +class TestModel_Pricing: + """ + Test Class for Pricing + """ + + def test_pricing_serialization(self): + """ + Test serialization/deserialization for Pricing + """ + + # Construct a json representation of a Pricing model + pricing_model_json = {} + pricing_model_json['amount'] = '100.0' + pricing_model_json['currency'] = 'USD' + pricing_model_json['unit'] = 'megabyte' + + # Construct a model instance of Pricing by calling from_dict on the json representation + pricing_model = Pricing.from_dict(pricing_model_json) + assert pricing_model != False + + # Construct a model instance of Pricing by calling from_dict on the json representation + pricing_model_dict = Pricing.from_dict(pricing_model_json).__dict__ + pricing_model2 = Pricing(**pricing_model_dict) + + # Verify the model instances are equivalent + assert pricing_model == pricing_model2 + + # Convert model instance back to dict and verify no loss of data + pricing_model_json2 = pricing_model.to_dict() + assert pricing_model_json2 == pricing_model_json + + +class TestModel_ProducerInputModel: + """ + Test Class for ProducerInputModel + """ + + def test_producer_input_model_serialization(self): + """ + Test serialization/deserialization for ProducerInputModel + """ + + # Construct dict forms of any model objects needed in order to build this model. + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + # Construct a json representation of a ProducerInputModel model + producer_input_model_model_json = {} + producer_input_model_model_json['engine_details'] = engine_details_model_model + producer_input_model_model_json['engines'] = [engine_details_model_model] + + # Construct a model instance of ProducerInputModel by calling from_dict on the json representation + producer_input_model_model = ProducerInputModel.from_dict(producer_input_model_model_json) + assert producer_input_model_model != False + + # Construct a model instance of ProducerInputModel by calling from_dict on the json representation + producer_input_model_model_dict = ProducerInputModel.from_dict(producer_input_model_model_json).__dict__ + producer_input_model_model2 = ProducerInputModel(**producer_input_model_model_dict) + + # Verify the model instances are equivalent + assert producer_input_model_model == producer_input_model_model2 + + # Convert model instance back to dict and verify no loss of data + producer_input_model_model_json2 = producer_input_model_model.to_dict() + assert producer_input_model_model_json2 == producer_input_model_model_json + + +class TestModel_PropertiesSchema: + """ + Test Class for PropertiesSchema + """ + + def test_properties_schema_serialization(self): + """ + Test serialization/deserialization for PropertiesSchema + """ + + # Construct a json representation of a PropertiesSchema model + properties_schema_model_json = {} + properties_schema_model_json['value'] = 'testString' + + # Construct a model instance of PropertiesSchema by calling from_dict on the json representation + properties_schema_model = PropertiesSchema.from_dict(properties_schema_model_json) + assert properties_schema_model != False + + # Construct a model instance of PropertiesSchema by calling from_dict on the json representation + properties_schema_model_dict = PropertiesSchema.from_dict(properties_schema_model_json).__dict__ + properties_schema_model2 = PropertiesSchema(**properties_schema_model_dict) + + # Verify the model instances are equivalent + assert properties_schema_model == properties_schema_model2 + + # Convert model instance back to dict and verify no loss of data + properties_schema_model_json2 = properties_schema_model.to_dict() + assert properties_schema_model_json2 == properties_schema_model_json + + +class TestModel_ProvidedCatalogWorkflows: + """ + Test Class for ProvidedCatalogWorkflows + """ + + def test_provided_catalog_workflows_serialization(self): + """ + Test serialization/deserialization for ProvidedCatalogWorkflows + """ + + # Construct dict forms of any model objects needed in order to build this model. + + workflow_definition_reference_model = {} # WorkflowDefinitionReference + workflow_definition_reference_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + provided_workflow_resource_model = {} # ProvidedWorkflowResource + provided_workflow_resource_model['definition'] = workflow_definition_reference_model + + # Construct a json representation of a ProvidedCatalogWorkflows model + provided_catalog_workflows_model_json = {} + provided_catalog_workflows_model_json['data_access'] = provided_workflow_resource_model + provided_catalog_workflows_model_json['request_new_product'] = provided_workflow_resource_model + + # Construct a model instance of ProvidedCatalogWorkflows by calling from_dict on the json representation + provided_catalog_workflows_model = ProvidedCatalogWorkflows.from_dict(provided_catalog_workflows_model_json) + assert provided_catalog_workflows_model != False + + # Construct a model instance of ProvidedCatalogWorkflows by calling from_dict on the json representation + provided_catalog_workflows_model_dict = ProvidedCatalogWorkflows.from_dict(provided_catalog_workflows_model_json).__dict__ + provided_catalog_workflows_model2 = ProvidedCatalogWorkflows(**provided_catalog_workflows_model_dict) + + # Verify the model instances are equivalent + assert provided_catalog_workflows_model == provided_catalog_workflows_model2 + + # Convert model instance back to dict and verify no loss of data + provided_catalog_workflows_model_json2 = provided_catalog_workflows_model.to_dict() + assert provided_catalog_workflows_model_json2 == provided_catalog_workflows_model_json + + +class TestModel_ProvidedWorkflowResource: + """ + Test Class for ProvidedWorkflowResource + """ + + def test_provided_workflow_resource_serialization(self): + """ + Test serialization/deserialization for ProvidedWorkflowResource + """ + + # Construct dict forms of any model objects needed in order to build this model. + + workflow_definition_reference_model = {} # WorkflowDefinitionReference + workflow_definition_reference_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a json representation of a ProvidedWorkflowResource model + provided_workflow_resource_model_json = {} + provided_workflow_resource_model_json['definition'] = workflow_definition_reference_model + + # Construct a model instance of ProvidedWorkflowResource by calling from_dict on the json representation + provided_workflow_resource_model = ProvidedWorkflowResource.from_dict(provided_workflow_resource_model_json) + assert provided_workflow_resource_model != False + + # Construct a model instance of ProvidedWorkflowResource by calling from_dict on the json representation + provided_workflow_resource_model_dict = ProvidedWorkflowResource.from_dict(provided_workflow_resource_model_json).__dict__ + provided_workflow_resource_model2 = ProvidedWorkflowResource(**provided_workflow_resource_model_dict) + + # Verify the model instances are equivalent + assert provided_workflow_resource_model == provided_workflow_resource_model2 + + # Convert model instance back to dict and verify no loss of data + provided_workflow_resource_model_json2 = provided_workflow_resource_model.to_dict() + assert provided_workflow_resource_model_json2 == provided_workflow_resource_model_json + + +class TestModel_RevokeAccessResponse: + """ + Test Class for RevokeAccessResponse + """ + + def test_revoke_access_response_serialization(self): + """ + Test serialization/deserialization for RevokeAccessResponse + """ + + # Construct a json representation of a RevokeAccessResponse model + revoke_access_response_model_json = {} + revoke_access_response_model_json['message'] = 'testString' + + # Construct a model instance of RevokeAccessResponse by calling from_dict on the json representation + revoke_access_response_model = RevokeAccessResponse.from_dict(revoke_access_response_model_json) + assert revoke_access_response_model != False + + # Construct a model instance of RevokeAccessResponse by calling from_dict on the json representation + revoke_access_response_model_dict = RevokeAccessResponse.from_dict(revoke_access_response_model_json).__dict__ + revoke_access_response_model2 = RevokeAccessResponse(**revoke_access_response_model_dict) + + # Verify the model instances are equivalent + assert revoke_access_response_model == revoke_access_response_model2 + + # Convert model instance back to dict and verify no loss of data + revoke_access_response_model_json2 = revoke_access_response_model.to_dict() + assert revoke_access_response_model_json2 == revoke_access_response_model_json + + +class TestModel_RevokeAccessStateResponse: + """ + Test Class for RevokeAccessStateResponse + """ + + def test_revoke_access_state_response_serialization(self): + """ + Test serialization/deserialization for RevokeAccessStateResponse + """ + + # Construct dict forms of any model objects needed in order to build this model. + + asset_model = {} # Asset + asset_model['metadata'] = {'anyKey': 'anyValue'} + asset_model['entity'] = {'anyKey': 'anyValue'} + + search_asset_pagination_info_model = {} # SearchAssetPaginationInfo + search_asset_pagination_info_model['query'] = 'ibm_data_product_revoke_access.state:(SCHEDULED OR FAILED)' + search_asset_pagination_info_model['limit'] = 1 + search_asset_pagination_info_model['bookmark'] = 'MQ==' + search_asset_pagination_info_model['include'] = 'entity' + search_asset_pagination_info_model['skip'] = 0 + + # Construct a json representation of a RevokeAccessStateResponse model + revoke_access_state_response_model_json = {} + revoke_access_state_response_model_json['results'] = [asset_model] + revoke_access_state_response_model_json['total_count'] = 42 + revoke_access_state_response_model_json['next'] = search_asset_pagination_info_model + + # Construct a model instance of RevokeAccessStateResponse by calling from_dict on the json representation + revoke_access_state_response_model = RevokeAccessStateResponse.from_dict(revoke_access_state_response_model_json) + assert revoke_access_state_response_model != False + + # Construct a model instance of RevokeAccessStateResponse by calling from_dict on the json representation + revoke_access_state_response_model_dict = RevokeAccessStateResponse.from_dict(revoke_access_state_response_model_json).__dict__ + revoke_access_state_response_model2 = RevokeAccessStateResponse(**revoke_access_state_response_model_dict) + + # Verify the model instances are equivalent + assert revoke_access_state_response_model == revoke_access_state_response_model2 + + # Convert model instance back to dict and verify no loss of data + revoke_access_state_response_model_json2 = revoke_access_state_response_model.to_dict() + assert revoke_access_state_response_model_json2 == revoke_access_state_response_model_json + + +class TestModel_Roles: + """ + Test Class for Roles + """ + + def test_roles_serialization(self): + """ + Test serialization/deserialization for Roles + """ + + # Construct a json representation of a Roles model + roles_model_json = {} + roles_model_json['role'] = 'owner' + + # Construct a model instance of Roles by calling from_dict on the json representation + roles_model = Roles.from_dict(roles_model_json) + assert roles_model != False + + # Construct a model instance of Roles by calling from_dict on the json representation + roles_model_dict = Roles.from_dict(roles_model_json).__dict__ + roles_model2 = Roles(**roles_model_dict) + + # Verify the model instances are equivalent + assert roles_model == roles_model2 + + # Convert model instance back to dict and verify no loss of data + roles_model_json2 = roles_model.to_dict() + assert roles_model_json2 == roles_model_json + + +class TestModel_SearchAssetPaginationInfo: + """ + Test Class for SearchAssetPaginationInfo + """ + + def test_search_asset_pagination_info_serialization(self): + """ + Test serialization/deserialization for SearchAssetPaginationInfo + """ + + # Construct a json representation of a SearchAssetPaginationInfo model + search_asset_pagination_info_model_json = {} + search_asset_pagination_info_model_json['query'] = 'ibm_data_product_revoke_access.state:(SCHEDULED OR FAILED)' + search_asset_pagination_info_model_json['limit'] = 1 + search_asset_pagination_info_model_json['bookmark'] = 'MQ==' + search_asset_pagination_info_model_json['include'] = 'entity' + search_asset_pagination_info_model_json['skip'] = 0 + + # Construct a model instance of SearchAssetPaginationInfo by calling from_dict on the json representation + search_asset_pagination_info_model = SearchAssetPaginationInfo.from_dict(search_asset_pagination_info_model_json) + assert search_asset_pagination_info_model != False + + # Construct a model instance of SearchAssetPaginationInfo by calling from_dict on the json representation + search_asset_pagination_info_model_dict = SearchAssetPaginationInfo.from_dict(search_asset_pagination_info_model_json).__dict__ + search_asset_pagination_info_model2 = SearchAssetPaginationInfo(**search_asset_pagination_info_model_dict) + + # Verify the model instances are equivalent + assert search_asset_pagination_info_model == search_asset_pagination_info_model2 + + # Convert model instance back to dict and verify no loss of data + search_asset_pagination_info_model_json2 = search_asset_pagination_info_model.to_dict() + assert search_asset_pagination_info_model_json2 == search_asset_pagination_info_model_json + + +class TestModel_ServiceIdCredentials: + """ + Test Class for ServiceIdCredentials + """ + + def test_service_id_credentials_serialization(self): + """ + Test serialization/deserialization for ServiceIdCredentials + """ + + # Construct a json representation of a ServiceIdCredentials model + service_id_credentials_model_json = {} + service_id_credentials_model_json['name'] = 'data-product-admin-service-id-API-key' + service_id_credentials_model_json['created_at'] = '2019-01-01T12:00:00Z' + + # Construct a model instance of ServiceIdCredentials by calling from_dict on the json representation + service_id_credentials_model = ServiceIdCredentials.from_dict(service_id_credentials_model_json) + assert service_id_credentials_model != False + + # Construct a model instance of ServiceIdCredentials by calling from_dict on the json representation + service_id_credentials_model_dict = ServiceIdCredentials.from_dict(service_id_credentials_model_json).__dict__ + service_id_credentials_model2 = ServiceIdCredentials(**service_id_credentials_model_dict) + + # Verify the model instances are equivalent + assert service_id_credentials_model == service_id_credentials_model2 + + # Convert model instance back to dict and verify no loss of data + service_id_credentials_model_json2 = service_id_credentials_model.to_dict() + assert service_id_credentials_model_json2 == service_id_credentials_model_json + + +class TestModel_UseCase: + """ + Test Class for UseCase + """ + + def test_use_case_serialization(self): + """ + Test serialization/deserialization for UseCase + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a UseCase model + use_case_model_json = {} + use_case_model_json['id'] = 'testString' + use_case_model_json['name'] = 'testString' + use_case_model_json['container'] = container_reference_model + + # Construct a model instance of UseCase by calling from_dict on the json representation + use_case_model = UseCase.from_dict(use_case_model_json) + assert use_case_model != False + + # Construct a model instance of UseCase by calling from_dict on the json representation + use_case_model_dict = UseCase.from_dict(use_case_model_json).__dict__ + use_case_model2 = UseCase(**use_case_model_dict) + + # Verify the model instances are equivalent + assert use_case_model == use_case_model2 + + # Convert model instance back to dict and verify no loss of data + use_case_model_json2 = use_case_model.to_dict() + assert use_case_model_json2 == use_case_model_json + + +class TestModel_Visualization: + """ + Test Class for Visualization + """ + + def test_visualization_serialization(self): + """ + Test serialization/deserialization for Visualization + """ + + # Construct a json representation of a Visualization model + visualization_model_json = {} + visualization_model_json['id'] = 'testString' + visualization_model_json['name'] = 'testString' + + # Construct a model instance of Visualization by calling from_dict on the json representation + visualization_model = Visualization.from_dict(visualization_model_json) + assert visualization_model != False + + # Construct a model instance of Visualization by calling from_dict on the json representation + visualization_model_dict = Visualization.from_dict(visualization_model_json).__dict__ + visualization_model2 = Visualization(**visualization_model_dict) + + # Verify the model instances are equivalent + assert visualization_model == visualization_model2 + + # Convert model instance back to dict and verify no loss of data + visualization_model_json2 = visualization_model.to_dict() + assert visualization_model_json2 == visualization_model_json + + +class TestModel_WorkflowDefinitionReference: + """ + Test Class for WorkflowDefinitionReference + """ + + def test_workflow_definition_reference_serialization(self): + """ + Test serialization/deserialization for WorkflowDefinitionReference + """ + + # Construct a json representation of a WorkflowDefinitionReference model + workflow_definition_reference_model_json = {} + workflow_definition_reference_model_json['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a model instance of WorkflowDefinitionReference by calling from_dict on the json representation + workflow_definition_reference_model = WorkflowDefinitionReference.from_dict(workflow_definition_reference_model_json) + assert workflow_definition_reference_model != False + + # Construct a model instance of WorkflowDefinitionReference by calling from_dict on the json representation + workflow_definition_reference_model_dict = WorkflowDefinitionReference.from_dict(workflow_definition_reference_model_json).__dict__ + workflow_definition_reference_model2 = WorkflowDefinitionReference(**workflow_definition_reference_model_dict) + + # Verify the model instances are equivalent + assert workflow_definition_reference_model == workflow_definition_reference_model2 + + # Convert model instance back to dict and verify no loss of data + workflow_definition_reference_model_json2 = workflow_definition_reference_model.to_dict() + assert workflow_definition_reference_model_json2 == workflow_definition_reference_model_json + + +# endregion +############################################################################## +# End of Model Tests +############################################################################## diff --git a/tests/src/dq_validator/__init__.py b/tests/src/dq_validator/__init__.py new file mode 100644 index 0000000..b646327 --- /dev/null +++ b/tests/src/dq_validator/__init__.py @@ -0,0 +1,20 @@ +# Copyright 2026 IBM Corporation +# Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# See the LICENSE file in the project root for license information. + +""" +Data quality validator tests +""" + +# This file is only here to get pylint to check the files in this directory diff --git a/tests/src/dq_validator/provider/test_assets.py b/tests/src/dq_validator/provider/test_assets.py new file mode 100644 index 0000000..15070a0 --- /dev/null +++ b/tests/src/dq_validator/provider/test_assets.py @@ -0,0 +1,445 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" + +import pytest +from unittest.mock import Mock, patch +import json + +from wxdi.dq_validator.provider import ProviderConfig, DQAssetsProvider + + +class TestDQAssetsProvider: + """Test suite for DQAssetsProvider class.""" + + @pytest.fixture + def config(self): + """Create a test configuration.""" + return ProviderConfig( + url="https://test-instance.com", + auth_token="Bearer test-token" + ) + + @pytest.fixture + def provider(self, config): + """Create a test DQAssetsProvider instance.""" + with patch('wxdi.dq_validator.provider.base_provider.Session') as mock_session_class: + mock_session = Mock() + mock_session_class.return_value = mock_session + provider = DQAssetsProvider(config) + yield provider + + # ==================== get_assets Tests ==================== + + def test_get_assets_with_project_id(self, provider): + """Test getting assets with project_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [ + { + "id": "asset-1", + "name": "Table1", + "type": "table" + }, + { + "id": "asset-2", + "name": "Column1", + "type": "column" + } + ], + "total_count": 2 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets(project_id="project-123") + + # Verify + assert "assets" in result + assert len(result["assets"]) == 2 + assert result["assets"][0]["id"] == "asset-1" + assert result["total_count"] == 2 + + # Verify the API call + provider.session.get.assert_called_once() + call_args = provider.session.get.call_args + + # Check URL + assert "https://test-instance.com/data_quality/v4/assets" in call_args[0][0] + assert "project_id=project-123" in call_args[0][0] + + # Check headers + headers = call_args[1]["headers"] + assert headers["Authorization"] == "Bearer test-token" + assert headers["Content-Type"] == "application/json" + + def test_get_assets_with_catalog_id(self, provider): + """Test getting assets with catalog_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [ + { + "id": "asset-3", + "name": "Table2", + "type": "table" + } + ], + "total_count": 1 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets(catalog_id="catalog-456") + + # Verify + assert "assets" in result + assert len(result["assets"]) == 1 + assert result["assets"][0]["id"] == "asset-3" + + # Verify the API call + call_args = provider.session.get.call_args + + # Check URL contains catalog_id + assert "catalog_id=catalog-456" in call_args[0][0] + assert "project_id" not in call_args[0][0] + + def test_get_assets_with_all_optional_params(self, provider): + """Test getting assets with all optional parameters.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [], + "total_count": 0, + "next": { + "start": "next-token" + } + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets( + project_id="project-123", + start="start-token", + limit=50, + include_children=True, + asset_type="column" + ) + + # Verify + assert "assets" in result + assert result["total_count"] == 0 + + # Verify the API call + call_args = provider.session.get.call_args + + # Check URL contains all parameters + assert "project_id=project-123" in call_args[0][0] + assert "start=start-token" in call_args[0][0] + assert "limit=50" in call_args[0][0] + assert "include_children=true" in call_args[0][0] + assert "type=column" in call_args[0][0] + + def test_get_assets_with_include_children_false(self, provider): + """Test getting assets with include_children=False.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [], + "total_count": 0 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets( + project_id="project-123", + include_children=False + ) + + # Verify + assert "assets" in result + + # Verify the API call + call_args = provider.session.get.call_args + + # Check URL contains include_children=false + assert "include_children=false" in call_args[0][0] + + def test_get_assets_with_limit(self, provider): + """Test getting assets with limit parameter.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [{"id": f"asset-{i}"} for i in range(10)], + "total_count": 10 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets( + project_id="project-123", + limit=10 + ) + + # Verify + assert len(result["assets"]) == 10 + + # Verify the API call + call_args = provider.session.get.call_args + + # Check URL contains limit + assert "limit=10" in call_args[0][0] + + def test_get_assets_with_start_token(self, provider): + """Test getting assets with start token for pagination.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [{"id": "asset-next"}], + "total_count": 1 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets( + project_id="project-123", + start="pagination-token-123" + ) + + # Verify + assert len(result["assets"]) == 1 + + # Verify the API call + call_args = provider.session.get.call_args + + # Check URL contains start token + assert "start=pagination-token-123" in call_args[0][0] + + def test_get_assets_with_asset_type_table(self, provider): + """Test getting assets filtered by type=table.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [ + {"id": "asset-1", "type": "table"}, + {"id": "asset-2", "type": "table"} + ], + "total_count": 2 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets( + project_id="project-123", + asset_type="table" + ) + + # Verify + assert len(result["assets"]) == 2 + assert all(asset["type"] == "table" for asset in result["assets"]) + + # Verify the API call + call_args = provider.session.get.call_args + + # Check URL contains type parameter + assert "type=table" in call_args[0][0] + + def test_get_assets_with_asset_type_column(self, provider): + """Test getting assets filtered by type=column.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [ + {"id": "asset-col-1", "type": "column"}, + {"id": "asset-col-2", "type": "column"} + ], + "total_count": 2 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets( + catalog_id="catalog-456", + asset_type="column" + ) + + # Verify + assert len(result["assets"]) == 2 + assert all(asset["type"] == "column" for asset in result["assets"]) + + # Verify the API call + call_args = provider.session.get.call_args + + # Check URL contains type parameter + assert "type=column" in call_args[0][0] + + def test_get_assets_missing_both_ids(self, provider): + """Test getting assets without project_id or catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.get_assets() + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_get_assets_both_ids_provided(self, provider): + """Test getting assets with both project_id and catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.get_assets( + project_id="project-123", + catalog_id="catalog-456" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + def test_get_assets_api_failure(self, provider): + """Test failed get assets request.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 404 + mock_response.text = "Project not found" + provider.session.get.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.get_assets(project_id="invalid-project") + + assert "Failed to get assets" in str(exc_info.value) + assert "404" in str(exc_info.value) + assert "Project not found" in str(exc_info.value) + + def test_get_assets_empty_result(self, provider): + """Test getting assets with no results.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [], + "total_count": 0 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets(project_id="project-empty") + + # Verify + assert result["assets"] == [] + assert result["total_count"] == 0 + + def test_get_assets_with_pagination_info(self, provider): + """Test getting assets with pagination information.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [{"id": f"asset-{i}"} for i in range(100)], + "total_count": 250, + "next": { + "start": "next-page-token" + } + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets( + project_id="project-123", + limit=100 + ) + + # Verify + assert len(result["assets"]) == 100 + assert result["total_count"] == 250 + assert "next" in result + assert result["next"]["start"] == "next-page-token" + + def test_get_assets_unauthorized(self, provider): + """Test getting assets with unauthorized access.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 401 + mock_response.text = "Unauthorized" + provider.session.get.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.get_assets(project_id="project-123") + + assert "Failed to get assets" in str(exc_info.value) + assert "401" in str(exc_info.value) + + def test_get_assets_server_error(self, provider): + """Test getting assets with server error.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 500 + mock_response.text = "Internal server error" + provider.session.get.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.get_assets(catalog_id="catalog-123") + + assert "Failed to get assets" in str(exc_info.value) + assert "500" in str(exc_info.value) + assert "Internal server error" in str(exc_info.value) + + def test_get_assets_with_complex_response(self, provider): + """Test getting assets with complex nested response.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [ + { + "id": "asset-complex-1", + "name": "ComplexTable", + "type": "table", + "metadata": { + "columns": ["col1", "col2"], + "row_count": 1000 + }, + "children": [ + {"id": "child-1", "type": "column"}, + {"id": "child-2", "type": "column"} + ] + } + ], + "total_count": 1 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets( + project_id="project-123", + include_children=True + ) + + # Verify + assert len(result["assets"]) == 1 + asset = result["assets"][0] + assert asset["id"] == "asset-complex-1" + assert "metadata" in asset + assert "children" in asset + assert len(asset["children"]) == 2 \ No newline at end of file diff --git a/tests/src/dq_validator/provider/test_cams.py b/tests/src/dq_validator/provider/test_cams.py new file mode 100644 index 0000000..05e2856 --- /dev/null +++ b/tests/src/dq_validator/provider/test_cams.py @@ -0,0 +1,513 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Test suite for CamsProvider module +""" + +import pytest +import json +from pathlib import Path +from wxdi.dq_validator.provider.cams import CamsProvider +from wxdi.dq_validator.provider.config import ProviderConfig +from wxdi.dq_validator.provider.data_asset_model import DataAsset + + +class TestCamsProvider: + """Test cases for CamsProvider class""" + + @pytest.fixture + def base_url(self): + """Base URL for API""" + return "https://api.example.com" + + @pytest.fixture + def auth_token(self): + """Authentication token""" + return "Bearer test-token-12345" + + @pytest.fixture + def project_id(self): + """Project ID""" + return "72d21c1d-499b-4784-a3c7-6f84507f9a20" + + @pytest.fixture + def catalog_id(self): + """Catalog ID""" + return "a1b2c3d4-e5f6-7890-abcd-ef1234567890" + + @pytest.fixture + def config_with_project(self, base_url, auth_token, project_id): + """Create ProviderConfig with project_id""" + return ProviderConfig(base_url, auth_token, project_id=project_id) + + @pytest.fixture + def config_with_catalog(self, base_url, auth_token, catalog_id): + """Create ProviderConfig with catalog_id""" + return ProviderConfig(base_url, auth_token, catalog_id=catalog_id) + + @pytest.fixture + def config_without_project_or_catalog(self, base_url, auth_token): + """Create ProviderConfig without project_id or catalog_id - should fail""" + return ProviderConfig(base_url, auth_token) + + @pytest.fixture + def data_asset_json(self): + """Load the data asset response JSON""" + json_path = Path(__file__).parent.parent.parent.parent / "data" / "data_asset_response.json" + with open(json_path, "r") as f: + return json.load(f) + + @pytest.fixture + def mock_data_asset(self, data_asset_json): + """Create DataAsset from JSON""" + return DataAsset.from_dict(data_asset_json) + + def test_provider_initialization(self, config_with_project): + """Test CamsProvider initialization""" + provider = CamsProvider(config_with_project) + assert provider.config == config_with_project + assert provider.config.url == "https://api.example.com" + assert provider.config.auth_token == "Bearer test-token-12345" + assert provider.config.project_id == "72d21c1d-499b-4784-a3c7-6f84507f9a20" + + def test_get_asset_by_id_success( + self, config_with_project, data_asset_json, mocker + ): + """Test successful asset retrieval""" + provider = CamsProvider(config_with_project) + + # Mock the session.get call + mock_response = mocker.Mock() + mock_response.text = json.dumps(data_asset_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + asset = provider.get_asset_by_id("6862f3ba-81f5-4122-8286-62bb4c5d6543") + + assert isinstance(asset, DataAsset) + assert asset.metadata.asset_id == "6862f3ba-81f5-4122-8286-62bb4c5d6543" + assert asset.metadata.name == "DEPARTMENT" + + def test_get_asset_by_id_with_project_id( + self, config_with_project, data_asset_json, mocker + ): + """Test that project_id is included in query parameters""" + provider = CamsProvider(config_with_project) + + mock_response = mocker.Mock() + mock_response.text = json.dumps(data_asset_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session_instance = mock_session.return_value + mock_session_instance.get.return_value = mock_response + + # Mock get_url_with_query_params to capture the call + mock_get_url = mocker.patch( + "wxdi.dq_validator.provider.cams.get_url_with_query_params" + ) + mock_get_url.return_value = ( + "https://api.example.com/v2/assets/asset-123?project_id=project-456" + ) + + provider.get_asset_by_id("asset-123") + + # Verify get_url_with_query_params was called with project_id + mock_get_url.assert_called_once() + call_args = mock_get_url.call_args + assert call_args[0][0] == "https://api.example.com/v2/assets/asset-123" + assert call_args[0][1]["project_id"] == "72d21c1d-499b-4784-a3c7-6f84507f9a20" + + def test_get_asset_by_id_without_project_id( + self, config_without_project_or_catalog, data_asset_json, mocker + ): + """Test asset retrieval without project_id""" + provider = CamsProvider(config_without_project_or_catalog) + + mock_response = mocker.Mock() + mock_response.text = json.dumps(data_asset_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + mock_get_url = mocker.patch( + "wxdi.dq_validator.provider.cams.get_url_with_query_params" + ) + mock_get_url.return_value = "https://api.example.com/v2/assets/asset-123" + + provider.get_asset_by_id("asset-123") + + # Verify get_url_with_query_params was called without project_id + mock_get_url.assert_called_once() + call_args = mock_get_url.call_args + assert "project_id" not in call_args[0][1] + + def test_get_asset_by_id_with_options( + self, config_with_project, data_asset_json, mocker + ): + """Test asset retrieval with additional options""" + provider = CamsProvider(config_with_project) + + mock_response = mocker.Mock() + mock_response.text = json.dumps(data_asset_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + mock_get_url = mocker.patch( + "wxdi.dq_validator.provider.cams.get_url_with_query_params" + ) + mock_get_url.return_value = "https://api.example.com/v2/assets/asset-123?project_id=project-456&include=metadata" + + options = {"include": "metadata"} + provider.get_asset_by_id("asset-123", options) + + # Verify both project_id and custom options are included + mock_get_url.assert_called_once() + call_args = mock_get_url.call_args + query_params = call_args[0][1] + assert query_params["project_id"] == "72d21c1d-499b-4784-a3c7-6f84507f9a20" + assert query_params["include"] == "metadata" + + def test_get_asset_by_id_url_construction( + self, config_with_project, data_asset_json, mocker + ): + """Test correct URL construction""" + provider = CamsProvider(config_with_project) + + mock_response = mocker.Mock() + mock_response.text = json.dumps(data_asset_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session_instance = mock_session.return_value + mock_session_instance.get.return_value = mock_response + + asset_id = "6862f3ba-81f5-4122-8286-62bb4c5d6543" + provider.get_asset_by_id(asset_id) + + # Verify the session.get was called + mock_session_instance.get.assert_called_once() + call_args = mock_session_instance.get.call_args + + # Check that headers were passed + assert "headers" in call_args[1] + assert call_args[1]["verify"] is False + + def test_get_asset_by_id_headers( + self, config_with_project, data_asset_json, mocker + ): + """Test that correct headers are sent""" + provider = CamsProvider(config_with_project) + + mock_response = mocker.Mock() + mock_response.text = json.dumps(data_asset_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session_instance = mock_session.return_value + mock_session_instance.get.return_value = mock_response + + # Mock get_request_headers to verify it's called + mock_get_headers = mocker.patch( + "wxdi.dq_validator.provider.cams.get_request_headers" + ) + mock_get_headers.return_value = { + "Authorization": "Bearer test-token-12345", + "Content-Type": "application/json", + } + + provider.get_asset_by_id("asset-123") + + # Verify get_request_headers was called with auth_token + mock_get_headers.assert_called_once_with("Bearer test-token-12345") + + # Verify session.get was called with the headers + mock_session_instance.get.assert_called_once() + call_args = mock_session_instance.get.call_args + assert call_args[1]["headers"] == { + "Authorization": "Bearer test-token-12345", + "Content-Type": "application/json", + } + + def test_get_asset_by_id_returns_data_asset( + self, config_with_project, data_asset_json, mocker + ): + """Test that DataAsset object is correctly constructed""" + provider = CamsProvider(config_with_project) + + mock_response = mocker.Mock() + mock_response.text = json.dumps(data_asset_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + asset = provider.get_asset_by_id("6862f3ba-81f5-4122-8286-62bb4c5d6543") + + # Verify DataAsset structure + assert isinstance(asset, DataAsset) + assert hasattr(asset, "metadata") + assert hasattr(asset, "entity") + assert hasattr(asset.entity, "data_asset") + assert hasattr(asset.entity, "column_info") + + # Verify specific data + assert len(asset.entity.data_asset.columns) == 5 + assert "DEPTNO" in asset.entity.column_info + assert "MGRNO" in asset.entity.column_info + + def test_get_asset_by_id_json_parsing(self, config_with_project, mocker): + """Test JSON parsing of response""" + provider = CamsProvider(config_with_project) + + # Create a minimal valid JSON response + minimal_json = { + "metadata": { + "project_id": "test-project", + "name": "TEST_TABLE", + "tags": [], + "asset_type": "data_asset", + "catalog_id": "test-catalog", + "created": 1234567890, + "created_at": "2024-01-01T00:00:00Z", + "owner_id": "owner-123", + "size": 0, + "version": 1, + "asset_state": "available", + "asset_attributes": [], + "asset_id": "asset-123", + "asset_category": "USER", + "creator_id": "creator-123", + }, + "entity": { + "data_asset": { + "columns": [], + "dataset": True, + "mime_type": "application/x-ibm-rel-table", + "properties": [], + }, + "column_info": {}, + "data_profile": {"attribute_classes": []}, + "key_analyses": { + "fk_defined": 0, + "pk_defined": 0, + "fk_assigned": 0, + "pk_assigned": 0, + "fk_suggested": 0, + "pk_suggested": 0, + "primary_keys": [], + "fk_defined_as_pk": 0, + "overlap_assigned": 0, + "fk_assigned_as_pk": 0, + "overlap_suggested": 0, + "fk_suggested_as_pk": 0, + "key_analysis_area_id": "area-123", + }, + "discovered_asset": {"extended_metadata": []}, + "dataview_visualization": {"jobs": {}}, + "metadata_enrichment_info": {"MDE_instrumented": False}, + "asset_data_quality_constraint": { + "asset_checks": [], + "rejected_checks": [], + "suggested_checks": [], + }, + "metadata_enrichment_area_info": { + "job_id": "job-123", + "area_id": "area-123", + "added_date": 1234567890, + "job_run_id": "run-123", + "last_enrichment_status": "finished", + "last_enrichment_timestamp": 1234567890, + }, + }, + "href": "/v2/assets/asset-123", + } + + mock_response = mocker.Mock() + mock_response.text = json.dumps(minimal_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + asset = provider.get_asset_by_id("asset-123") + + assert isinstance(asset, DataAsset) + assert asset.metadata.name == "TEST_TABLE" + assert asset.metadata.asset_id == "asset-123" + + def test_get_asset_by_id_http_error(self, config_with_project, mocker): + """Test handling of HTTP errors (404, 500, etc.)""" + provider = CamsProvider(config_with_project) + + # Mock a 404 response + mock_response = mocker.Mock() + mock_response.status_code = 404 + mock_response.ok = False + mock_response.text = json.dumps({"error": "Asset not found"}) + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + # The implementation checks response.ok and raises ValueError + with pytest.raises(ValueError) as exc_info: + provider.get_asset_by_id("non-existent-asset") + + # Verify the error message + assert "Could not find data asset" in str(exc_info.value) + assert "non-existent-asset" in str(exc_info.value) + assert config_with_project.project_id in str(exc_info.value) + + def test_get_asset_by_id_invalid_json(self, config_with_project, mocker): + """Test handling of invalid JSON response""" + provider = CamsProvider(config_with_project) + + # Mock a response with invalid JSON + mock_response = mocker.Mock() + mock_response.text = "This is not valid JSON" + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + # Should raise JSONDecodeError + with pytest.raises(json.JSONDecodeError): + provider.get_asset_by_id("asset-123") + + def test_get_asset_by_id_malformed_response(self, config_with_project, mocker): + """Test handling of malformed response (valid JSON but invalid structure)""" + provider = CamsProvider(config_with_project) + + # Mock a response with valid JSON but missing required fields + malformed_json = {"some_field": "some_value"} + mock_response = mocker.Mock() + mock_response.text = json.dumps(malformed_json) + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + # Should raise ValidationError from Pydantic + with pytest.raises(Exception): # Pydantic ValidationError + provider.get_asset_by_id("asset-123") + + def test_get_asset_by_id_network_error(self, config_with_project, mocker): + """Test handling of network errors""" + provider = CamsProvider(config_with_project) + + # Mock a network error + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.side_effect = ConnectionError("Network error") + + # Should raise ConnectionError + with pytest.raises(ConnectionError): + provider.get_asset_by_id("asset-123") + + def test_get_asset_by_id_timeout(self, config_with_project, mocker): + """Test handling of timeout errors""" + provider = CamsProvider(config_with_project) + + # Mock a timeout error + from requests.exceptions import Timeout + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.side_effect = Timeout("Request timeout") + + # Should raise Timeout + with pytest.raises(Timeout): + provider.get_asset_by_id("asset-123") + + def test_config_with_catalog_id(self, config_with_catalog, catalog_id): + """Test ProviderConfig initialization with catalog_id""" + provider = CamsProvider(config_with_catalog) + assert provider.config.catalog_id == catalog_id + assert provider.config.project_id is None + + def test_get_asset_by_id_with_catalog_id( + self, config_with_catalog, data_asset_json, mocker + ): + """Test that catalog_id is included in query parameters""" + provider = CamsProvider(config_with_catalog) + + mock_response = mocker.Mock() + mock_response.text = json.dumps(data_asset_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session_instance = mock_session.return_value + mock_session_instance.get.return_value = mock_response + + # Mock get_url_with_query_params to capture the call + mock_get_url = mocker.patch( + "wxdi.dq_validator.provider.cams.get_url_with_query_params" + ) + mock_get_url.return_value = ( + "https://api.example.com/v2/assets/asset-123?catalog_id=catalog-456" + ) + + provider.get_asset_by_id("asset-123") + + # Verify get_url_with_query_params was called with catalog_id + mock_get_url.assert_called_once() + call_args = mock_get_url.call_args + assert call_args[0][0] == "https://api.example.com/v2/assets/asset-123" + assert call_args[0][1]["catalog_id"] == "a1b2c3d4-e5f6-7890-abcd-ef1234567890" + assert "project_id" not in call_args[0][1] + + def test_get_asset_by_id_catalog_error_message(self, config_with_catalog, mocker): + """Test error message includes catalog_id when using catalog""" + provider = CamsProvider(config_with_catalog) + + # Mock a 404 response + mock_response = mocker.Mock() + mock_response.status_code = 404 + mock_response.ok = False + mock_response.text = json.dumps({"error": "Asset not found"}) + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + with pytest.raises(ValueError) as exc_info: + provider.get_asset_by_id("non-existent-asset") + + # Verify the error message includes catalog context + assert "Could not find data asset" in str(exc_info.value) + assert "non-existent-asset" in str(exc_info.value) + assert "catalog" in str(exc_info.value) + assert config_with_catalog.catalog_id in str(exc_info.value) + + def test_get_asset_by_id_with_catalog_and_options( + self, config_with_catalog, data_asset_json, mocker + ): + """Test asset retrieval with catalog_id and additional options""" + # Mock a timeout error + from requests.exceptions import Timeout + + provider = CamsProvider(config_with_catalog) + + mock_response = mocker.Mock() + mock_response.text = json.dumps(data_asset_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + mock_session.return_value.get.side_effect = [mocker.DEFAULT, Timeout("Request timeout")] + + mock_get_url = mocker.patch( + "wxdi.dq_validator.provider.cams.get_url_with_query_params" + ) + mock_get_url.return_value = "https://api.example.com/v2/assets/asset-123?catalog_id=catalog-456&include=metadata" + + options = {"include": "metadata"} + provider.get_asset_by_id("asset-123", options) + + # Verify both catalog_id and custom options are included + mock_get_url.assert_called_once() + call_args = mock_get_url.call_args + query_params = call_args[0][1] + assert query_params["catalog_id"] == "a1b2c3d4-e5f6-7890-abcd-ef1234567890" + assert query_params["include"] == "metadata" + assert "project_id" not in query_params + + # Should raise Timeout with side effect set above + with pytest.raises(Timeout): + provider.get_asset_by_id("asset-123") + + +# Made with Bob diff --git a/tests/src/dq_validator/provider/test_checks.py b/tests/src/dq_validator/provider/test_checks.py new file mode 100644 index 0000000..8bd3462 --- /dev/null +++ b/tests/src/dq_validator/provider/test_checks.py @@ -0,0 +1,796 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" + +import pytest +from unittest.mock import Mock, patch +import json + +from wxdi.dq_validator.provider import ProviderConfig, ChecksProvider + + +class TestChecksProvider: + """Test suite for ChecksProvider class.""" + + @pytest.fixture + def config(self): + """Create a test configuration.""" + return ProviderConfig( + url="https://test-instance.com", + auth_token="Bearer test-token" + ) + + @pytest.fixture + def provider(self, config): + """Create a test ChecksProvider instance.""" + with patch('wxdi.dq_validator.provider.base_provider.Session') as mock_session_class: + mock_session = Mock() + mock_session_class.return_value = mock_session + provider = ChecksProvider(config) + yield provider + + # ==================== create_check Tests ==================== + + def test_create_check_with_project_id(self, provider): + """Test creating a check with project_id - returns check ID only.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + expected_body = { + "id": "check-123", + "name": "check_uniqueness_of_id", + "type": "uniqueness", + "dimension": { + "id": "dimension-456" + }, + "native_id": "asset-789/check-123" + } + mock_response.text = json.dumps(expected_body) + provider.session.post.return_value = mock_response + + # Execute + result = provider.create_check( + name="check_uniqueness_of_id", + dimension_id="dimension-456", + native_id="asset-789/check-123", + check_type="uniqueness", + project_id="project-123" + ) + + # Verify - should return only check ID (string) + assert isinstance(result, str) + assert result == "check-123" + + # Verify the API call + provider.session.post.assert_called_once() + call_args = provider.session.post.call_args + + # Check URL + assert "https://test-instance.com/data_quality/v4/checks" in call_args[0][0] + assert "project_id=project-123" in call_args[0][0] + + # Check headers + headers = call_args[1]["headers"] + assert headers["Authorization"] == "Bearer test-token" + assert headers["Content-Type"] == "application/json" + + # Check payload + payload = json.loads(call_args[1]["data"]) + assert payload["name"] == "check_uniqueness_of_id" + assert payload["type"] == "uniqueness" + assert payload["dimension"]["id"] == "dimension-456" + assert payload["native_id"] == "asset-789/check-123" + assert json.loads(payload["details"])["origin"] == "SDK" + + def test_create_check_with_catalog_id(self, provider): + """Test creating a check with catalog_id""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + expected_body = { + "id": "check-456", + "name": "check_completeness", + "type": "completeness" + } + mock_response.text = json.dumps(expected_body) + provider.session.post.return_value = mock_response + + # Execute + result = provider.create_check( + name="check_completeness", + dimension_id="dimension-789", + native_id="asset-111/check-456", + catalog_id="catalog-999" + ) + + # Verify - should return only check ID (string) + assert isinstance(result, str) + assert result == "check-456" + + # Verify the API call + call_args = provider.session.post.call_args + + # Check URL contains catalog_id + assert "catalog_id=catalog-999" in call_args[0][0] + assert "project_id" not in call_args[0][0] + + def test_create_check_without_check_type(self, provider): + """Test creating a check without check_type (should default to name) - returns check ID only.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + expected_body = { + "id": "check-789", + "name": "format_check", + "type": "format_check" + } + mock_response.text = json.dumps(expected_body) + provider.session.post.return_value = mock_response + + # Execute + result = provider.create_check( + name="format_check", + dimension_id="dimension-111", + native_id="asset-222/check-789", + project_id="project-123" + ) + + # Verify - should return only check ID (string) + assert isinstance(result, str) + assert result == "check-789" + + # Verify the API call + call_args = provider.session.post.call_args + payload = json.loads(call_args[1]["data"]) + + # Check that type defaults to name + assert payload["type"] == "format_check" + assert payload["name"] == "format_check" + + def test_create_check_missing_both_ids(self, provider): + """Test creating a check without project_id or catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.create_check( + name="check_test", + dimension_id="dimension-123", + native_id="asset-456/check-789" + ) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_create_check_both_ids_provided(self, provider): + """Test creating a check with both project_id and catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.create_check( + name="check_test", + dimension_id="dimension-123", + native_id="asset-456/check-789", + project_id="project-123", + catalog_id="catalog-456" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + def test_create_check_api_failure(self, provider): + """Test failed check creation.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 400 + mock_response.text = "Bad request" + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.create_check( + name="check_test", + dimension_id="dimension-123", + native_id="asset-456/check-789", + project_id="project-123" + ) + + assert "Failed to create check" in str(exc_info.value) + assert "400" in str(exc_info.value) + assert "Bad request" in str(exc_info.value) + + def test_create_check_missing_id_in_response(self, provider): + """Test check creation with missing id in response.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "name": "check_test", + "type": "test" + # Missing "id" field + }) + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.create_check( + name="check_test", + dimension_id="dimension-123", + native_id="asset-456/check-789", + project_id="project-123" + ) + + assert "Check ID not found in response" in str(exc_info.value) + + def test_create_check_with_special_characters_in_native_id(self, provider): + """Test creating a check with special characters in native_id - returns check ID only.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + expected_body = { + "id": "check-special-123", + "name": "check_format", + "native_id": "asset-abc-123/check-xyz-456" + } + mock_response.text = json.dumps(expected_body) + provider.session.post.return_value = mock_response + + # Execute + result = provider.create_check( + name="check_format", + dimension_id="dimension-999", + native_id="asset-abc-123/check-xyz-456", + project_id="project-123" + ) + + # Verify - should return only check ID (string) + assert isinstance(result, str) + assert result == "check-special-123" + + # Verify the payload contains the special native_id + call_args = provider.session.post.call_args + payload = json.loads(call_args[1]["data"]) + assert payload["native_id"] == "asset-abc-123/check-xyz-456" + + def test_create_check_with_parent_check_id(self, provider): + """Test creating a check with parent_check_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + expected_body = { + "id": "child-check-123", + "name": "check_format_email", + "type": "format", + "parent": { + "id": "parent-check-456" + } + } + mock_response.text = json.dumps(expected_body) + provider.session.post.return_value = mock_response + + # Execute + result = provider.create_check( + name="check_format_email", + dimension_id="dimension-789", + native_id="asset-111/email/format", + check_type="format", + project_id="project-123", + parent_check_id="parent-check-456" + ) + + # Verify - should return only check ID (string) + assert isinstance(result, str) + assert result == "child-check-123" + + # Verify the API call includes parent in payload + provider.session.post.assert_called_once() + call_args = provider.session.post.call_args + + # Check payload includes parent + payload = json.loads(call_args[1]["data"]) + assert "parent" in payload + assert payload["parent"]["id"] == "parent-check-456" + assert payload["native_id"] == "asset-111/email/format" + + # ==================== _create_check_full Tests ==================== + + def test_create_check_full_with_project_id(self, provider): + """Test _create_check_full returns full check body with project_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + expected_body = { + "id": "check-full-123", + "name": "check_uniqueness_of_id", + "type": "uniqueness", + "native_id": "asset-456/check-789", + "dimension": { + "id": "dimension-123" + }, + "details": '{"origin": "SDK"}' + } + mock_response.text = json.dumps(expected_body) + provider.session.post.return_value = mock_response + + # Execute + result = provider._create_check_full( + name="check_uniqueness_of_id", + dimension_id="dimension-123", + native_id="asset-456/check-789", + check_type="uniqueness", + project_id="project-123" + ) + + # Verify - should return full check body (dict) + assert isinstance(result, dict) + assert result["id"] == "check-full-123" + assert result["name"] == "check_uniqueness_of_id" + assert result["type"] == "uniqueness" + assert result["native_id"] == "asset-456/check-789" + assert result["dimension"]["id"] == "dimension-123" + + # Verify the API call + provider.session.post.assert_called_once() + call_args = provider.session.post.call_args + + # Check URL + assert "https://test-instance.com/data_quality/v4/checks" in call_args[0][0] + assert "project_id=project-123" in call_args[0][0] + + # Check payload + payload = json.loads(call_args[1]["data"]) + assert payload["name"] == "check_uniqueness_of_id" + assert payload["type"] == "uniqueness" + assert payload["dimension"]["id"] == "dimension-123" + assert json.loads(payload["details"])["origin"] == "SDK" + + def test_create_check_full_with_catalog_id(self, provider): + """Test _create_check_full returns full check body with catalog_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + expected_body = { + "id": "check-full-456", + "name": "check_completeness", + "type": "completeness", + "native_id": "asset-111/check-456" + } + mock_response.text = json.dumps(expected_body) + provider.session.post.return_value = mock_response + + # Execute + result = provider._create_check_full( + name="check_completeness", + dimension_id="dimension-789", + native_id="asset-111/check-456", + catalog_id="catalog-999" + ) + + # Verify - should return full check body (dict) + assert isinstance(result, dict) + assert result["id"] == "check-full-456" + assert result["name"] == "check_completeness" + assert result["type"] == "completeness" + + # Verify the API call + call_args = provider.session.post.call_args + + # Check URL contains catalog_id + assert "catalog_id=catalog-999" in call_args[0][0] + assert "project_id" not in call_args[0][0] + + def test_create_check_full_with_parent_check_id(self, provider): + """Test _create_check_full returns full check body with parent.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + expected_body = { + "id": "child-check-full-123", + "name": "check_format_email", + "type": "format", + "parent": { + "id": "parent-check-456", + "name": "parent_check" + } + } + mock_response.text = json.dumps(expected_body) + provider.session.post.return_value = mock_response + + # Execute + result = provider._create_check_full( + name="check_format_email", + dimension_id="dimension-789", + native_id="asset-111/email/format", + check_type="format", + project_id="project-123", + parent_check_id="parent-check-456" + ) + + # Verify - should return full check body with parent + assert isinstance(result, dict) + assert result["id"] == "child-check-full-123" + assert result["parent"]["id"] == "parent-check-456" + assert result["parent"]["name"] == "parent_check" + + # Verify the API call includes parent in payload + provider.session.post.assert_called_once() + call_args = provider.session.post.call_args + + # Check payload includes parent + payload = json.loads(call_args[1]["data"]) + assert "parent" in payload + assert payload["parent"]["id"] == "parent-check-456" + + def test_create_check_full_without_check_type(self, provider): + """Test _create_check_full defaults check_type to name.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + expected_body = { + "id": "check-full-789", + "name": "format_check", + "type": "format_check" + } + mock_response.text = json.dumps(expected_body) + provider.session.post.return_value = mock_response + + # Execute + result = provider._create_check_full( + name="format_check", + dimension_id="dimension-111", + native_id="asset-222/check-789", + project_id="project-123" + ) + + # Verify - should return full check body + assert isinstance(result, dict) + assert result["id"] == "check-full-789" + assert result["name"] == "format_check" + assert result["type"] == "format_check" + + # Verify the API call + call_args = provider.session.post.call_args + payload = json.loads(call_args[1]["data"]) + + # Check that type defaults to name + assert payload["type"] == "format_check" + assert payload["name"] == "format_check" + + def test_create_check_full_missing_both_ids(self, provider): + """Test _create_check_full without project_id or catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider._create_check_full( + name="check_test", + dimension_id="dimension-123", + native_id="asset-456/check-789" + ) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_create_check_full_both_ids_provided(self, provider): + """Test _create_check_full with both project_id and catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider._create_check_full( + name="check_test", + dimension_id="dimension-123", + native_id="asset-456/check-789", + project_id="project-123", + catalog_id="catalog-456" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + def test_create_check_full_api_failure(self, provider): + """Test _create_check_full with failed API request.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 500 + mock_response.text = "Internal server error" + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider._create_check_full( + name="check_test", + dimension_id="dimension-123", + native_id="asset-456/check-789", + project_id="project-123" + ) + + assert "Failed to create check" in str(exc_info.value) + assert "500" in str(exc_info.value) + assert "Internal server error" in str(exc_info.value) + + def test_create_check_full_missing_id_in_response(self, provider): + """Test _create_check_full with missing id in response.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "name": "check_test", + "type": "test" + # Missing "id" field + }) + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider._create_check_full( + name="check_test", + dimension_id="dimension-123", + native_id="asset-456/check-789", + project_id="project-123" + ) + + assert "Check ID not found in response" in str(exc_info.value) + + # ==================== get_checks Tests ==================== + + def test_get_checks_with_project_id(self, provider): + """Test getting checks with project_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "checks": [ + { + "id": "check-1", + "name": "case_check", + "type": "case", + "asset": { + "id": "asset-123" + } + }, + { + "id": "check-2", + "name": "case_check_2", + "type": "case", + "asset": { + "id": "asset-123" + } + } + ] + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_checks( + dq_asset_id="asset-123", + check_type="case", + project_id="project-456" + ) + + # Verify + assert len(result) == 2 + assert result[0]["id"] == "check-1" + assert result[1]["id"] == "check-2" + assert result[0]["type"] == "case" + + # Verify the API call + provider.session.get.assert_called_once() + call_args = provider.session.get.call_args + + # Check URL + assert "https://test-instance.com/data_quality/v4/checks" in call_args[0][0] + assert "asset.id=asset-123" in call_args[0][0] + assert "type=case" in call_args[0][0] + assert "project_id=project-456" in call_args[0][0] + assert "include_children=true" in call_args[0][0] + + # Check headers + headers = call_args[1]["headers"] + assert headers["Authorization"] == "Bearer test-token" + + def test_get_checks_with_catalog_id(self, provider): + """Test getting checks with catalog_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "checks": [ + { + "id": "check-3", + "type": "completeness" + } + ] + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_checks( + dq_asset_id="asset-789", + check_type="completeness", + catalog_id="catalog-999" + ) + + # Verify + assert len(result) == 1 + assert result[0]["id"] == "check-3" + + # Verify the API call + call_args = provider.session.get.call_args + + # Check URL contains catalog_id + assert "catalog_id=catalog-999" in call_args[0][0] + assert "project_id" not in call_args[0][0] + + def test_get_checks_with_include_children_false(self, provider): + """Test getting checks with include_children=False.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "checks": [] + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_checks( + dq_asset_id="asset-111", + check_type="format", + project_id="project-222", + include_children=False + ) + + # Verify + assert len(result) == 0 + + # Verify the API call + call_args = provider.session.get.call_args + + # Check URL contains include_children=false + assert "include_children=false" in call_args[0][0] + + def test_get_checks_empty_result(self, provider): + """Test getting checks with no results.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "checks": [] + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_checks( + dq_asset_id="asset-nonexistent", + check_type="unknown", + project_id="project-123" + ) + + # Verify + assert result == [] + assert len(result) == 0 + + def test_get_checks_missing_both_ids(self, provider): + """Test getting checks without project_id or catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.get_checks( + dq_asset_id="asset-123", + check_type="case" + ) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_get_checks_both_ids_provided(self, provider): + """Test getting checks with both project_id and catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.get_checks( + dq_asset_id="asset-123", + check_type="case", + project_id="project-123", + catalog_id="catalog-456" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + def test_get_checks_api_failure(self, provider): + """Test failed get checks request.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 404 + mock_response.text = "Asset not found" + provider.session.get.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.get_checks( + dq_asset_id="asset-invalid", + check_type="case", + project_id="project-123" + ) + + assert "Failed to get checks" in str(exc_info.value) + assert "404" in str(exc_info.value) + assert "Asset not found" in str(exc_info.value) + + def test_get_checks_missing_checks_in_response(self, provider): + """Test get checks with missing 'checks' field in response.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "data": [] + # Missing "checks" field + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_checks( + dq_asset_id="asset-123", + check_type="case", + project_id="project-123" + ) + + # Verify - should return empty list when checks field is missing + assert result == [] + + def test_get_checks_with_multiple_check_types(self, provider): + """Test getting checks filters by specific check_type.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "checks": [ + { + "id": "check-format-1", + "type": "format" + }, + { + "id": "check-format-2", + "type": "format" + } + ] + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_checks( + dq_asset_id="asset-multi", + check_type="format", + project_id="project-123" + ) + + # Verify + assert len(result) == 2 + assert all(check["type"] == "format" for check in result) + + # Verify the API call includes the check_type filter + call_args = provider.session.get.call_args + assert "type=format" in call_args[0][0] + + def test_get_checks_with_comparison_check_type(self, provider): + """Test getting checks with comparison check type.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "checks": [ + { + "id": "check-comparison-1", + "type": "comparison", + "name": "compare_columns" + } + ] + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_checks( + dq_asset_id="asset-comparison", + check_type="comparison", + project_id="project-123" + ) + + # Verify + assert len(result) == 1 + assert result[0]["type"] == "comparison" + assert result[0]["name"] == "compare_columns" \ No newline at end of file diff --git a/tests/src/dq_validator/provider/test_config.py b/tests/src/dq_validator/provider/test_config.py new file mode 100644 index 0000000..3158db3 --- /dev/null +++ b/tests/src/dq_validator/provider/test_config.py @@ -0,0 +1,284 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Test suite for ProviderConfig module +""" + +import pytest +from unittest.mock import Mock, patch +from wxdi.dq_validator.provider.config import ProviderConfig +from wxdi.common.auth.auth_config import AuthConfig, EnvironmentType + + +class TestProviderConfig: + """Test cases for ProviderConfig class""" + + @pytest.fixture + def base_url(self): + """Base URL for API""" + return "https://api.example.com" + + @pytest.fixture + def auth_token(self): + """Authentication token""" + return "Bearer test-token-12345" + + @pytest.fixture + def project_id(self): + """Project ID""" + return "72d21c1d-499b-4784-a3c7-6f84507f9a20" + + @pytest.fixture + def catalog_id(self): + """Catalog ID""" + return "a1b2c3d4-e5f6-7890-abcd-ef1234567890" + + @pytest.fixture + def auth_config_ibm_cloud(self): + """AuthConfig for IBM Cloud""" + return AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key="test-api-key-12345" + ) + + @pytest.fixture + def auth_config_aws_cloud(self): + """AuthConfig for AWS Cloud""" + return AuthConfig( + environment_type=EnvironmentType.AWS_CLOUD, + api_key="test-api-key-12345", + account_id="test-account-id" + ) + + @pytest.fixture + def auth_config_gov_cloud(self): + """AuthConfig for Government Cloud""" + return AuthConfig( + environment_type=EnvironmentType.GOV_CLOUD, + api_key="test-api-key-12345" + ) + + @pytest.fixture + def auth_config_on_prem(self): + """AuthConfig for On-Premises""" + return AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url="https://on-prem.example.com", + username="test-user", + api_key="test-api-key" + ) + + def test_config_with_auth_token_only(self, base_url, auth_token): + """Test ProviderConfig initialization with auth_token only""" + config = ProviderConfig(base_url, auth_token=auth_token) + + assert config.url == base_url + assert config._auth_token == auth_token + assert config.project_id is None + assert config.catalog_id is None + assert config.auth_provider is None + assert config.auth_token == auth_token + + def test_config_with_project_id(self, base_url, auth_token, project_id): + """Test ProviderConfig initialization with project_id""" + config = ProviderConfig(base_url, auth_token=auth_token, project_id=project_id) + + assert config.url == base_url + assert config.project_id == project_id + assert config.catalog_id is None + + def test_config_with_catalog_id(self, base_url, auth_token, catalog_id): + """Test ProviderConfig initialization with catalog_id""" + config = ProviderConfig(base_url, auth_token=auth_token, catalog_id=catalog_id) + + assert config.url == base_url + assert config.catalog_id == catalog_id + assert config.project_id is None + + def test_config_with_auth_config_ibm_cloud(self, base_url, auth_config_ibm_cloud): + """Test ProviderConfig initialization with AuthConfig for IBM Cloud""" + with patch('wxdi.dq_validator.provider.config.AuthProvider') as mock_auth_provider: + mock_provider_instance = Mock() + mock_provider_instance.get_token.return_value = "mocked-token-12345" + mock_auth_provider.return_value = mock_provider_instance + + config = ProviderConfig(base_url, auth_config=auth_config_ibm_cloud) + + assert config.url == base_url + assert config.auth_provider is not None + assert config._auth_token is None + mock_auth_provider.assert_called_once_with(auth_config_ibm_cloud) + + # Test that auth_token property calls get_token + token = config.auth_token + assert token == "mocked-token-12345" + mock_provider_instance.get_token.assert_called_once() + + def test_config_with_auth_config_aws_cloud(self, base_url, auth_config_aws_cloud): + """Test ProviderConfig initialization with AuthConfig for AWS Cloud""" + with patch('wxdi.dq_validator.provider.config.AuthProvider') as mock_auth_provider: + mock_provider_instance = Mock() + mock_provider_instance.get_token.return_value = "aws-token-12345" + mock_auth_provider.return_value = mock_provider_instance + + config = ProviderConfig(base_url, auth_config=auth_config_aws_cloud) + + assert config.auth_provider is not None + mock_auth_provider.assert_called_once_with(auth_config_aws_cloud) + + token = config.auth_token + assert token == "aws-token-12345" + + def test_config_with_auth_config_gov_cloud(self, base_url, auth_config_gov_cloud): + """Test ProviderConfig initialization with AuthConfig for Government Cloud""" + with patch('wxdi.dq_validator.provider.config.AuthProvider') as mock_auth_provider: + mock_provider_instance = Mock() + mock_provider_instance.get_token.return_value = "gov-token-12345" + mock_auth_provider.return_value = mock_provider_instance + + config = ProviderConfig(base_url, auth_config=auth_config_gov_cloud) + + assert config.auth_provider is not None + token = config.auth_token + assert token == "gov-token-12345" + + def test_config_with_auth_config_on_prem(self, base_url, auth_config_on_prem): + """Test ProviderConfig initialization with AuthConfig for On-Premises""" + with patch('wxdi.dq_validator.provider.config.AuthProvider') as mock_auth_provider: + mock_provider_instance = Mock() + mock_provider_instance.get_token.return_value = "on-prem-token-12345" + mock_auth_provider.return_value = mock_provider_instance + + config = ProviderConfig(base_url, auth_config=auth_config_on_prem) + + assert config.auth_provider is not None + token = config.auth_token + assert token == "on-prem-token-12345" + + def test_config_with_both_auth_token_and_auth_config( + self, base_url, auth_token, auth_config_ibm_cloud + ): + """Test ProviderConfig with both auth_token and auth_config (auth_config takes precedence)""" + with patch('wxdi.dq_validator.provider.config.AuthProvider') as mock_auth_provider: + mock_provider_instance = Mock() + mock_provider_instance.get_token.return_value = "config-token-12345" + mock_auth_provider.return_value = mock_provider_instance + + config = ProviderConfig( + base_url, + auth_token=auth_token, + auth_config=auth_config_ibm_cloud + ) + + # auth_config should take precedence + assert config.auth_provider is not None + assert config._auth_token == auth_token + + # When getting token, auth_provider should be used first + token = config.auth_token + assert token == "config-token-12345" + mock_provider_instance.get_token.assert_called_once() + + def test_config_auth_token_property_with_auth_provider( + self, base_url, auth_config_ibm_cloud + ): + """Test auth_token property when auth_provider is set""" + with patch('wxdi.dq_validator.provider.config.AuthProvider') as mock_auth_provider: + mock_provider_instance = Mock() + mock_provider_instance.get_token.return_value = "provider-token" + mock_auth_provider.return_value = mock_provider_instance + + config = ProviderConfig(base_url, auth_config=auth_config_ibm_cloud) + + # First call + token1 = config.auth_token + assert token1 == "provider-token" + + # Second call - should call get_token again (no caching) + token2 = config.auth_token + assert token2 == "provider-token" + assert mock_provider_instance.get_token.call_count == 2 + + def test_config_auth_token_property_with_static_token(self, base_url, auth_token): + """Test auth_token property when only static token is set""" + config = ProviderConfig(base_url, auth_token=auth_token) + + token = config.auth_token + assert token == auth_token + + def test_config_auth_token_property_no_auth(self, base_url): + """Test auth_token property raises error when no authentication is provided""" + config = ProviderConfig(base_url) + + assert config.auth_token == '' + + def test_config_get_auth_token_no_auth(self, base_url): + """Test auth_token property raises error when no authentication is provided""" + config = ProviderConfig(base_url) + + with pytest.raises(ValueError) as exc_info: + _ = config.get_auth_token() + + assert "No authentication token provided" in str(exc_info.value) + + def test_config_with_all_parameters( + self, base_url, auth_token, project_id, catalog_id, auth_config_ibm_cloud + ): + """Test ProviderConfig with all parameters""" + with patch('wxdi.dq_validator.provider.config.AuthProvider') as mock_auth_provider: + mock_provider_instance = Mock() + returned_token = "full-config-token" + mock_provider_instance.get_token.return_value = returned_token + mock_auth_provider.return_value = mock_provider_instance + + config = ProviderConfig( + base_url, + auth_token=auth_token, + project_id=project_id, + catalog_id=catalog_id, + auth_config=auth_config_ibm_cloud + ) + + assert config.url == base_url + assert config._auth_token == auth_token + assert config.project_id == project_id + assert config.catalog_id == catalog_id + assert config.auth_provider is not None + assert config.auth_token == returned_token + + def test_config_auth_provider_none_when_no_auth_config(self, base_url, auth_token): + """Test that auth_provider is None when auth_config is not provided""" + config = ProviderConfig(base_url, auth_token=auth_token) + + assert config.auth_provider is None + + def test_config_with_project_and_catalog( + self, base_url, auth_token, project_id, catalog_id + ): + """Test ProviderConfig with both project_id and catalog_id""" + config = ProviderConfig( + base_url, + auth_token=auth_token, + project_id=project_id, + catalog_id=catalog_id + ) + + assert config.project_id == project_id + assert config.catalog_id == catalog_id + assert config.auth_token == auth_token + + +# Made with Bob \ No newline at end of file diff --git a/tests/src/dq_validator/provider/test_dimensions.py b/tests/src/dq_validator/provider/test_dimensions.py new file mode 100644 index 0000000..396fc04 --- /dev/null +++ b/tests/src/dq_validator/provider/test_dimensions.py @@ -0,0 +1,227 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" + +import pytest +from unittest.mock import Mock, patch +import json + +from wxdi.dq_validator.provider import ProviderConfig, DimensionsProvider + + +class TestDimensionsProvider: + """Test suite for DimensionsProvider class.""" + + @pytest.fixture + def config(self): + """Create a test configuration.""" + return ProviderConfig( + url="https://test-instance.com", + auth_token="Bearer test-token" + ) + + @pytest.fixture + def provider(self, config): + """Create a test DimensionsProvider instance.""" + with patch('wxdi.dq_validator.provider.base_provider.Session') as mock_session_class: + mock_session = Mock() + mock_session_class.return_value = mock_session + provider = DimensionsProvider(config) + yield provider + + # ==================== search_dimension Tests ==================== + + def test_search_dimension_success(self, provider): + """Test successful dimension search.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "dimensions": [ + { + "id": "371114cd-5516-4691-8b2e-1e66edf66486", + "name": "Completeness", + "description": "Data is complete if it contains all required values.", + "is_default": True + } + ] + }) + provider.session.post.return_value = mock_response + + # Execute + result = provider.search_dimension("Completeness") + + # Verify + assert result == "371114cd-5516-4691-8b2e-1e66edf66486" + + # Verify the API call + provider.session.post.assert_called_once() + call_args = provider.session.post.call_args + + # Check URL + assert "https://test-instance.com/data_quality/v4/search_dq_dimension" in call_args[0][0] + assert "name=Completeness" in call_args[0][0] + + # Check headers + headers = call_args[1]["headers"] + assert headers["Authorization"] == "Bearer test-token" + assert headers["Content-Type"] == "application/json" + + def test_search_dimension_case_insensitive(self, provider): + """Test case-insensitive dimension search.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "dimensions": [ + { + "id": "accuracy-id-123", + "name": "Accuracy", + "description": "Data accuracy dimension" + } + ] + }) + provider.session.post.return_value = mock_response + + # Execute with lowercase + result = provider.search_dimension("accuracy") + + # Verify - should match case-insensitively + assert result == "accuracy-id-123" + + def test_search_dimension_multiple_results(self, provider): + """Test searching when API returns multiple dimensions.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "dimensions": [ + { + "id": "completeness-id", + "name": "Completeness", + "description": "Completeness dimension" + }, + { + "id": "accuracy-id", + "name": "Accuracy", + "description": "Accuracy dimension" + }, + { + "id": "validity-id", + "name": "Validity", + "description": "Validity dimension" + } + ] + }) + provider.session.post.return_value = mock_response + + # Execute - search for Accuracy + result = provider.search_dimension("Accuracy") + + # Verify - should find the correct one + assert result == "accuracy-id" + + def test_search_dimension_api_failure(self, provider): + """Test failed dimension search request.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 500 + mock_response.text = "Internal server error" + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.search_dimension("Completeness") + + assert "Failed to get dimension" in str(exc_info.value) + assert "500" in str(exc_info.value) + + def test_search_dimension_not_found_empty_list(self, provider): + """Test searching when API returns empty list.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "dimensions": [] + }) + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.search_dimension("NonExistent") + + assert "Dimension with name 'NonExistent' not found" in str(exc_info.value) + + def test_search_dimension_not_found_no_match(self, provider): + """Test searching when name doesn't match any result.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "dimensions": [ + { + "id": "completeness-id", + "name": "Completeness", + "description": "Completeness dimension" + } + ] + }) + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.search_dimension("InvalidDimension") + + assert "Dimension with name 'InvalidDimension' not found in results" in str(exc_info.value) + + def test_search_dimension_missing_id_in_response(self, provider): + """Test searching when ID is missing in response.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "dimensions": [ + { + "name": "Completeness", + "description": "Completeness dimension" + # Missing "id" field + } + ] + }) + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.search_dimension("Completeness") + + assert "Dimension ID not found in response" in str(exc_info.value) + + def test_search_dimension_missing_dimensions_key(self, provider): + """Test searching when response doesn't have 'dimensions' key.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "data": [] + # Missing "dimensions" key + }) + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.search_dimension("Completeness") + + assert "Dimension with name 'Completeness' not found" in str(exc_info.value) \ No newline at end of file diff --git a/tests/src/dq_validator/provider/test_dq_search.py b/tests/src/dq_validator/provider/test_dq_search.py new file mode 100644 index 0000000..dbaa81a --- /dev/null +++ b/tests/src/dq_validator/provider/test_dq_search.py @@ -0,0 +1,279 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Tests for DQSearchProvider +""" + +import pytest +from unittest.mock import Mock, patch, MagicMock +from wxdi.dq_validator.provider import ProviderConfig, DQSearchProvider + + +@pytest.fixture +def config(): + """Create a test configuration""" + return ProviderConfig( + url="https://test.example.com", + auth_token="Bearer test-token", + project_id="test-project-id" + ) + + +@pytest.fixture +def provider(config): + """Create a DQSearchProvider instance""" + with patch('wxdi.dq_validator.provider.base_provider.Session') as mock_session_class: + mock_session = Mock() + mock_session_class.return_value = mock_session + provider = DQSearchProvider(config) + yield provider + + +class TestSearchDQCheck: + """Tests for search_dq_check method""" + + def test_search_dq_check_success_with_project_id(self, provider): + """Test successful DQ check search with project_id""" + mock_response = Mock() + mock_response.ok = True + mock_response.text = '''{ + "id": "ad277842-dea7-44ef-8e4b-d940df0f79aa", + "account_id": "999", + "created_at": "2025-12-19T06:37:23.519Z", + "dimension": { + "id": "ec453723-669c-48bb-82c1-11b69b3b8c93", + "name": "Validity" + }, + "name": "Format check", + "native_id": "b2debda2-6ab9-4a39-8c23-17954e004dcf/7377e2cd-ac0e-4833-8760-fd0e8cb682aa", + "wkc_container_id": "24419069-d649-45cb-a2c1-64d6eed650d5", + "type": "format" + }''' + provider.session.post.return_value = mock_response + + result = provider.search_dq_check( + native_id="b2debda2-6ab9-4a39-8c23-17954e004dcf/7377e2cd-ac0e-4833-8760-fd0e8cb682aa", + check_type="format", + project_id="24419069-d649-45cb-a2c1-64d6eed650d5" + ) + + assert result['id'] == "ad277842-dea7-44ef-8e4b-d940df0f79aa" + assert result['name'] == "Format check" + assert result['type'] == "format" + assert result['native_id'] == "b2debda2-6ab9-4a39-8c23-17954e004dcf/7377e2cd-ac0e-4833-8760-fd0e8cb682aa" + + # Verify project_id is in the URL + call_args = provider.session.post.call_args + assert "project_id=24419069-d649-45cb-a2c1-64d6eed650d5" in call_args[0][0] + + def test_search_dq_check_success_with_catalog_id(self, provider): + """Test successful DQ check search with catalog_id""" + mock_response = Mock() + mock_response.ok = True + mock_response.text = '''{ + "id": "ad277842-dea7-44ef-8e4b-d940df0f79aa", + "name": "Format check", + "type": "format" + }''' + provider.session.post.return_value = mock_response + + result = provider.search_dq_check( + native_id="test-native-id", + check_type="format", + catalog_id="catalog-123" + ) + + assert result['id'] == "ad277842-dea7-44ef-8e4b-d940df0f79aa" + + # Verify catalog_id is in the URL + call_args = provider.session.post.call_args + assert "catalog_id=catalog-123" in call_args[0][0] + assert "project_id" not in call_args[0][0] + + def test_search_dq_check_missing_both_ids(self, provider): + """Test DQ check search without project_id or catalog_id""" + with pytest.raises(ValueError) as exc_info: + provider.search_dq_check( + native_id="test-id", + check_type="format" + ) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_search_dq_check_both_ids_provided(self, provider): + """Test DQ check search with both project_id and catalog_id""" + with pytest.raises(ValueError) as exc_info: + provider.search_dq_check( + native_id="test-id", + check_type="format", + project_id="project-123", + catalog_id="catalog-123" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + def test_search_dq_check_failure(self, provider): + """Test DQ check search failure""" + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 404 + mock_response.text = "Not found" + provider.session.post.return_value = mock_response + + with pytest.raises(ValueError) as exc_info: + provider.search_dq_check( + native_id="invalid-id", + check_type="format", + project_id="test-project" + ) + + assert "Failed to search DQ check" in str(exc_info.value) + assert "404" in str(exc_info.value) + + def test_search_dq_check_with_include_children(self, provider): + """Test DQ check search with include_children parameter""" + mock_response = Mock() + mock_response.ok = True + mock_response.text = '{"id": "test-id", "type": "format"}' + provider.session.post.return_value = mock_response + + provider.search_dq_check( + native_id="test-native-id", + check_type="format", + project_id="test-project", + include_children=False + ) + + # Verify the URL contains include_children=false + call_args = provider.session.post.call_args + assert "include_children=false" in call_args[0][0] + + +class TestSearchDQAsset: + """Tests for search_dq_asset method""" + + def test_search_dq_asset_success_with_project_id(self, provider): + """Test successful DQ asset search with project_id""" + mock_response = Mock() + mock_response.ok = True + mock_response.text = '''{ + "id": "1488a413-99f9-4bed-906d-c33b505d5728", + "account_id": "999", + "created_at": "2026-01-28T14:08:08.380Z", + "name": "RTN", + "native_id": "b2debda2-6ab9-4a39-8c23-17954e004dcf/RTN", + "wkc_container_id": "24419069-d649-45cb-a2c1-64d6eed650d5", + "type": "column" + }''' + provider.session.post.return_value = mock_response + + result = provider.search_dq_asset( + native_id="b2debda2-6ab9-4a39-8c23-17954e004dcf/RTN", + project_id="24419069-d649-45cb-a2c1-64d6eed650d5", + asset_type="column" + ) + + assert result['id'] == "1488a413-99f9-4bed-906d-c33b505d5728" + assert result['name'] == "RTN" + assert result['type'] == "column" + assert result['native_id'] == "b2debda2-6ab9-4a39-8c23-17954e004dcf/RTN" + + # Verify project_id is in the URL + call_args = provider.session.post.call_args + assert "project_id=24419069-d649-45cb-a2c1-64d6eed650d5" in call_args[0][0] + + def test_search_dq_asset_success_with_catalog_id(self, provider): + """Test successful DQ asset search with catalog_id""" + mock_response = Mock() + mock_response.ok = True + mock_response.text = '''{ + "id": "test-asset-id", + "name": "Test Asset", + "type": "column" + }''' + provider.session.post.return_value = mock_response + + result = provider.search_dq_asset( + native_id="test-native-id", + catalog_id="catalog-123", + asset_type="column" + ) + + assert result['id'] == "test-asset-id" + + # Verify catalog_id is in the URL + call_args = provider.session.post.call_args + assert "catalog_id=catalog-123" in call_args[0][0] + assert "project_id" not in call_args[0][0] + + def test_search_dq_asset_missing_both_ids(self, provider): + """Test DQ asset search without project_id or catalog_id""" + with pytest.raises(ValueError) as exc_info: + provider.search_dq_asset( + native_id="test-id" + ) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_search_dq_asset_both_ids_provided(self, provider): + """Test DQ asset search with both project_id and catalog_id""" + with pytest.raises(ValueError) as exc_info: + provider.search_dq_asset( + native_id="test-id", + project_id="project-123", + catalog_id="catalog-123" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + def test_search_dq_asset_failure(self, provider): + """Test DQ asset search failure""" + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 404 + mock_response.text = "Not found" + provider.session.post.return_value = mock_response + + with pytest.raises(ValueError) as exc_info: + provider.search_dq_asset( + native_id="invalid-id", + project_id="test-project" + ) + + assert "Failed to search DQ asset" in str(exc_info.value) + assert "404" in str(exc_info.value) + + def test_search_dq_asset_with_optional_params(self, provider): + """Test DQ asset search with optional parameters""" + mock_response = Mock() + mock_response.ok = True + mock_response.text = '{"id": "test-id", "type": "column"}' + provider.session.post.return_value = mock_response + + provider.search_dq_asset( + native_id="test-native-id", + project_id="test-project", + asset_type="table", + include_children=False, + get_actual_asset=True + ) + + # Verify the URL contains the parameters + call_args = provider.session.post.call_args + url = call_args[0][0] + assert "type=table" in url + assert "include_children=false" in url + assert "get_actual_asset=true" in url \ No newline at end of file diff --git a/tests/src/dq_validator/provider/test_glossary.py b/tests/src/dq_validator/provider/test_glossary.py new file mode 100644 index 0000000..464c6c3 --- /dev/null +++ b/tests/src/dq_validator/provider/test_glossary.py @@ -0,0 +1,697 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Comprehensive unit tests for the GlossaryProvider module. + +This test suite validates all aspects of the GlossaryProvider including: + - Configuration validation + - Fetching published artifacts by ID + - Fetching terms by version ID + - Response parsing and model creation + - Error handling (HTTP errors, network errors, invalid responses) + - Query parameter handling + +Test Coverage: + - TestProviderConfig: Configuration class validation + - TestGlossaryProvider: Provider functionality + - TestGlossaryTermModel: Response model parsing + - TestUtilityFunctions: Helper function validation + +Running Tests: + Run all glossary tests: + $ pytest tests/src/provider/test_glossary.py -v + + Run specific test: + $ pytest tests/src/provider/test_glossary.py::TestGlossaryProvider::test_get_published_artifact_success -v + + Run with coverage: + $ pytest tests/src/provider/test_glossary.py --cov=dq_validator.provider --cov-report=html + +Note: + Tests use pytest-mock to avoid actual network calls. + Test data is loaded from tests/data/ directory. + +See Also: + - src/dq_validator/provider/glossary.py: GlossaryProvider implementation + - src/dq_validator/provider/response_model.py: Response models +""" + +import json +import pytest +from pathlib import Path +from wxdi.dq_validator.provider.glossary import GlossaryProvider +from wxdi.dq_validator.provider.config import ProviderConfig +from wxdi.dq_validator.provider.response_model import GlossaryTerm +from wxdi.dq_validator.utils import get_request_headers, get_url_with_query_params + + +# Test constants +MOCK_URL = "https://test.example.com" +MOCK_AUTH_TOKEN = "Bearer test-token-123" +MOCK_PROJECT_ID = "test-project-id" +MOCK_TERM_ID = "30d1b847-0aa9-4840-a182-dd157fe977a0" +MOCK_VERSION_ID = "bdeef8cc-d9ab-4822-b3df-cef82b4de538_0" + +# Get test data directory +TEST_DATA_DIR = Path(__file__).parent.parent.parent.parent / "data" + + +def load_test_data(filename: str) -> str: + """Load test data from JSON file""" + file_path = TEST_DATA_DIR / filename + with open(file_path, "r") as f: + return f.read() + + +# ============================================================================ +# ProviderConfig Tests +# ============================================================================ + + +class TestProviderConfig: + """Test ProviderConfig class""" + + def test_config_with_all_parameters(self): + """Test creating config with all parameters""" + config = ProviderConfig( + url=MOCK_URL, auth_token=MOCK_AUTH_TOKEN, project_id=MOCK_PROJECT_ID + ) + assert config.url == MOCK_URL + assert config.auth_token == MOCK_AUTH_TOKEN + assert config.project_id == MOCK_PROJECT_ID + + def test_config_without_project_id(self): + """Test creating config without project_id""" + config = ProviderConfig(url=MOCK_URL, auth_token=MOCK_AUTH_TOKEN) + assert config.url == MOCK_URL + assert config.auth_token == MOCK_AUTH_TOKEN + assert config.project_id is None + + def test_config_attributes_accessible(self): + """Test that config attributes are accessible""" + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN, MOCK_PROJECT_ID) + assert hasattr(config, "url") + assert hasattr(config, "auth_token") + assert hasattr(config, "project_id") + + +# ============================================================================ +# Utility Functions Tests +# ============================================================================ + + +class TestUtilityFunctions: + """Test utility functions""" + + def test_get_request_headers_with_auth(self): + """Test getting request headers with auth token""" + headers = get_request_headers(MOCK_AUTH_TOKEN) + assert headers["Authorization"] == MOCK_AUTH_TOKEN + assert headers["Content-Type"] == "application/json" + + def test_get_request_headers_custom_content_type(self): + """Test getting request headers with custom content type""" + headers = get_request_headers(MOCK_AUTH_TOKEN, "text/plain") + assert headers["Authorization"] == MOCK_AUTH_TOKEN + assert headers["Content-Type"] == "text/plain" + + def test_get_request_headers_no_auth(self): + """Test getting request headers without auth token""" + headers = get_request_headers("") + assert "Authorization" not in headers + assert headers["Content-Type"] == "application/json" + + def test_get_url_with_query_params(self): + """Test adding query parameters to URL""" + url = "https://example.com/api" + params = {"key1": "value1", "key2": "value2"} + result = get_url_with_query_params(url, params) + assert "key1=value1" in result + assert "key2=value2" in result + assert result.startswith(url) + + def test_get_url_without_query_params(self): + """Test URL without query parameters""" + url = "https://example.com/api" + result = get_url_with_query_params(url, None) + assert result == url + + +# ============================================================================ +# GlossaryProvider Tests +# ============================================================================ + + +class TestGlossaryProvider: + """Test GlossaryProvider class""" + + def test_provider_initialization(self): + """Test provider initialization with config""" + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + assert provider.config == config + assert provider.config.url == MOCK_URL + assert provider.config.auth_token == MOCK_AUTH_TOKEN + + def test_get_published_artifact_success(self, mocker): + """Test successful retrieval of published artifact""" + # Load test data + test_data = load_test_data("term_latest_version.json") + + # Mock response + mock_response = mocker.MagicMock() + mock_response.text = test_data + mock_response.status_code = 200 + + # Mock Session.get + mock_session = mocker.MagicMock() + mock_session.get.return_value = mock_response + mocker.patch( + "wxdi.dq_validator.provider.base_provider.Session", return_value=mock_session + ) + + # Create provider and call method + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + result = provider.get_published_artifact_by_id(MOCK_TERM_ID) + + # Verify + assert isinstance(result, GlossaryTerm) + assert result.metadata.artifact_id == MOCK_TERM_ID + assert result.metadata.name == "mango" + assert result.metadata.state == "PUBLISHED" + + # Verify URL was constructed correctly + expected_url = ( + f"{MOCK_URL}/v3/governance_artifact_types/glossary_term/{MOCK_TERM_ID}" + ) + mock_session.get.assert_called_once() + call_args = mock_session.get.call_args + assert call_args[0][0] == expected_url + + def test_get_published_artifact_with_options(self, mocker): + """Test retrieval with query options""" + test_data = load_test_data("term_latest_version.json") + + mock_response = mocker.MagicMock() + mock_response.text = test_data + mock_response.status_code = 200 + + mock_session = mocker.MagicMock() + mock_session.get.return_value = mock_response + mocker.patch( + "wxdi.dq_validator.provider.base_provider.Session", return_value=mock_session + ) + + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + options = {"include": "metadata", "limit": "10"} + result = provider.get_published_artifact_by_id(MOCK_TERM_ID, options) + + # Verify result + assert isinstance(result, GlossaryTerm) + + # Verify URL includes query parameters + call_args = mock_session.get.call_args + called_url = call_args[0][0] + assert "include=metadata" in called_url + assert "limit=10" in called_url + + def test_get_term_by_version_id_success(self, mocker): + """Test successful retrieval of term by version ID""" + # Load test data with full details + test_data = load_test_data("term_response.json") + + mock_response = mocker.MagicMock() + mock_response.text = test_data + mock_response.status_code = 200 + + mock_session = mocker.MagicMock() + mock_session.get.return_value = mock_response + mocker.patch( + "wxdi.dq_validator.provider.base_provider.Session", return_value=mock_session + ) + + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + result = provider.get_term_by_version_id(MOCK_TERM_ID, MOCK_VERSION_ID) + + # Verify + assert isinstance(result, GlossaryTerm) + assert result.metadata.artifact_id == MOCK_TERM_ID + assert result.metadata.version_id == MOCK_VERSION_ID + assert result.metadata.name == "mango" + assert result.metadata.effective_start_date is not None + + # Verify extended attributes + assert result.entity is not None + assert len(result.entity.extended_attribute_groups.dq_constraints) == 2 + + # Verify URL was constructed correctly + expected_url = ( + f"{MOCK_URL}/v3/glossary_terms/{MOCK_TERM_ID}/versions/{MOCK_VERSION_ID}" + ) + mock_session.get.assert_called_once() + call_args = mock_session.get.call_args + assert call_args[0][0] == expected_url + + def test_get_term_by_version_id_with_options(self, mocker): + """Test retrieval by version ID with query options""" + test_data = load_test_data("term_response.json") + + mock_response = mocker.MagicMock() + mock_response.text = test_data + mock_response.status_code = 200 + + mock_session = mocker.MagicMock() + mock_session.get.return_value = mock_response + mocker.patch( + "wxdi.dq_validator.provider.base_provider.Session", return_value=mock_session + ) + + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + options = {"included_extended_attribute_groups": "dq_constraints"} + result = provider.get_term_by_version_id(MOCK_TERM_ID, MOCK_VERSION_ID, options) + + # Verify result + assert isinstance(result, GlossaryTerm) + + # Verify URL includes query parameters + call_args = mock_session.get.call_args + called_url = call_args[0][0] + assert "included_extended_attribute_groups=dq_constraints" in called_url + + def test_get_term_by_version_id_draft(self, mocker): + """Test successful retrieval of term by version ID""" + # Load test data with full details + test_data = load_test_data("term_draft.json") + + mock_response = mocker.MagicMock() + mock_response.text = test_data + mock_response.status_code = 200 + + mock_session = mocker.MagicMock() + mock_session.get.return_value = mock_response + mocker.patch( + "wxdi.dq_validator.provider.base_provider.Session", return_value=mock_session + ) + + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + result = provider.get_term_by_version_id(MOCK_TERM_ID, MOCK_VERSION_ID) + + # Verify + assert isinstance(result, GlossaryTerm) + assert result.metadata.artifact_id == MOCK_TERM_ID + assert result.metadata.version_id == MOCK_VERSION_ID + assert result.metadata.name == "mango" + assert result.metadata.effective_start_date is None + assert result.metadata.draft_mode is not None + assert result.metadata.workflow_id is not None + assert result.metadata.workflow_state is not None + assert result.metadata.state == "DRAFT_HISTORY" + + # Verify extended attributes + assert result.entity.extended_attribute_groups is None + + # Verify URL was constructed correctly + expected_url = ( + f"{MOCK_URL}/v3/glossary_terms/{MOCK_TERM_ID}/versions/{MOCK_VERSION_ID}" + ) + mock_session.get.assert_called_once() + call_args = mock_session.get.call_args + assert call_args[0][0] == expected_url + + def test_request_headers_included(self, mocker): + """Test that authorization headers are included in requests""" + test_data = load_test_data("term_latest_version.json") + + mock_response = mocker.MagicMock() + mock_response.text = test_data + + mock_session = mocker.MagicMock() + mock_session.get.return_value = mock_response + mocker.patch( + "wxdi.dq_validator.provider.base_provider.Session", return_value=mock_session + ) + + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + provider.get_published_artifact_by_id(MOCK_TERM_ID) + + # Verify headers were passed as keyword argument + call_args = mock_session.get.call_args + assert "headers" in call_args.kwargs + headers = call_args.kwargs["headers"] + assert "Authorization" in headers + assert headers["Authorization"] == MOCK_AUTH_TOKEN + + +# ============================================================================ +# GlossaryTerm Model Tests +# ============================================================================ + + +class TestGlossaryTermModel: + """Test GlossaryTerm response model""" + + def test_parse_latest_version_response(self): + """Test parsing latest version response""" + test_data = load_test_data("term_latest_version.json") + term = GlossaryTerm.from_json(test_data) + + assert term.metadata.artifact_id == MOCK_TERM_ID + assert term.metadata.version_id == MOCK_VERSION_ID + assert term.metadata.name == "mango" + assert term.metadata.state == "PUBLISHED" + + def test_parse_full_term_response(self): + """Test parsing full term response with DQ constraints""" + test_data = load_test_data("term_response.json") + term = GlossaryTerm.from_json(test_data) + + # Verify metadata + assert term.metadata.artifact_id == MOCK_TERM_ID + assert term.metadata.name == "mango" + + # Verify DQ constraints + dq_constraints = term.entity.extended_attribute_groups.dq_constraints + assert len(dq_constraints) == 2 + + # First constraint: data_type + first_constraint = dq_constraints[0] + assert first_constraint.metadata.type == "data_type" + assert len(first_constraint.check) == 2 + + # Second constraint: length + second_constraint = dq_constraints[1] + assert second_constraint.metadata.type == "length" + assert len(second_constraint.check) == 2 + + def test_model_to_dict(self): + """Test converting model to dictionary""" + test_data = load_test_data("term_latest_version.json") + term = GlossaryTerm.from_json(test_data) + + result_dict = term.to_dict() + assert isinstance(result_dict, dict) + assert "metadata" in result_dict + assert "entity" in result_dict + assert result_dict["metadata"]["artifact_id"] == MOCK_TERM_ID + + def test_model_to_json(self): + """Test converting model to JSON string""" + test_data = load_test_data("term_latest_version.json") + term = GlossaryTerm.from_json(test_data) + + result_json = term.to_json() + assert isinstance(result_json, str) + + # Verify it's valid JSON + parsed = json.loads(result_json) + assert parsed["metadata"]["artifact_id"] == MOCK_TERM_ID + + def test_model_from_dict(self): + """Test creating model from dictionary""" + test_data = load_test_data("term_latest_version.json") + data_dict = json.loads(test_data) + + term = GlossaryTerm.from_dict(data_dict) + assert isinstance(term, GlossaryTerm) + assert term.metadata.artifact_id == MOCK_TERM_ID + + def test_dq_constraint_check_values(self): + """Test extracting check constraint values""" + test_data = load_test_data("term_response.json") + term = GlossaryTerm.from_json(test_data) + + # Get first constraint (data_type) + first_constraint = term.entity.extended_attribute_groups.dq_constraints[0] + + # Verify check values + checks = {check.name: check for check in first_constraint.check} + assert "data_type" in checks + assert checks["data_type"].value == "STRING" + assert "length" in checks + assert checks["length"].numeric_value == 80 + + # Get second constraint (length) + second_constraint = term.entity.extended_attribute_groups.dq_constraints[1] + checks = {check.name: check for check in second_constraint.check} + assert "min" in checks + assert checks["min"].numeric_value == 3 + assert "max" in checks + assert checks["max"].numeric_value == 80 + + +# ============================================================================ +# Integration Tests +# ============================================================================ + + +class TestGlossaryProviderIntegration: + """Integration tests for GlossaryProvider workflow""" + + def test_full_workflow_get_latest_then_version(self, mocker): + """Test complete workflow: get latest version, then get full details""" + # Load test data + latest_data = load_test_data("term_latest_version.json") + full_data = load_test_data("term_response.json") + + # Mock responses + mock_response_latest = mocker.MagicMock() + mock_response_latest.text = latest_data + + mock_response_full = mocker.MagicMock() + mock_response_full.text = full_data + + # Mock Session to return different responses + mock_session = mocker.MagicMock() + mock_session.get.side_effect = [mock_response_latest, mock_response_full] + mocker.patch( + "wxdi.dq_validator.provider.base_provider.Session", return_value=mock_session + ) + + # Create provider + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Step 1: Get latest version + latest_term = provider.get_published_artifact_by_id(MOCK_TERM_ID) + assert latest_term.metadata.version_id == MOCK_VERSION_ID + + # Step 2: Get full details using version_id + full_term = provider.get_term_by_version_id( + MOCK_TERM_ID, latest_term.metadata.version_id + ) + + # Verify full details + assert full_term.metadata.artifact_id == MOCK_TERM_ID + assert len(full_term.entity.extended_attribute_groups.dq_constraints) == 2 + + +# ============================================================================ +# Error Handling Tests +# ============================================================================ + + +class TestGlossaryProviderErrorHandling: + """Test error handling in GlossaryProvider""" + + def test_get_published_artifact_http_error(self, mocker): + """Test handling of HTTP errors (404, 500, etc.) in get_published_artifact_by_id""" + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a 404 response + mock_response = mocker.Mock() + mock_response.status_code = 404 + mock_response.ok = False + mock_response.text = json.dumps({"error": "Artifact not found"}) + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + # The implementation checks response.ok and raises ValueError + with pytest.raises(ValueError) as exc_info: + provider.get_published_artifact_by_id("non-existent-artifact") + + # Verify the error message contains artifact_id + assert "Cannot get artifact" in str(exc_info.value) + assert "non-existent-artifact" in str(exc_info.value) + + def test_get_term_by_version_id_http_error(self, mocker): + """Test handling of HTTP errors in get_term_by_version_id""" + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a 404 response + mock_response = mocker.Mock() + mock_response.status_code = 404 + mock_response.ok = False + mock_response.text = json.dumps({"error": "Version not found"}) + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + # The implementation checks response.ok and raises ValueError + with pytest.raises(ValueError) as exc_info: + provider.get_term_by_version_id("artifact-123", "version-456") + + # Verify the error message contains both artifact_id and version_id + assert "Cannot get artifact" in str(exc_info.value) + assert "artifact-123" in str(exc_info.value) + assert "version-456" in str(exc_info.value) + + def test_get_published_artifact_invalid_json(self, mocker): + """Test handling of invalid JSON response in get_published_artifact_by_id""" + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a response with invalid JSON + mock_response = mocker.Mock() + mock_response.ok = True + mock_response.text = "This is not valid JSON" + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + # Should raise JSONDecodeError + with pytest.raises(json.JSONDecodeError): + provider.get_published_artifact_by_id(MOCK_TERM_ID) + + def test_get_term_by_version_id_invalid_json(self, mocker): + """Test handling of invalid JSON response in get_term_by_version_id""" + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a response with invalid JSON + mock_response = mocker.Mock() + mock_response.ok = True + mock_response.text = "This is not valid JSON" + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + # Should raise JSONDecodeError + with pytest.raises(json.JSONDecodeError): + provider.get_term_by_version_id(MOCK_TERM_ID, MOCK_VERSION_ID) + + def test_get_published_artifact_malformed_response(self, mocker): + """Test handling of malformed response in get_published_artifact_by_id""" + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a response with valid JSON but missing required fields + malformed_json = {"some_field": "some_value"} + mock_response = mocker.Mock() + mock_response.ok = True + mock_response.text = json.dumps(malformed_json) + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + # Should raise ValidationError from Pydantic + with pytest.raises(Exception): # Pydantic ValidationError + provider.get_published_artifact_by_id(MOCK_TERM_ID) + + def test_get_term_by_version_id_malformed_response(self, mocker): + """Test handling of malformed response in get_term_by_version_id""" + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a response with valid JSON but missing required fields + malformed_json = {"some_field": "some_value"} + mock_response = mocker.Mock() + mock_response.ok = True + mock_response.text = json.dumps(malformed_json) + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + # Should raise ValidationError from Pydantic + with pytest.raises(Exception): # Pydantic ValidationError + provider.get_term_by_version_id(MOCK_TERM_ID, MOCK_VERSION_ID) + + def test_get_published_artifact_network_error(self, mocker): + """Test handling of network errors in get_published_artifact_by_id""" + from requests.exceptions import ConnectionError + + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a network error + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.side_effect = ConnectionError( + "Network unreachable" + ) + + # Should raise ConnectionError + with pytest.raises(ConnectionError): + provider.get_published_artifact_by_id(MOCK_TERM_ID) + + def test_get_term_by_version_id_network_error(self, mocker): + """Test handling of network errors in get_term_by_version_id""" + from requests.exceptions import ConnectionError + + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a network error + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.side_effect = ConnectionError( + "Network unreachable" + ) + + # Should raise ConnectionError + with pytest.raises(ConnectionError): + provider.get_term_by_version_id(MOCK_TERM_ID, MOCK_VERSION_ID) + + def test_get_published_artifact_timeout(self, mocker): + """Test handling of timeout errors in get_published_artifact_by_id""" + from requests.exceptions import Timeout + + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a timeout error + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.side_effect = Timeout("Request timed out") + + # Should raise Timeout + with pytest.raises(Timeout): + provider.get_published_artifact_by_id(MOCK_TERM_ID) + + def test_get_term_by_version_id_timeout(self, mocker): + """Test handling of timeout errors in get_term_by_version_id""" + from requests.exceptions import Timeout + + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a timeout error + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.side_effect = Timeout("Request timed out") + + # Should raise Timeout + with pytest.raises(Timeout): + provider.get_term_by_version_id(MOCK_TERM_ID, MOCK_VERSION_ID) + + +# Made with Bob diff --git a/tests/src/dq_validator/provider/test_issues.py b/tests/src/dq_validator/provider/test_issues.py new file mode 100644 index 0000000..80ee668 --- /dev/null +++ b/tests/src/dq_validator/provider/test_issues.py @@ -0,0 +1,1129 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +import pytest +from unittest.mock import Mock, patch, MagicMock +import json + +from wxdi.dq_validator.provider import ProviderConfig, IssuesProvider + + +class TestIssuesProvider: + """Test suite for IssuesProvider class.""" + + @pytest.fixture + def config(self): + """Create a test configuration.""" + return ProviderConfig( + url="https://test-instance.com", + auth_token="Bearer test-token" + ) + + @pytest.fixture + def provider(self, config): + """Create a test IssuesProvider instance.""" + with patch('wxdi.dq_validator.provider.base_provider.Session') as mock_session_class: + mock_session = Mock() + mock_session_class.return_value = mock_session + provider = IssuesProvider(config) + yield provider + + def test_update_issue_values_both_fields_with_project_id(self, provider): + """Test updating both occurrences and tested records with project_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issue_id": "issue-123", + "number_of_occurrences": 777, + "number_of_tested_records": 1100, + "status": "updated" + }) + provider.session.patch.return_value = mock_response + + # Execute + result = provider.update_issue_values("issue-123", occurrences=10, tested_records=100, project_id="project-123") + + # Verify + assert result["issue_id"] == "issue-123" + assert result["number_of_occurrences"] == 777 + assert result["number_of_tested_records"] == 1100 + assert result["status"] == "updated" + + # Verify the API call + provider.session.patch.assert_called_once() + call_args = provider.session.patch.call_args + + # Check URL + assert "https://test-instance.com/data_quality/v4/issues/issue-123" in call_args[0][0] + assert "project_id=project-123" in call_args[0][0] + + # Check headers + headers = call_args[1]["headers"] + assert headers["Authorization"] == "Bearer test-token" + assert headers["Content-Type"] == "application/json-patch+json" + + # Check payload - should have both operations + payload = json.loads(call_args[1]["data"]) + assert len(payload) == 2 + assert payload[0]["op"] == "add" + assert payload[0]["path"] == "/number_of_occurrences" + assert payload[0]["value"] == 10 + assert payload[1]["op"] == "add" + assert payload[1]["path"] == "/number_of_tested_records" + assert payload[1]["value"] == 100 + + def test_update_issue_values_both_fields_with_catalog_id(self, provider): + """Test updating both occurrences and tested records with catalog_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issue_id": "issue-123", + "number_of_occurrences": 777, + "number_of_tested_records": 1100, + "status": "updated" + }) + provider.session.patch.return_value = mock_response + + # Execute + result = provider.update_issue_values("issue-123", occurrences=10, tested_records=100, catalog_id="catalog-456") + + # Verify + assert result["issue_id"] == "issue-123" + + # Verify the API call + call_args = provider.session.patch.call_args + + # Check URL contains catalog_id + assert "catalog_id=catalog-456" in call_args[0][0] + assert "project_id" not in call_args[0][0] + + def test_update_issue_values_missing_both_ids(self, provider): + """Test updating without project_id or catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.update_issue_values("issue-123", occurrences=10, tested_records=100) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_update_issue_values_both_ids_provided(self, provider): + """Test updating with both project_id and catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.update_issue_values( + "issue-123", + occurrences=10, + tested_records=100, + project_id="project-123", + catalog_id="catalog-456" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + + def test_update_issue_values_with_replace_operation(self, provider): + """Test updating with replace operation.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issue_id": "issue-999", + "number_of_occurrences": 100, + "number_of_tested_records": 1000, + "status": "updated" + }) + provider.session.patch.return_value = mock_response + + # Execute + result = provider.update_issue_values( + "issue-999", + occurrences=100, + tested_records=1000, + project_id="project-123", + operation="replace" + ) + + # Verify + assert result["issue_id"] == "issue-999" + assert result["number_of_occurrences"] == 100 + assert result["number_of_tested_records"] == 1000 + + # Verify the API call + call_args = provider.session.patch.call_args + payload = json.loads(call_args[1]["data"]) + assert len(payload) == 2 + assert payload[0]["op"] == "replace" + assert payload[0]["path"] == "/number_of_occurrences" + assert payload[0]["value"] == 100 + assert payload[1]["op"] == "replace" + assert payload[1]["path"] == "/number_of_tested_records" + assert payload[1]["value"] == 1000 + + def test_update_issue_values_failure(self, provider): + """Test failed update of issue values.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 404 + mock_response.text = "Issue not found" + provider.session.patch.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.update_issue_values("invalid-issue", occurrences=10, tested_records=100, project_id="project-123") + + assert "Failed to update issue metrics for issue invalid-issue" in str(exc_info.value) + assert "404" in str(exc_info.value) + + def test_update_issue_values_with_zero_values(self, provider): + """Test updating with zero values (should be included).""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issue_id": "issue-000", + "number_of_occurrences": 0, + "number_of_tested_records": 0, + "status": "updated" + }) + provider.session.patch.return_value = mock_response + + # Execute + result = provider.update_issue_values("issue-000", occurrences=0, tested_records=0, project_id="project-123") + + # Verify + assert result["number_of_occurrences"] == 0 + assert result["number_of_tested_records"] == 0 + + # Verify the API call includes both zero values + call_args = provider.session.patch.call_args + payload = json.loads(call_args[1]["data"]) + assert len(payload) == 2 + assert payload[0]["value"] == 0 + assert payload[1]["value"] == 0 + + def test_get_issue_id_success_with_project_id(self, provider): + """Test successful retrieval of issue ID with project_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "id": "b8f4252b-cd35-4668-9b35-4635bfc6e2e0", + "account_id": "999", + "created_at": "2025-12-19T06:37:24.955Z", + "check": { + "id": "ad277842-dea7-44ef-8e4b-d940df0f79aa", + "account_id": "999", + "created_at": "2025-12-19T06:37:23.519Z", + "dimension": { + "id": "ec453723-669c-48bb-82c1-11b69b3b8c93", + "name": "Validity", + "description": "Data is valid if it conforms to the syntax (format, type, range) of its definition.", + "is_default": True + }, + "name": "Format check", + "native_id": "b2debda2-6ab9-4a39-8c23-17954e004dcf/7377e2cd-ac0e-4833-8760-fd0e8cb682aa", + "wkc_container_id": "24419069-d649-45cb-a2c1-64d6eed650d5", + "parent": { + "id": "f3fca1af-f00f-42b7-af42-f0965673237e" + }, + "type": "format" + }, + "reported_for": { + "id": "1488a413-99f9-4bed-906d-c33b505d5728", + "native_id": "b2debda2-6ab9-4a39-8c23-17954e004dcf/RTN" + }, + "number_of_occurrences": 0, + "number_of_tested_records": 1000, + "percent_occurrences": 0, + "status": "actual", + "ignored": False, + "details": "{\"sampling\":{\"size\":1000,\"min_records\":0,\"type\":\"SEQUENTIAL\"}}", + "archived_issues": [] + }) + provider.session.post.return_value = mock_response + + # Execute + result = provider.get_issue_id( + reported_for_id="1488a413-99f9-4bed-906d-c33b505d5728", + dq_check_id="ad277842-dea7-44ef-8e4b-d940df0f79aa", + project_id="24419069-d649-45cb-a2c1-64d6eed650d5" + ) + + # Verify + assert result == "b8f4252b-cd35-4668-9b35-4635bfc6e2e0" + + # Verify the API call + provider.session.post.assert_called_once() + call_args = provider.session.post.call_args + url = call_args[0][0] + + # Check URL contains required parameters + assert "project_id=24419069-d649-45cb-a2c1-64d6eed650d5" in url + assert "reported_for.id=1488a413-99f9-4bed-906d-c33b505d5728" in url + assert "check.id=ad277842-dea7-44ef-8e4b-d940df0f79aa" in url + + def test_get_issue_id_success_with_catalog_id(self, provider): + """Test successful retrieval of issue ID with catalog_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "id": "issue-id-789", + "catalog_id": "catalog-123" + }) + provider.session.post.return_value = mock_response + + # Execute + result = provider.get_issue_id( + reported_for_id="asset-id", + dq_check_id="check-id", + catalog_id="catalog-123" + ) + + # Verify + assert result == "issue-id-789" + + # Verify the API call + call_args = provider.session.post.call_args + url = call_args[0][0] + + # Check URL contains catalog_id + assert "catalog_id=catalog-123" in url + assert "project_id" not in url + + def test_get_issue_id_missing_both_ids(self, provider): + """Test get_issue_id without project_id or catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.get_issue_id( + reported_for_id="asset-id", + dq_check_id="check-id" + ) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_get_issue_id_both_ids_provided(self, provider): + """Test get_issue_id with both project_id and catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.get_issue_id( + reported_for_id="asset-id", + dq_check_id="check-id", + project_id="project-123", + catalog_id="catalog-456" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + def test_get_issue_id_failure(self, provider): + """Test failed retrieval of issue ID.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 404 + mock_response.text = "Not found" + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.get_issue_id( + reported_for_id="invalid-asset", + dq_check_id="invalid-check", + project_id="invalid-project" + ) + + assert "Failed to search for issue" in str(exc_info.value) + assert "404" in str(exc_info.value) + + def test_update_issue_metrics(self, provider): + """Test updating issue metrics using CAMS asset and check IDs.""" + with patch('wxdi.dq_validator.provider.dq_search.DQSearchProvider') as mock_search_provider_class: + # Setup mock search provider + mock_search_provider = Mock() + mock_search_provider_class.return_value = mock_search_provider + + # Mock search_dq_asset response + mock_search_provider.search_dq_asset.return_value = { + "id": "dq-asset-123", + "name": "Test Asset", + "type": "column" + } + + # Mock get_issues response (replaces search_dq_check and get_issue_id) + mock_get_issues_response = Mock() + mock_get_issues_response.ok = True + mock_get_issues_response.text = json.dumps({ + "issues": [ + { + "id": "issue-789", + "check": { + "id": "dq-check-456", + "native_id": "cams-asset-abc/cams-check-def" + }, + "project_id": "project-123" + } + ], + "total_count": 1 + }) + provider.session.get.return_value = mock_get_issues_response + + # Mock update_issue_values response + mock_update_response = Mock() + mock_update_response.ok = True + mock_update_response.text = json.dumps({ + "issue_id": "issue-789", + "number_of_occurrences": 10, + "number_of_tested_records": 100 + }) + provider.session.patch.return_value = mock_update_response + + # Execute + result = provider.update_issue_metrics( + asset_id="cams-asset-abc", + check_id="cams-check-def", + occurrences=10, + tested_records=100, + column_name="test_column", + check_type="format", + project_id="project-123", + asset_type="column" + ) + + # Verify + assert result["issue_id"] == "issue-789" + assert result["number_of_occurrences"] == 10 + assert result["number_of_tested_records"] == 100 + + # Verify search_dq_asset was called correctly + mock_search_provider.search_dq_asset.assert_called_once_with( + native_id="cams-asset-abc/test_column", + project_id="project-123", + catalog_id=None, + asset_type="column" + ) + + # Verify get_issues (GET request) was called instead of search_dq_check + provider.session.get.assert_called_once() + call_args = provider.session.get.call_args + url = call_args[0][0] + assert "reported_for.id=dq-asset-123" in url + assert "type=format" in url + assert "include_children=false" in url + + def test_update_issue_metrics_with_check_native_id(self, provider): + """Test updating issue metrics using check_native_id instead of asset_id and check_id.""" + with patch('wxdi.dq_validator.provider.dq_search.DQSearchProvider') as mock_search_provider_class: + # Setup mock search provider + mock_search_provider = Mock() + mock_search_provider_class.return_value = mock_search_provider + + # Mock search_dq_asset response + mock_search_provider.search_dq_asset.return_value = { + "id": "dq-asset-456", + "name": "Test Asset", + "type": "column" + } + + # Mock get_issues response + mock_get_issues_response = Mock() + mock_get_issues_response.ok = True + mock_get_issues_response.text = json.dumps({ + "issues": [ + { + "id": "issue-999", + "check": { + "id": "dq-check-789", + "native_id": "asset-abc/column_name/case" + }, + "project_id": "project-456" + } + ], + "total_count": 1 + }) + provider.session.get.return_value = mock_get_issues_response + + # Mock update_issue_values response + mock_update_response = Mock() + mock_update_response.ok = True + mock_update_response.text = json.dumps({ + "issue_id": "issue-999", + "number_of_occurrences": 25, + "number_of_tested_records": 250 + }) + provider.session.patch.return_value = mock_update_response + + # Execute - using check_native_id instead of asset_id and check_id + result = provider.update_issue_metrics( + check_native_id="asset-abc/column_name/case", + occurrences=25, + tested_records=250, + column_name="column_name", + check_type="case", + project_id="project-456", + asset_type="column" + ) + + # Verify + assert result["issue_id"] == "issue-999" + assert result["number_of_occurrences"] == 25 + assert result["number_of_tested_records"] == 250 + + # Verify search_dq_asset was called with extracted asset_id + mock_search_provider.search_dq_asset.assert_called_once_with( + native_id="asset-abc/column_name", + project_id="project-456", + catalog_id=None, + asset_type="column" + ) + + def test_validate_and_resolve_ids_with_asset_and_check_ids(self, provider): + """Test _validate_and_resolve_ids with asset_id and check_id provided.""" + asset_id, check_id, check_native_id = provider._validate_and_resolve_ids( + asset_id="asset-123", + check_id="check-456", + check_native_id=None + ) + + assert asset_id == "asset-123" + assert check_id == "check-456" + assert check_native_id == "asset-123/check-456" + + def test_validate_and_resolve_ids_with_check_native_id(self, provider): + """Test _validate_and_resolve_ids with check_native_id provided.""" + asset_id, check_id, check_native_id = provider._validate_and_resolve_ids( + asset_id=None, + check_id=None, + check_native_id="asset-abc/check-def" + ) + + assert asset_id == "asset-abc" + assert check_id == "check-def" + assert check_native_id == "asset-abc/check-def" + + def test_validate_and_resolve_ids_with_check_native_id_containing_slashes(self, provider): + """Test _validate_and_resolve_ids with check_native_id containing multiple slashes.""" + asset_id, check_id, check_native_id = provider._validate_and_resolve_ids( + asset_id=None, + check_id=None, + check_native_id="asset-123/column/check-type" + ) + + assert asset_id == "asset-123" + assert check_id == "column/check-type" + assert check_native_id == "asset-123/column/check-type" + + def test_validate_and_resolve_ids_with_both_provided(self, provider): + """Test _validate_and_resolve_ids when both asset_id/check_id and check_native_id are provided.""" + # When both are provided, the original values are kept (no parsing or construction happens) + asset_id, check_id, check_native_id = provider._validate_and_resolve_ids( + asset_id="asset-111", + check_id="check-222", + check_native_id="asset-999/check-888" + ) + + # Original values should be returned as-is + assert asset_id == "asset-111" + assert check_id == "check-222" + assert check_native_id == "asset-999/check-888" + + def test_validate_and_resolve_ids_with_neither_provided(self, provider): + """Test _validate_and_resolve_ids raises error when neither IDs are provided.""" + with pytest.raises(ValueError) as exc_info: + provider._validate_and_resolve_ids( + asset_id=None, + check_id=None, + check_native_id=None + ) + + assert "Either (asset_id and check_id) or check_native_id must be provided" in str(exc_info.value) + + def test_validate_and_resolve_ids_with_only_asset_id(self, provider): + """Test _validate_and_resolve_ids raises error when only asset_id is provided.""" + with pytest.raises(ValueError) as exc_info: + provider._validate_and_resolve_ids( + asset_id="asset-123", + check_id=None, + check_native_id=None + ) + + assert "Either (asset_id and check_id) or check_native_id must be provided" in str(exc_info.value) + + def test_validate_and_resolve_ids_with_only_check_id(self, provider): + """Test _validate_and_resolve_ids raises error when only check_id is provided.""" + with pytest.raises(ValueError) as exc_info: + provider._validate_and_resolve_ids( + asset_id=None, + check_id="check-456", + check_native_id=None + ) + + assert "Either (asset_id and check_id) or check_native_id must be provided" in str(exc_info.value) + + def test_validate_and_resolve_ids_with_invalid_check_native_id_format(self, provider): + """Test _validate_and_resolve_ids raises error for invalid check_native_id format.""" + with pytest.raises(ValueError) as exc_info: + provider._validate_and_resolve_ids( + asset_id=None, + check_id=None, + check_native_id="invalid-format-no-slash" + ) + + assert "Invalid check_native_id format (missing /)" in str(exc_info.value) + assert "invalid-format-no-slash" in str(exc_info.value) + + def test_get_issues_with_catalog_id(self, provider): + """Test getting issues with catalog_id and check_id filter.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issues": [ + { + "id": "issue-1", + "check": { + "id": "check-1", + "native_id": "asset-id/065c2b72-4600-4d15-8c48-298a2abf66cd" + }, + "check_name": "Completeness Check", + "number_of_occurrences": 5, + "number_of_tested_records": 100 + }, + { + "id": "issue-2", + "check": { + "id": "check-2", + "native_id": "asset-id/other-check-id" + }, + "check_name": "Format Check", + "number_of_occurrences": 3, + "number_of_tested_records": 100 + } + ], + "total_count": 2 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_issues( + dq_asset_id="08b139ca-35a6-4b61-b87b-aa832870d89c", + check_type="completeness", + check_id="065c2b72-4600-4d15-8c48-298a2abf66cd", + catalog_id="07708fd8-8d77-4a07-a01b-0132130bce0e", + limit=20, + latest_only=True, + include_children=False, + sort_by="check_name", + sort_direction="asc" + ) + + # Verify - should return only the matching issue + assert result is not None + assert result["id"] == "issue-1" + assert result["check"]["id"] == "check-1" + + # Verify the API call + provider.session.get.assert_called_once() + call_args = provider.session.get.call_args + url = call_args[0][0] + + # Check URL contains required parameters + assert "catalog_id=07708fd8-8d77-4a07-a01b-0132130bce0e" in url + assert "reported_for.id=08b139ca-35a6-4b61-b87b-aa832870d89c" in url + assert "type=completeness" in url + assert "limit=20" in url + assert "latest_only=true" in url + assert "include_children=false" in url + assert "sort_by=check_name" in url + assert "sort_direction=asc" in url + + def test_get_issues_with_project_id(self, provider): + """Test getting issues with project_id and check_id filter.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issues": [ + { + "id": "issue-3", + "check": { + "id": "check-3", + "native_id": "asset-id/test-check-id" + }, + "check_name": "Range Check", + "number_of_occurrences": 10, + "number_of_tested_records": 200 + } + ], + "total_count": 1 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_issues( + dq_asset_id="asset-id-123", + check_type="range", + check_id="test-check-id", + project_id="project-456" + ) + + # Verify - should return the matching issue + assert result is not None + assert result["id"] == "issue-3" + + # Verify the API call + call_args = provider.session.get.call_args + url = call_args[0][0] + + # Check URL contains project_id + assert "project_id=project-456" in url + assert "catalog_id" not in url + assert "include_children=false" in url + + def test_get_issues_missing_both_ids(self, provider): + """Test get_issues without project_id or catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.get_issues( + dq_asset_id="asset-id", + check_type="completeness", + check_id="test-check-id" + ) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_get_issues_both_ids_provided(self, provider): + """Test get_issues with both project_id and catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.get_issues( + dq_asset_id="asset-id", + check_type="completeness", + check_id="test-check-id", + project_id="project-123", + catalog_id="catalog-456" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + def test_get_issues_failure(self, provider): + """Test failed retrieval of issues.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 404 + mock_response.text = "Not found" + provider.session.get.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.get_issues( + dq_asset_id="invalid-asset", + check_type="completeness", + check_id="test-check-id", + project_id="invalid-project" + ) + + assert "Failed to get issues" in str(exc_info.value) + assert "404" in str(exc_info.value) + + def test_get_issues_with_minimal_params(self, provider): + """Test getting issues with minimal parameters (using defaults).""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issues": [ + { + "id": "issue-4", + "check": { + "id": "check-4", + "native_id": "asset-id/minimal-check-id" + } + } + ], + "total_count": 1 + }) + provider.session.get.return_value = mock_response + + # Execute with minimal params + result = provider.get_issues( + dq_asset_id="asset-id", + check_type="datatype", + check_id="minimal-check-id", + catalog_id="catalog-123" + ) + + # Verify - should return the matching issue + assert result is not None + assert result["id"] == "issue-4" + + # Verify the API call includes default values + call_args = provider.session.get.call_args + url = call_args[0][0] + + assert "limit=20" in url + assert "latest_only=true" in url + assert "include_children=false" in url + assert "sort_by=check_name" in url + assert "sort_direction=asc" in url + + def test_get_issues_no_match_found(self, provider): + """Test get_issues when no matching check_id is found.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issues": [ + { + "id": "issue-5", + "check": { + "id": "check-5", + "native_id": "asset-id/different-check-id" + } + } + ], + "total_count": 1 + }) + provider.session.get.return_value = mock_response + + # Execute with a check_id that doesn't match + result = provider.get_issues( + dq_asset_id="asset-id", + check_type="format", + check_id="non-existent-check-id", + catalog_id="catalog-123" + ) + + # Verify - should return None when no match found + assert result is None + + def test_create_issue_with_project_id(self, provider): + """Test creating an issue with project_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "id": "046605b5-48d9-489e-b846-8ef96a7a1aba", + "check": { + "id": "6be18374-573a-4cf8-8ab7-e428506e428b" + }, + "reported_for": { + "id": "894d01fd-bdfc-4a4f-b68b-62751e06e06a" + }, + "number_of_occurrences": 123, + "number_of_tested_records": 456789, + "status": "actual", + "ignored": False + }) + provider.session.post.return_value = mock_response + + # Execute + result = provider.create_issue( + dq_check_id="6be18374-573a-4cf8-8ab7-e428506e428b", + reported_for_id="894d01fd-bdfc-4a4f-b68b-62751e06e06a", + number_of_occurrences=123, + number_of_tested_records=456789, + project_id="project-123" + ) + + # Verify + assert result == "046605b5-48d9-489e-b846-8ef96a7a1aba" + + # Verify the API call + provider.session.post.assert_called_once() + call_args = provider.session.post.call_args + + # Check URL + assert "https://test-instance.com/data_quality/v4/issues" in call_args[0][0] + assert "project_id=project-123" in call_args[0][0] + + # Check payload + payload = json.loads(call_args[1]["data"]) + assert payload["check"]["id"] == "6be18374-573a-4cf8-8ab7-e428506e428b" + assert payload["reported_for"]["id"] == "894d01fd-bdfc-4a4f-b68b-62751e06e06a" + assert payload["number_of_occurrences"] == 123 + assert payload["number_of_tested_records"] == 456789 + + def test_create_issue_missing_both_ids(self, provider): + """Test creating an issue without project_id or catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.create_issue( + dq_check_id="check-123", + reported_for_id="asset-456", + number_of_occurrences=10, + number_of_tested_records=100 + ) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_create_issue_failure(self, provider): + """Test failed creation of issue.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 400 + mock_response.text = "Bad request" + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.create_issue( + dq_check_id="invalid-check", + reported_for_id="invalid-asset", + number_of_occurrences=10, + number_of_tested_records=100, + project_id="project-123" + ) + + assert "Failed to create issue" in str(exc_info.value) + assert "400" in str(exc_info.value) + + def test_create_issues_bulk_with_project_id(self, provider): + """Test creating multiple issues in bulk with project_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issues": [ + { + "id": "issue-bulk-1", + "check": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/Validity", + "type": "format" + }, + "status": "aggregation" + }, + { + "id": "issue-bulk-2", + "check": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/sample3", + "type": "format" + }, + "status": "actual" + } + ], + "assets": [ + { + "id": "asset-bulk-1", + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f", + "type": "data_asset" + }, + { + "id": "asset-bulk-2", + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/NAME", + "type": "column" + } + ] + }) + provider.session.post.return_value = mock_response + + # Prepare bulk payload + bulk_payload = { + "issues": [ + { + "check": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/Validity", + "type": "format" + }, + "reported_for": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f", + "type": "data_asset" + }, + "number_of_occurrences": 200, + "number_of_tested_records": 1000, + "status": "aggregation", + "ignored": False + }, + { + "check": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/sample3", + "type": "format" + }, + "reported_for": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/NAME", + "type": "column" + }, + "number_of_occurrences": 200, + "number_of_tested_records": 1000, + "status": "actual", + "ignored": False + } + ], + "assets": [ + { + "name": "ACCOUNT_HOLDERS.csv", + "type": "data_asset", + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f", + "weight": 1 + }, + { + "name": "NAME", + "type": "column", + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/NAME", + "parent": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f", + "type": "data_asset" + }, + "weight": 1 + } + ], + "existing_checks": [ + { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/Validity", + "type": "format" + }, + { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/sample3", + "type": "format" + } + ] + } + + # Execute + result = provider.create_issues_bulk( + payload=bulk_payload, + project_id="project-123", + incremental_reporting=False, + refresh_assets=False + ) + + # Verify + assert "issues" in result + assert len(result["issues"]) == 2 + assert result["issues"][0]["id"] == "issue-bulk-1" + assert result["issues"][1]["id"] == "issue-bulk-2" + + # Verify the API call + provider.session.post.assert_called_once() + call_args = provider.session.post.call_args + + # Check URL + assert "https://test-instance.com/data_quality/v4/create_issues" in call_args[0][0] + assert "project_id=project-123" in call_args[0][0] + assert "incremental_reporting=false" in call_args[0][0] + assert "refresh_assets=false" in call_args[0][0] + + # Check payload + payload = json.loads(call_args[1]["data"]) + assert len(payload["issues"]) == 2 + assert len(payload["assets"]) == 2 + assert len(payload["existing_checks"]) == 2 + + def test_create_issues_bulk_with_catalog_id_and_incremental_reporting(self, provider): + """Test creating bulk issues with catalog_id and incremental_reporting=True.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issues": [ + {"id": "issue-1", "status": "aggregation"}, + {"id": "issue-2", "status": "actual"} + ] + }) + provider.session.post.return_value = mock_response + + # Minimal bulk payload + bulk_payload = { + "issues": [ + { + "check": {"native_id": "asset/check1", "type": "format"}, + "reported_for": {"native_id": "asset", "type": "data_asset"}, + "number_of_occurrences": 10, + "number_of_tested_records": 100, + "status": "aggregation", + "ignored": False + } + ], + "assets": [ + {"name": "Test Asset", "type": "data_asset", "native_id": "asset", "weight": 1} + ], + "existing_checks": [ + {"native_id": "asset/check1", "type": "format"} + ] + } + + # Execute + result = provider.create_issues_bulk( + payload=bulk_payload, + catalog_id="catalog-456", + incremental_reporting=True, + refresh_assets=True + ) + + # Verify + assert "issues" in result + assert len(result["issues"]) == 2 + + # Verify the API call + call_args = provider.session.post.call_args + url = call_args[0][0] + + # Check URL contains catalog_id and boolean params + assert "catalog_id=catalog-456" in url + assert "project_id" not in url + assert "incremental_reporting=true" in url + assert "refresh_assets=true" in url + + def test_create_issues_bulk_missing_both_ids(self, provider): + """Test creating bulk issues without project_id or catalog_id.""" + bulk_payload = { + "issues": [], + "assets": [], + "existing_checks": [] + } + + with pytest.raises(ValueError) as exc_info: + provider.create_issues_bulk(payload=bulk_payload) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_create_issues_bulk_both_ids_provided(self, provider): + """Test creating bulk issues with both project_id and catalog_id.""" + bulk_payload = { + "issues": [], + "assets": [], + "existing_checks": [] + } + + with pytest.raises(ValueError) as exc_info: + provider.create_issues_bulk( + payload=bulk_payload, + project_id="project-123", + catalog_id="catalog-456" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + def test_create_issues_bulk_failure(self, provider): + """Test failed bulk issue creation.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 400 + mock_response.text = "Invalid payload" + provider.session.post.return_value = mock_response + + bulk_payload = { + "issues": [], + "assets": [], + "existing_checks": [] + } + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.create_issues_bulk( + payload=bulk_payload, + project_id="project-123" + ) + + assert "Failed to create issues in bulk" in str(exc_info.value) + assert "400" in str(exc_info.value) + assert "Invalid payload" in str(exc_info.value) \ No newline at end of file diff --git a/tests/src/dq_validator/provider/test_thread_safety.py b/tests/src/dq_validator/provider/test_thread_safety.py new file mode 100644 index 0000000..1f9ae19 --- /dev/null +++ b/tests/src/dq_validator/provider/test_thread_safety.py @@ -0,0 +1,129 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Tests for thread safety of provider classes +""" + +import pytest +import threading +from unittest.mock import Mock, patch +from wxdi.dq_validator.provider import ProviderConfig, DQSearchProvider, IssuesProvider + + +class TestThreadSafety: + """Test suite for thread safety of provider classes.""" + + @pytest.fixture + def config(self): + """Create a test configuration.""" + return ProviderConfig( + url="https://test-instance.com", + auth_token="Bearer test-token" + ) + + def test_dq_search_provider_thread_local_sessions(self, config): + """Test that each thread gets its own session in DQSearchProvider.""" + with patch('wxdi.dq_validator.provider.base_provider.Session') as mock_session_class: + provider = DQSearchProvider(config) + + # Track session instances created + sessions_created = [] + + def mock_session_factory(): + session = Mock() + sessions_created.append(session) + return session + + mock_session_class.side_effect = mock_session_factory + + # Access session from main thread + main_session = provider.session + assert len(sessions_created) == 1 + + # Access session from another thread + thread_sessions = [] + + def access_session(): + thread_sessions.append(provider.session) + + thread = threading.Thread(target=access_session) + thread.start() + thread.join() + + # Should have created a second session for the new thread + assert len(sessions_created) == 2 + assert main_session is sessions_created[0] + assert thread_sessions[0] is sessions_created[1] + assert main_session is not thread_sessions[0] + + def test_issues_provider_thread_local_sessions(self, config): + """Test that each thread gets its own session in IssuesProvider.""" + with patch('wxdi.dq_validator.provider.base_provider.Session') as mock_session_class: + provider = IssuesProvider(config) + + # Track session instances created + sessions_created = [] + + def mock_session_factory(): + session = Mock() + sessions_created.append(session) + return session + + mock_session_class.side_effect = mock_session_factory + + # Access session from main thread + main_session = provider.session + assert len(sessions_created) == 1 + + # Access session from another thread + thread_sessions = [] + + def access_session(): + thread_sessions.append(provider.session) + + thread = threading.Thread(target=access_session) + thread.start() + thread.join() + + # Should have created a second session for the new thread + assert len(sessions_created) == 2 + assert main_session is sessions_created[0] + assert thread_sessions[0] is sessions_created[1] + assert main_session is not thread_sessions[0] + + def test_dq_search_provider_session_reuse_within_thread(self, config): + """Test that session is reused within the same thread.""" + with patch('wxdi.dq_validator.provider.base_provider.Session') as mock_session_class: + provider = DQSearchProvider(config) + + sessions_created = [] + + def mock_session_factory(): + session = Mock() + sessions_created.append(session) + return session + + mock_session_class.side_effect = mock_session_factory + + # Access session multiple times in the same thread + session1 = provider.session + session2 = provider.session + session3 = provider.session + + # Should only create one session + assert len(sessions_created) == 1 + assert session1 is session2 + assert session2 is session3 \ No newline at end of file diff --git a/tests/src/dq_validator/test_case_check.py b/tests/src/dq_validator/test_case_check.py new file mode 100644 index 0000000..94cf648 --- /dev/null +++ b/tests/src/dq_validator/test_case_check.py @@ -0,0 +1,258 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for CaseCheck +""" + +import pytest +from datetime import date +from wxdi.dq_validator.checks.case_check import CaseCheck, ColumnCaseEnum +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + + +class TestCaseCheckInitialization: + """Test CaseCheck initialization and parameter validation""" + + def test_init_with_no_parameter(self): + """Test initialization without a paramater defaults to CaseCheckEnum : ANY_CASE """ + check = CaseCheck() + assert check.case_type == ColumnCaseEnum.ANY_CASE + + def test_init_with_parameter(self): + """Test initialization with a specific case type""" + check = CaseCheck(case_type=ColumnCaseEnum.UPPER_CASE) + assert check.case_type == ColumnCaseEnum.UPPER_CASE + + def test_init_with_different_datatype_raises_error(self): + """Test initialization with a different datatype raises ValueError""" + with pytest.raises(ValueError) as exc_info: + CaseCheck("AnyCase") + assert "case_type must be a ColumnCaseEnum" in str(exc_info.value) + + def test_get_check_name(self): + """Test get_check_name returns correct name""" + check = CaseCheck() + assert check.get_check_name() == "case_check" + + def test_get_dimension(self): + """Test get_dimension returns correct dimension""" + check = CaseCheck() + assert check.get_dimension() == DataQualityDimension.CONSISTENCY + + def test_set_dimension(self): + """Test set_dimension changes the dimension""" + check = CaseCheck() + assert check.get_dimension() == DataQualityDimension.CONSISTENCY + + check.set_dimension(DataQualityDimension.VALIDITY) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + +class TestCaseCheckHelperMethods: + """Test helper methods for word and sentence delimiters""" + + def test_is_word_delimiter(self): + """Test detection of word delimiters (non-alphabetic characters)""" + check = CaseCheck() + assert check._is_word_delimiter(" ") is True + assert check._is_word_delimiter("-") is True + assert check._is_word_delimiter("_") is True + assert check._is_word_delimiter("1") is True + assert check._is_word_delimiter("'") is True + assert check._is_word_delimiter(",") is True + + def test_not_word_delimiter(self): + """Test detection of word delimiters fails for alphabetic characters""" + check = CaseCheck() + assert check._is_word_delimiter("a") is False + assert check._is_word_delimiter("Z") is False + + def test_is_sentence_delimiter(self): + """Test detection of sentence boundaries (. ! ?) """ + check = CaseCheck() + assert check._is_sentence_delimiter(".") is True + assert check._is_sentence_delimiter("!") is True + assert check._is_sentence_delimiter("?") is True + + def test_not_sentence_delimiter(self): + """Test detection of sentence boundaries (. ! ?) fails """ + check = CaseCheck() + assert check._is_sentence_delimiter(" ") is False + assert check._is_sentence_delimiter(",") is False + assert check._is_sentence_delimiter("a") is False + assert check._is_sentence_delimiter("\n") is False + + +class TestCaseCheckValidation: + """Test validation logic for various case scenarios""" + + def test_any_case_passes(self): + """Test AnyCase accepts any string format""" + check = CaseCheck(ColumnCaseEnum.ANY_CASE) + context = {'column_name': 'description'} + result = check.validate("anything GOES here 123!", context) + assert result is None + + def test_upper_case_passes(self): + """Test UpperCase validation success""" + check = CaseCheck(ColumnCaseEnum.UPPER_CASE) + context = {'column_name': 'country_code'} + result = check.validate("USA IS IN WEST", context) + assert result is None + + def test_upper_case_fails(self): + """Test UpperCase validation failure""" + check = CaseCheck(ColumnCaseEnum.UPPER_CASE) + context = {'column_name': 'country_code'} + result = check.validate("USA is in west", context) + assert result is not None + assert "country_code does not follow UpperCase" in result.message + + def test_lower_case_passes(self): + """Test LowerCase validation success""" + check = CaseCheck(ColumnCaseEnum.LOWER_CASE) + context = {'column_name': 'email'} + result = check.validate("test@example.com", context) + assert result is None + + def test_lower_case_fails(self): + """Test LowerCase validation failure""" + check = CaseCheck(ColumnCaseEnum.LOWER_CASE) + context = {'column_name': 'email'} + result = check.validate("Test@example.com", context) + assert result is not None + assert "email does not follow LowerCase" in result.message + + def test_name_case_passes(self): + """Test NameCase validation success""" + check = CaseCheck(ColumnCaseEnum.NAME_CASE) + context = {'column_name': 'full_name'} + result = check.validate("Jean-Luc O'Niel", context) + assert result is None + + def test_name_case_fails(self): + """Test NameCase validation failure""" + check = CaseCheck(ColumnCaseEnum.NAME_CASE) + context = {'column_name': 'full_name'} + result = check.validate("john Doe", context) + assert result is not None + result = check.validate("JoHn Doe", context) + assert result is not None + assert "full_name does not follow NameCase" in result.message + + def test_sentence_case_passes(self): + """Test SentenceCase validation success""" + check = CaseCheck(ColumnCaseEnum.SENTENCE_CASE) + context = {'column_name': 'comment'} + result = check.validate("This is a sentence.", context) + assert result is None + result = check.validate("First sentence. Second sentence!", context) + assert result is None + + def test_sentence_case_fails(self): + """Test SentenceCase validation failure""" + check = CaseCheck(ColumnCaseEnum.SENTENCE_CASE) + context = {'column_name': 'comment'} + result = check.validate("First sentence. second sentence.", context) + assert result is not None + assert "comment does not follow SentenceCase" in result.message + + def test_none_value_fails(self): + """Test None input returns error""" + check = CaseCheck(ColumnCaseEnum.UPPER_CASE) + context = {'column_name': 'username'} + result = check.validate(None, context) + assert result is not None + assert "username is None, cannot check the case" in result.message + + +class TestCaseCheckNonStringTypes: + """Test validation of different non-string datatypes via auto-conversion""" + + def test_int_input_passes_upper_case(self): + """Test Integer input to string passes UpperCase.""" + check = CaseCheck(ColumnCaseEnum.UPPER_CASE) + context = {'column_name': 'id_number'} + result = check.validate(100, context) + assert result is None + + def test_float_input_passes_lower_case(self): + """Test Float input to string passes LowerCase.""" + check = CaseCheck(ColumnCaseEnum.LOWER_CASE) + context = {'column_name': 'score'} + result = check.validate(99.9, context) + assert result is None + + def test_boolean_true_fails_upper_case(self): + """Test Bool input to string fails UpperCase.""" + check = CaseCheck(ColumnCaseEnum.UPPER_CASE) + context = {'column_name': 'is_active'} + result = check.validate(True, context) + assert result is not None + # Verify result.value maintains the original boolean type + assert isinstance(result.value, bool) + assert "is_active does not follow UpperCase" in result.message + + def test_boolean_false_passes_name_case(self): + """Test Bool input to string passes NameCase """ + check = CaseCheck(ColumnCaseEnum.NAME_CASE) + context = {'column_name': 'flag'} + result = check.validate(False, context) + assert result is None + + def test_date_input_passes_sentence_case(self): + """Test Date input passes SentenceCase.""" + check = CaseCheck(ColumnCaseEnum.SENTENCE_CASE) + context = {'column_name': 'created_at'} + result = check.validate(date(2024, 1, 1), context) + assert result is None + + def test_list_input_fails_lower_case(self): + """Test List input to string fails LowerCase.""" + check = CaseCheck(ColumnCaseEnum.LOWER_CASE) + context = {'column_name': 'tags'} + result = check.validate([1, "A"], context) + assert result is not None + assert "tags" in result.message + assert "does not follow LowerCase" in result.message + + def test_custom_object_conversion(self): + """Test a custom object whose str() representation fails UpperCase""" + class MockObj: + def __str__(self): + return "lower" + + check = CaseCheck(ColumnCaseEnum.UPPER_CASE) + result = check.validate(MockObj(), {'column_name': 'obj'}) + assert result is not None + assert "obj does not follow UpperCase" in result.message + + +class TestCaseCheckEdgeCases: + """Test edge cases for CaseCheck""" + + def test_empty_string_passes(self): + """Test that empty strings pass case validation """ + check = CaseCheck(ColumnCaseEnum.UPPER_CASE) + result = check.validate("", {'column_name': 'test'}) + result is None + + def test_repr(self): + """Test __repr__ output""" + check = CaseCheck(ColumnCaseEnum.NAME_CASE) + repr_str = repr(check) + assert "CaseCheck" in repr_str + assert "case_type=NameCase" in repr_str \ No newline at end of file diff --git a/tests/src/dq_validator/test_comparison_check.py b/tests/src/dq_validator/test_comparison_check.py new file mode 100644 index 0000000..526d24c --- /dev/null +++ b/tests/src/dq_validator/test_comparison_check.py @@ -0,0 +1,491 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for ComparisonCheck +""" + +import pytest +from datetime import datetime, date +from wxdi.dq_validator.checks.comparison_check import ComparisonCheck, ComparisonOperator +from wxdi.dq_validator.metadata import AssetMetadata, ColumnMetadata, DataType +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + + +class TestComparisonCheckInitialization: + """Test ComparisonCheck initialization and parameter validation""" + + def test_init_with_enum_operator_and_value(self): + """Test initialization with enum operator and target value""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=18) + assert check.operator == ComparisonOperator.GREATER_THAN + assert check.target_value == 18 + assert check.target_column is None + assert check.is_column_comparison == False + + def test_init_with_string_operator_and_value(self): + """Test initialization with string operator and target value""" + check = ComparisonCheck(operator='>', target_value=18) + assert check.operator == ComparisonOperator.GREATER_THAN + assert check.target_value == 18 + + def test_init_with_enum_operator_and_column(self): + """Test initialization with enum operator and target column""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_column='min_value') + assert check.operator == ComparisonOperator.GREATER_THAN + assert check.target_column == 'min_value' + assert check.target_value is None + assert check.is_column_comparison == True + + def test_init_all_operators_as_enum(self): + """Test initialization with all operator enums""" + operators = [ + ComparisonOperator.GREATER_THAN, + ComparisonOperator.LESS_THAN, + ComparisonOperator.GREATER_THAN_OR_EQUAL, + ComparisonOperator.LESS_THAN_OR_EQUAL, + ComparisonOperator.EQUAL, + ComparisonOperator.NOT_EQUAL + ] + for op in operators: + check = ComparisonCheck(operator=op, target_value=10) + assert check.operator == op + + def test_init_all_operators_as_string(self): + """Test initialization with all operator strings""" + operators = ['>', '<', '>=', '<=', '==', '!='] + for op in operators: + check = ComparisonCheck(operator=op, target_value=10) + assert check.operator.value == op + + def test_init_invalid_string_operator_raises_error(self): + """Test that invalid string operator raises ValueError""" + with pytest.raises(ValueError) as exc_info: + ComparisonCheck(operator='<>', target_value=10) + assert "Invalid operator '<>'" in str(exc_info.value) + + def test_init_invalid_operator_type_raises_error(self): + """Test that invalid operator type raises TypeError""" + with pytest.raises(TypeError) as exc_info: + ComparisonCheck(operator=123, target_value=10) # type: ignore[arg-type] + assert "operator must be ComparisonOperator or str" in str(exc_info.value) + + def test_init_no_target_raises_error(self): + """Test that no target raises ValueError""" + with pytest.raises(ValueError) as exc_info: + ComparisonCheck(operator=ComparisonOperator.GREATER_THAN) + assert "Either target_column or target_value must be specified" in str(exc_info.value) + + def test_init_both_targets_raises_error(self): + """Test that both targets raises ValueError""" + with pytest.raises(ValueError) as exc_info: + ComparisonCheck( + operator=ComparisonOperator.GREATER_THAN, + target_column='col1', + target_value=10 + ) + assert "Cannot specify both target_column and target_value" in str(exc_info.value) + + def test_get_check_name(self): + """Test get_check_name returns correct name""" + check = ComparisonCheck(operator='>', target_value=10) + assert check.get_check_name() == "comparison_check" + + def test_get_dimension(self): + """Test get_dimension returns correct dimension""" + check = ComparisonCheck(operator='>', target_value=10) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + def test_set_dimension(self): + """Test set_dimension changes the dimension""" + check = ComparisonCheck(operator='>', target_value=10) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + check.set_dimension(DataQualityDimension.CONSISTENCY) + assert check.get_dimension() == DataQualityDimension.CONSISTENCY + + +class TestComparisonCheckColumnToValue: + """Test ComparisonCheck with column-to-value comparisons""" + + def test_greater_than_passes(self): + """Test greater than comparison passes""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=18) + context = {'column_name': 'age'} + result = check.validate(25, context) + assert result is None + + def test_greater_than_fails(self): + """Test greater than comparison fails""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=18) + context = {'column_name': 'age'} + result = check.validate(15, context) + assert result is not None + assert "age (15) > 18 failed" in result.message + + def test_less_than_passes(self): + """Test less than comparison passes""" + check = ComparisonCheck(operator=ComparisonOperator.LESS_THAN, target_value=100) + context = {'column_name': 'score'} + result = check.validate(75, context) + assert result is None + + def test_less_than_fails(self): + """Test less than comparison fails""" + check = ComparisonCheck(operator=ComparisonOperator.LESS_THAN, target_value=100) + context = {'column_name': 'score'} + result = check.validate(150, context) + assert result is not None + assert "score (150) < 100 failed" in result.message + + def test_greater_than_or_equal_passes_equal(self): + """Test >= passes when equal""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN_OR_EQUAL, target_value=18) + context = {'column_name': 'age'} + result = check.validate(18, context) + assert result is None + + def test_greater_than_or_equal_passes_greater(self): + """Test >= passes when greater""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN_OR_EQUAL, target_value=18) + context = {'column_name': 'age'} + result = check.validate(25, context) + assert result is None + + def test_greater_than_or_equal_fails(self): + """Test >= fails when less""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN_OR_EQUAL, target_value=18) + context = {'column_name': 'age'} + result = check.validate(17, context) + assert result is not None + assert "age (17) >= 18 failed" in result.message + + def test_less_than_or_equal_passes_equal(self): + """Test <= passes when equal""" + check = ComparisonCheck(operator=ComparisonOperator.LESS_THAN_OR_EQUAL, target_value=100) + context = {'column_name': 'discount'} + result = check.validate(100, context) + assert result is None + + def test_less_than_or_equal_passes_less(self): + """Test <= passes when less""" + check = ComparisonCheck(operator=ComparisonOperator.LESS_THAN_OR_EQUAL, target_value=100) + context = {'column_name': 'discount'} + result = check.validate(75, context) + assert result is None + + def test_less_than_or_equal_fails(self): + """Test <= fails when greater""" + check = ComparisonCheck(operator=ComparisonOperator.LESS_THAN_OR_EQUAL, target_value=100) + context = {'column_name': 'discount'} + result = check.validate(150, context) + assert result is not None + assert "discount (150) <= 100 failed" in result.message + + def test_equal_passes(self): + """Test == passes when equal""" + check = ComparisonCheck(operator=ComparisonOperator.EQUAL, target_value=200) + context = {'column_name': 'status_code'} + result = check.validate(200, context) + assert result is None + + def test_equal_fails(self): + """Test == fails when not equal""" + check = ComparisonCheck(operator=ComparisonOperator.EQUAL, target_value=200) + context = {'column_name': 'status_code'} + result = check.validate(404, context) + assert result is not None + assert "status_code (404) == 200 failed" in result.message + + def test_not_equal_passes(self): + """Test != passes when not equal""" + check = ComparisonCheck(operator=ComparisonOperator.NOT_EQUAL, target_value=0) + context = {'column_name': 'amount'} + result = check.validate(100, context) + assert result is None + + def test_not_equal_fails(self): + """Test != fails when equal""" + check = ComparisonCheck(operator=ComparisonOperator.NOT_EQUAL, target_value=0) + context = {'column_name': 'amount'} + result = check.validate(0, context) + assert result is not None + assert "amount (0) != 0 failed" in result.message + + +class TestComparisonCheckColumnToColumn: + """Test ComparisonCheck with column-to-column comparisons""" + + def setup_method(self): + """Set up metadata for column-to-column tests""" + self.metadata = AssetMetadata( + table_name='test', + columns=[ + ColumnMetadata('salary', DataType.DECIMAL), + ColumnMetadata('min_salary', DataType.DECIMAL) + ] + ) + + def test_column_to_column_greater_than_passes(self): + """Test column-to-column > passes""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_column='min_salary') + record = [60000.00, 50000.00] + context = {'column_name': 'salary', 'record': record, 'metadata': self.metadata} + result = check.validate(60000.00, context) + assert result is None + + def test_column_to_column_greater_than_fails(self): + """Test column-to-column > fails""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_column='min_salary') + record = [45000.00, 50000.00] + context = {'column_name': 'salary', 'record': record, 'metadata': self.metadata} + result = check.validate(45000.00, context) + assert result is not None + assert "salary (45000.0) > min_salary (50000.0) failed" in result.message + + def test_column_to_column_target_not_found(self): + """Test error when target column not found""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_column='nonexistent') + record = [60000.00, 50000.00] + context = {'column_name': 'salary', 'record': record, 'metadata': self.metadata} + result = check.validate(60000.00, context) + assert result is not None + assert "Target column 'nonexistent' not found" in result.message + + def test_column_to_column_target_is_none(self): + """Test error when target column value is None""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_column='min_salary') + record = [60000.00, None] + context = {'column_name': 'salary', 'record': record, 'metadata': self.metadata} + result = check.validate(60000.00, context) + assert result is not None + assert "min_salary=None" in result.message + + +class TestComparisonCheckStringComparison: + """Test ComparisonCheck with string values""" + + def test_string_greater_than_passes(self): + """Test string lexicographic > passes""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value='apple') + context = {'column_name': 'fruit'} + result = check.validate('banana', context) + assert result is None # 'banana' > 'apple' + + def test_string_less_than_passes(self): + """Test string lexicographic < passes""" + check = ComparisonCheck(operator=ComparisonOperator.LESS_THAN, target_value='zebra') + context = {'column_name': 'word'} + result = check.validate('apple', context) + assert result is None # 'apple' < 'zebra' + + def test_string_equal_passes(self): + """Test string == passes""" + check = ComparisonCheck(operator=ComparisonOperator.EQUAL, target_value='test') + context = {'column_name': 'value'} + result = check.validate('test', context) + assert result is None + + +class TestComparisonCheckDateComparison: + """Test ComparisonCheck with date and datetime values""" + + def test_date_greater_than_passes(self): + """Test date > passes""" + check = ComparisonCheck( + operator=ComparisonOperator.GREATER_THAN, + target_value=date(2024, 1, 1) + ) + context = {'column_name': 'end_date'} + result = check.validate(date(2024, 12, 31), context) + assert result is None + + def test_date_less_than_passes(self): + """Test date < passes""" + check = ComparisonCheck( + operator=ComparisonOperator.LESS_THAN, + target_value=date(2024, 12, 31) + ) + context = {'column_name': 'start_date'} + result = check.validate(date(2024, 1, 1), context) + assert result is None + + def test_datetime_greater_than_passes(self): + """Test datetime > passes""" + check = ComparisonCheck( + operator=ComparisonOperator.GREATER_THAN, + target_value=datetime(2024, 1, 1, 0, 0, 0) + ) + context = {'column_name': 'timestamp'} + result = check.validate(datetime(2024, 6, 15, 12, 0, 0), context) + assert result is None + + def test_datetime_less_than_or_equal_passes(self): + """Test datetime <= passes""" + check = ComparisonCheck( + operator=ComparisonOperator.LESS_THAN_OR_EQUAL, + target_value=datetime(2024, 12, 31, 23, 59, 59) + ) + context = {'column_name': 'timestamp'} + result = check.validate(datetime(2024, 6, 15, 12, 0, 0), context) + assert result is None + + +class TestComparisonCheckBooleanComparison: + """Test ComparisonCheck with boolean values""" + + def test_boolean_equal_true_passes(self): + """Test boolean == True passes""" + check = ComparisonCheck(operator=ComparisonOperator.EQUAL, target_value=True) + context = {'column_name': 'is_active'} + result = check.validate(True, context) + assert result is None + + def test_boolean_equal_false_passes(self): + """Test boolean == False passes""" + check = ComparisonCheck(operator=ComparisonOperator.EQUAL, target_value=False) + context = {'column_name': 'is_active'} + result = check.validate(False, context) + assert result is None + + def test_boolean_not_equal_passes(self): + """Test boolean != passes""" + check = ComparisonCheck(operator=ComparisonOperator.NOT_EQUAL, target_value=False) + context = {'column_name': 'is_active'} + result = check.validate(True, context) + assert result is None + + def test_boolean_greater_than_passes(self): + """Test boolean > passes (True > False in Python)""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=False) + context = {'column_name': 'flag'} + result = check.validate(True, context) + assert result is None + + +class TestComparisonCheckNoneHandling: + """Test ComparisonCheck with None values""" + + def test_none_value_fails(self): + """Test None value returns error""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=50000) + context = {'column_name': 'salary'} + result = check.validate(None, context) + assert result is not None + assert "is None, cannot perform comparison" in result.message + + +class TestComparisonCheckTypeMismatch: + """Test ComparisonCheck with type mismatches""" + + def test_type_mismatch_string_vs_int_fails(self): + """Test type mismatch between string and int fails""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=50000) + context = {'column_name': 'salary'} + result = check.validate('60000', context) + assert result is not None + assert "Type mismatch" in result.message + assert "str" in result.message + assert "int" in result.message + + def test_type_mismatch_int_vs_float_fails(self): + """Test type mismatch between int and float fails""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=50000) + context = {'column_name': 'value'} + result = check.validate(60000.0, context) + assert result is not None + assert "Type mismatch" in result.message + + def test_type_mismatch_column_to_column(self): + """Test type mismatch in column-to-column comparison""" + metadata = AssetMetadata( + table_name='test', + columns=[ + ColumnMetadata('value1', DataType.STRING), + ColumnMetadata('value2', DataType.INTEGER) + ] + ) + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_column='value2') + record = ['100', 50] + context = {'column_name': 'value1', 'record': record, 'metadata': metadata} + result = check.validate('100', context) + assert result is not None + assert "Type mismatch" in result.message + + +class TestComparisonCheckFloatComparison: + """Test ComparisonCheck with float values""" + + def test_float_greater_than_passes(self): + """Test float > passes""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=50.5) + context = {'column_name': 'amount'} + result = check.validate(75.3, context) + assert result is None + + def test_float_less_than_passes(self): + """Test float < passes""" + check = ComparisonCheck(operator=ComparisonOperator.LESS_THAN, target_value=100.0) + context = {'column_name': 'amount'} + result = check.validate(99.99, context) + assert result is None + + def test_float_equal_passes(self): + """Test float == passes""" + check = ComparisonCheck(operator=ComparisonOperator.EQUAL, target_value=3.14) + context = {'column_name': 'pi'} + result = check.validate(3.14, context) + assert result is None + + +class TestComparisonCheckEdgeCases: + """Test ComparisonCheck edge cases""" + + def test_zero_comparison(self): + """Test comparison with zero""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=0) + context = {'column_name': 'value'} + result = check.validate(1, context) + assert result is None + + def test_negative_numbers(self): + """Test comparison with negative numbers""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=-10) + context = {'column_name': 'temperature'} + result = check.validate(-5, context) + assert result is None + + def test_very_large_numbers(self): + """Test comparison with very large numbers""" + check = ComparisonCheck(operator=ComparisonOperator.LESS_THAN, target_value=10**100) + context = {'column_name': 'big_number'} + result = check.validate(10**50, context) + assert result is None + + def test_repr(self): + """Test __repr__ method""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=10) + repr_str = repr(check) + assert "ComparisonCheck" in repr_str + assert ">" in repr_str + assert "10" in repr_str + + def test_repr_with_column(self): + """Test __repr__ method with target column""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_column='min_value') + repr_str = repr(check) + assert "ComparisonCheck" in repr_str + assert "min_value" in repr_str + diff --git a/tests/src/dq_validator/test_completeness_check.py b/tests/src/dq_validator/test_completeness_check.py new file mode 100644 index 0000000..83dfe19 --- /dev/null +++ b/tests/src/dq_validator/test_completeness_check.py @@ -0,0 +1,131 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +import pytest +from wxdi.dq_validator.checks.completeness_check import CompletenessCheck +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + +class TestCompletenessCheckInitialization: + """Comprehensive tests for the CompletenessCheck class""" + + def test_init_without_missing_values_allowed(self): + """Test default initialization (missing values should NOT be allowed)""" + check = CompletenessCheck() + assert check.missing_values_allowed is False + + def test_init_with_missing_values_allowed(self): + """Test initialization with missing_values_allowed explicitly""" + check = CompletenessCheck(missing_values_allowed=True) + assert check.missing_values_allowed is True + + def test_get_check_name(self): + """Ensure check name is correct""" + check = CompletenessCheck() + assert check.get_check_name() == "completeness_check" + + def test_get_dimension(self): + """Test get_dimension returns correct dimension""" + check = CompletenessCheck() + assert check.get_dimension() == DataQualityDimension.COMPLETENESS + + def test_set_dimension(self): + """Test set_dimension changes the dimension""" + check = CompletenessCheck() + assert check.get_dimension() == DataQualityDimension.COMPLETENESS + + check.set_dimension(DataQualityDimension.VALIDITY) + assert check.get_dimension() == DataQualityDimension.VALIDITY + +class TestValidationMissingValuesNotAllowed: + """ Test completeness check for various value inputs""" + + def test_validate_string_passes(self): + """Test non-empty string should pass""" + check = CompletenessCheck(missing_values_allowed=False) + result = check.validate("Hello", {"column_name": "name"}) + assert result is None + + def test_validate_numerical_passes(self): + """Test numerical non-empty values should pass""" + check = CompletenessCheck(missing_values_allowed=False) + result = check.validate(0, {"column_name": "score"}) + assert result is None + + def test_validate_boolean_passes(self): + """Booleans should pass""" + check = CompletenessCheck(missing_values_allowed=False) + result = check.validate(False, {"column_name": "is_valid"}) + assert result is None + + def test_validate_none_fails(self): + """None should fail when not allowed""" + check = CompletenessCheck(missing_values_allowed=False) + result = check.validate(None, {"column_name": "user_id"}) + assert result is not None + assert "user_id is missing" in result.message + assert result.value is None + + def test_validate_empty_string_fails(self): + """Purely empty string should fail""" + check = CompletenessCheck(missing_values_allowed=False) + result = check.validate("", {"column_name": "email"}) + assert result is not None + + def test_validate_white_space_string_passes(self): + """Strings with only spaces should fail""" + check = CompletenessCheck(missing_values_allowed=False) + result = check.validate(" ", {"column_name": "comments"}) + assert result is None + + +class TestValidationMissingValuesAllowed: + + def test_missing_allowed_string_passes(self): + """Test non-empty string should pass""" + check = CompletenessCheck(missing_values_allowed=True) + result = check.validate("Hello", {"column_name": "name"}) + assert result is None + + def test_missing_allowed_numerical_passes(self): + """Test numerical non-empty values should pass""" + check = CompletenessCheck(missing_values_allowed=True) + result = check.validate(0, {"column_name": "score"}) + assert result is None + + def test_missing_allowed_boolean_passes(self): + """Booleans should pass""" + check = CompletenessCheck(missing_values_allowed=True) + result = check.validate(False, {"column_name": "is_valid"}) + assert result is None + + def test_missing_allowed_none_passes(self): + """None should pass when missing_values_allowed is True""" + check = CompletenessCheck(missing_values_allowed=True) + result = check.validate(None, {"column_name": "optional_field"}) + assert result is None + + def test_missing_allowed_whitespace_passes(self): + """Whitespace strings should pass when missing_values_allowed""" + check = CompletenessCheck(missing_values_allowed=True) + result = check.validate(" ", {"column_name": "notes"}) + assert result is None + + +class TestValidateEdgeCases: + + def test_repr(self): + """Test the string representation""" + check = CompletenessCheck(missing_values_allowed=True) + assert "missing_values_allowed=True" in repr(check) \ No newline at end of file diff --git a/tests/src/dq_validator/test_data_asset_model.py b/tests/src/dq_validator/test_data_asset_model.py new file mode 100644 index 0000000..fd247a7 --- /dev/null +++ b/tests/src/dq_validator/test_data_asset_model.py @@ -0,0 +1,187 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Test suite for DataAsset model +""" + +import json +import pytest +from pathlib import Path +from wxdi.dq_validator.provider.data_asset_model import DataAsset + + +class TestDataAssetModel: + """Test cases for DataAsset Pydantic model""" + + @pytest.fixture + def data_asset_json(self): + """Load the data asset response JSON""" + json_path = Path(__file__).parent.parent.parent / "data" / "data_asset_response.json" + with open(json_path, "r") as f: + return json.load(f) + + def test_parse_data_asset_response(self, data_asset_json): + """Test parsing the complete data asset response""" + data_asset = DataAsset.from_dict(data_asset_json) + + assert data_asset is not None + assert isinstance(data_asset, DataAsset) + + def test_metadata_fields(self, data_asset_json): + """Test metadata fields are correctly parsed""" + data_asset = DataAsset.from_dict(data_asset_json) + + assert data_asset.metadata.asset_id == "6862f3ba-81f5-4122-8286-62bb4c5d6543" + assert data_asset.metadata.name == "DEPARTMENT" + assert data_asset.metadata.asset_type == "data_asset" + assert data_asset.metadata.project_id == "72d21c1d-499b-4784-a3c7-6f84507f9a20" + + def test_columns_parsed(self, data_asset_json): + """Test that columns are correctly parsed""" + data_asset = DataAsset.from_dict(data_asset_json) + + columns = data_asset.entity.data_asset.columns + assert len(columns) == 5 + + # Check first column + assert columns[0].name == "DEPTNO" + assert columns[0].type.type == "char" + assert columns[0].type.length == 3 + assert columns[0].type.nullable is False + + def test_column_info_parsed(self, data_asset_json): + """Test that column_info is correctly parsed""" + data_asset = DataAsset.from_dict(data_asset_json) + + column_info = data_asset.entity.column_info + assert "DEPTNO" in column_info + assert "MGRNO" in column_info + assert "LOCATION" in column_info + + def test_column_checks_parsed(self, data_asset_json): + """Test that column checks are correctly parsed""" + data_asset = DataAsset.from_dict(data_asset_json) + + # DEPTNO has 4 checks + deptno_info = data_asset.entity.column_info["DEPTNO"] + assert len(deptno_info.column_checks) == 4 + + # Check types + check_types = [check.metadata.type for check in deptno_info.column_checks] + assert "format" in check_types + assert "uniqueness" in check_types + assert "completeness" in check_types + assert "data_class" in check_types + + def test_data_class_info(self, data_asset_json): + """Test data class information""" + data_asset = DataAsset.from_dict(data_asset_json) + + deptno_info = data_asset.entity.column_info["DEPTNO"] + assert deptno_info.data_class.selected_data_class.name == "ICD-10" + assert len(deptno_info.data_class.suggested_classes) == 2 + + def test_inferred_type(self, data_asset_json): + """Test inferred type information""" + data_asset = DataAsset.from_dict(data_asset_json) + + mgrno_info = data_asset.entity.column_info["MGRNO"] + assert mgrno_info.inferred_type.type == "INT8" + assert mgrno_info.inferred_type.display_name == "tinyint" + assert mgrno_info.inferred_type.precision == 3 + + def test_column_without_checks(self, data_asset_json): + """Test column without any checks (LOCATION)""" + data_asset = DataAsset.from_dict(data_asset_json) + + location_info = data_asset.entity.column_info["LOCATION"] + assert len(location_info.column_checks) == 0 + assert location_info.data_class.selected_data_class.name == "NoClassDetected" + + def test_asset_properties(self, data_asset_json): + """Test asset properties""" + data_asset = DataAsset.from_dict(data_asset_json) + + properties = data_asset.entity.data_asset.properties + assert len(properties) == 2 + + schema_prop = next(p for p in properties if p.name == "schema_name") + assert schema_prop.value == "DB2INST1" + + table_prop = next(p for p in properties if p.name == "table_name") + assert table_prop.value == "DEPARTMENT" + + def test_check_constraint_details(self, data_asset_json): + """Test detailed check constraint information""" + data_asset = DataAsset.from_dict(data_asset_json) + + # Get format check from MGRNO + mgrno_info = data_asset.entity.column_info["MGRNO"] + format_check = next( + c for c in mgrno_info.column_checks + if c.metadata.type == "format" + ) + + assert len(format_check.check) == 1 + assert format_check.check[0].name == "formats" + assert format_check.check[0].list_value == ["999999"] + + def test_completeness_check(self, data_asset_json): + """Test completeness check parsing""" + data_asset = DataAsset.from_dict(data_asset_json) + + deptno_info = data_asset.entity.column_info["DEPTNO"] + completeness_check = next( + c for c in deptno_info.column_checks + if c.metadata.type == "completeness" + ) + + assert len(completeness_check.check) == 1 + assert completeness_check.check[0].name == "missing_values_allowed" + assert completeness_check.check[0].boolean_value is False + + def test_range_check(self, data_asset_json): + """Test range check parsing""" + data_asset = DataAsset.from_dict(data_asset_json) + + mgrno_info = data_asset.entity.column_info["MGRNO"] + range_check = next( + c for c in mgrno_info.column_checks + if c.metadata.type == "range" + ) + + assert len(range_check.check) == 2 + range_type = next(c for c in range_check.check if c.name == "range_type") + assert range_type.value == "number" + + min_value = next(c for c in range_check.check if c.name == "min") + assert min_value.numeric_value == 0 + + def test_case_check(self, data_asset_json): + """Test case check parsing""" + data_asset = DataAsset.from_dict(data_asset_json) + + deptname_info = data_asset.entity.column_info["DEPTNAME"] + case_check = next( + c for c in deptname_info.column_checks + if c.metadata.type == "case" + ) + + assert len(case_check.check) == 1 + assert case_check.check[0].name == "case_type" + assert case_check.check[0].value == "UpperCase" + +# Made with Bob diff --git a/tests/src/dq_validator/test_datatype_check.py b/tests/src/dq_validator/test_datatype_check.py new file mode 100644 index 0000000..eb90200 --- /dev/null +++ b/tests/src/dq_validator/test_datatype_check.py @@ -0,0 +1,424 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for DataType Check +""" + +from wxdi.dq_validator.checks.datatype_check import DataTypeCheck +from wxdi.dq_validator.datatypes import DataType, DataTypeEnum +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + + +class TestDataTypeCheckInitialization: + """Tests for the datatype_check class initialization""" + + def test_init_with_datatype(self): + """Test initialization with a completely intitialized datatype object""" + expected = DataType(DataTypeEnum.INT32) + check = DataTypeCheck(expected) + assert check.expected_type == expected + + def test_init_without_datatype(self): + """Test initialization without a datatype object""" + check = DataTypeCheck() + assert check.expected_type == None + + def test_init_with_empty_datatype(self): + """Test initialization with an empty datatype object""" + expected = DataType() + check = DataTypeCheck(expected) + assert check.expected_type.dtype == DataTypeEnum.UNKNOWN + + def test_get_check_name(self): + """Test get_check_name returns correct name""" + check = DataTypeCheck(DataType(DataTypeEnum.STRING, length=10)) + assert check.get_check_name() == "datatype_check" + + def test_get_dimension(self): + """Test get_dimension returns correct dimension""" + check = DataTypeCheck(DataType(DataTypeEnum.STRING, length=10)) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + def test_set_dimension(self): + """Test set_dimension changes the dimension""" + check = DataTypeCheck(DataType(DataTypeEnum.STRING, length=10)) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + check.set_dimension(DataQualityDimension.CONSISTENCY) + assert check.get_dimension() == DataQualityDimension.CONSISTENCY + + +class TestDataTypeCheckValidation: + """Test validation results for empty and non-empty values""" + + def test_empty_datatype_with_value_passes(self): + """UNKNOWN of empty datatype doesnt match integer""" + expected = DataType() + check = DataTypeCheck(expected) + result = check.validate(123, {"column_name": "age"}) + assert result is not None + assert "not compatible" in result.message + + def test_matching_datatype_passes(self): + """Value matches expected type""" + expected = DataType(DataTypeEnum.INT32) + check = DataTypeCheck(expected) + result = check.validate(100, {"column_name": "count"}) + assert result is None + + def test_mismatching_datatype_fails(self): + """Value inferred as STRING but expected INT""" + expected = DataType(DataTypeEnum.INT32) + check = DataTypeCheck(expected) + result = check.validate("abc", {"column_name": "count"}) + assert result is not None + assert "not compatible" in result.message + + def test_empty_datatype_with_none_passes(self): + """None value allowed for an empty expected type""" + expected = DataType() + check = DataTypeCheck(expected) + result = check.validate(None, {"column_name": "optional"}) + assert result is None + + def test_no_datatype_with_none_passes(self): + """With no expected type any value is allowed""" + check = DataTypeCheck(None) + result = check.validate(None, {"column_name": "optional"}) + assert result is None + + def test_datatype_with_none_passes(self): + """Null value allowed regardless of datatype""" + expected = DataType(DataTypeEnum.DATE) + check = DataTypeCheck(expected) + result = check.validate(None, {"column_name": "created_at"}) + assert result is None + + +class TestIntegerCompatibility: + """Test validation results for various values against expected Integer types""" + + def test_int8_pass(self): + """Test int8 value passes""" + check = DataTypeCheck(DataType(DataTypeEnum.INT8)) + result = check.validate(127, {"column_name": "age"}) + assert result is None + + def test_int8_fail_overflow(self): + """Test int8 value fails due to overflow""" + check = DataTypeCheck(DataType(DataTypeEnum.INT8)) + result = check.validate(200, {"column_name": "age"}) + assert result is not None + assert "not compatible" in result.message + + def test_int16_accepts_int8(self): + """Test expected int16 accepts int8""" + check = DataTypeCheck(DataType(DataTypeEnum.INT16)) + result = check.validate(100, {"column_name": "score"}) + assert result is None + + def test_int32_accepts_int16(self): + """Test expected int32 accepts int16""" + check = DataTypeCheck(DataType(DataTypeEnum.INT32)) + result = check.validate(30000, {"column_name": "population"}) + assert result is None + + def test_int64_accepts_int32(self): + """Test expected int64 accepts int32""" + check = DataTypeCheck(DataType(DataTypeEnum.INT64)) + result = check.validate(2000000000, {"column_name": "id"}) + assert result is None + + def test_int32_fail_decimal(self): + """Test int32 should reject decimal values""" + check = DataTypeCheck(DataType(DataTypeEnum.INT32)) + result = check.validate(12.34, {"column_name": "price"}) + assert result is not None + assert "not compatible" in result.message + + def test_int32_fail_non_numeric_string(self): + """Test int32 should reject non-numeric strings""" + check = DataTypeCheck(DataType(DataTypeEnum.INT32)) + result = check.validate("abc", {"column_name": "age"}) + assert result is not None + assert "not compatible" in result.message + + def test_int_accepts_numeric_string(self): + """Numeric string value is parsed to compatible numerical type""" + check = DataTypeCheck(DataType(DataTypeEnum.INT32)) + result = check.validate("12345", {"column_name": "age"}) + assert result is None + + + +class TestDecimalCompatibility: + """Test validation results for various values against expected Decimal types""" + + def test_decimal_exact(self): + """Test decimal value passes""" + expected = DataType(DataTypeEnum.DECIMAL, precision=5, scale=2) + check = DataTypeCheck(expected) + result = check.validate("123.45", {"column_name": "price"}) + assert result is None + + def test_decimal_scale_too_large(self): + """Test decimal rejects scale overflow""" + expected = DataType(DataTypeEnum.DECIMAL, precision=5, scale=2) + check = DataTypeCheck(expected) + result = check.validate("12.345", {"column_name": "price"}) + assert result is not None + + def test_decimal_integer_allowed(self): + """Test decimal accepts integer""" + expected = DataType(DataTypeEnum.DECIMAL, precision=10, scale=2) + check = DataTypeCheck(expected) + result = check.validate(100, {"column_name": "amount"}) + assert result is None + + def test_decimal_precision_overflow(self): + """Test decimal rejects precision overflow""" + expected = DataType(DataTypeEnum.DECIMAL, precision=4, scale=2) + check = DataTypeCheck(expected) + result = check.validate("123.45", {"column_name": "amount"}) + assert result is not None + + def test_decimal_rejects_date(self): + """Testcdecimal should reject date values""" + expected = DataType(DataTypeEnum.DECIMAL, precision=10, scale=2) + check = DataTypeCheck(expected) + result = check.validate("2024-01-15", {"column_name": "price"}) + assert result is not None + assert "not compatible" in result.message + + +class TestStringCompatibility: + """Test validation results for various values against expected String types""" + + def test_string_within_length(self): + check = DataTypeCheck(DataType(DataTypeEnum.STRING, length=10)) + assert check.validate("hello", {"column_name": "name"}) is None + + def test_string_exact_length(self): + check = DataTypeCheck(DataType(DataTypeEnum.STRING, length=5)) + assert check.validate("hello", {"column_name": "name"}) is None + + def test_string_too_long(self): + check = DataTypeCheck(DataType(DataTypeEnum.STRING, length=5)) + result = check.validate("toolong", {"column_name": "name"}) + assert result is not None + assert "not compatible" in result.message + + def test_string_numeric_passes(self): + """Input string value can be parsed to Numerical value""" + check = DataTypeCheck(DataType(DataTypeEnum.INT64, length=10)) + result = check.validate("12345", {"column_name": "code"}) + assert result is None + + def test_string_rejects_date(self): + """String constraint should accept appropriate date values""" + check = DataTypeCheck(DataType(DataTypeEnum.STRING, length=10)) + result = check.validate("2024-01-15", {"column_name": "dob"}) + assert result is not None + assert "not compatible" in result.message + + +class TestDateTimeCompatibility: + """Test validation results for various values against expected DateTime types""" + + def test_date_string(self): + """Test date value passes""" + check = DataTypeCheck(DataType(DataTypeEnum.DATE)) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is None + + def test_invalid_date_string(self): + """Test date value fails""" + check = DataTypeCheck(DataType(DataTypeEnum.DATE)) + result = check.validate("hello", {"column_name": "dob"}) + assert result is not None + assert "not compatible" in result.message + + def test_time_string(self): + """Test time value passes""" + check = DataTypeCheck(DataType(DataTypeEnum.TIME)) + result = check.validate("14:23:01", {"column_name": "login_time"}) + assert result is None + + def test_timestamp_string(self): + """Test timestamp value passes""" + check = DataTypeCheck(DataType(DataTypeEnum.TIMESTAMP)) + result = check.validate("2024-01-10 14:23:01", {"column_name": "created_at"}) + assert result is None + + def test_timestamp_against_date(self): + """Test timestamp value fails against expected date""" + check = DataTypeCheck(DataType(DataTypeEnum.DATE)) + result = check.validate("2024-01-01 10:11:12", {"column_name": "created"}) + assert result is not None + + +class TestDateFormats: + """Test validation results for various date formats""" + + def test_date_dash_format(self): + """YYYY-MM-DD format should be detected as DATE""" + check = DataTypeCheck(DataType(DataTypeEnum.DATE)) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is None + + def test_date_slash_format(self): + """MM/YYYY format should be detected as DATE""" + check = DataTypeCheck(DataType(DataTypeEnum.DATE)) + result = check.validate("01/2024", {"column_name": "dob"}) + assert result is None + + def test_date_dot_format(self): + """DD.MM.YYYY format should be detected as DATE""" + check = DataTypeCheck(DataType(DataTypeEnum.DATE)) + result = check.validate("10.01.2024", {"column_name": "dob"}) + assert result is None + + def test_date_month_name(self): + """Mon-DD-YY format should be detected as DATE""" + check = DataTypeCheck(DataType(DataTypeEnum.DATE)) + result = check.validate("Jan-10-24", {"column_name": "dob"}) + assert result is None + + def test_date_textual(self): + """Month DD, YYYY format should be detected as DATE""" + check = DataTypeCheck(DataType(DataTypeEnum.DATE)) + result = check.validate("Jan 10, 2024", {"column_name": "dob"}) + assert result is None + + +class TestTimeFormats: + """Test validation results for various time formats""" + + def test_time_24h_seconds(self): + """24-hour HH:MM:SS format should be detected as TIME""" + check = DataTypeCheck(DataType(DataTypeEnum.TIME)) + result = check.validate("14:23:01", {"column_name": "login_time"}) + assert result is None + + def test_time_24h_no_seconds(self): + """24-hour HH:MM format should be detected as TIME""" + check = DataTypeCheck(DataType(DataTypeEnum.TIME)) + result = check.validate("09:45", {"column_name": "login_time"}) + assert result is None + + def test_time_milliseconds(self): + """Time with milliseconds should be detected as TIME""" + check = DataTypeCheck(DataType(DataTypeEnum.TIME)) + result = check.validate("14:23:01.123", {"column_name": "login_time"}) + assert result is None + + def test_time_12h_with_am_pm(self): + """12-hour time with AM/PM should be detected as TIME""" + check = DataTypeCheck(DataType(DataTypeEnum.TIME)) + result = check.validate("02:23 PM", {"column_name": "login_time"}) + assert result is None + + def test_time_12h_with_seconds(self): + """12-hour time with seconds and AM/PM should be detected as TIME""" + check = DataTypeCheck(DataType(DataTypeEnum.TIME)) + result = check.validate("02:23:01PM", {"column_name": "login_time"}) + assert result is None + + def test_time_with_timezone(self): + """Time with timezone offset should be detected as TIME""" + check = DataTypeCheck(DataType(DataTypeEnum.TIME)) + result = check.validate("14:23:01+0530", {"column_name": "login_time"}) + assert result is None + + +class TestTimeStampFormats: + """Test validation results for various timestamp formats""" + + def test_timestamp_standard(self): + """Standard YYYY-MM-DD HH:MM:SS format should be detected as TIMESTAMP""" + check = DataTypeCheck(DataType(DataTypeEnum.TIMESTAMP)) + result = check.validate("2024-01-10 14:23:01", {"column_name": "created_at"}) + assert result is None + + def test_timestamp_iso_t(self): + """ISO T-separated timestamp should be detected as TIMESTAMP""" + check = DataTypeCheck(DataType(DataTypeEnum.TIMESTAMP)) + result = check.validate("2024-01-10T14:23:01", {"column_name": "created_at"}) + assert result is None + + def test_timestamp_with_millis(self): + """Timestamp with milliseconds should be detected as TIMESTAMP""" + check = DataTypeCheck(DataType(DataTypeEnum.TIMESTAMP)) + result = check.validate("2024-01-10 14:23:01.456", {"column_name": "created_at"}) + assert result is None + + def test_timestamp_with_timezone(self): + """Timestamp with timezone offset should be detected as TIMESTAMP""" + check = DataTypeCheck(DataType(DataTypeEnum.TIMESTAMP)) + result = check.validate("2024-01-10 14:23:01+0530", {"column_name": "created_at"}) + assert result is None + + def test_timestamp_12h_clock(self): + """12-hour timestamp with AM/PM should be detected as TIMESTAMP""" + check = DataTypeCheck(DataType(DataTypeEnum.TIMESTAMP)) + result = check.validate("2024-01-10 02:23:01PM", {"column_name": "created_at"}) + assert result is None + + def test_timestamp_european(self): + """European DD-MM-YYYY HH:MM timestamp should be detected as TIMESTAMP""" + check = DataTypeCheck(DataType(DataTypeEnum.TIMESTAMP)) + result = check.validate("10-01-2024 14:23", {"column_name": "created_at"}) + assert result is None + + def test_timestamp_us_format(self): + """US MM-DD-YYYY HH:MM:SS timestamp should be detected as TIMESTAMP""" + check = DataTypeCheck(DataType(DataTypeEnum.TIMESTAMP)) + result = check.validate("01-10-2024 14:23:01", {"column_name": "created_at"}) + assert result is None + + +class TestCrossTypeFailures: + """Test failure of values against cross-types""" + + def test_string_vs_int(self): + """Test string fails against int32""" + check = DataTypeCheck(DataType(DataTypeEnum.INT32)) + result = check.validate("abc", {"column_name": "age"}) + assert result is not None + + def test_date_vs_decimal(self): + """Test date fails against decimal""" + check = DataTypeCheck(DataType(DataTypeEnum.DECIMAL, precision=5, scale=2)) + result = check.validate("2024-01-01", {"column_name": "amount"}) + assert result is not None + + +class TestFormatAndTypeIntegration: + """Test proper initialization of parameters in datatypes""" + + def test_inferred_format_is_set(self): + """Test initialization of inferred_type and inferred_format""" + check = DataTypeCheck(DataType(DataTypeEnum.DATE)) + engine = check.engine + check.validate("2024-01-10", {"column_name": "dob"}) + assert engine.inferred_type.dtype == DataTypeEnum.DATE + assert engine.inferred_format is not None + + def test_repr(self): + """Test __repr__ method""" + check = DataTypeCheck(DataType(DataTypeEnum.INT32)) + assert "datatype_check" in repr(check) + assert "int32" in repr(check) \ No newline at end of file diff --git a/tests/src/dq_validator/test_format_check.py b/tests/src/dq_validator/test_format_check.py new file mode 100644 index 0000000..76f530a --- /dev/null +++ b/tests/src/dq_validator/test_format_check.py @@ -0,0 +1,357 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for Format Check +""" + +from wxdi.dq_validator.checks.format_check import FormatCheck, FormatConstraintType +from wxdi.dq_validator.datetime_formats import DateTimeFormats +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + +class TestFormatCheckInitialization: + """Test FormatCheck initialization""" + + def test_init_valid_formats(self): + """Test initialization with ValidFormats and format set passes""" + check = FormatCheck(FormatConstraintType.ValidFormats, {DateTimeFormats.ISO_DATE}) + assert check.constraint_type == FormatConstraintType.ValidFormats + assert DateTimeFormats.ISO_DATE in check.formats + + def test_init_invalid_formats(self): + """Test initialization with InvalidFormats and format set""" + check = FormatCheck(FormatConstraintType.InvalidFormats, {"ABx"}) + assert check.constraint_type == FormatConstraintType.InvalidFormats + + def test_init_multiple_formats(self): + """Test formats correctly stores multiple formats in the set""" + formats = {DateTimeFormats.ISO_DATE, DateTimeFormats.UK_DATE, "AAA"} + check = FormatCheck(FormatConstraintType.ValidFormats, formats) + assert check.constraint_type == FormatConstraintType.ValidFormats + assert check.formats == formats + assert len(check.formats) == 3 + + def test_init_without_formats(self): + """Test initialization without format set""" + check = FormatCheck(constraint_type=FormatConstraintType.InvalidFormats) + assert check.constraint_type == FormatConstraintType.InvalidFormats + assert len(check.formats) == 0 + + def test_init_without_constraint_type(self): + """Test initialization without FormatConstraintType""" + check = FormatCheck(formats={DateTimeFormats.ISO_DATE}) + assert check.constraint_type == FormatConstraintType.ValidFormats + assert DateTimeFormats.ISO_DATE in check.formats + + def test_init_without_parameters(self): + """Test initialization without any parameters""" + check = FormatCheck() + assert check.constraint_type == FormatConstraintType.ValidFormats + assert len(check.formats) == 0 + + def test_get_check_name(self): + """Test get_check_name returns correct name""" + check = FormatCheck(FormatConstraintType.ValidFormats, set()) + assert check.get_check_name() == "format_check" + + def test_get_dimension(self): + """Test get_dimension returns correct dimension""" + check = FormatCheck(FormatConstraintType.ValidFormats, set()) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + def test_set_dimension(self): + """Test set_dimension changes the dimension""" + check = FormatCheck(FormatConstraintType.ValidFormats, set()) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + check.set_dimension(DataQualityDimension.CONSISTENCY) + assert check.get_dimension() == DataQualityDimension.CONSISTENCY + + +class TestValidFormats: + """Test ValidFormats validation where value must match one of the allowed formats""" + + def test_valid_date_format_passes(self): + """Date format in allowed list should pass""" + check = FormatCheck(FormatConstraintType.ValidFormats, {DateTimeFormats.ISO_DATE}) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is None + + def test_invalid_date_format_fails(self): + """Date not in allowed list should fail""" + check = FormatCheck(FormatConstraintType.ValidFormats, {DateTimeFormats.UK_DATE}) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is not None + assert "violates" in result.message + + def test_valid_time_passes(self): + """Time in allowed list should pass""" + check = FormatCheck(FormatConstraintType.ValidFormats, {DateTimeFormats.TIME_24H}) + result = check.validate("14:23:01", {"column_name": "login_time"}) + assert result is None + + def test_invalid_time_fails(self): + """Time not in allowed list should fail""" + check = FormatCheck(FormatConstraintType.ValidFormats, {"%hh:%nn %a"}) + result = check.validate("14:23:01", {"column_name": "login_time"}) + assert result is not None + assert "violates" in result.message + + def test_valid_timestamp_passes(self): + """Timestamp in allowed list should pass""" + check = FormatCheck(FormatConstraintType.ValidFormats, {"%yyyy-%mm-%dd %HH:%nn:%ss"}) + result = check.validate("2024-01-10 14:23:01", {"column_name": "created_at"}) + assert result is None + + +class TestComplexFormatScenarios: + """High-complexity format validation scenarios""" + + def test_complex_alphanumeric_with_symbols_valid(self): + """Mixed case, digits and symbols should match expected format""" + check = FormatCheck( + FormatConstraintType.ValidFormats, + {"Aa@99#Aa"} + ) + result = check.validate("Ab@12#Xy", {"column_name": "password"}) + assert result is None + + def test_email_like_string_valid_format(self): + """Email-like structure should match detected format""" + check = FormatCheck( + FormatConstraintType.ValidFormats, + {"aaaa@aaaa.aaa"} + ) + result = check.validate("user@mail.com", {"column_name": "email"}) + assert result is None + + def test_ideographic_common_sandwich_valid(self): + """Common script between ideographs should be treated as ideographic""" + check = FormatCheck( + FormatConstraintType.ValidFormats, + {"III"} + ) + result = check.validate("中·国", {"column_name": "country"}) + assert result is None + + def test_emoji_surrogate_valid_format(self): + """Emoji (surrogate pair) should be preserved as-is""" + check = FormatCheck( + FormatConstraintType.ValidFormats, + {"😀"} + ) + result = check.validate("😀", {"column_name": "icon"}) + assert result is None + + def test_string_longer_than_255_maps_to_na(self): + """Strings longer than 255 UTF-16 units should map to """ + long_value = "a" * 256 + check = FormatCheck( + FormatConstraintType.ValidFormats, + {""} + ) + result = check.validate(long_value, {"column_name": "payload"}) + assert result is None + + def test_numeric_value_vs_numeric_string_difference(self): + """Numeric value should not be treated the same as numeric string""" + check = FormatCheck( + FormatConstraintType.ValidFormats, + {"999"} + ) + result = check.validate(123, {"column_name": "code"}) + assert result is None + + +class TestInvalidFormats: + """Test InvalidFormats validation where value must not match any forbidden format""" + + def test_forbidden_date_fails(self): + """Forbidden date format should fail""" + check = FormatCheck(FormatConstraintType.InvalidFormats, {DateTimeFormats.ISO_DATE}) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is not None + assert "violates" in result.message + + def test_allowed_date_passes(self): + """Date not in forbidden list should pass""" + check = FormatCheck(FormatConstraintType.InvalidFormats, {DateTimeFormats.UK_DATE}) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is None + + def test_forbidden_time_fails(self): + """Forbidden time format should fail""" + check = FormatCheck(FormatConstraintType.InvalidFormats, {DateTimeFormats.TIME_24H}) + result = check.validate("14:23:01", {"column_name": "time"}) + assert result is not None + assert "violates" in result.message + + def test_allowed_time_passes(self): + """Time not in forbidden list should pass""" + check = FormatCheck(FormatConstraintType.InvalidFormats, {"%hh:%nn %a"}) + result = check.validate("14:23:01", {"column_name": "time"}) + assert result is None + + +class TestNoDetectedFormat: + """Test behavior when InferredTypeEngine cannot detect any format""" + + def test_string_with_no_format_valid_formats(self): + """Non-date string should fail ValidFormats""" + check = FormatCheck(FormatConstraintType.ValidFormats, {"%yyyy-%nn-%dd"}) + result = check.validate("hello", {"column_name": "name"}) + assert result is not None + assert "violates" in result.message + + def test_string_with_no_format_invalid_formats(self): + """Non-date string should pass InvalidFormats""" + check = FormatCheck(FormatConstraintType.InvalidFormats, {"%yyyy-%nn-%dd"}) + result = check.validate("hello", {"column_name": "name"}) + assert result is None + + +class TestEmptyAndNAFormats: + """Tests for and format handling""" + + def test_empty_string_allowed_when_empty_format_present(self): + """Empty string should map to and pass when allowed""" + check = FormatCheck(FormatConstraintType.ValidFormats, {""}) + result = check.validate("", {"column_name": "comment"}) + assert result is None + + def test_empty_string_rejected_when_empty_format_not_allowed(self): + """Empty string should fail ValidFormats when not allowed""" + check = FormatCheck(FormatConstraintType.ValidFormats, {"AAA"}) + result = check.validate("", {"column_name": "comment"}) + assert result is not None + assert "violates" in result.message + + def test_empty_string_blocked_by_invalid_formats(self): + """Empty string should fail InvalidFormats when is forbidden""" + check = FormatCheck(FormatConstraintType.InvalidFormats, {""}) + result = check.validate("", {"column_name": "comment"}) + assert result is not None + assert "violates" in result.message + + def test_long_string_maps_to_na_and_passes_when_allowed(self): + """String longer than 255 UTF-16 units should map to and pass when allowed""" + long_value = "A" * 300 + check = FormatCheck(FormatConstraintType.ValidFormats, {""}) + result = check.validate(long_value, {"column_name": "payload"}) + assert result is None + + def test_long_string_rejected_when_na_not_allowed(self): + """String mapping to should fail ValidFormats when not allowed""" + long_value = "A" * 300 + check = FormatCheck(FormatConstraintType.ValidFormats, {"AAA"}) + result = check.validate(long_value, {"column_name": "payload"}) + assert result is not None + assert "violates" in result.message + + def test_na_blocked_by_invalid_formats(self): + """ should fail InvalidFormats when explicitly forbidden""" + long_value = "A" * 300 + check = FormatCheck(FormatConstraintType.InvalidFormats, {""}) + result = check.validate(long_value, {"column_name": "payload"}) + assert result is not None + assert "violates" in result.message + + +class TestIdeographicCommonSandwich: + """Tests ideograph + COMMON + ideograph → III behavior""" + + def test_han_common_han(self): + """Han ideographs with COMMON in between should map to III""" + check = FormatCheck(FormatConstraintType.ValidFormats, {"III"}) + result = check.validate("中·国", {"column_name": "value"}) + assert result is None + + def test_hiragana_common_hiragana(self): + """Hiragana with COMMON in between should map to III""" + check = FormatCheck(FormatConstraintType.ValidFormats, {"III"}) + result = check.validate("あ・い", {"column_name": "value"}) + assert result is None + + def test_katakana_common_katakana(self): + """Katakana with COMMON in between should map to III""" + check = FormatCheck(FormatConstraintType.ValidFormats, {"III"}) + result = check.validate("カ・タ", {"column_name": "value"}) + assert result is None + + def test_hangul_common_hangul(self): + """Hangul with COMMON in between should map to III""" + check = FormatCheck(FormatConstraintType.ValidFormats, {"III"}) + result = check.validate("한·국", {"column_name": "value"}) + assert result is None + + + +class TestFormatCheckConstraintEdgeCases: + """Test FormatCheck behavior when constraint type or format sets are missing or empty""" + + def test_valid_formats_empty_rejects_date(self): + """With ValidFormats and empty set, any detected format should fail""" + check = FormatCheck(FormatConstraintType.ValidFormats, set()) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is not None + assert "violates" in result.message + + def test_invalid_formats_empty_allows_date(self): + """With InvalidFormats and empty set, any detected format should pass""" + check = FormatCheck(FormatConstraintType.InvalidFormats, set()) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is None + + def test_valid_formats_none_rejects(self): + """formats=None behaves like empty set for ValidFormats""" + check = FormatCheck(FormatConstraintType.ValidFormats, None) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is not None + assert "violates" in result.message + + def test_invalid_formats_none_allows(self): + """formats=None behaves like empty set for InvalidFormats""" + check = FormatCheck(FormatConstraintType.InvalidFormats, None) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is None + + +class TestNumericValues: + """Test behavior when numeric values are passed""" + + def test_numeric_value_valid_formats(self): + """Numbers should fail ValidFormats for the date format""" + check = FormatCheck(FormatConstraintType.ValidFormats, {DateTimeFormats.ISO_DATE}) + result = check.validate(123, {"column_name": "value"}) + assert result is not None + assert "violates" in result.message + + def test_numeric_value_invalid_formats(self): + """Numbers should pass in case of invalid date formats""" + check = FormatCheck(FormatConstraintType.InvalidFormats, {DateTimeFormats.ISO_DATE}) + result = check.validate(123, {"column_name": "value"}) + assert result is None + +class TestTimeZones: + """Test handling of timezone and Zulu (Z) timestamps""" + + def test_zulu_timestamp(self): + """Zulu timestamp should match %z-based formats""" + check = FormatCheck( + FormatConstraintType.ValidFormats, + {"%yyyy-%mm-%dd %HH:%nn:%ssZ"} + ) + result = check.validate("2024-01-10 14:23:01Z", {"column_name": "ts"}) + assert result is None diff --git a/tests/src/dq_validator/test_integration.py b/tests/src/dq_validator/test_integration.py new file mode 100644 index 0000000..44cdba1 --- /dev/null +++ b/tests/src/dq_validator/test_integration.py @@ -0,0 +1,466 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Integration tests for Data Intelligence SDK +""" + +import math +import pytest +from wxdi.dq_validator import ( + AssetMetadata, + ColumnMetadata, + DataType, + Validator, + ValidationRule, + ComparisonCheck, + ComparisonOperator, + ValidValuesCheck, + LengthCheck, +) + + +class TestBasicIntegration: + """Test basic integration scenarios""" + + def setup_method(self): + """Set up test metadata""" + self.metadata = AssetMetadata( + table_name="test_table", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("name", DataType.STRING, length=100), + ColumnMetadata("age", DataType.INTEGER), + ColumnMetadata("status", DataType.STRING, length=20), + ], + ) + + def test_single_rule_single_check_passes(self): + """Test single rule with single check passes""" + validator = Validator(self.metadata) + validator.add_rule( + ValidationRule("name").add_check(LengthCheck(min_length=2, max_length=50)) + ) + + record = [1, "John Doe", 25, "active"] + result = validator.validate(record) + + assert result.is_valid + assert result.score == "1/1" + assert math.isclose(result.pass_rate, 100.0) + assert len(result.errors) == 0 + + def test_single_rule_single_check_fails(self): + """Test single rule with single check fails""" + validator = Validator(self.metadata) + validator.add_rule( + ValidationRule("name").add_check(LengthCheck(min_length=10, max_length=50)) + ) + + record = [1, "John", 25, "active"] + result = validator.validate(record) + + assert not result.is_valid + assert result.score == "0/1" + assert math.isclose(result.pass_rate, 0.0, abs_tol=1e-9) + assert len(result.errors) == 1 + assert result.errors[0].column_name == "name" + + def test_multiple_rules_all_pass(self): + """Test multiple rules all pass""" + validator = Validator(self.metadata) + validator.add_rule( + ValidationRule("name").add_check(LengthCheck(min_length=2, max_length=50)) + ) + validator.add_rule( + ValidationRule("age").add_check( + ComparisonCheck(operator=">=", target_value=18) + ) + ) + validator.add_rule( + ValidationRule("status").add_check(ValidValuesCheck(["active", "inactive"])) + ) + + record = [1, "John Doe", 25, "active"] + result = validator.validate(record) + + assert result.is_valid + assert result.score == "3/3" + assert math.isclose(result.pass_rate, 100.0) + + def test_multiple_rules_some_fail(self): + """Test multiple rules with some failures""" + validator = Validator(self.metadata) + validator.add_rule( + ValidationRule("name").add_check(LengthCheck(min_length=2, max_length=50)) + ) + validator.add_rule( + ValidationRule("age").add_check( + ComparisonCheck(operator=">=", target_value=18) + ) + ) + validator.add_rule( + ValidationRule("status").add_check(ValidValuesCheck(["active", "inactive"])) + ) + + record = [1, "John Doe", 15, "pending"] # age < 18, status invalid + result = validator.validate(record) + + assert not result.is_valid + assert result.score == "1/3" + assert result.pass_rate == pytest.approx(33.33, rel=0.1) + assert len(result.errors) == 2 + + def test_single_rule_multiple_checks(self): + """Test single rule with multiple checks""" + validator = Validator(self.metadata) + validator.add_rule( + ValidationRule("age") + .add_check(ComparisonCheck(operator=">=", target_value=18)) + .add_check(ComparisonCheck(operator="<=", target_value=65)) + ) + + record = [1, "John Doe", 30, "active"] + result = validator.validate(record) + + assert result.is_valid + assert result.score == "2/2" + assert math.isclose(result.pass_rate, 100.0) + + +class TestBatchValidation: + """Test batch validation scenarios""" + + def setup_method(self): + """Set up test metadata and validator""" + self.metadata = AssetMetadata( + table_name="users", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("username", DataType.STRING), + ColumnMetadata("age", DataType.INTEGER), + ], + ) + + self.validator = Validator(self.metadata) + self.validator.add_rule( + ValidationRule("username").add_check( + LengthCheck(min_length=3, max_length=20) + ) + ) + self.validator.add_rule( + ValidationRule("age").add_check( + ComparisonCheck(operator=">=", target_value=18) + ) + ) + + def test_batch_all_pass(self): + """Test batch validation with all records passing""" + records = [[1, "alice", 25], [2, "bob", 30], [3, "charlie", 22]] + + results = self.validator.validate_batch(records) + + assert len(results) == 3 + assert all(r.is_valid for r in results) + assert all(r.score == "2/2" for r in results) + + def test_batch_some_fail(self): + """Test batch validation with some failures""" + records = [ + [1, "alice", 25], # Pass + [2, "ab", 30], # Fail: username too short + [3, "charlie", 15], # Fail: age < 18 + ] + + results = self.validator.validate_batch(records) + + assert len(results) == 3 + assert results[0].is_valid + assert not results[1].is_valid + assert not results[2].is_valid + + # Check specific errors + assert len(results[1].errors) == 1 + assert results[1].errors[0].column_name == "username" + + assert len(results[2].errors) == 1 + assert results[2].errors[0].column_name == "age" + + def test_batch_record_indices(self): + """Test batch validation tracks record indices""" + records = [ + [1, "alice", 25], + [2, "bob", 30], + ] + + results = self.validator.validate_batch(records) + + assert results[0].record_index == 0 + assert results[1].record_index == 1 + + +class TestComplexScenarios: + """Test complex validation scenarios""" + + def test_column_to_column_comparison(self): + """Test column-to-column comparison""" + metadata = AssetMetadata( + table_name="transactions", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("amount", DataType.DECIMAL), + ColumnMetadata("min_amount", DataType.DECIMAL), + ], + ) + + validator = Validator(metadata) + validator.add_rule( + ValidationRule("amount").add_check( + ComparisonCheck(operator=">", target_column="min_amount") + ) + ) + + # Pass case + record_pass = [1, 100.00, 50.00] + result_pass = validator.validate(record_pass) + assert result_pass.is_valid + + # Fail case + record_fail = [2, 30.00, 50.00] + result_fail = validator.validate(record_fail) + assert not result_fail.is_valid + assert ( + "amount (30.0) > min_amount (50.0) failed" in result_fail.errors[0].message + ) + + def test_case_insensitive_validation(self): + """Test case-insensitive validation""" + metadata = AssetMetadata( + table_name="data", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("status", DataType.STRING), + ], + ) + + validator = Validator(metadata) + validator.add_rule( + ValidationRule("status").add_check( + ValidValuesCheck(["active", "inactive"], case_sensitive=False) + ) + ) + + # Test various cases + test_cases = [ + [1, "active"], + [2, "ACTIVE"], + [3, "Active"], + [4, "AcTiVe"], + ] + + results = validator.validate_batch(test_cases) + assert all(r.is_valid for r in results) + + def test_multiple_checks_per_field(self): + """Test multiple checks on same field""" + metadata = AssetMetadata( + table_name="data", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("age", DataType.INTEGER), + ], + ) + + validator = Validator(metadata) + validator.add_rule( + ValidationRule("age") + .add_check(ComparisonCheck(operator=">=", target_value=18)) + .add_check(ComparisonCheck(operator="<=", target_value=65)) + .add_check(ComparisonCheck(operator="!=", target_value=0)) + ) + + # All checks pass + record_pass = [1, 30] + result_pass = validator.validate(record_pass) + assert result_pass.is_valid + assert result_pass.score == "3/3" + + # Some checks fail + record_fail = [2, 70] # > 65 + result_fail = validator.validate(record_fail) + assert not result_fail.is_valid + assert result_fail.score == "2/3" + + def test_length_check_with_integers(self): + """Test length check converts integers to strings""" + metadata = AssetMetadata( + table_name="data", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ], + ) + + validator = Validator(metadata) + validator.add_rule( + ValidationRule("id").add_check(LengthCheck(min_length=4, max_length=6)) + ) + + # Pass: 12345 -> "12345" (length 5) + result_pass = validator.validate([12345]) + assert result_pass.is_valid + + # Fail: 12 -> "12" (length 2) + result_fail = validator.validate([12]) + assert not result_fail.is_valid + + +class TestValidationResultDetails: + """Test ValidationResult details""" + + def test_result_to_dict(self): + """Test ValidationResult.to_dict()""" + metadata = AssetMetadata( + table_name="test", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("name", DataType.STRING), + ], + ) + + validator = Validator(metadata) + validator.add_rule(ValidationRule("name").add_check(LengthCheck(min_length=5))) + + record = [1, "abc"] + result = validator.validate(record) + + result_dict = result.to_dict() + + assert "record_index" in result_dict + assert "is_valid" in result_dict + assert "score" in result_dict + assert "pass_rate" in result_dict + assert "total_checks" in result_dict + assert "passed_checks" in result_dict + assert "failed_checks" in result_dict + assert "errors" in result_dict + + assert result_dict["is_valid"] == False + assert result_dict["total_checks"] == 1 + assert result_dict["failed_checks"] == 1 + assert len(result_dict["errors"]) == 1 + + def test_error_to_dict(self): + """Test ValidationError.to_dict()""" + metadata = AssetMetadata( + table_name="test", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("age", DataType.INTEGER), + ], + ) + + validator = Validator(metadata) + validator.add_rule( + ValidationRule("age").add_check( + ComparisonCheck(operator=">=", target_value=18) + ) + ) + + record = [1, 15] + result = validator.validate(record) + + error_dict = result.errors[0].to_dict() + + assert "column" in error_dict + assert "check" in error_dict + assert "message" in error_dict + assert "value" in error_dict + assert "expected" in error_dict + + assert error_dict["column"] == "age" + assert error_dict["check"] == "comparison_check" + + +class TestEdgeCases: + """Test edge cases and error conditions""" + + def test_empty_record_array(self): + """Test validation with empty record array""" + metadata = AssetMetadata( + table_name="test", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ], + ) + + validator = Validator(metadata) + validator.add_rule( + ValidationRule("id").add_check( + ComparisonCheck(operator=">", target_value=0) + ) + ) + + # Empty record should cause error + record = [] + result = validator.validate(record) + + assert not result.is_valid + assert len(result.errors) > 0 + + def test_validator_with_no_rules(self): + """Test validator with no rules""" + metadata = AssetMetadata( + table_name="test", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ], + ) + + validator = Validator(metadata) + + record = [1] + result = validator.validate(record) + + assert result.is_valid + assert result.total_checks == 0 + assert result.score == "0/0" + + def test_fluent_api_chaining(self): + """Test fluent API method chaining""" + metadata = AssetMetadata( + table_name="test", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("name", DataType.STRING), + ColumnMetadata("age", DataType.INTEGER), + ], + ) + + # Chain multiple add_rule calls + validator = ( + Validator(metadata) + .add_rule(ValidationRule("name").add_check(LengthCheck(min_length=2))) + .add_rule( + ValidationRule("age").add_check( + ComparisonCheck(operator=">=", target_value=0) + ) + ) + ) + + record = [1, "John", 25] + result = validator.validate(record) + + assert result.is_valid + assert result.total_checks == 2 diff --git a/tests/src/dq_validator/test_issue_reporting.py b/tests/src/dq_validator/test_issue_reporting.py new file mode 100644 index 0000000..760d819 --- /dev/null +++ b/tests/src/dq_validator/test_issue_reporting.py @@ -0,0 +1,1609 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" + +""" +Tests for IssueReporter utility. +""" + +import pytest +from unittest.mock import Mock, patch, MagicMock, call +from wxdi.dq_validator.issue_reporting import IssueReporter +from wxdi.dq_validator.provider import ProviderConfig +from wxdi.dq_validator import Validator, ValidationRule, AssetMetadata, ColumnMetadata +from wxdi.dq_validator.checks import ( + FormatCheck, CompletenessCheck, ComparisonCheck, + LengthCheck, RangeCheck, RegexCheck, CaseCheck, + DataTypeCheck, ValidValuesCheck +) +from wxdi.dq_validator.metadata import DataType +from wxdi.dq_validator.checks.comparison_check import ComparisonOperator +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + + +@pytest.fixture +def config(): + """Create a test configuration.""" + return ProviderConfig( + url="https://test-instance.com", + auth_token="Bearer test-token", + project_id="project-123" + ) + + +@pytest.fixture +def reporter(config): + """Create a test IssueReporter instance with mocked providers.""" + with patch('wxdi.dq_validator.issue_reporting.ChecksProvider'), \ + patch('wxdi.dq_validator.issue_reporting.IssuesProvider'), \ + patch('wxdi.dq_validator.issue_reporting.DimensionsProvider'), \ + patch('wxdi.dq_validator.issue_reporting.DQAssetsProvider'), \ + patch('wxdi.dq_validator.issue_reporting.CamsProvider'), \ + patch('wxdi.dq_validator.issue_reporting.DQSearchProvider'): + reporter = IssueReporter(config) + yield reporter + + +@pytest.fixture +def sample_validator(): + """Create a sample validator for testing""" + metadata = AssetMetadata('test_table', [ + ColumnMetadata('email', DataType.STRING), + ColumnMetadata('name', DataType.STRING), + ColumnMetadata('age', DataType.INTEGER), + ColumnMetadata('salary', DataType.FLOAT) + ]) + validator = Validator(metadata) + validator.add_rule(ValidationRule('email').add_check(FormatCheck(formats={'email'}))) + validator.add_rule(ValidationRule('name').add_check(CompletenessCheck()).add_check(LengthCheck(min_length=2))) + validator.add_rule(ValidationRule('age').add_check(RangeCheck(min_value=0, max_value=120))) + validator.add_rule(ValidationRule('salary').add_check(ComparisonCheck( + operator=ComparisonOperator.GREATER_THAN, + target_column='min_salary' + ))) + return validator + + +class TestIssueReporterInitialization: + """Test IssueReporter initialization""" + + def test_init_creates_all_providers(self, config): + """Test that initialization creates all required providers.""" + with patch('wxdi.dq_validator.issue_reporting.ChecksProvider') as mock_checks, \ + patch('wxdi.dq_validator.issue_reporting.IssuesProvider') as mock_issues, \ + patch('wxdi.dq_validator.issue_reporting.DimensionsProvider') as mock_dimensions, \ + patch('wxdi.dq_validator.issue_reporting.DQAssetsProvider') as mock_assets, \ + patch('wxdi.dq_validator.issue_reporting.DQSearchProvider') as mock_search, \ + patch('wxdi.dq_validator.issue_reporting.CamsProvider') as mock_cams: + + reporter = IssueReporter(config) + + # Verify all providers were created with correct config + mock_checks.assert_called_once_with(config) + mock_issues.assert_called_once_with(config) + mock_dimensions.assert_called_once_with(config) + mock_assets.assert_called_once_with(config) + mock_search.assert_called_once_with(config) + mock_cams.assert_called_once_with(config) + + # Verify config is stored + assert reporter.config == config + + # Verify all provider attributes exist + assert hasattr(reporter, 'check_provider') + assert hasattr(reporter, 'issues_provider') + assert hasattr(reporter, 'dimension_provider') + assert hasattr(reporter, 'asset_provider') + assert hasattr(reporter, 'search_provider') + assert hasattr(reporter, 'cams_provider') + + +class TestMapCheckNameToCheckType: + """Test map_check_name_to_check_type static method""" + + def test_format_check(self): + """Test mapping format_check to check type.""" + result = IssueReporter.map_check_name_to_check_type("format_check") + assert result == "format" + + def test_completeness_check(self): + """Test mapping completeness_check to check type.""" + result = IssueReporter.map_check_name_to_check_type("completeness_check") + assert result == "completeness" + + def test_comparison_check(self): + """Test mapping comparison_check to check type.""" + result = IssueReporter.map_check_name_to_check_type("comparison_check") + assert result == "comparison" + + def test_valid_values_check(self): + """Test mapping valid_values_check to check type.""" + result = IssueReporter.map_check_name_to_check_type("valid_values_check") + assert result == "possible_values" + + def test_unknown_check(self): + """Test mapping unknown check name returns None.""" + result = IssueReporter.map_check_name_to_check_type("unknown_check") + assert result is None + + +class TestMapCheckNameToCpdName: + """Test map_check_name_to_cpd_name static method""" + + def test_format_check(self): + """Test mapping format_check to CPD name.""" + result = IssueReporter.map_check_name_to_cpd_name("format_check") + assert result == "Format check" + + def test_completeness_check(self): + """Test mapping completeness_check to CPD name.""" + result = IssueReporter.map_check_name_to_cpd_name("completeness_check") + assert result == "Completeness check" + + def test_case_check(self): + """Test mapping case_check to CPD name.""" + result = IssueReporter.map_check_name_to_cpd_name("case_check") + assert result == "Capitalization style check" + + def test_valid_values_check(self): + """Test mapping valid_values_check to CPD name.""" + result = IssueReporter.map_check_name_to_cpd_name("valid_values_check") + assert result == "Possible values check" + + def test_unknown_check(self): + """Test mapping unknown check name returns None.""" + result = IssueReporter.map_check_name_to_cpd_name("unknown_check") + assert result is None + + +class TestGetCheckFromValidator: + """Test get_check_from_validator static method""" + + def test_get_check_found(self, sample_validator): + """Test getting check from validator when it exists.""" + check = IssueReporter.get_check_from_validator( + sample_validator, + "email", + "format_check" + ) + + assert check is not None + assert isinstance(check, FormatCheck) + assert check.get_check_name() == "format_check" + + def test_get_completeness_check(self, sample_validator): + """Test getting completeness check.""" + check = IssueReporter.get_check_from_validator( + sample_validator, + "name", + "completeness_check" + ) + + assert check is not None + assert isinstance(check, CompletenessCheck) + + def test_get_range_check(self, sample_validator): + """Test getting range check.""" + check = IssueReporter.get_check_from_validator( + sample_validator, + "age", + "range_check" + ) + + assert check is not None + assert isinstance(check, RangeCheck) + + def test_get_comparison_check(self, sample_validator): + """Test getting comparison check.""" + check = IssueReporter.get_check_from_validator( + sample_validator, + "salary", + "comparison_check" + ) + + assert check is not None + assert isinstance(check, ComparisonCheck) + + def test_check_not_found_wrong_column(self, sample_validator): + """Test getting check with wrong column name.""" + check = IssueReporter.get_check_from_validator( + sample_validator, + "nonexistent_column", + "format_check" + ) + + assert check is None + + def test_check_not_found_wrong_check_name(self, sample_validator): + """Test getting check with wrong check name.""" + check = IssueReporter.get_check_from_validator( + sample_validator, + "email", + "nonexistent_check" + ) + + assert check is None + + def test_multiple_checks_on_column(self, sample_validator): + """Test getting specific check when column has multiple checks.""" + # name column has both completeness and length checks + completeness_check = IssueReporter.get_check_from_validator( + sample_validator, + "name", + "completeness_check" + ) + length_check = IssueReporter.get_check_from_validator( + sample_validator, + "name", + "length_check" + ) + + assert completeness_check is not None + assert isinstance(completeness_check, CompletenessCheck) + assert length_check is not None + assert isinstance(length_check, LengthCheck) + + def test_empty_validator(self): + """Test with validator that has no rules.""" + metadata = AssetMetadata('empty_table', [ + ColumnMetadata('col1', DataType.STRING) + ]) + empty_validator = Validator(metadata) + + check = IssueReporter.get_check_from_validator( + empty_validator, + "col1", + "format_check" + ) + + assert check is None + + +class TestGetCheckId: + """Test get_check_id method""" + + def test_get_check_id_success(self, reporter): + """Test successful check ID retrieval.""" + reporter.search_provider.search_dq_check = Mock(return_value={ + "id": "check-id-123", + "native_id": "asset-id/column/check-type", + "type": "format" + }) + + result = reporter.get_check_id( + check_native_id="asset-id/column/check-type", + check_type="format", + project_id="project-123" + ) + + assert result == "check-id-123" + reporter.search_provider.search_dq_check.assert_called_once_with( + native_id="asset-id/column/check-type", + check_type="format", + project_id="project-123", + catalog_id=None + ) + + def test_get_check_id_with_catalog_id(self, reporter): + """Test check ID retrieval with catalog_id.""" + reporter.search_provider.search_dq_check = Mock(return_value={ + "id": "check-id-456" + }) + + result = reporter.get_check_id( + check_native_id="asset-id/column/check-type", + check_type="completeness", + catalog_id="catalog-789" + ) + + assert result == "check-id-456" + reporter.search_provider.search_dq_check.assert_called_once_with( + native_id="asset-id/column/check-type", + check_type="completeness", + project_id=None, + catalog_id="catalog-789" + ) + + def test_get_check_id_not_found(self, reporter): + """Test check ID retrieval when check not found.""" + reporter.search_provider.search_dq_check = Mock(side_effect=ValueError("Not found")) + + result = reporter.get_check_id( + check_native_id="nonexistent", + check_type="format", + project_id="project-123" + ) + + assert result is None + + def test_get_check_id_exception_handling(self, reporter): + """Test that exceptions are caught and None is returned.""" + reporter.search_provider.search_dq_check = Mock(side_effect=Exception("Unexpected error")) + + result = reporter.get_check_id( + check_native_id="asset-id/column/check", + check_type="format", + project_id="project-123" + ) + + assert result is None + + def test_get_check_id_missing_id_in_response(self, reporter): + """Test when response doesn't contain 'id' field.""" + reporter.search_provider.search_dq_check = Mock(return_value={ + "native_id": "asset-id/column/check-type" + # Missing 'id' field + }) + + result = reporter.get_check_id( + check_native_id="asset-id/column/check-type", + check_type="format", + project_id="project-123" + ) + + assert result is None + + +class TestCreateCheck: + """Test create_check method""" + + def test_create_check_format(self, reporter, sample_validator): + """Test creating a format check.""" + # Setup mocks + reporter.dimension_provider.search_dimension = Mock(return_value="dimension-id-123") + reporter.check_provider._create_check_full = Mock(return_value={ + "id": "check-id-456", + "name": "Format check", + "type": "format", + "native_id": "asset-789/format/Validity" + }) + + # Get the format check from validator + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + # Execute + result = reporter.create_check( + asset_id="asset-789", + column_name="email", + check_obj=format_check, + project_id="project-123" + ) + + # Verify - now returns dict + assert isinstance(result, dict) + assert result["id"] == "check-id-456" + + # Verify dimension search was called + reporter.dimension_provider.search_dimension.assert_called_once() + + # Verify check creation was called with correct parameters + reporter.check_provider._create_check_full.assert_called_once() + call_args = reporter.check_provider._create_check_full.call_args + assert call_args[1]['name'] == "Format check" + assert call_args[1]['dimension_id'] == "dimension-id-123" + # When parent_check_id is None, native_id format is: asset_id/check_type/DimensionName + assert "asset-789/format/Validity" in call_args[1]['native_id'] + assert call_args[1]['check_type'] == "format" + assert call_args[1]['project_id'] == "project-123" + + def test_create_check_completeness(self, reporter, sample_validator): + """Test creating a completeness check.""" + reporter.dimension_provider.search_dimension = Mock(return_value="dimension-id-comp") + reporter.check_provider._create_check_full = Mock(return_value={ + "id": "check-id-comp", + "name": "Completeness check", + "type": "completeness", + "native_id": "asset-999/completeness/Completeness" + }) + + completeness_check = IssueReporter.get_check_from_validator( + sample_validator, "name", "completeness_check" + ) + + result = reporter.create_check( + asset_id="asset-999", + column_name="name", + check_obj=completeness_check, + project_id="project-456" + ) + + assert isinstance(result, dict) + assert result["id"] == "check-id-comp" + + call_args = reporter.check_provider._create_check_full.call_args + assert call_args[1]['name'] == "Completeness check" + assert call_args[1]['check_type'] == "completeness" + # When parent_check_id is None, native_id format is: asset_id/check_type/DimensionName + assert "asset-999/completeness/Completeness" in call_args[1]['native_id'] + + def test_create_check_comparison_with_target_column(self, reporter, sample_validator): + """Test creating a comparison check with target column and parent_id.""" + reporter.dimension_provider.search_dimension = Mock(return_value="dimension-id-comp") + reporter.check_provider._create_check_full = Mock(return_value={ + "id": "check-id-comparison", + "name": "Comparison check", + "type": "comparison", + "native_id": "asset-comp/comparison/salary/greaterThan/min_salary" + }) + + comparison_check = IssueReporter.get_check_from_validator( + sample_validator, "salary", "comparison_check" + ) + + result = reporter.create_check( + asset_id="asset-comp", + column_name="salary", + check_obj=comparison_check, + project_id="project-789", + parent_id="parent-check-id" # Need parent_id for column name in native_id + ) + + assert isinstance(result, dict) + assert result["id"] == "check-id-comparison" + + call_args = reporter.check_provider._create_check_full.call_args + # When parent_id is provided, native_id includes column name and operator details + native_id = call_args[1]['native_id'] + assert "asset-comp/comparison/salary/" in native_id + assert "greaterThan" in native_id + assert "min_salary" in native_id + assert call_args[1]['parent_check_id'] == "parent-check-id" + + def test_create_check_with_catalog_id(self, reporter, sample_validator): + """Test creating a check with catalog_id instead of project_id.""" + reporter.dimension_provider.search_dimension = Mock(return_value="dimension-id-cat") + reporter.check_provider._create_check_full = Mock(return_value={ + "id": "check-id-cat", + "name": "Format check", + "type": "format" + }) + + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + result = reporter.create_check( + asset_id="asset-cat", + column_name="email", + check_obj=format_check, + catalog_id="catalog-999" + ) + + assert isinstance(result, dict) + assert result["id"] == "check-id-cat" + + call_args = reporter.check_provider._create_check_full.call_args + assert call_args[1]['catalog_id'] == "catalog-999" + assert call_args[1]['project_id'] is None + + def test_create_check_lowercase_column_name(self, reporter, sample_validator): + """Test that column name is converted to lowercase in native_id when parent_id is provided.""" + reporter.dimension_provider.search_dimension = Mock(return_value="dim-id") + reporter.check_provider._create_check_full = Mock(return_value={ + "id": "check-id", + "name": "Format check", + "type": "format", + "native_id": "asset-id/format/email/" + }) + + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + reporter.create_check( + asset_id="asset-id", + column_name="EMAIL", # Uppercase + check_obj=format_check, + project_id="project-id", + parent_id="parent-id" # Need parent_id for column name to be in native_id + ) + + call_args = reporter.check_provider._create_check_full.call_args + native_id = call_args[1]['native_id'] + # Should be lowercase + assert "/email/" in native_id + assert "/EMAIL/" not in native_id + + +class TestValidateAndPrepareCheckData: + """Test _validate_and_prepare_check_data method""" + + def test_validate_skip_no_failures(self, reporter, sample_validator): + """Test that validation returns None when there are no failures.""" + stats = {'failed': 0, 'total': 100} + + result = reporter._validate_and_prepare_check_data( + column_name="email", + check_name="format_check", + stats=stats, + data_asset_entity=Mock(), + assets_map={}, + validator=sample_validator + ) + + assert result is None + + def test_validate_skip_unmapped_check_type(self, reporter, sample_validator): + """Test that validation returns None for unmapped check types.""" + stats = {'failed': 10, 'total': 100} + + result = reporter._validate_and_prepare_check_data( + column_name="email", + check_name="unknown_check", # Not in mapping + stats=stats, + data_asset_entity=Mock(), + assets_map={}, + validator=sample_validator + ) + + assert result is None + + def test_validate_skip_missing_column_info(self, reporter, sample_validator): + """Test that validation returns None when column info is missing.""" + stats = {'failed': 10, 'total': 100} + data_asset_entity = Mock() + data_asset_entity.column_info = {} # Empty column info + + result = reporter._validate_and_prepare_check_data( + column_name="email", + check_name="format_check", + stats=stats, + data_asset_entity=data_asset_entity, + assets_map={}, + validator=sample_validator + ) + + assert result is None + + def test_validate_skip_missing_column_asset_id(self, reporter, sample_validator): + """Test that validation returns None when column asset ID not found.""" + stats = {'failed': 10, 'total': 100} + data_asset_entity = Mock() + data_asset_entity.column_info = {"email": Mock()} + assets_map = {} # Empty map + + result = reporter._validate_and_prepare_check_data( + column_name="email", + check_name="format_check", + stats=stats, + data_asset_entity=data_asset_entity, + assets_map=assets_map, + validator=sample_validator + ) + + assert result is None + + def test_validate_skip_check_not_in_validator(self, reporter, sample_validator): + """Test that validation returns None when check not found in validator.""" + stats = {'failed': 10, 'total': 100} + data_asset_entity = Mock() + data_asset_entity.column_info = {"email": Mock()} + assets_map = {"email": {"id": "column-asset-id"}} + + result = reporter._validate_and_prepare_check_data( + column_name="email", + check_name="range_check", # email doesn't have range check + stats=stats, + data_asset_entity=data_asset_entity, + assets_map=assets_map, + validator=sample_validator + ) + + assert result is None + + def test_validate_success(self, reporter, sample_validator): + """Test successful validation returns tuple with all data.""" + stats = {'failed': 10, 'total': 100} + data_asset_entity = Mock() + data_asset_entity.column_info = {"email": Mock()} + assets_map = {"email": {"id": "column-asset-id-123", "name": "email"}} + + result = reporter._validate_and_prepare_check_data( + column_name="email", + check_name="format_check", + stats=stats, + data_asset_entity=data_asset_entity, + assets_map=assets_map, + validator=sample_validator + ) + + assert result is not None + check_type, column_id, check_obj, occurrences, total = result + assert check_type == "format" + assert column_id == "column-asset-id-123" + assert isinstance(check_obj, FormatCheck) + assert occurrences == 10 + assert total == 100 + + +class TestCreateCheckAndIssue: + """Test _create_check_and_issue method""" + + def test_create_check_and_issue_success(self, reporter, sample_validator): + """Test successful check and issue creation.""" + # Mock handle_parent to return a parent check dict + reporter.handle_parent = Mock(return_value={"id": "parent-check-id", "type": "format"}) + reporter.create_check = Mock(return_value={ + "id": "check-id-123", + "name": "Format check", + "type": "format" + }) + reporter.issues_provider.create_issue = Mock() + + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column" + } + } + + result = reporter._create_check_and_issue( + asset_id="asset-id", + column_name="email", + column_id="column-asset-id", + check_name="format_check", + check_obj=format_check, + number_of_occurrences=10, + total_records=100, + project_id="project-123", + assets_map=assets_map + ) + + assert result is True + reporter.handle_parent.assert_called_once() + reporter.create_check.assert_called_once() + reporter.issues_provider.create_issue.assert_called_once_with( + dq_check_id="check-id-123", + reported_for_id="column-asset-id", + number_of_occurrences=10, + number_of_tested_records=100, + project_id="project-123", + catalog_id=None + ) + + def test_create_check_and_issue_409_conflict_update_success(self, reporter, sample_validator): + """Test handling 409 conflict by updating existing check.""" + # Mock handle_parent to return a parent check dict + reporter.handle_parent = Mock(return_value={"id": "parent-check-id", "type": "format"}) + # First call raises 409, then get_checks returns existing check + reporter.create_check = Mock( + side_effect=ValueError("409 Conflict: Check already exists") + ) + reporter.check_provider.get_checks = Mock(return_value=[ + {"id": "existing-check-id", "type": "format", "native_id": "asset-id/email/format"} + ]) + reporter.issues_provider.update_issue_metrics = Mock() + + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column" + } + } + + result = reporter._create_check_and_issue( + asset_id="asset-id", + column_name="email", + column_id="column-asset-id", + check_name="format_check", + check_obj=format_check, + number_of_occurrences=10, + total_records=100, + project_id="project-123", + assets_map=assets_map + ) + + assert result is True + reporter.check_provider.get_checks.assert_called_once_with( + dq_asset_id="column-asset-id", + check_type="format", + project_id="project-123" + ) + reporter.issues_provider.update_issue_metrics.assert_called_once() + + def test_create_check_and_issue_409_conflict_check_not_found(self, reporter, sample_validator): + """Test handling 409 conflict when existing check cannot be found.""" + reporter.handle_parent = Mock(return_value=None) + reporter.create_check = Mock( + side_effect=ValueError("409 Conflict: Check already exists") + ) + reporter.check_provider.get_checks = Mock(return_value=[ + {"id": "other-check-id", "type": "completeness"} # Different type + ]) + + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column" + } + } + + result = reporter._create_check_and_issue( + asset_id="asset-id", + column_name="email", + column_id="column-asset-id", + check_name="format_check", + check_obj=format_check, + number_of_occurrences=10, + total_records=100, + project_id="project-123", + assets_map=assets_map + ) + + assert result is False + + def test_create_check_and_issue_non_409_error(self, reporter, sample_validator): + """Test handling non-409 errors.""" + reporter.handle_parent = Mock(return_value=None) + reporter.create_check = Mock( + side_effect=ValueError("500 Internal Server Error") + ) + + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column" + } + } + + result = reporter._create_check_and_issue( + asset_id="asset-id", + column_name="email", + column_id="column-asset-id", + check_name="format_check", + check_obj=format_check, + number_of_occurrences=10, + total_records=100, + project_id="project-123", + assets_map=assets_map + ) + + assert result is False + + def test_create_check_and_issue_unmapped_check_type(self, reporter, sample_validator): + """Test handling unmapped check type.""" + # Create a mock check with unknown type + unknown_check = Mock() + unknown_check.get_check_name = Mock(return_value="unknown_check") + + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column" + } + } + + result = reporter._create_check_and_issue( + asset_id="asset-id", + column_name="email", + column_id="column-asset-id", + check_name="unknown_check", + check_obj=unknown_check, + number_of_occurrences=10, + total_records=100, + project_id="project-123", + assets_map=assets_map + ) + + assert result is False + + +class TestHandleParent: + """Test handle_parent method""" + + def test_handle_parent_found_existing(self, reporter, sample_validator): + """Test finding existing parent check.""" + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + # Mock search_provider to return existing check + reporter.search_provider.search_dq_check = Mock(return_value={ + "id": "parent-check-id", + "native_id": "asset-id/format/Accuracy", + "type": "format" + }) + + result = reporter.handle_parent( + asset_id="asset-id", + check_obj=format_check, + project_id="project-123" + ) + + assert result is not None + assert result["id"] == "parent-check-id" + assert "_newly_created" not in result + reporter.search_provider.search_dq_check.assert_called_once() + + def test_handle_parent_not_found_creates_new(self, reporter, sample_validator): + """Test creating new parent check when not found.""" + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + # Mock search to raise exception (not found) + reporter.search_provider.search_dq_check = Mock( + side_effect=Exception("Check not found") + ) + + # Mock create_check to return new check + reporter.create_check = Mock(return_value={ + "id": "new-parent-check-id", + "native_id": "asset-id/format/Accuracy", + "type": "format" + }) + + result = reporter.handle_parent( + asset_id="asset-id", + check_obj=format_check, + project_id="project-123" + ) + + assert result is not None + assert result["id"] == "new-parent-check-id" + assert result["_newly_created"] is True + reporter.create_check.assert_called_once_with( + asset_id="asset-id", + column_name=None, + check_obj=format_check, + project_id="project-123", + catalog_id=None, + parent_id=None + ) + + def test_handle_parent_creation_fails(self, reporter, sample_validator): + """Test when both search and creation fail - should raise RuntimeError.""" + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + # Mock search to raise exception + reporter.search_provider.search_dq_check = Mock( + side_effect=Exception("Check not found") + ) + + # Mock create_check to also raise exception + reporter.create_check = Mock( + side_effect=Exception("Creation failed") + ) + + # Should raise RuntimeError when parent creation fails + with pytest.raises(RuntimeError) as exc_info: + reporter.handle_parent( + asset_id="asset-id", + check_obj=format_check, + project_id="project-123" + ) + + assert "Failed to create parent check" in str(exc_info.value) + assert "Creation failed" in str(exc_info.value) + + def test_handle_parent_with_catalog_id(self, reporter, sample_validator): + """Test handle_parent with catalog_id.""" + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + reporter.search_provider.search_dq_check = Mock(return_value={ + "id": "parent-check-id", + "native_id": "asset-id/format/Validity", + "type": "format" + }) + + result = reporter.handle_parent( + asset_id="asset-id", + check_obj=format_check, + project_id=None, + catalog_id="catalog-123" + ) + + assert result is not None + reporter.search_provider.search_dq_check.assert_called_once_with( + native_id="asset-id/format/Validity", + check_type="format", + project_id=None, + catalog_id="catalog-123", + include_children=False + ) + + +class TestCreateBulkIssues: + """Test create_bulk_issues method""" + + def test_create_bulk_issues_success(self, reporter): + """Test successful bulk issue creation.""" + parent_check = { + "id": "parent-check-id", + "native_id": "asset-id/format/Accuracy", + "type": "format" + } + + child_check = { + "id": "child-check-id", + "native_id": "asset-id/email/format", + "type": "format" + } + + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column", + "native_id": "schema.table.email", + "parent": { + "id": "parent-asset-id" + }, + "weight": 1 + }, + "table": { + "id": "parent-asset-id", + "name": "table", + "type": "data_asset", + "native_id": "schema.table", + "weight": 1 + } + } + + reporter.issues_provider.create_issues_bulk = Mock(return_value={ + "created": 2 + }) + + result = reporter.create_bulk_issues( + parent_check=parent_check, + child_check=child_check, + column_name="email", + assets_map=assets_map, + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + assert result is not None + reporter.issues_provider.create_issues_bulk.assert_called_once() + + # Verify the payload structure + call_args = reporter.issues_provider.create_issues_bulk.call_args + payload = call_args.kwargs["payload"] + + assert len(payload["issues"]) == 2 + assert len(payload["assets"]) == 2 + assert len(payload["existing_checks"]) == 2 + assert payload["issues"][0]["status"] == "aggregation" + assert payload["issues"][1]["status"] == "actual" + + def test_create_bulk_issues_column_not_found(self, reporter): + """Test when column asset not found in assets_map.""" + parent_check = {"id": "parent-check-id"} + child_check = {"id": "child-check-id"} + assets_map = {} + + with pytest.raises(ValueError, match="Column asset not found"): + reporter.create_bulk_issues( + parent_check=parent_check, + child_check=child_check, + column_name="email", + assets_map=assets_map, + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + def test_create_bulk_issues_parent_id_not_found(self, reporter): + """Test when parent asset ID not found in column asset.""" + parent_check = {"id": "parent-check-id"} + child_check = {"id": "child-check-id"} + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column" + # Missing parent + } + } + + with pytest.raises(ValueError, match="Parent asset ID not found"): + reporter.create_bulk_issues( + parent_check=parent_check, + child_check=child_check, + column_name="email", + assets_map=assets_map, + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + def test_create_bulk_issues_parent_asset_not_in_map(self, reporter): + """Test when parent asset not found in assets_map.""" + parent_check = {"id": "parent-check-id"} + child_check = {"id": "child-check-id"} + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column", + "parent": { + "id": "parent-asset-id" + } + } + # Parent asset not in map + } + + with pytest.raises(ValueError, match="Parent asset not found in assets_map"): + reporter.create_bulk_issues( + parent_check=parent_check, + child_check=child_check, + column_name="email", + assets_map=assets_map, + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + def test_create_bulk_issues_parent_native_id_missing(self, reporter): + """Test when parent asset native_id is missing.""" + parent_check = {"id": "parent-check-id"} + child_check = {"id": "child-check-id"} + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column", + "parent": { + "id": "parent-asset-id" + } + }, + "table": { + "id": "parent-asset-id", + "name": "table", + "type": "data_asset" + # Missing native_id + } + } + + with pytest.raises(ValueError, match="Parent asset native_id not found"): + reporter.create_bulk_issues( + parent_check=parent_check, + child_check=child_check, + column_name="email", + assets_map=assets_map, + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + def test_create_bulk_issues_api_failure(self, reporter): + """Test when bulk API call fails.""" + parent_check = { + "id": "parent-check-id", + "native_id": "asset-id/format/Accuracy", + "type": "format" + } + + child_check = { + "id": "child-check-id", + "native_id": "asset-id/email/format", + "type": "format" + } + + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column", + "native_id": "schema.table.email", + "parent": { + "id": "parent-asset-id" + }, + "weight": 1 + }, + "table": { + "id": "parent-asset-id", + "name": "table", + "type": "data_asset", + "native_id": "schema.table", + "weight": 1 + } + } + + reporter.issues_provider.create_issues_bulk = Mock( + side_effect=Exception("API Error") + ) + + with pytest.raises(Exception, match="API Error"): + reporter.create_bulk_issues( + parent_check=parent_check, + child_check=child_check, + column_name="email", + assets_map=assets_map, + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + +class TestHelperMethods: + """Test refactored helper methods""" + + def test_find_existing_check_success(self, reporter): + """Test successfully finding an existing check.""" + reporter.check_provider.get_checks = Mock(return_value=[ + {"id": "check-1", "type": "format", "native_id": "asset/email/format"}, + {"id": "check-2", "type": "completeness", "native_id": "asset/email/completeness"} + ]) + + result = reporter._find_existing_check( + column_id="column-asset-id", + check_type="format", + project_id="project-123" + ) + + assert result is not None + assert result[0] == "check-1" + assert result[1] == "asset/email/format" + + def test_find_existing_check_not_found(self, reporter): + """Test when check type doesn't match.""" + reporter.check_provider.get_checks = Mock(return_value=[ + {"id": "check-1", "type": "completeness"} + ]) + + result = reporter._find_existing_check( + column_id="column-asset-id", + check_type="format", + project_id="project-123" + ) + + assert result is None + + def test_find_existing_check_api_error(self, reporter): + """Test when API call fails.""" + reporter.check_provider.get_checks = Mock( + side_effect=ValueError("API Error") + ) + + result = reporter._find_existing_check( + column_id="column-asset-id", + check_type="format", + project_id="project-123" + ) + + assert result is None + + def test_update_existing_check_metrics_success(self, reporter): + """Test successful metric update.""" + reporter.issues_provider.update_issue_metrics = Mock() + + result = reporter._update_existing_check_metrics( + existing_check_native_id="asset/email/format", + number_of_occurrences=10, + total_records=100, + column_name="email", + check_type="format", + project_id="project-123" + ) + + assert result is True + reporter.issues_provider.update_issue_metrics.assert_called_once() + + def test_update_existing_check_metrics_failure(self, reporter): + """Test when metric update fails.""" + reporter.issues_provider.update_issue_metrics = Mock( + side_effect=ValueError("Update failed") + ) + + result = reporter._update_existing_check_metrics( + existing_check_native_id="asset/email/format", + number_of_occurrences=10, + total_records=100, + column_name="email", + check_type="format", + project_id="project-123" + ) + + assert result is False + + def test_handle_409_conflict_success(self, reporter): + """Test successful 409 conflict handling.""" + reporter._find_existing_check = Mock(return_value=("check-id", "native-id")) + reporter._update_existing_check_metrics = Mock(return_value=True) + + result = reporter._handle_409_conflict( + column_id="column-asset-id", + check_type="format", + number_of_occurrences=10, + total_records=100, + column_name="email", + check_name="format_check", + asset_id="asset-id", + project_id="project-123" + ) + + assert result is True + + def test_handle_409_conflict_check_not_found(self, reporter): + """Test 409 conflict when check not found.""" + reporter._find_existing_check = Mock(return_value=None) + + result = reporter._handle_409_conflict( + column_id="column-asset-id", + check_type="format", + number_of_occurrences=10, + total_records=100, + column_name="email", + check_name="format_check", + asset_id="asset-id", + project_id="project-123" + ) + + assert result is False + + def test_handle_409_conflict_update_fails_fallback(self, reporter): + """Test 409 conflict when update fails but fallback succeeds.""" + reporter._find_existing_check = Mock(return_value=("check-id", "native-id")) + reporter._update_existing_check_metrics = Mock(return_value=False) + reporter._handle_update_failure = Mock() + + result = reporter._handle_409_conflict( + column_id="column-asset-id", + check_type="format", + number_of_occurrences=10, + total_records=100, + column_name="email", + check_name="format_check", + asset_id="asset-id", + project_id="project-123" + ) + + assert result is True + reporter._handle_update_failure.assert_called_once() + + +class TestHandleUpdateFailure: + """Test _handle_update_failure method""" + + def test_handle_update_failure_issue_not_found_creates_issue(self, reporter): + """Test creating issue when update fails with 'Issue not found'.""" + reporter.get_check_id = Mock(return_value="check-id-123") + reporter.issues_provider.create_issue = Mock() + + error = ValueError("Issue not found for check") + + result = reporter._handle_update_failure( + error=error, + asset_id="asset-id", + check_type="format", + column_name="email", + column_id="column-asset-id", + number_of_occurrences=10, + total_records=100, + project_id="project-123", + check_id="check-id" + ) + + assert result is True + reporter.get_check_id.assert_called_once_with( + check_native_id="asset-id/check-id", + check_type="format", + project_id="project-123" + ) + reporter.issues_provider.create_issue.assert_called_once() + + def test_handle_update_failure_issue_id_not_found(self, reporter): + """Test creating issue when update fails with 'Issue ID not found'.""" + reporter.get_check_id = Mock(return_value="check-id-456") + reporter.issues_provider.create_issue = Mock() + + error = ValueError("Issue ID not found in response") + + result = reporter._handle_update_failure( + error=error, + asset_id="asset-id", + check_type="completeness", + column_name="name", + column_id="column-asset-id", + number_of_occurrences=5, + total_records=50, + project_id="project-123", + check_id="check-id" + ) + + assert result is True + reporter.issues_provider.create_issue.assert_called_once() + + def test_handle_update_failure_check_id_not_found(self, reporter): + """Test when check_id cannot be found.""" + reporter.get_check_id = Mock(return_value=None) + + error = ValueError("Issue not found") + + result = reporter._handle_update_failure( + error=error, + asset_id="asset-id", + check_type="format", + column_name="email", + column_id="column-asset-id", + number_of_occurrences=10, + total_records=100, + project_id="project-123", + check_id="check-id" + ) + + assert result is True + reporter.get_check_id.assert_called_once() + + def test_handle_update_failure_different_error(self, reporter): + """Test handling different error types.""" + error = ValueError("Different error message") + + result = reporter._handle_update_failure( + error=error, + asset_id="asset-id", + check_type="format", + column_name="email", + column_id="column-asset-id", + number_of_occurrences=10, + total_records=100, + project_id="project-123", + check_id="check-id" + ) + + assert result is True + + +class TestHandleExistingCheck: + """Test _handle_existing_check method""" + + def test_handle_existing_check_success(self, reporter): + """Test successful handling of existing check.""" + column_info = Mock() + check = Mock() + check.metadata = Mock() + check.metadata.type = "format" + check.metadata.check_id = "check-id-123" + column_info.column_checks = [check] + + reporter.issues_provider.update_issue_metrics = Mock() + + result = reporter._handle_existing_check( + column_info=column_info, + check_type="format", + asset_id="asset-id", + column_name="email", + column_id="column-asset-id", + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + assert result is True + reporter.issues_provider.update_issue_metrics.assert_called_once_with( + asset_id="asset-id", + check_id="check-id-123", + occurrences=10, + tested_records=100, + column_name="email", + check_type="format", + project_id="project-123", + asset_type="column", + operation="add" + ) + + def test_handle_existing_check_type_mismatch(self, reporter): + """Test when check type doesn't match.""" + column_info = Mock() + check = Mock() + check.metadata = Mock() + check.metadata.type = "completeness" # Different type + check.metadata.check_id = "check-id-123" + column_info.column_checks = [check] + + result = reporter._handle_existing_check( + column_info=column_info, + check_type="format", # Looking for format + asset_id="asset-id", + column_name="email", + column_id="column-asset-id", + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + assert result is False + + def test_handle_existing_check_no_check_id(self, reporter): + """Test when check has no check_id.""" + column_info = Mock() + check = Mock() + check.metadata = Mock() + check.metadata.type = "format" + check.metadata.check_id = None # No check_id + column_info.column_checks = [check] + + result = reporter._handle_existing_check( + column_info=column_info, + check_type="format", + asset_id="asset-id", + column_name="email", + column_id="column-asset-id", + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + assert result is False + + def test_handle_existing_check_update_failure(self, reporter): + """Test handling update failure.""" + column_info = Mock() + check = Mock() + check.metadata = Mock() + check.metadata.type = "format" + check.metadata.check_id = "check-id-123" + column_info.column_checks = [check] + + reporter.issues_provider.update_issue_metrics = Mock( + side_effect=ValueError("Update failed") + ) + reporter._handle_update_failure = Mock(return_value=True) + + result = reporter._handle_existing_check( + column_info=column_info, + check_type="format", + asset_id="asset-id", + column_name="email", + column_id="column-asset-id", + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + assert result is True + reporter._handle_update_failure.assert_called_once() + + +class TestReportIssues: + """Test update_issues method""" + + def test_report_issues_success(self, reporter, sample_validator): + """Test successful update of issues.""" + # Mock dependencies + reporter.cams_provider.get_asset_by_id = Mock() + data_asset = Mock() + data_asset_entity = Mock() + column_info = Mock() + column_info.column_checks = [] + data_asset_entity.column_info = {"email": column_info} + data_asset.entity = data_asset_entity + reporter.cams_provider.get_asset_by_id.return_value = data_asset + + reporter.asset_provider.get_assets = Mock(return_value={ + "assets": [{"name": "email", "id": "column-asset-id"}] + }) + + reporter._create_check_and_issue = Mock(return_value=True) + + # Create combined stats in nested dict format + combined_stats = { + "email": { + "format_check": {"failed": 10, "total": 100} + } + } + + reporter.report_issues( + asset_id="asset-id", + stats=combined_stats, + validator=sample_validator + ) + + reporter._create_check_and_issue.assert_called_once() + + def test_report_issues_with_existing_checks(self, reporter, sample_validator): + """Test update with existing checks.""" + # Mock dependencies + reporter.cams_provider.get_asset_by_id = Mock() + data_asset = Mock() + data_asset_entity = Mock() + column_info = Mock() + check = Mock() + check.metadata = Mock() + check.metadata.type = "format" + check.metadata.check_id = "check-id-123" + column_info.column_checks = [check] + data_asset_entity.column_info = {"email": column_info} + data_asset.entity = data_asset_entity + reporter.cams_provider.get_asset_by_id.return_value = data_asset + + reporter.asset_provider.get_assets = Mock(return_value={ + "assets": [{"name": "email", "id": "column-asset-id"}] + }) + + reporter._handle_existing_check = Mock(return_value=True) + + # Create combined stats in nested dict format + combined_stats = { + "email": { + "format_check": {"failed": 10, "total": 100} + } + } + + reporter.report_issues( + asset_id="asset-id", + stats=combined_stats, + validator=sample_validator + ) + + reporter._handle_existing_check.assert_called_once() + + def test_report_issues_check_not_handled_creates_new(self, reporter, sample_validator): + """Test creating new check when existing check not handled.""" + # Mock dependencies + reporter.cams_provider.get_asset_by_id = Mock() + data_asset = Mock() + data_asset_entity = Mock() + column_info = Mock() + check = Mock() + check.metadata = Mock() + check.metadata.type = "completeness" # Different type + check.metadata.check_id = "check-id-123" + column_info.column_checks = [check] + data_asset_entity.column_info = {"email": column_info} + data_asset.entity = data_asset_entity + reporter.cams_provider.get_asset_by_id.return_value = data_asset + + reporter.asset_provider.get_assets = Mock(return_value={ + "assets": [{"name": "email", "id": "column-asset-id"}] + }) + + reporter._handle_existing_check = Mock(return_value=False) + reporter._create_check_and_issue = Mock(return_value=True) + + # Create combined stats in nested dict format + combined_stats = { + "email": { + "format_check": {"failed": 10, "total": 100} + } + } + + reporter.report_issues( + asset_id="asset-id", + stats=combined_stats, + validator=sample_validator + ) + + reporter._handle_existing_check.assert_called_once() + reporter._create_check_and_issue.assert_called_once() \ No newline at end of file diff --git a/tests/src/dq_validator/test_length_check.py b/tests/src/dq_validator/test_length_check.py new file mode 100644 index 0000000..66c57ed --- /dev/null +++ b/tests/src/dq_validator/test_length_check.py @@ -0,0 +1,307 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for LengthCheck +""" + +import pytest +from wxdi.dq_validator.checks.length_check import LengthCheck +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + + +class TestLengthCheckInitialization: + """Test LengthCheck initialization and parameter validation""" + + def test_init_with_min_length_only(self): + """Test initialization with only min_length""" + check = LengthCheck(min_length=5) + assert check.min_length == 5 + assert check.max_length is None + + def test_init_with_max_length_only(self): + """Test initialization with only max_length""" + check = LengthCheck(max_length=10) + assert check.min_length is None + assert check.max_length == 10 + + def test_init_with_both_lengths(self): + """Test initialization with both min and max length""" + check = LengthCheck(min_length=3, max_length=20) + assert check.min_length == 3 + assert check.max_length == 20 + + def test_init_no_parameters_raises_error(self): + """Test that initialization without parameters raises ValueError""" + with pytest.raises(ValueError) as exc_info: + LengthCheck() + assert "At least one of min_length or max_length must be specified" in str(exc_info.value) + + def test_init_negative_min_length_raises_error(self): + """Test that negative min_length raises ValueError""" + with pytest.raises(ValueError) as exc_info: + LengthCheck(min_length=-1) + assert "min_length cannot be negative" in str(exc_info.value) + + def test_init_negative_max_length_raises_error(self): + """Test that negative max_length raises ValueError""" + with pytest.raises(ValueError) as exc_info: + LengthCheck(max_length=-5) + assert "max_length cannot be negative" in str(exc_info.value) + + def test_init_min_greater_than_max_raises_error(self): + """Test that min_length > max_length raises ValueError""" + with pytest.raises(ValueError) as exc_info: + LengthCheck(min_length=10, max_length=5) + assert "min_length (10) cannot be greater than max_length (5)" in str(exc_info.value) + + def test_get_check_name(self): + """Test get_check_name returns correct name""" + check = LengthCheck(min_length=1) + assert check.get_check_name() == "length_check" + + def test_get_dimension(self): + """Test get_dimension returns correct dimension""" + check = LengthCheck(min_length=1) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + def test_set_dimension(self): + """Test set_dimension changes the dimension""" + check = LengthCheck(min_length=1) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + check.set_dimension(DataQualityDimension.CONSISTENCY) + assert check.get_dimension() == DataQualityDimension.CONSISTENCY + + +class TestLengthCheckStringValidation: + """Test LengthCheck with string values""" + + def test_string_within_range_passes(self): + """Test string within min/max range passes""" + check = LengthCheck(min_length=3, max_length=20) + context = {'column_name': 'username'} + result = check.validate('john_doe', context) + assert result is None + + def test_string_too_short_fails(self): + """Test string shorter than min_length fails""" + check = LengthCheck(min_length=3, max_length=20) + context = {'column_name': 'username'} + result = check.validate('ab', context) + assert result is not None + assert result.column_name == 'username' + assert result.check_name == 'length_check' + assert "length (2) is less than minimum (3)" in result.message + + def test_string_too_long_fails(self): + """Test string longer than max_length fails""" + check = LengthCheck(min_length=3, max_length=20) + context = {'column_name': 'username'} + result = check.validate('a' * 25, context) + assert result is not None + assert "length (25) exceeds maximum (20)" in result.message + + def test_empty_string_with_min_zero_passes(self): + """Test empty string passes when min_length is 0""" + check = LengthCheck(min_length=0, max_length=100) + context = {'column_name': 'optional_field'} + result = check.validate('', context) + assert result is None + + def test_empty_string_with_min_one_fails(self): + """Test empty string fails when min_length is 1""" + check = LengthCheck(min_length=1, max_length=100) + context = {'column_name': 'required_field'} + result = check.validate('', context) + assert result is not None + assert "length (0) is less than minimum (1)" in result.message + + def test_exact_length_required_passes(self): + """Test exact length requirement passes""" + check = LengthCheck(min_length=2, max_length=2) + context = {'column_name': 'country_code'} + result = check.validate('US', context) + assert result is None + + def test_exact_length_required_fails(self): + """Test exact length requirement fails""" + check = LengthCheck(min_length=2, max_length=2) + context = {'column_name': 'country_code'} + result = check.validate('USA', context) + assert result is not None + assert "length (3) exceeds maximum (2)" in result.message + + def test_unicode_characters(self): + """Test Unicode characters are counted correctly""" + check = LengthCheck(min_length=1, max_length=10) + context = {'column_name': 'name'} + result = check.validate('日本語', context) + assert result is None # Length is 3 characters + + def test_whitespace_only_string(self): + """Test whitespace-only string is counted""" + check = LengthCheck(min_length=1, max_length=10) + context = {'column_name': 'text'} + result = check.validate(' ', context) + assert result is None # Length is 3 + + +class TestLengthCheckNonStringValidation: + """Test LengthCheck with non-string values (converted to string)""" + + def test_integer_value_passes(self): + """Test integer value is converted to string""" + check = LengthCheck(min_length=3, max_length=10) + context = {'column_name': 'id'} + result = check.validate(12345, context) + assert result is None # str(12345) = "12345", length = 5 + + def test_integer_value_fails(self): + """Test integer value fails when string length is out of range""" + check = LengthCheck(min_length=1, max_length=5) + context = {'column_name': 'number'} + result = check.validate(123456789, context) + assert result is not None + assert "length (9) exceeds maximum (5)" in result.message + + def test_float_value_passes(self): + """Test float value is converted to string""" + check = LengthCheck(min_length=3, max_length=10) + context = {'column_name': 'amount'} + result = check.validate(123.45, context) + assert result is None # str(123.45) = "123.45", length = 6 + + def test_boolean_true_passes(self): + """Test boolean True is converted to string""" + check = LengthCheck(min_length=4, max_length=5) + context = {'column_name': 'flag'} + result = check.validate(True, context) + assert result is None # str(True) = "True", length = 4 + + def test_boolean_false_passes(self): + """Test boolean False is converted to string""" + check = LengthCheck(min_length=5, max_length=5) + context = {'column_name': 'flag'} + result = check.validate(False, context) + assert result is None # str(False) = "False", length = 5 + + def test_list_value_passes(self): + """Test list value is converted to string""" + check = LengthCheck(min_length=5, max_length=50) + context = {'column_name': 'data'} + result = check.validate([1, 2, 3], context) + assert result is None # str([1, 2, 3]) = "[1, 2, 3]", length = 9 + + def test_dict_value_passes(self): + """Test dict value is converted to string""" + check = LengthCheck(min_length=5, max_length=50) + context = {'column_name': 'config'} + result = check.validate({'a': 1}, context) + assert result is None # str({'a': 1}) = "{'a': 1}", length varies + + +class TestLengthCheckNoneHandling: + """Test LengthCheck with None values""" + + def test_none_value_fails(self): + """Test None value returns error""" + check = LengthCheck(min_length=3, max_length=20) + context = {'column_name': 'username'} + result = check.validate(None, context) + assert result is not None + assert result.column_name == 'username' + assert "is None, cannot check length" in result.message + + +class TestLengthCheckMinLengthOnly: + """Test LengthCheck with only min_length specified""" + + def test_min_length_only_passes(self): + """Test validation passes when only min_length is specified""" + check = LengthCheck(min_length=10) + context = {'column_name': 'description'} + result = check.validate('a' * 100, context) + assert result is None # No max limit + + def test_min_length_only_fails(self): + """Test validation fails when below min_length""" + check = LengthCheck(min_length=10) + context = {'column_name': 'description'} + result = check.validate('short', context) + assert result is not None + assert "length (5) is less than minimum (10)" in result.message + + +class TestLengthCheckMaxLengthOnly: + """Test LengthCheck with only max_length specified""" + + def test_max_length_only_passes(self): + """Test validation passes when only max_length is specified""" + check = LengthCheck(max_length=10) + context = {'column_name': 'code'} + result = check.validate('ABC', context) + assert result is None # No min limit + + def test_max_length_only_fails(self): + """Test validation fails when exceeds max_length""" + check = LengthCheck(max_length=10) + context = {'column_name': 'code'} + result = check.validate('a' * 15, context) + assert result is not None + assert "length (15) exceeds maximum (10)" in result.message + + +class TestLengthCheckEdgeCases: + """Test LengthCheck edge cases""" + + def test_zero_min_length(self): + """Test min_length of 0 is valid""" + check = LengthCheck(min_length=0, max_length=10) + context = {'column_name': 'field'} + result = check.validate('', context) + assert result is None + + def test_zero_max_length(self): + """Test max_length of 0 requires empty string""" + check = LengthCheck(min_length=0, max_length=0) + context = {'column_name': 'field'} + result = check.validate('', context) + assert result is None + + def test_zero_max_length_fails_with_content(self): + """Test max_length of 0 fails with non-empty string""" + check = LengthCheck(min_length=0, max_length=0) + context = {'column_name': 'field'} + result = check.validate('a', context) + assert result is not None + assert "length (1) exceeds maximum (0)" in result.message + + def test_large_min_length(self): + """Test very large min_length""" + check = LengthCheck(min_length=1000) + context = {'column_name': 'field'} + result = check.validate('a' * 999, context) + assert result is not None + assert "length (999) is less than minimum (1000)" in result.message + + def test_repr(self): + """Test __repr__ method""" + check = LengthCheck(min_length=5, max_length=10) + repr_str = repr(check) + assert "LengthCheck" in repr_str + assert "5" in repr_str + assert "10" in repr_str + diff --git a/tests/src/dq_validator/test_pandas_validator.py b/tests/src/dq_validator/test_pandas_validator.py new file mode 100644 index 0000000..63a9ed1 --- /dev/null +++ b/tests/src/dq_validator/test_pandas_validator.py @@ -0,0 +1,535 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for PandasValidator + +Tests the Pandas DataFrame integration including: +- Initialization and configuration +- Summary statistics calculation +- Validation column addition +- Invalid/valid row filtering +- Column expansion +- Chunked processing +- Error handling +""" + +import math +import pytest +import sys +from typing import Dict, Any + +# Check if pandas is available +try: + import pandas as pd + + PANDAS_AVAILABLE = True +except ImportError: + PANDAS_AVAILABLE = False + +from wxdi.dq_validator import ( + AssetMetadata, + ColumnMetadata, + DataType, + Validator, + ValidationRule, +) +from wxdi.dq_validator.checks import ( + LengthCheck, + ValidValuesCheck, + ComparisonCheck, + ComparisonOperator, +) + +if PANDAS_AVAILABLE: + from wxdi.dq_validator.integrations import PandasValidator + + +# Skip all tests if pandas is not available +pytestmark = pytest.mark.skipif(not PANDAS_AVAILABLE, reason="pandas not installed") + + +@pytest.fixture +def sample_metadata(): + """Create sample metadata for testing""" + return AssetMetadata( + table_name="test_table", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("name", DataType.STRING, length=50), + ColumnMetadata("age", DataType.INTEGER), + ColumnMetadata("status", DataType.STRING, length=20), + ], + ) + + +@pytest.fixture +def sample_validator(sample_metadata): + """Create sample validator with rules""" + validator = Validator(sample_metadata) + + # Add validation rules + validator.add_rule( + ValidationRule("name").add_check(LengthCheck(min_length=2, max_length=50)) + ) + validator.add_rule( + ValidationRule("age").add_check( + ComparisonCheck( + operator=ComparisonOperator.GREATER_THAN_OR_EQUAL, target_value=18 + ) + ) + ) + validator.add_rule( + ValidationRule("status").add_check( + ValidValuesCheck(["active", "inactive"], case_sensitive=False) + ) + ) + + return validator + + +@pytest.fixture +def sample_dataframe(): + """Create sample DataFrame for testing""" + return pd.DataFrame( + { + "id": [1, 2, 3, 4, 5], + "name": ["Alice", "B", "Charlie", "David", "Eve"], + "age": [25, 30, 17, 35, 40], + "status": ["active", "ACTIVE", "inactive", "pending", "Active"], + } + ) + + +class TestPandasValidatorInitialization: + """Test PandasValidator initialization""" + + def test_basic_initialization(self, sample_validator): + """Test basic initialization with default parameters""" + pandas_validator = PandasValidator(sample_validator) + + assert pandas_validator.validator == sample_validator + assert pandas_validator.chunk_size == 10000 + assert pandas_validator.column_prefix == "dq_" + + def test_custom_chunk_size(self, sample_validator): + """Test initialization with custom chunk size""" + pandas_validator = PandasValidator(sample_validator, chunk_size=5000) + + assert pandas_validator.chunk_size == 5000 + + def test_custom_column_prefix(self, sample_validator): + """Test initialization with custom column prefix""" + pandas_validator = PandasValidator( + sample_validator, column_prefix="validation_" + ) + + assert pandas_validator.column_prefix == "validation_" + + def test_invalid_chunk_size(self, sample_validator): + """Test initialization with invalid chunk size""" + with pytest.raises(ValueError, match="chunk_size must be positive"): + PandasValidator(sample_validator, chunk_size=0) + + with pytest.raises(ValueError, match="chunk_size must be positive"): + PandasValidator(sample_validator, chunk_size=-100) + + +class TestSummaryStatistics: + """Test summary statistics calculation""" + + def test_basic_summary(self, sample_validator, sample_dataframe): + """Test basic summary statistics""" + pandas_validator = PandasValidator(sample_validator) + summary = pandas_validator.get_summary_statistics(sample_dataframe) + + assert isinstance(summary, dict) + assert "total_rows" in summary + assert "valid_rows" in summary + assert "invalid_rows" in summary + assert "pass_rate" in summary + assert "total_checks" in summary + assert "passed_checks" in summary + assert "failed_checks" in summary + + assert summary["total_rows"] == 5 + assert summary["valid_rows"] + summary["invalid_rows"] == 5 + assert 0 <= summary["pass_rate"] <= 100 + + def test_all_valid_rows(self, sample_validator): + """Test summary with all valid rows""" + df = pd.DataFrame( + { + "id": [1, 2, 3], + "name": ["Alice", "Bob", "Charlie"], + "age": [25, 30, 35], + "status": ["active", "inactive", "active"], + } + ) + + pandas_validator = PandasValidator(sample_validator) + summary = pandas_validator.get_summary_statistics(df) + + assert summary["total_rows"] == 3 + assert summary["valid_rows"] == 3 + assert summary["invalid_rows"] == 0 + assert math.isclose(summary["pass_rate"], 100.0) + + def test_all_invalid_rows(self, sample_validator): + """Test summary with all invalid rows""" + df = pd.DataFrame( + { + "id": [1, 2, 3], + "name": ["A", "B", "C"], # All too short + "age": [15, 16, 17], # All under 18 + "status": ["pending", "deleted", "archived"], # All invalid + } + ) + + pandas_validator = PandasValidator(sample_validator) + summary = pandas_validator.get_summary_statistics(df) + + assert summary["total_rows"] == 3 + assert summary["valid_rows"] == 0 + assert summary["invalid_rows"] == 3 + assert math.isclose(summary["pass_rate"], 0.0, abs_tol=1e-9) + + def test_empty_dataframe(self, sample_validator): + """Test summary with empty DataFrame""" + df = pd.DataFrame(columns=["id", "name", "age", "status"]) + + pandas_validator = PandasValidator(sample_validator) + summary = pandas_validator.get_summary_statistics(df) + + assert summary["total_rows"] == 0 + assert summary["valid_rows"] == 0 + assert summary["invalid_rows"] == 0 + assert math.isclose(summary["pass_rate"], 0.0, abs_tol=1e-9) + + def test_chunked_processing(self, sample_validator): + """Test summary with chunked processing""" + # Create DataFrame larger than chunk size + df = pd.DataFrame( + { + "id": range(1, 101), + "name": ["Alice"] * 100, + "age": [25] * 100, + "status": ["active"] * 100, + } + ) + + pandas_validator = PandasValidator(sample_validator, chunk_size=30) + summary = pandas_validator.get_summary_statistics(df) + + assert summary["total_rows"] == 100 + assert summary["valid_rows"] == 100 + + +class TestAddValidationColumn: + """Test adding validation column to DataFrame""" + + def test_basic_validation_column(self, sample_validator, sample_dataframe): + """Test adding validation column""" + pandas_validator = PandasValidator(sample_validator) + df_validated = pandas_validator.add_validation_column(sample_dataframe) + + # Check that original columns are preserved + assert all(col in df_validated.columns for col in sample_dataframe.columns) + + # Check that validation column is added + assert "dq_validation_result" in df_validated.columns + + # Check that DataFrame has same number of rows + assert len(df_validated) == len(sample_dataframe) + + def test_validation_result_structure(self, sample_validator, sample_dataframe): + """Test structure of validation result""" + pandas_validator = PandasValidator(sample_validator) + df_validated = pandas_validator.add_validation_column(sample_dataframe) + + # Check first validation result + result = df_validated["dq_validation_result"].iloc[0] + + assert isinstance(result, dict) + assert "is_valid" in result + assert "score" in result + assert "pass_rate" in result + assert "total_checks" in result + assert "passed_checks" in result + assert "failed_checks" in result + assert "error_count" in result + assert "errors" in result + + assert isinstance(result["is_valid"], bool) + assert isinstance(result["score"], str) + assert isinstance(result["pass_rate"], float) + assert isinstance(result["errors"], list) + + def test_custom_column_prefix(self, sample_validator, sample_dataframe): + """Test validation column with custom prefix""" + pandas_validator = PandasValidator(sample_validator, column_prefix="val_") + df_validated = pandas_validator.add_validation_column(sample_dataframe) + + assert "val_validation_result" in df_validated.columns + assert "dq_validation_result" not in df_validated.columns + + def test_column_conflict_detection(self, sample_validator): + """Test detection of column name conflicts""" + df = pd.DataFrame( + { + "id": [1, 2], + "name": ["Alice", "Bob"], + "age": [25, 30], + "status": ["active", "inactive"], + "dq_validation_result": ["existing", "data"], # Conflict! + } + ) + + pandas_validator = PandasValidator(sample_validator) + + with pytest.raises(ValueError, match="already exists"): + pandas_validator.add_validation_column(df) + + +class TestInvalidRowFiltering: + """Test filtering invalid rows""" + + def test_get_invalid_rows(self, sample_validator, sample_dataframe): + """Test getting invalid rows""" + pandas_validator = PandasValidator(sample_validator) + invalid_df = pandas_validator.get_invalid_rows(sample_dataframe) + + # Should have validation column + assert "dq_validation_result" in invalid_df.columns + + # All rows should be invalid + for _, row in invalid_df.iterrows(): + assert row["dq_validation_result"]["is_valid"] is False + + def test_get_valid_rows(self, sample_validator, sample_dataframe): + """Test getting valid rows""" + pandas_validator = PandasValidator(sample_validator) + valid_df = pandas_validator.get_valid_rows(sample_dataframe) + + # Should have validation column + assert "dq_validation_result" in valid_df.columns + + # All rows should be valid + for _, row in valid_df.iterrows(): + assert row["dq_validation_result"]["is_valid"] is True + + def test_no_invalid_rows(self, sample_validator): + """Test when there are no invalid rows""" + df = pd.DataFrame( + { + "id": [1, 2, 3], + "name": ["Alice", "Bob", "Charlie"], + "age": [25, 30, 35], + "status": ["active", "inactive", "active"], + } + ) + + pandas_validator = PandasValidator(sample_validator) + invalid_df = pandas_validator.get_invalid_rows(df) + + assert len(invalid_df) == 0 + + def test_no_valid_rows(self, sample_validator): + """Test when there are no valid rows""" + df = pd.DataFrame( + { + "id": [1, 2, 3], + "name": ["A", "B", "C"], + "age": [15, 16, 17], + "status": ["pending", "deleted", "archived"], + } + ) + + pandas_validator = PandasValidator(sample_validator) + valid_df = pandas_validator.get_valid_rows(df) + + assert len(valid_df) == 0 + + +class TestColumnExpansion: + """Test expanding validation column""" + + def test_basic_expansion(self, sample_validator, sample_dataframe): + """Test basic column expansion""" + pandas_validator = PandasValidator(sample_validator) + df_validated = pandas_validator.add_validation_column(sample_dataframe) + df_expanded = pandas_validator.expand_validation_column(df_validated) + + # Check that expanded columns exist + assert "dq_is_valid" in df_expanded.columns + assert "dq_score" in df_expanded.columns + assert "dq_pass_rate" in df_expanded.columns + assert "dq_total_checks" in df_expanded.columns + assert "dq_passed_checks" in df_expanded.columns + assert "dq_failed_checks" in df_expanded.columns + assert "dq_error_count" in df_expanded.columns + assert "dq_errors" in df_expanded.columns + + # Original validation column should be removed + assert "dq_validation_result" not in df_expanded.columns + + # Original columns should be preserved + assert all(col in df_expanded.columns for col in sample_dataframe.columns) + + def test_expansion_with_custom_prefix(self, sample_validator, sample_dataframe): + """Test expansion with custom prefix""" + pandas_validator = PandasValidator(sample_validator, column_prefix="val_") + df_validated = pandas_validator.add_validation_column(sample_dataframe) + df_expanded = pandas_validator.expand_validation_column(df_validated) + + assert "val_is_valid" in df_expanded.columns + assert "val_score" in df_expanded.columns + assert "val_pass_rate" in df_expanded.columns + + def test_expansion_without_validation_column( + self, sample_validator, sample_dataframe + ): + """Test expansion when validation column doesn't exist""" + pandas_validator = PandasValidator(sample_validator) + + with pytest.raises( + ValueError, match="does not contain validation result column" + ): + pandas_validator.expand_validation_column(sample_dataframe) + + +class TestEdgeCases: + """Test edge cases and error handling""" + + def test_dataframe_with_missing_columns(self, sample_validator): + """Test DataFrame missing required columns""" + df = pd.DataFrame( + { + "id": [1, 2], + "name": ["Alice", "Bob"], + # Missing 'age' and 'status' columns + } + ) + + pandas_validator = PandasValidator(sample_validator) + + # Should raise error when trying to validate + with pytest.raises(Exception): + pandas_validator.get_summary_statistics(df) + + def test_dataframe_with_null_values(self, sample_validator): + """Test DataFrame with null values""" + df = pd.DataFrame( + { + "id": [1, 2, 3], + "name": ["Alice", None, "Charlie"], + "age": [25, 30, None], + "status": ["active", "inactive", None], + } + ) + + pandas_validator = PandasValidator(sample_validator) + summary = pandas_validator.get_summary_statistics(df) + + # Should handle nulls gracefully + assert summary["total_rows"] == 3 + assert summary["invalid_rows"] > 0 # Nulls should cause failures + + def test_large_dataframe_chunked_processing(self, sample_validator): + """Test chunked processing with large DataFrame""" + # Create large DataFrame + df = pd.DataFrame( + { + "id": range(1, 10001), + "name": ["Alice"] * 10000, + "age": [25] * 10000, + "status": ["active"] * 10000, + } + ) + + pandas_validator = PandasValidator(sample_validator, chunk_size=1000) + summary = pandas_validator.get_summary_statistics(df) + + assert summary["total_rows"] == 10000 + assert summary["valid_rows"] == 10000 + + def test_string_representation(self, sample_validator): + """Test string representation of validator""" + pandas_validator = PandasValidator(sample_validator, chunk_size=5000) + str_repr = str(pandas_validator) + + assert "PandasValidator" in str_repr + assert "5000" in str_repr + + +class TestIntegrationScenarios: + """Test complete integration scenarios""" + + def test_complete_workflow(self, sample_validator, sample_dataframe): + """Test complete validation workflow""" + pandas_validator = PandasValidator(sample_validator) + + # Step 1: Get summary + summary = pandas_validator.get_summary_statistics(sample_dataframe) + assert summary["total_rows"] == 5 + + # Step 2: Add validation column + df_validated = pandas_validator.add_validation_column(sample_dataframe) + assert "dq_validation_result" in df_validated.columns + + # Step 3: Filter invalid rows + invalid_df = pandas_validator.get_invalid_rows(sample_dataframe) + assert len(invalid_df) == summary["invalid_rows"] + + # Step 4: Expand columns + df_expanded = pandas_validator.expand_validation_column(df_validated) + assert "dq_is_valid" in df_expanded.columns + + def test_multiple_validations_same_dataframe( + self, sample_validator, sample_dataframe + ): + """Test running multiple validations on same DataFrame""" + pandas_validator = PandasValidator(sample_validator) + + # Run validation multiple times + summary1 = pandas_validator.get_summary_statistics(sample_dataframe) + summary2 = pandas_validator.get_summary_statistics(sample_dataframe) + + # Results should be consistent + assert summary1 == summary2 + + def test_validation_with_different_chunk_sizes(self, sample_validator): + """Test that different chunk sizes produce same results""" + df = pd.DataFrame( + { + "id": range(1, 101), + "name": ["Alice"] * 100, + "age": [25] * 100, + "status": ["active"] * 100, + } + ) + + validator1 = PandasValidator(sample_validator, chunk_size=10) + validator2 = PandasValidator(sample_validator, chunk_size=50) + + summary1 = validator1.get_summary_statistics(df) + summary2 = validator2.get_summary_statistics(df) + + # Results should be identical regardless of chunk size + assert summary1 == summary2 + + +if __name__ == "__main__": + pytest.main([__file__, "-v"]) diff --git a/tests/src/dq_validator/test_range_check.py b/tests/src/dq_validator/test_range_check.py new file mode 100644 index 0000000..ed6f8fb --- /dev/null +++ b/tests/src/dq_validator/test_range_check.py @@ -0,0 +1,380 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for RangeCheck +""" + +import pytest +from decimal import Decimal +from datetime import datetime, date +from wxdi.dq_validator.checks.range_check import RangeCheck +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + + +class TestRangeCheckInitialization: + """Test RangeCheck initialization and parameter validation""" + + def test_init_with_min_value_only(self): + """Test initialization with only min_value""" + check = RangeCheck(min_value=10) + assert isinstance(check.min_value, Decimal) + assert check.min_value == Decimal("10") + assert check.max_value is None + + def test_init_with_max_value_only(self): + """Test initialization with only max_value""" + check = RangeCheck(max_value=20) + assert isinstance(check.max_value, Decimal) + assert check.min_value is None + assert check.max_value == Decimal("20") + + def test_init_with_both_values(self): + """Test successful initialization with min and max""" + check = RangeCheck(min_value=10, max_value=20) + assert isinstance(check.min_value, Decimal) + assert isinstance(check.max_value, Decimal) + assert check.min_value == Decimal("10") + assert check.max_value == Decimal("20") + + def test_init_with_no_parameters_raises_error(self): + """Test that missing min or max raises ValueError""" + with pytest.raises(ValueError) as exc_info: + RangeCheck() + assert "At least one of min_value or max_value must be specified" in str( + exc_info.value + ) + + def test_init_min_greater_than_max_raises_error(self): + """Test that min_value > max_value raises ValueError""" + with pytest.raises(ValueError) as exc_info: + RangeCheck(min_value=20, max_value=10) + assert "min_value (20) cannot be greater than max_value (10)" in str( + exc_info.value + ) + + def test_init_incompatible_numeric_and_string_raises_error(self): + """Test initialization failure when min is Decimal and max is string""" + with pytest.raises(TypeError) as exc_info: + RangeCheck(min_value=10, max_value="20") + assert "Incompatible types of min_value and max_value" in str(exc_info.value) + + def test_init_incompatible_datetime_and_decimal_raises_error(self): + """Test initialization failure when min is datetime and max is Decimal""" + with pytest.raises(TypeError) as exc_info: + RangeCheck(min_value=date(2024, 1, 1), max_value=500.5) + assert "Incompatible types of min_value and max_value" in str(exc_info.value) + + def test_init_incompatible_string_and_datetime_raises_error(self): + """Test initialization failure when min is string and max is datetime""" + with pytest.raises(TypeError) as exc_info: + RangeCheck(min_value="2024-01-01", max_value=date(2024, 12, 31)) + assert "Incompatible types of min_value and max_value" in str(exc_info.value) + + def test_get_check_name(self): + """Test get_check_name returns correct name""" + check = RangeCheck(min_value=10, max_value=20) + assert check.get_check_name() == "range_check" + + def test_get_dimension(self): + """Test get_dimension returns correct dimension""" + check = RangeCheck(min_value=10, max_value=20) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + def test_set_dimension(self): + """Test set_dimension changes the dimension""" + check = RangeCheck(min_value=10, max_value=20) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + check.set_dimension(DataQualityDimension.CONSISTENCY) + assert check.get_dimension() == DataQualityDimension.CONSISTENCY + + +class TestRangeCheckNormalization: + """Test normalization logic via class initialization""" + + def test_normalize_numeric_init(self): + """Test normalization of int and float values to Decimal""" + check = RangeCheck(min_value=10, max_value=20.5) + assert isinstance(check.min_value, Decimal) + assert isinstance(check.max_value, Decimal) + assert check.min_value == Decimal("10") + assert check.max_value == Decimal("20.5") + + def test_normalize_decimal_init(self): + """Test normalization of default Decimal values to Decimal""" + check = RangeCheck(min_value=Decimal("10"), max_value=Decimal("20.5")) + assert isinstance(check.min_value, Decimal) + assert isinstance(check.max_value, Decimal) + assert check.min_value == Decimal("10") + assert check.max_value == Decimal("20.5") + + def test_normalize_date_init(self): + """Test normalization of date and default datetime objects to datetime""" + check = RangeCheck( + min_value=date(2024, 1, 1), + max_value=datetime(2024, 12, 31, 23, 59, 59, 999999), + ) + assert isinstance(check.min_value, datetime) + assert isinstance(check.max_value, datetime) + assert check.min_value == datetime(2024, 1, 1, 0, 0, 0) + assert check.max_value == datetime(2024, 12, 31, 23, 59, 59, 999999) + assert check.max_value.microsecond == 999999 + + def test_normalize_string_fallback_init(self): + """Test non-numeric and non-date inputs fallback to string during init""" + check = RangeCheck(min_value="alpha", max_value="omega") + assert isinstance(check.min_value, str) + assert isinstance(check.max_value, str) + assert check.min_value == "alpha" + assert check.max_value == "omega" + + def test_normalization_none_init(self): + """Test normalization of None inputs return None""" + check = RangeCheck(min_value="alpha") + assert isinstance(check.min_value, str) + assert check.min_value == "alpha" + assert check.max_value == None + + +class TestRangeCheckValidation: + """Test validation logic for various scenarios""" + + def test_range_passes(self): + """Test value within range passes""" + check = RangeCheck(min_value=10, max_value=20) + context = {"column_name": "score"} + assert check.validate(15, context) is None + assert check.validate(10, context) is None # Boundary inclusive + assert check.validate(20, context) is None # Boundary inclusive + + def test_less_than_min_fails(self): + """Test value below minimum fails""" + check = RangeCheck(min_value=10, max_value=20) + context = {"column_name": "score"} + result = check.validate(5, context) + assert result is not None + assert "score (5) is less than minimum (10)" in result.message + + def test_greater_than_max_fails(self): + """Test value above maximum fails""" + check = RangeCheck(min_value=10, max_value=20) + context = {"column_name": "score"} + result = check.validate(25, context) + assert result is not None + assert "score (25) is greater than maximum (20)" in result.message + + def test_none_value_fails(self): + """Test None value fails check""" + check = RangeCheck(min_value="alpha", max_value="gamma") + context = {"column_name": "title"} + result = check.validate(None, context) + assert result is not None + assert "title is None, cannot check to be within range" in result.message + + def test_range_empty_string_passes(self): + """Passes because '' is not less than the min '' and not greater than 'z'""" + check = RangeCheck(min_value="", max_value="z") + context = {"column_name": "code"} + result = check.validate("", context) + assert result is None + + def test_range_empty_string_fails(self): + """Fails because '' is lexicographically less than 'a'""" + check = RangeCheck(min_value="a", max_value="z") + context = {"column_name": "code"} + result = check.validate("", context) + assert result is not None + assert "code () is less than minimum (a)" in result.message + + def test_range_whitespace_passes(self): + """Passes because ' ' (two spaces) is 'greater' than ' ' (one space)""" + check = RangeCheck(min_value=" ", max_value="z") + context = {"column_name": "padded_val"} + result = check.validate(" ", context) + assert result is None + + def test_range_whitespace_fails(self): + """Fails because a space is lexicographically less than the character '0'""" + check = RangeCheck(min_value="0", max_value="9") + context = {"column_name": "digit_check"} + result = check.validate(" ", context) + assert result is not None + assert "digit_check ( ) is less than minimum (0)" in result.message + + +class TestRangeCheckDataTypes: + """Test range check with different data types""" + + def test_date_range_passes(self): + """Test validation passes for date value in range""" + check = RangeCheck(date(2024, 1, 1), date(2024, 1, 31)) + context = {"column_name": "created_at"} + result = check.validate(date(2024, 1, 15), context) + assert result is None + + def test_date_range_fails(self): + """Test validation fails for date value out of range""" + check = RangeCheck(date(2024, 1, 1), date(2024, 1, 31)) + context = {"column_name": "created_at"} + result = check.validate(date(2024, 2, 1), context) + assert result is not None + assert ( + "created_at (2024-02-01 00:00:00) is greater than maximum (2024-01-31 00:00:00)" + in result.message + ) + + def test_datetime_range_passes(self): + """Test validation passes for datetime value in range""" + check = RangeCheck(datetime(2024, 1, 1, 0, 0), datetime(2024, 1, 1, 23, 59)) + context = {"column_name": "timestamp"} + result = check.validate(datetime(2024, 1, 1, 12, 0), context) + assert result is None + + def test_datetime_range_fails(self): + """Test validation fails for datetime value out of range""" + check = RangeCheck(datetime(2024, 1, 1, 0, 0), datetime(2024, 1, 1, 12, 0)) + context = {"column_name": "timestamp"} + val = datetime(2024, 1, 1, 15, 0) + result = check.validate(val, context) + assert result is not None + assert ( + "timestamp (2024-01-01 15:00:00) is greater than maximum (2024-01-01 12:00:00)" + in result.message + ) + + def test_string_range_passes(self): + """Test validation passes for string value in range""" + check = RangeCheck("apple", "cherry") + context = {"column_name": "fruit"} + result = check.validate("banana", context) + assert result is None + + def test_string_range_fails(self): + """Test validation fails for string value out of range""" + check = RangeCheck("apple", "cherry") + context = {"column_name": "fruit"} + result = check.validate("date", context) + assert result is not None + assert "fruit (date) is greater than maximum (cherry)" in result.message + + def test_int_range_passes(self): + """Test validation passes for integer value in range""" + check = RangeCheck(1, 10) + context = {"column_name": "count"} + result = check.validate(5, context) + assert result is None + + def test_int_range_fails(self): + """Test validation fails for integer value out of range""" + check = RangeCheck(1, 10) + context = {"column_name": "count"} + result = check.validate(15, context) + assert result is not None + assert "count (15) is greater than maximum (10)" in result.message + + def test_float_range_passes(self): + """Test validation passes for float value in range""" + check = RangeCheck(1.0, 5.5) + context = {"column_name": "rating"} + result = check.validate(3.2, context) + assert result is None + + def test_float_range_fails(self): + """Test validation fails for float value out of range""" + check = RangeCheck(1.0, 5.5) + context = {"column_name": "rating"} + result = check.validate(0.5, context) + assert result is not None + assert "rating (0.5) is less than minimum (1.0)" in result.message + + def test_decimal_range_passes(self): + """Test validation passes for decimal value in range""" + check = RangeCheck(Decimal("10.00"), Decimal("20.00")) + context = {"column_name": "price"} + result = check.validate(Decimal("15.50"), context) + assert result is None + + def test_decimal_range_fails(self): + """Test validation fails for decimal value out of range""" + check = RangeCheck(Decimal("10.00"), Decimal("20.00")) + context = {"column_name": "price"} + result = check.validate(Decimal("25.00"), context) + assert result is not None + assert "price (25.00) is greater than maximum (20.00)" in result.message + + +class TestRangeCheckTypeMismatch: + """Test for type mismatches between values and range boundaries""" + + def test_decimal_range_with_string_value_fails(self): + """Test validation error when comparing a string value against a numeric range""" + check = RangeCheck(min_value=10, max_value=20) + context = {"column_name": "age"} + result = check.validate("25", context) + assert result is not None + assert "Cannot compare str with min_value Decimal" in result.message + + def test_datetime_range_with_decimal_value_fails(self): + """Test validation error when comparing a numeric value against a datetime range""" + check = RangeCheck(min_value=date(2024, 1, 1), max_value=date(2024, 12, 31)) + context = {"column_name": "created_at"} + result = check.validate(500, context) + assert result is not None + assert "Cannot compare Decimal with min_value datetime" in result.message + + def test_string_range_with_datetime_value_fails(self): + """Test validation error when comparing a datetime value against a string range""" + check = RangeCheck(min_value="aaa", max_value="zzz") + context = {"column_name": "code"} + result = check.validate(datetime(2024, 1, 1), context) + assert result is not None + assert "Cannot compare datetime with min_value str" in result.message + + def test_decimal_range_with_datetime_value_fails(self): + """Test validation error when comparing a datetime value against a numeric range""" + check = RangeCheck(min_value=0, max_value=100) + context = {"column_name": "score"} + result = check.validate(datetime(2025, 1, 1), context) + assert result is not None + assert "Cannot compare datetime with min_value Decimal" in result.message + + def test_datetime_range_with_string_value_fails(self): + """Test validation error when comparing a string value against a datetime range""" + check = RangeCheck(min_value=date(2024, 1, 1), max_value=date(2024, 12, 31)) + context = {"column_name": "deadline"} + result = check.validate("2024-06-01", context) + assert result is not None + assert "Cannot compare str with min_value datetime" in result.message + + def test_string_range_with_decimal_value_fails(self): + """Test validation error when comparing a numeric value against a string range""" + check = RangeCheck(min_value="A", max_value="Z") + context = {"column_name": "category_code"} + result = check.validate(10.5, context) + assert result is not None + assert "Cannot compare Decimal with min_value str" in result.message + + +class TestRangeCheckEdgeCases: + """Test edge cases for RangeCheck""" + + def test_repr(self): + """Test __repr__ output""" + check = RangeCheck(5, 15) + repr_str = repr(check) + assert "RangeCheck" in repr_str + assert "min=5" in repr_str + assert "max=15" in repr_str diff --git a/tests/src/dq_validator/test_regex_check.py b/tests/src/dq_validator/test_regex_check.py new file mode 100644 index 0000000..704758d --- /dev/null +++ b/tests/src/dq_validator/test_regex_check.py @@ -0,0 +1,280 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for RegexCheck +""" + +import pytest +import re +from wxdi.dq_validator.checks.regex_check import RegexCheck +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + +class TestRegexCheckInitialization: + """Test RegexCheck initialization and parameter validation""" + + def test_init_with_only_pattern(self): + """Test successful initialization with only a valid pattern""" + pattern = r"^[A-Z]{3}$" + check = RegexCheck(pattern = r"^[A-Z]{3}$") + assert check.pattern == pattern + assert check.case_sensitive is True + assert isinstance(check._compiled_pattern, re.Pattern) + + def test_init_with_only_case_sensitive_raises_error(self): + """Test failed initialization with only a case_sensitive""" + with pytest.raises(ValueError) as exc_info: + RegexCheck(case_sensitive=True) + assert "pattern must be a non-empty string" in str(exc_info.value) + + def test_init_with_both_parameters(self): + """Test successful initialization with only a valid pattern""" + pattern = r"^[A-Z]{3}$" + check = RegexCheck(pattern = r"^[A-Z]{3}$", case_sensitive = False) + assert check.pattern == pattern + assert check.case_sensitive is False + assert isinstance(check._compiled_pattern, re.Pattern) + + def test_init_with_no_parameter_raises_error(self): + """Test failed initialization with no parameter""" + with pytest.raises(ValueError) as exc_info: + RegexCheck() + assert "pattern must be a non-empty string" in str(exc_info.value) + + def test_init_empty_string_raises_error(self): + """Test that empty string pattern raises ValueError""" + with pytest.raises(ValueError) as exc_info: + RegexCheck(pattern="" , case_sensitive = True) + assert "pattern must be a non-empty string" in str(exc_info.value) + + def test_init_invalid_type_raises_error(self): + """Test that non-string pattern raises ValueError""" + with pytest.raises(ValueError) as exc_info: + RegexCheck(pattern=123) + assert "pattern must be a non-empty string" in str(exc_info.value) + + def test_init_invalid_regex_syntax_raises_error(self): + """Test that syntactically incorrect regex raises ValueError""" + invalid_pattern = "[0-9" # Missing closing bracket + with pytest.raises(ValueError) as exc_info: + RegexCheck(pattern=invalid_pattern) + assert f"Invalid regex pattern '{invalid_pattern}'" in str(exc_info.value) + + def test_get_check_name(self): + """Test get_check_name returns correct name""" + check = RegexCheck(pattern=r"^[A-Z]{3}$") + assert check.get_check_name() == "regex_check" + + def test_get_dimension(self): + """Test get_dimension returns correct dimension""" + check = RegexCheck(pattern=r"^[A-Z]{3}$") + assert check.get_dimension() == DataQualityDimension.VALIDITY + + def test_set_dimension(self): + """Test set_dimension changes the dimension""" + check = RegexCheck(pattern=r"^[A-Z]{3}$") + assert check.get_dimension() == DataQualityDimension.VALIDITY + + check.set_dimension(DataQualityDimension.CONSISTENCY) + assert check.get_dimension() == DataQualityDimension.CONSISTENCY + + +class TestRegexCheckValidation: + """Test validation logic for regex matching""" + + def test_regex_match_passes(self): + """Test that value matching the pattern passes""" + # Pattern for a simple email-like structure + check = RegexCheck(pattern=r"^\w+@\w+\.com$") + context = {'column_name': 'email'} + result = check.validate("user@test.com", context) + assert result is None + + def test_regex_no_match_fails(self): + """Test that value not matching the pattern returns ValidationError""" + check = RegexCheck(pattern=r"^\d{3}-\d{2}$") # Format: 000-00 + context = {'column_name': 'code'} + val = "123-A" + result = check.validate(val, context) + assert result is not None + assert r"code ('123-A') does not match pattern '^\d{3}-\d{2}$'" in result.message + + def test_regex_partial_match_passes(self): + """Test that .search() allows partial matches unless anchored""" + # Pattern is not anchored with ^ or $ + check = RegexCheck(pattern=r"cat") + context = {'column_name': 'sentence'} + result = check.validate("the category", context) + assert result is None + + def test_regex_partial_match_fails(self): + """Test that .search() fails when the pattern is nowhere in the string""" + check = RegexCheck(pattern=r"cat") + context = {'column_name': 'sentence'} + result = check.validate("the dog runs", context) + assert result is not None + assert "sentence ('the dog runs') does not match pattern 'cat'" in result.message + + def test_case_sensitive_match_passes(self): + """Test passes when case_sensitive is True (default)""" + check = RegexCheck(pattern=r"^abc$") + context = {'column_name': 'test'} + result = check.validate("abc", context) + assert result is None + + def test_case_sensitive_match_fails(self): + """Test fails when case_sensitive is True (default)""" + check = RegexCheck(pattern=r"^abc$") + context = {'column_name': 'test'} + result = check.validate("ABC", context) + assert result is not None + assert "test ('ABC') does not match pattern '^abc$'" in result.message + + def test_case_insensitive_match_passes(self): + """Test passes when case_sensitive is False""" + check = RegexCheck(pattern=r"^abc$", case_sensitive=False) + context = {'column_name': 'test'} + result = check.validate("abc", context) + assert result is None + result = check.validate("ABC", context) + assert result is None + result = check.validate("aBc", context) + assert result is None + + def test_case_insensitive_match_fails(self): + """Test fails when case_sensitive is False""" + check = RegexCheck(pattern=r"^abc$", case_sensitive=False) + context = {'column_name': 'test'} + result = check.validate("XabcY", context) + assert result is not None + assert "test ('XabcY') does not match pattern '^abc$'" in result.message + + def test_empty_string_passes(self): + """Test that an empty string passes if the pattern allows it (e.g., zero or more)""" + check = RegexCheck(pattern=r"^\d*$") + context = {'column_name': 'optional_id'} + result = check.validate("", context) + assert result is None + + def test_empty_string_fails(self): + """Test that an empty string fails if the pattern requires characters""" + check = RegexCheck(pattern=r"^\d+$") + context = {'column_name': 'required_id'} + result = check.validate("", context) + assert result is not None + assert "does not match pattern" in result.message + + def test_whitespace_string_passes(self): + """Test that whitespace passes if the pattern allows it (Note: .strip() makes this '')""" + check = RegexCheck(pattern=r"^$") + context = {'column_name': 'blank_space'} + result = check.validate(" ", context) + assert result is None + + def test_whitespace_string_fails(self): + """Test that whitespace fails if the pattern expects actual content""" + check = RegexCheck(pattern=r"^\w+$") + context = {'column_name': 'username'} + result = check.validate(" ", context) + assert result is not None + assert "('') does not match pattern" in result.message + + def test_regex_none_value_fails(self): + """Test that None value matching the pattern fails""" + check = RegexCheck(pattern=r"^\w+@\w+\.com$") + context = {'column_name': 'email'} + result = check.validate(None, context) + assert result is not None + assert "email is None, cannot perform regex check" in result.message + + +class TestRegexCheckValueDataTypes: + """Test validation of different data types for the value parameter""" + + def test_validate_integer_passes(self): + """Matches when the integer string representation fits the pattern""" + check = RegexCheck(pattern=r"^\d{3}$") + result = check.validate(123, {'column_name': 'id'}) + assert result is None + + def test_validate_integer_fails(self): + """Fails when the integer string representation doesn't fit (too many digits)""" + check = RegexCheck(pattern=r"^\d{2}$") + result = check.validate(123, {'column_name': 'id'}) + assert result is not None + assert "does not match pattern" in result.message + + def test_validate_float_passes(self): + """Matches a decimal string representation""" + check = RegexCheck(pattern=r"^\d+\.\d+$") + result = check.validate(99.99, {'column_name': 'price'}) + assert result is None + + def test_validate_float_fails(self): + """Fails when float contains unexpected characters (like the dot) for a digit-only pattern""" + check = RegexCheck(pattern=r"^\d+$") + result = check.validate(99.9, {'column_name': 'price'}) + assert result is not None + + def test_validate_boolean_passes(self): + """Matches 'True' or 'False' (Python's capitalized string format)""" + check = RegexCheck(pattern=r"^(True|False)$") + result = check.validate(True, {'column_name': 'active'}) + assert result is None + + def test_validate_boolean_fails(self): + check = RegexCheck(pattern=r"^true$", case_sensitive=True) + result = check.validate(True, {'column_name': 'active'}) + assert result is not None + + def test_validate_list_passes(self): + """Matches the literal string representation of a list""" + check = RegexCheck(pattern=r"^\[.*\]$") + result = check.validate([1, 2], {'column_name': 'tags'}) + assert result is None + + def test_validate_list_fails(self): + """Fails when list contents don't match the specific pattern""" + check = RegexCheck(pattern=r"^\[\d\]$") + result = check.validate([1, 2], {'column_name': 'tags'}) + assert result is not None + + +class TestRegexCheckEdgeCases: + """Test edge cases and special regex scenarios""" + + def test_regex_with_special_characters(self): + """Test regex containing special characters and escape sequences""" + check = RegexCheck(pattern=r"^\$ \d+\.\d{2}$") # Format: $ 10.00 + context = {'column_name': 'price'} + result = check.validate("$ 10.99", context) + assert result is None + result = check.validate("10.99", context) + assert result is not None + + def test_validate_none_custom_message(self): + """Verify the specific error message for None inputs""" + check = RegexCheck(pattern=r"^None$") + result = check.validate(None, {'column_name': 'val'}) + assert "is None, cannot perform regex check" in result.message + + + def test_repr(self): + """Test __repr__ output""" + check = RegexCheck(pattern=r"\d+", case_sensitive=False) + repr_str = repr(check) + assert "RegexCheck" in repr_str + assert "pattern='\\d+'" in repr_str + assert "case_sensitive=False" in repr_str \ No newline at end of file diff --git a/tests/src/dq_validator/test_result_consolidator.py b/tests/src/dq_validator/test_result_consolidator.py new file mode 100644 index 0000000..e6d11d9 --- /dev/null +++ b/tests/src/dq_validator/test_result_consolidator.py @@ -0,0 +1,471 @@ +# Copyright 2026 IBM Corporation +# Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# See the LICENSE file in the project root for license information. + +""" +Tests for ValidationResultConsolidated utility. +""" + +import pytest +from wxdi.dq_validator.result import ValidationResult +from wxdi.dq_validator.result_consolidator import ValidationResultConsolidated +from wxdi.dq_validator.base import ValidationError +from wxdi.dq_validator import Validator, ValidationRule, AssetMetadata, ColumnMetadata +from wxdi.dq_validator.checks import LengthCheck, CompletenessCheck +from wxdi.dq_validator.metadata import DataType + + +@pytest.fixture +def sample_validator(): + """Create a sample validator for testing""" + metadata = AssetMetadata('test_table', [ + ColumnMetadata('email', DataType.STRING), + ColumnMetadata('name', DataType.STRING) + ]) + validator = Validator(metadata) + # Add multiple checks to each column for comprehensive testing + validator.add_rule(ValidationRule('email').add_check(CompletenessCheck()).add_check(LengthCheck(min_length=5))) + validator.add_rule(ValidationRule('name').add_check(CompletenessCheck()).add_check(LengthCheck(min_length=2))) + return validator + + +class TestValidationResultConsolidated: + """Test ValidationResultConsolidated class""" + + def test_init_with_error_storage(self, sample_validator): + """Test initialization with error storage enabled""" + consolidator = ValidationResultConsolidated(sample_validator, store_errors=True) + assert consolidator.store_errors is True + assert consolidator.total_records == 0 + assert consolidator.valid_records == 0 + assert consolidator.invalid_records == 0 + + def test_init_without_error_storage(self, sample_validator): + """Test initialization with error storage disabled""" + consolidator = ValidationResultConsolidated(sample_validator, store_errors=False) + assert consolidator.store_errors is False + assert consolidator.total_records == 0 + + def test_add_valid_result(self, sample_validator): + """Test adding a valid result""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["test@example.com"], 0) + result.total_checks = 2 + result.passed_checks = 2 + result.failed_checks = 0 + + consolidator.add_result(result) + + assert consolidator.total_records == 1 + assert consolidator.valid_records == 1 + assert consolidator.invalid_records == 0 + + def test_add_invalid_result(self, sample_validator): + """Test adding an invalid result""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["invalid"], 0) + result.total_checks = 2 + result.passed_checks = 1 + result.failed_checks = 1 + + error = ValidationError( + column_name="email", + check_name="completeness_check", + message="Invalid email format", + value="invalid", + expected="valid email" + ) + result.add_error(error) + + consolidator.add_result(result) + + assert consolidator.total_records == 1 + assert consolidator.valid_records == 0 + assert consolidator.invalid_records == 1 + + def test_add_multiple_results(self, sample_validator): + """Test adding multiple results""" + consolidator = ValidationResultConsolidated(sample_validator) + + # Valid result + result1 = ValidationResult(["test@example.com"], 0) + result1.total_checks = 2 + result1.passed_checks = 2 + + # Invalid result + result2 = ValidationResult(["invalid"], 1) + result2.total_checks = 2 + result2.passed_checks = 1 + error = ValidationError("email", "completeness_check", "Invalid", "invalid") + result2.add_error(error) + + consolidator.add_results([result1, result2]) + + assert consolidator.total_records == 2 + assert consolidator.valid_records == 1 + assert consolidator.invalid_records == 1 + + def test_get_overall_statistics(self, sample_validator): + """Test getting overall statistics""" + consolidator = ValidationResultConsolidated(sample_validator) + + # Add some results + for i in range(10): + result = ValidationResult([f"test{i}@example.com"], i) + result.total_checks = 2 + if i < 7: # 7 valid, 3 invalid + result.passed_checks = 2 + else: + result.passed_checks = 1 + error = ValidationError("email", "completeness_check", "Invalid", f"test{i}") + result.add_error(error) + consolidator.add_result(result) + + stats = consolidator.get_overall_statistics() + + assert stats['total_records'] == 10 + assert stats['valid_records'] == 7 + assert stats['invalid_records'] == 3 + assert stats['pass_rate'] == 70.0 + assert stats['total_errors'] == 3 + + def test_get_column_statistics_single(self, sample_validator): + """Test getting statistics for a single column""" + consolidator = ValidationResultConsolidated(sample_validator) + + # Add results with errors on 'email' column + for i in range(5): + result = ValidationResult([f"test{i}"], i) + result.total_checks = 2 + result.passed_checks = 1 + error = ValidationError("email", "completeness_check", "Invalid", f"test{i}") + result.add_error(error) + consolidator.add_result(result) + + stats = consolidator.get_column_statistics('email') + + # Email has 2 checks (completeness_check and length_check), so 5 records * 2 checks = 10 total + assert stats['failed'] == 5 # 5 completeness_check failures + assert stats['passed'] == 5 # 5 length_check passes + assert stats['total'] == 10 # 5 records * 2 checks + + def test_get_column_statistics_all(self, sample_validator): + """Test getting statistics for all columns""" + consolidator = ValidationResultConsolidated(sample_validator) + + # Add errors for different columns + result1 = ValidationResult(["test"], 0) + result1.total_checks = 3 + result1.passed_checks = 1 + result1.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + result1.add_error(ValidationError("name", "length_check", "Too short", "a")) + + consolidator.add_result(result1) + + stats = consolidator.get_column_statistics() + + assert 'email' in stats + assert 'name' in stats + assert stats['email']['failed'] == 1 + assert stats['name']['failed'] == 1 + + def test_get_check_statistics_single(self, sample_validator): + """Test getting statistics for a single check type""" + consolidator = ValidationResultConsolidated(sample_validator) + + # Add results with completeness_check errors + for i in range(3): + result = ValidationResult([f"test{i}"], i) + result.total_checks = 2 + result.passed_checks = 1 + error = ValidationError("email", "completeness_check", "Invalid", f"test{i}") + result.add_error(error) + consolidator.add_result(result) + + stats = consolidator.get_check_statistics('completeness_check') + + # completeness_check is on both email and name columns, so 3 records * 2 columns = 6 total + assert stats['failed'] == 3 # 3 email completeness failures + assert stats['passed'] == 3 # 3 name completeness passes + assert stats['total'] == 6 # 3 records * 2 columns + + def test_get_check_statistics_all(self, sample_validator): + """Test getting statistics for all check types""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["test"], 0) + result.total_checks = 3 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + result.add_error(ValidationError("name", "length_check", "Too short", "a")) + + consolidator.add_result(result) + + stats = consolidator.get_check_statistics() + + assert 'completeness_check' in stats + assert 'length_check' in stats + assert stats['completeness_check']['failed'] == 1 + assert stats['length_check']['failed'] == 1 + + def test_get_combined_statistics_both_specified(self, sample_validator): + """Test getting combined statistics with both column and check specified""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["test"], 0) + result.total_checks = 2 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + + consolidator.add_result(result) + + stats = consolidator.get_combined_statistics('email', 'completeness_check') + + assert stats['failed'] == 1 + assert stats['total'] == 1 + + def test_get_combined_statistics_column_only(self, sample_validator): + """Test getting combined statistics with only column specified""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["test"], 0) + result.total_checks = 3 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + result.add_error(ValidationError("email", "length_check", "Too short", "a")) + + consolidator.add_result(result) + + stats = consolidator.get_combined_statistics(column_name='email') + + assert 'completeness_check' in stats + assert 'length_check' in stats + assert stats['completeness_check']['failed'] == 1 + assert stats['length_check']['failed'] == 1 + + def test_get_combined_statistics_check_only(self, sample_validator): + """Test getting combined statistics with only check specified""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["test"], 0) + result.total_checks = 3 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + result.add_error(ValidationError("name", "completeness_check", "Invalid", "123")) + + consolidator.add_result(result) + + stats = consolidator.get_combined_statistics(check_name='completeness_check') + + assert 'email' in stats + assert 'name' in stats + assert stats['email']['failed'] == 1 + assert stats['name']['failed'] == 1 + + def test_get_combined_statistics_all(self, sample_validator): + """Test getting all combined statistics""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["test"], 0) + result.total_checks = 3 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + result.add_error(ValidationError("name", "length_check", "Too short", "a")) + + consolidator.add_result(result) + + stats = consolidator.get_combined_statistics() + + assert 'email' in stats + assert 'name' in stats + assert 'completeness_check' in stats['email'] + assert 'length_check' in stats['name'] + + def test_get_errors_by_column_with_storage(self, sample_validator): + """Test getting errors by column when storage is enabled""" + consolidator = ValidationResultConsolidated(sample_validator, store_errors=True) + + result = ValidationResult(["test"], 0) + result.total_checks = 2 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + + consolidator.add_result(result) + + errors = consolidator.get_errors_by_column('email') + + assert len(errors) == 1 + assert errors[0]['column'] == 'email' + assert errors[0]['check'] == 'completeness_check' + assert errors[0]['record_index'] == 0 + + def test_get_errors_by_column_without_storage(self, sample_validator): + """Test getting errors by column when storage is disabled""" + consolidator = ValidationResultConsolidated(sample_validator, store_errors=False) + + result = ValidationResult(["test"], 0) + result.total_checks = 2 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + + consolidator.add_result(result) + + with pytest.raises(RuntimeError, match="Error details not available"): + consolidator.get_errors_by_column('email') + + def test_get_errors_by_check_with_storage(self, sample_validator): + """Test getting errors by check when storage is enabled""" + consolidator = ValidationResultConsolidated(sample_validator, store_errors=True) + + result = ValidationResult(["test"], 0) + result.total_checks = 2 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + + consolidator.add_result(result) + + errors = consolidator.get_errors_by_check('completeness_check') + + assert len(errors) == 1 + assert errors[0]['check'] == 'completeness_check' + + def test_get_errors_by_check_without_storage(self, sample_validator): + """Test getting errors by check when storage is disabled""" + consolidator = ValidationResultConsolidated(sample_validator, store_errors=False) + + with pytest.raises(RuntimeError, match="Error details not available"): + consolidator.get_errors_by_check('completeness_check') + + def test_get_errors_by_column_and_check(self, sample_validator): + """Test getting errors by column and check combination""" + consolidator = ValidationResultConsolidated(sample_validator, store_errors=True) + + result = ValidationResult(["test"], 0) + result.total_checks = 3 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + result.add_error(ValidationError("email", "length_check", "Too short", "a")) + + consolidator.add_result(result) + + errors = consolidator.get_errors_by_column_and_check('email', 'completeness_check') + + assert len(errors) == 1 + assert errors[0]['column'] == 'email' + assert errors[0]['check'] == 'completeness_check' + + def test_get_all_errors_with_storage(self, sample_validator): + """Test getting all errors when storage is enabled""" + consolidator = ValidationResultConsolidated(sample_validator, store_errors=True) + + result = ValidationResult(["test"], 0) + result.total_checks = 3 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + result.add_error(ValidationError("name", "length_check", "Too short", "a")) + + consolidator.add_result(result) + + errors = consolidator.get_all_errors() + + assert len(errors) == 2 + + def test_get_all_errors_without_storage(self, sample_validator): + """Test getting all errors when storage is disabled""" + consolidator = ValidationResultConsolidated(sample_validator, store_errors=False) + + with pytest.raises(RuntimeError, match="Error details not available"): + consolidator.get_all_errors() + + def test_get_columns(self, sample_validator): + """Test getting list of validated columns""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["test"], 0) + result.total_checks = 3 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + result.add_error(ValidationError("name", "length_check", "Too short", "a")) + + consolidator.add_result(result) + + columns = consolidator.get_columns() + + assert 'email' in columns + assert 'name' in columns + assert len(columns) == 2 + + def test_get_checks(self, sample_validator): + """Test getting list of executed checks""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["test"], 0) + result.total_checks = 3 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + result.add_error(ValidationError("name", "length_check", "Too short", "a")) + + consolidator.add_result(result) + + checks = consolidator.get_checks() + + assert 'completeness_check' in checks + assert 'length_check' in checks + assert len(checks) == 2 + + def test_to_dict(self, sample_validator): + """Test converting consolidator to dictionary""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["test"], 0) + result.total_checks = 2 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + + consolidator.add_result(result) + + data = consolidator.to_dict() + + assert 'overall' in data + assert 'by_column' in data + assert 'by_check' in data + assert 'combined' in data + assert 'columns' in data + assert 'checks' in data + assert 'error_count' in data + + def test_repr(self, sample_validator): + """Test string representation""" + consolidator = ValidationResultConsolidated(sample_validator) + + result1 = ValidationResult(["test"], 0) + result1.total_checks = 2 + result1.passed_checks = 2 + + result2 = ValidationResult(["invalid"], 1) + result2.total_checks = 2 + result2.passed_checks = 1 + result2.add_error(ValidationError("email", "completeness_check", "Invalid", "invalid")) + + consolidator.add_results([result1, result2]) + + repr_str = repr(consolidator) + + assert 'ValidationResultConsolidated' in repr_str + assert 'records=2' in repr_str + assert 'valid=1' in repr_str + assert 'invalid=1' in repr_str + assert 'errors=1' in repr_str \ No newline at end of file diff --git a/tests/src/dq_validator/test_rule_loader.py b/tests/src/dq_validator/test_rule_loader.py new file mode 100644 index 0000000..94ab8e2 --- /dev/null +++ b/tests/src/dq_validator/test_rule_loader.py @@ -0,0 +1,559 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Test suite for RuleLoader module +""" + +import pytest +import json +from pathlib import Path +from wxdi.dq_validator.rule_loader import RuleLoader +from wxdi.dq_validator.metadata import AssetMetadata, ColumnMetadata, DataType +from wxdi.dq_validator.validator import Validator +from wxdi.dq_validator.provider.response_model import ( + GlossaryTerm, + Metadata, + GlossaryTermEntity, + ExtendedAttributeGroups, +) +from wxdi.dq_validator.provider.constraint_model import ( + DataQualityConstraint, + ConstraintMetadata, + CheckConstraint, + CheckType, +) +from wxdi.dq_validator.provider.data_asset_model import DataAsset +from datetime import datetime + + +class TestRuleLoader: + """Test cases for RuleLoader class""" + + @pytest.fixture + def base_url(self): + """Base URL for API""" + return "https://api.example.com" + + @pytest.fixture + def auth_token(self): + """Authentication token""" + return "test-token-12345" + + @pytest.fixture + def rule_loader(self, base_url, auth_token): + """Create RuleLoader instance""" + return RuleLoader(base_url, auth_token) + + @pytest.fixture + def asset_metadata(self): + """Create sample asset metadata""" + columns = [ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("name", DataType.STRING, length=100), + ColumnMetadata("age", DataType.INTEGER), + ] + return AssetMetadata("users", columns) + + @pytest.fixture + def mock_glossary_term_basic(self): + """Create a basic glossary term without DQ constraints""" + return GlossaryTerm( + metadata=Metadata( + artifact_type="glossary_term", + artifact_id="term-123", + version_id="version-456", + source_repository_id="repo-789", + global_id="global-123", + effective_start_date=datetime(2026, 1, 1), + created_by="user1", + created_at=datetime(2026, 1, 1), + modified_by="user1", + modified_at=datetime(2026, 1, 1), + revision="0", + state="PUBLISHED", + ), + entity=GlossaryTermEntity( + abbreviations=[], + state="PUBLISHED", + default_locale_id="en-US", + reference_copy=False, + steward_ids=[], + steward_group_ids=[], + custom_relationships=[], + custom_attributes=[], + ), + ) + + @pytest.fixture + def mock_glossary_term_with_constraints(self): + """Create a glossary term with DQ constraints""" + # Create completeness constraint + completeness_constraint = DataQualityConstraint( + metadata=ConstraintMetadata(type=CheckType.COMPLETENESS), + origin=[], + check=[CheckConstraint(name="missing_values_allowed", boolean_value=False)], + ) + + # Create range constraint + range_constraint = DataQualityConstraint( + metadata=ConstraintMetadata(type=CheckType.RANGE), + origin=[], + check=[ + CheckConstraint(name="min", numeric_value=0), + CheckConstraint(name="max", numeric_value=120), + ], + ) + + # Create data type constraint + datatype_constraint = DataQualityConstraint( + metadata=ConstraintMetadata(type=CheckType.DATA_TYPE), + origin=[], + check=[ + CheckConstraint(name="data_type", value="integer"), + CheckConstraint(name="length", numeric_value=10), + ], + ) + + return GlossaryTerm( + metadata=Metadata( + artifact_type="glossary_term", + artifact_id="term-123", + version_id="version-456", + source_repository_id="repo-789", + global_id="global-123", + effective_start_date=datetime(2026, 1, 1), + created_by="user1", + created_at=datetime(2026, 1, 1), + modified_by="user1", + modified_at=datetime(2026, 1, 1), + revision="0", + state="PUBLISHED", + ), + entity=GlossaryTermEntity( + abbreviations=[], + state="PUBLISHED", + default_locale_id="en-US", + reference_copy=False, + steward_ids=[], + steward_group_ids=[], + custom_relationships=[], + custom_attributes=[], + extended_attribute_groups=ExtendedAttributeGroups( + dq_constraints=[ + completeness_constraint, + range_constraint, + datatype_constraint, + ] + ), + ), + ) + + def test_rule_loader_initialization(self, base_url, auth_token): + """Test RuleLoader initialization""" + loader = RuleLoader(base_url, auth_token) + assert loader.base_url == base_url + assert loader.auth_token == auth_token + + def test_load_from_glossary_no_version_id( + self, rule_loader, asset_metadata, mocker + ): + """Test load_from_glossary raises ValueError when no version_id""" + # Mock the provider to return a term without version_id + mock_term = mocker.Mock() + mock_term.metadata.version_id = None + + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.GlossaryProvider") + mock_provider.return_value.get_published_artifact_by_id.return_value = mock_term + + with pytest.raises(ValueError, match="No version_id found"): + rule_loader.load_from_glossary("term-123", "age", asset_metadata) + + def test_load_from_glossary_no_constraints( + self, rule_loader, asset_metadata, mock_glossary_term_basic, mocker + ): + """Test load_from_glossary with no DQ constraints""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.GlossaryProvider") + mock_provider.return_value.get_published_artifact_by_id.return_value = ( + mock_glossary_term_basic + ) + mock_provider.return_value.get_term_by_version_id.return_value = ( + mock_glossary_term_basic + ) + + validator = rule_loader.load_from_glossary("term-123", "age", asset_metadata) + + assert isinstance(validator, Validator) + assert validator.metadata == asset_metadata + assert len(validator.rules) == 0 + + def test_load_from_glossary_with_constraints( + self, + rule_loader, + asset_metadata, + mock_glossary_term_basic, + mock_glossary_term_with_constraints, + mocker, + ): + """Test load_from_glossary with DQ constraints""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.GlossaryProvider") + mock_provider.return_value.get_published_artifact_by_id.return_value = ( + mock_glossary_term_basic + ) + mock_provider.return_value.get_term_by_version_id.return_value = ( + mock_glossary_term_with_constraints + ) + + validator = rule_loader.load_from_glossary("term-123", "age", asset_metadata) + + assert isinstance(validator, Validator) + assert validator.metadata == asset_metadata + assert len(validator.rules) == 1 + + rule = validator.rules[0] + assert rule.column_name == "age" + assert len(rule.checks) == 3 + + def test_load_from_glossary_provider_config( + self, rule_loader, asset_metadata, mock_glossary_term_basic, mocker + ): + """Test that GlossaryProvider is initialized with correct config""" + mock_provider_class = mocker.patch("wxdi.dq_validator.rule_loader.GlossaryProvider") + mock_provider_config = mocker.patch("wxdi.dq_validator.rule_loader.ProviderConfig") + + mock_provider_instance = mocker.Mock() + mock_provider_instance.get_published_artifact_by_id.return_value = ( + mock_glossary_term_basic + ) + mock_provider_instance.get_term_by_version_id.return_value = ( + mock_glossary_term_basic + ) + mock_provider_class.return_value = mock_provider_instance + + rule_loader.load_from_glossary("term-123", "age", asset_metadata) + + # Verify ProviderConfig was called with correct parameters + mock_provider_config.assert_called_once_with( + rule_loader.base_url, rule_loader.auth_token + ) + + # Verify GlossaryProvider was initialized with the config + mock_provider_class.assert_called_once() + + def test_load_from_glossary_api_calls( + self, + rule_loader, + asset_metadata, + mock_glossary_term_basic, + mock_glossary_term_with_constraints, + mocker, + ): + """Test that correct API calls are made""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.GlossaryProvider") + mock_instance = mock_provider.return_value + + mock_instance.get_published_artifact_by_id.return_value = ( + mock_glossary_term_basic + ) + mock_instance.get_term_by_version_id.return_value = ( + mock_glossary_term_with_constraints + ) + + rule_loader.load_from_glossary("term-123", "age", asset_metadata) + + # Verify get_published_artifact_by_id was called + mock_instance.get_published_artifact_by_id.assert_called_once_with("term-123") + + # Verify get_term_by_version_id was called with correct parameters + mock_instance.get_term_by_version_id.assert_called_once_with( + "term-123", + "version-456", + {"included_extended_attribute_groups": "dq_constraints"}, + ) + + def test_load_from_glossary_constraint_filtering( + self, + rule_loader, + asset_metadata, + mock_glossary_term_basic, + mocker, + ): + """Test that None checks are filtered out""" + # Create a constraint with unsupported type that returns None from to_check() + unsupported_constraint = DataQualityConstraint( + metadata=ConstraintMetadata(type=CheckType.UNIQUENESS), # Not supported + origin=[], + check=[], + ) + + mock_term_with_none = GlossaryTerm( + metadata=mock_glossary_term_basic.metadata, + entity=GlossaryTermEntity( + abbreviations=[], + state="PUBLISHED", + default_locale_id="en-US", + reference_copy=False, + steward_ids=[], + steward_group_ids=[], + custom_relationships=[], + custom_attributes=[], + extended_attribute_groups=ExtendedAttributeGroups( + dq_constraints=[unsupported_constraint] + ), + ), + ) + + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.GlossaryProvider") + mock_provider.return_value.get_published_artifact_by_id.return_value = ( + mock_glossary_term_basic + ) + mock_provider.return_value.get_term_by_version_id.return_value = ( + mock_term_with_none + ) + + validator = rule_loader.load_from_glossary("term-123", "age", asset_metadata) + + # Should have no rules since the constraint returned None + assert len(validator.rules) == 0 + + +class TestRuleLoaderCams: + """Test cases for RuleLoader.load_from_cams method""" + + @pytest.fixture + def base_url(self): + """Base URL for API""" + return "https://api.example.com" + + @pytest.fixture + def auth_token(self): + """Authentication token""" + return "test-token-12345" + + @pytest.fixture + def project_id(self): + """Project ID""" + return "72d21c1d-499b-4784-a3c7-6f84507f9a20" + + @pytest.fixture + def rule_loader(self, base_url, auth_token): + """Create RuleLoader instance""" + return RuleLoader(base_url, auth_token) + + @pytest.fixture + def data_asset_json(self): + """Load the data asset response JSON""" + json_path = Path(__file__).parent.parent.parent / "data" / "data_asset_response.json" + with open(json_path, "r") as f: + return json.load(f) + + @pytest.fixture + def mock_data_asset(self, data_asset_json): + """Create DataAsset from JSON""" + return DataAsset.from_dict(data_asset_json) + + def test_load_from_cams_with_auto_metadata( + self, rule_loader, project_id, mock_data_asset, mocker + ): + """Test load_from_cams with automatic metadata extraction""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.CamsProvider") + mock_provider.return_value.get_asset_by_id.return_value = mock_data_asset + + validator = rule_loader.load_from_cams( + "6862f3ba-81f5-4122-8286-62bb4c5d6543", project_id + ) + + assert isinstance(validator, Validator) + assert validator.metadata.table_name == "DEPARTMENT" + assert len(validator.metadata.columns) == 5 + + # Check column names + column_names = [col.name for col in validator.metadata.columns] + assert "DEPTNO" in column_names + assert "DEPTNAME" in column_names + assert "MGRNO" in column_names + assert "ADMRDEPT" in column_names + assert "LOCATION" in column_names + + def test_load_from_cams_with_provided_metadata( + self, rule_loader, project_id, mock_data_asset, mocker + ): + """Test load_from_cams with user-provided metadata""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.CamsProvider") + mock_provider.return_value.get_asset_by_id.return_value = mock_data_asset + + # Provide custom metadata + custom_metadata = AssetMetadata( + "CUSTOM_TABLE", + [ + ColumnMetadata("DEPTNO", DataType.STRING, length=3), + ColumnMetadata("DEPTNAME", DataType.STRING, length=36), + ], + ) + + validator = rule_loader.load_from_cams( + "6862f3ba-81f5-4122-8286-62bb4c5d6543", project_id, custom_metadata + ) + + assert isinstance(validator, Validator) + assert validator.metadata.table_name == "CUSTOM_TABLE" + assert len(validator.metadata.columns) == 2 + + def test_load_from_cams_rules_created( + self, rule_loader, project_id, mock_data_asset, mocker + ): + """Test that validation rules are created from column checks""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.CamsProvider") + mock_provider.return_value.get_asset_by_id.return_value = mock_data_asset + + validator = rule_loader.load_from_cams( + "6862f3ba-81f5-4122-8286-62bb4c5d6543", project_id + ) + + # Should have rules for columns with checks + # DEPTNO has 4 checks, MGRNO has 3, ADMRDEPT has 3, DEPTNAME has 3 + # LOCATION has 0 checks + assert len(validator.rules) == 4 + + # Check that rules are for the correct columns + rule_columns = [rule.column_name for rule in validator.rules] + assert "DEPTNO" in rule_columns + assert "MGRNO" in rule_columns + assert "ADMRDEPT" in rule_columns + assert "DEPTNAME" in rule_columns + assert "LOCATION" not in rule_columns # No checks for this column + + def test_load_from_cams_check_types( + self, rule_loader, project_id, mock_data_asset, mocker + ): + """Test that different check types are correctly loaded""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.CamsProvider") + mock_provider.return_value.get_asset_by_id.return_value = mock_data_asset + + validator = rule_loader.load_from_cams( + "6862f3ba-81f5-4122-8286-62bb4c5d6543", project_id + ) + + # Find DEPTNO rule (has format, uniqueness, completeness, data_class checks) + deptno_rule = next(r for r in validator.rules if r.column_name == "DEPTNO") + # Note: uniqueness and data_class checks return None, so only 2 checks should be added + assert len(deptno_rule.checks) == 2 + + # Find DEPTNAME rule (has uniqueness, completeness, case checks) + deptname_rule = next( + r for r in validator.rules if r.column_name == "DEPTNAME" + ) + # Note: uniqueness check returns None, so only 2 checks should be added + assert len(deptname_rule.checks) == 2 + + def test_load_from_cams_provider_config( + self, rule_loader, project_id, mock_data_asset, mocker + ): + """Test that CamsProvider is initialized with correct config""" + mock_provider_class = mocker.patch("wxdi.dq_validator.rule_loader.CamsProvider") + mock_provider_config = mocker.patch("wxdi.dq_validator.rule_loader.ProviderConfig") + + mock_provider_instance = mocker.Mock() + mock_provider_instance.get_asset_by_id.return_value = mock_data_asset + mock_provider_class.return_value = mock_provider_instance + + rule_loader.load_from_cams( + "6862f3ba-81f5-4122-8286-62bb4c5d6543", project_id + ) + + # Verify ProviderConfig was called with correct parameters + mock_provider_config.assert_called_once_with( + rule_loader.base_url, rule_loader.auth_token, project_id=project_id + ) + + def test_load_from_cams_column_metadata_extraction( + self, rule_loader, project_id, mock_data_asset, mocker + ): + """Test that column metadata is correctly extracted from data asset""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.CamsProvider") + mock_provider.return_value.get_asset_by_id.return_value = mock_data_asset + + validator = rule_loader.load_from_cams( + "6862f3ba-81f5-4122-8286-62bb4c5d6543", project_id + ) + + # Check DEPTNO column metadata (uses inferred type) + deptno_col = validator.metadata.get_column("DEPTNO") + assert deptno_col.name == "DEPTNO" + assert deptno_col.data_type == DataType.STRING + assert deptno_col.length == 3 + assert deptno_col.nullable is False + + # Check MGRNO column metadata (uses inferred type INT8 -> INTEGER) + mgrno_col = validator.metadata.get_column("MGRNO") + assert mgrno_col.name == "MGRNO" + assert mgrno_col.data_type == DataType.INTEGER + assert mgrno_col.length == 6 + assert mgrno_col.precision == 3 + assert mgrno_col.nullable is True + + def test_load_from_cams_type_conversion( + self, rule_loader, project_id, mock_data_asset, mocker + ): + """Test CAMS type to DataType conversion""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.CamsProvider") + mock_provider.return_value.get_asset_by_id.return_value = mock_data_asset + + validator = rule_loader.load_from_cams( + "6862f3ba-81f5-4122-8286-62bb4c5d6543", project_id + ) + + # Check various type conversions + # STRING type (from inferred type) + deptno_col = validator.metadata.get_column("DEPTNO") + assert deptno_col.data_type == DataType.STRING + + # INTEGER type (from INT8 inferred type) + mgrno_col = validator.metadata.get_column("MGRNO") + assert mgrno_col.data_type == DataType.INTEGER + + def test_load_from_cams_skips_unsupported_checks( + self, rule_loader, project_id, mock_data_asset, mocker + ): + """Test that unsupported check types are skipped""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.CamsProvider") + mock_provider.return_value.get_asset_by_id.return_value = mock_data_asset + + validator = rule_loader.load_from_cams( + "6862f3ba-81f5-4122-8286-62bb4c5d6543", project_id + ) + + # DEPTNO has uniqueness and data_class checks which return None + # So it should have 2 checks instead of 4 + deptno_rule = next(r for r in validator.rules if r.column_name == "DEPTNO") + assert len(deptno_rule.checks) == 2 + + def test_load_from_cams_column_without_checks( + self, rule_loader, project_id, mock_data_asset, mocker + ): + """Test that columns without checks don't create rules""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.CamsProvider") + mock_provider.return_value.get_asset_by_id.return_value = mock_data_asset + + validator = rule_loader.load_from_cams( + "6862f3ba-81f5-4122-8286-62bb4c5d6543", project_id + ) + + # LOCATION has no checks, so no rule should be created + location_rules = [r for r in validator.rules if r.column_name == "LOCATION"] + assert len(location_rules) == 0 + + +# Made with Bob diff --git a/tests/src/dq_validator/test_spark_validator.py b/tests/src/dq_validator/test_spark_validator.py new file mode 100644 index 0000000..012790a --- /dev/null +++ b/tests/src/dq_validator/test_spark_validator.py @@ -0,0 +1,805 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for SparkValidator + +Tests the PySpark DataFrame integration including: +- Initialization and configuration +- Summary statistics calculation (distributed) +- Validation column addition (UDF-based) +- Invalid/valid row filtering +- Column expansion +- Validation report writing +- Error sampling +- Distributed processing +""" + +import math +import pytest +import sys +from typing import Dict, Any +import tempfile +import shutil + +# Check if pyspark is available +try: + from pyspark.sql import SparkSession + from pyspark.sql.types import StructType, StructField, IntegerType, StringType + + PYSPARK_AVAILABLE = True +except ImportError: + PYSPARK_AVAILABLE = False + +from wxdi.dq_validator import ( + AssetMetadata, + ColumnMetadata, + DataType, + Validator, + ValidationRule, +) +from wxdi.dq_validator.checks import ( + LengthCheck, + ValidValuesCheck, + ComparisonCheck, + ComparisonOperator, +) + +if PYSPARK_AVAILABLE: + from wxdi.dq_validator.integrations import SparkValidator + + +# Skip all tests if pyspark is not available +pytestmark = pytest.mark.skipif(not PYSPARK_AVAILABLE, reason="pyspark not installed") + + +@pytest.fixture(scope="module") +def spark(): + """Create Spark session for testing""" + if not PYSPARK_AVAILABLE: + return None + + # Set Python executable and path for Spark workers + import os + from pathlib import Path + + python_path = sys.executable + os.environ["PYSPARK_PYTHON"] = python_path + os.environ["PYSPARK_DRIVER_PYTHON"] = python_path + + # Add src directory to PYTHONPATH so workers can import dq_validator + src_path = str(Path(__file__).parent.parent / "src") + current_pythonpath = os.environ.get("PYTHONPATH", "") + if current_pythonpath: + os.environ["PYTHONPATH"] = f"{src_path}{os.pathsep}{current_pythonpath}" + else: + os.environ["PYTHONPATH"] = src_path + + spark = ( + SparkSession.builder.appName("DataQualityTests") + .master("local[2]") + .config("spark.sql.shuffle.partitions", "2") + .config("spark.python.worker.reuse", "false") + .config("spark.sql.execution.pyspark.udf.faulthandler.enabled", "true") + .config("spark.python.worker.faulthandler.enabled", "true") + .getOrCreate() + ) + + yield spark + + spark.stop() + + +@pytest.fixture +def sample_metadata(): + """Create sample metadata for testing""" + return AssetMetadata( + table_name="test_table", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("name", DataType.STRING, length=50), + ColumnMetadata("age", DataType.INTEGER), + ColumnMetadata("status", DataType.STRING, length=20), + ], + ) + + +@pytest.fixture +def sample_validator(sample_metadata): + """Create sample validator with rules""" + validator = Validator(sample_metadata) + + # Add validation rules + validator.add_rule( + ValidationRule("name").add_check(LengthCheck(min_length=2, max_length=50)) + ) + validator.add_rule( + ValidationRule("age").add_check( + ComparisonCheck( + operator=ComparisonOperator.GREATER_THAN_OR_EQUAL, target_value=18 + ) + ) + ) + validator.add_rule( + ValidationRule("status").add_check( + ValidValuesCheck(["active", "inactive"], case_sensitive=False) + ) + ) + + return validator + + +@pytest.fixture +def sample_dataframe(spark): + """Create sample Spark DataFrame for testing""" + if not PYSPARK_AVAILABLE: + return None + + data = [ + (1, "Alice", 25, "active"), + (2, "B", 30, "ACTIVE"), + (3, "Charlie", 17, "inactive"), + (4, "David", 35, "pending"), + (5, "Eve", 40, "Active"), + ] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + return spark.createDataFrame(data, schema) + + +class TestSparkValidatorInitialization: + """Test SparkValidator initialization""" + + def test_basic_initialization(self, sample_validator): + """Test basic initialization with default parameters""" + spark_validator = SparkValidator(sample_validator) + + assert spark_validator.validator == sample_validator + assert spark_validator.column_prefix == "dq_" + + def test_custom_column_prefix(self, sample_validator): + """Test initialization with custom column prefix""" + spark_validator = SparkValidator(sample_validator, column_prefix="validation_") + + assert spark_validator.column_prefix == "validation_" + + def test_string_representation(self, sample_validator): + """Test string representation of validator""" + spark_validator = SparkValidator(sample_validator) + str_repr = str(spark_validator) + + assert "SparkValidator" in str_repr + + +class TestSummaryStatistics: + """Test summary statistics calculation with distributed aggregation""" + + def test_basic_summary(self, sample_validator, sample_dataframe): + """Test basic summary statistics""" + spark_validator = SparkValidator(sample_validator) + summary = spark_validator.get_summary_statistics(sample_dataframe) + + assert isinstance(summary, dict) + assert "total_rows" in summary + assert "valid_rows" in summary + assert "invalid_rows" in summary + assert "pass_rate" in summary + assert "total_checks" in summary + assert "passed_checks" in summary + assert "failed_checks" in summary + + assert summary["total_rows"] == 5 + assert summary["valid_rows"] + summary["invalid_rows"] == 5 + assert 0 <= summary["pass_rate"] <= 100 + + def test_all_valid_rows(self, spark, sample_validator): + """Test summary with all valid rows""" + data = [ + (1, "Alice", 25, "active"), + (2, "Bob", 30, "inactive"), + (3, "Charlie", 35, "active"), + ] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + summary = spark_validator.get_summary_statistics(df) + + assert summary["total_rows"] == 3 + assert summary["valid_rows"] == 3 + assert summary["invalid_rows"] == 0 + assert math.isclose(summary["pass_rate"], 100.0) + + def test_all_invalid_rows(self, spark, sample_validator): + """Test summary with all invalid rows""" + data = [ + (1, "A", 15, "pending"), + (2, "B", 16, "deleted"), + (3, "C", 17, "archived"), + ] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + summary = spark_validator.get_summary_statistics(df) + + assert summary["total_rows"] == 3 + assert summary["valid_rows"] == 0 + assert summary["invalid_rows"] == 3 + assert math.isclose(summary["pass_rate"], 0.0, abs_tol=1e-9) + + def test_empty_dataframe(self, spark, sample_validator): + """Test summary with empty DataFrame""" + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + df = spark.createDataFrame([], schema) + + spark_validator = SparkValidator(sample_validator) + summary = spark_validator.get_summary_statistics(df) + + assert summary["total_rows"] == 0 + assert summary["valid_rows"] == 0 + assert summary["invalid_rows"] == 0 + assert math.isclose(summary["pass_rate"], 0.0, abs_tol=1e-9) + + def test_large_dataframe(self, spark, sample_validator): + """Test summary with large DataFrame (distributed processing)""" + # Create large dataset + data = [(i, "Alice", 25, "active") for i in range(1, 1001)] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + summary = spark_validator.get_summary_statistics(df) + + assert summary["total_rows"] == 1000 + assert summary["valid_rows"] == 1000 + + +class TestAddValidationColumn: + """Test adding validation column using Spark UDF""" + + def test_basic_validation_column(self, sample_validator, sample_dataframe): + """Test adding validation column""" + spark_validator = SparkValidator(sample_validator) + df_validated = spark_validator.add_validation_column(sample_dataframe) + + # Check that original columns are preserved + original_cols = sample_dataframe.columns + assert all(col in df_validated.columns for col in original_cols) + + # Check that validation column is added + assert "dq_validation_result" in df_validated.columns + + # Check that DataFrame has same number of rows + assert df_validated.count() == sample_dataframe.count() + + def test_validation_result_structure(self, sample_validator, sample_dataframe): + """Test structure of validation result""" + spark_validator = SparkValidator(sample_validator) + df_validated = spark_validator.add_validation_column(sample_dataframe) + + # Get schema of validation column + validation_col = df_validated.schema["dq_validation_result"] + + # Check that it's a struct type + assert validation_col.dataType.typeName() == "struct" + + # Check field names + field_names = [f.name for f in validation_col.dataType.fields] + assert "is_valid" in field_names + assert "score" in field_names + assert "pass_rate" in field_names + assert "total_checks" in field_names + assert "passed_checks" in field_names + assert "failed_checks" in field_names + assert "error_count" in field_names + assert "errors" in field_names + + def test_custom_column_prefix(self, sample_validator, sample_dataframe): + """Test validation column with custom prefix""" + spark_validator = SparkValidator(sample_validator, column_prefix="val_") + df_validated = spark_validator.add_validation_column(sample_dataframe) + + assert "val_validation_result" in df_validated.columns + assert "dq_validation_result" not in df_validated.columns + + def test_column_conflict_detection(self, spark, sample_validator): + """Test detection of column name conflicts""" + data = [ + (1, "Alice", 25, "active", "existing_data"), + (2, "Bob", 30, "inactive", "existing_data"), + ] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + StructField("dq_validation_result", StringType(), True), # Conflict! + ] + ) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + + with pytest.raises(ValueError, match="already exists"): + spark_validator.add_validation_column(df) + + +class TestInvalidRowFiltering: + """Test filtering invalid rows with distributed operations""" + + def test_get_invalid_rows(self, sample_validator, sample_dataframe): + """Test getting invalid rows""" + spark_validator = SparkValidator(sample_validator) + invalid_df = spark_validator.get_invalid_rows(sample_dataframe) + + # Should have validation column + assert "dq_validation_result" in invalid_df.columns + + # Collect and check all rows are invalid + rows = invalid_df.select("dq_validation_result.is_valid").collect() + for row in rows: + assert row["is_valid"] is False + + def test_get_valid_rows(self, sample_validator, sample_dataframe): + """Test getting valid rows""" + spark_validator = SparkValidator(sample_validator) + valid_df = spark_validator.get_valid_rows(sample_dataframe) + + # Should have validation column + assert "dq_validation_result" in valid_df.columns + + # Collect and check all rows are valid + rows = valid_df.select("dq_validation_result.is_valid").collect() + for row in rows: + assert row["is_valid"] is True + + def test_no_invalid_rows(self, spark, sample_validator): + """Test when there are no invalid rows""" + data = [ + (1, "Alice", 25, "active"), + (2, "Bob", 30, "inactive"), + (3, "Charlie", 35, "active"), + ] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + invalid_df = spark_validator.get_invalid_rows(df) + + assert invalid_df.count() == 0 + + def test_no_valid_rows(self, spark, sample_validator): + """Test when there are no valid rows""" + data = [ + (1, "A", 15, "pending"), + (2, "B", 16, "deleted"), + (3, "C", 17, "archived"), + ] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + valid_df = spark_validator.get_valid_rows(df) + + assert valid_df.count() == 0 + + +class TestColumnExpansion: + """Test expanding validation column""" + + def test_basic_expansion(self, sample_validator, sample_dataframe): + """Test basic column expansion""" + spark_validator = SparkValidator(sample_validator) + df_validated = spark_validator.add_validation_column(sample_dataframe) + df_expanded = spark_validator.expand_validation_column(df_validated) + + # Check that expanded columns exist + assert "dq_is_valid" in df_expanded.columns + assert "dq_score" in df_expanded.columns + assert "dq_pass_rate" in df_expanded.columns + assert "dq_total_checks" in df_expanded.columns + assert "dq_passed_checks" in df_expanded.columns + assert "dq_failed_checks" in df_expanded.columns + assert "dq_error_count" in df_expanded.columns + assert "dq_errors" in df_expanded.columns + + # Original validation column should be removed + assert "dq_validation_result" not in df_expanded.columns + + # Original columns should be preserved + original_cols = sample_dataframe.columns + assert all(col in df_expanded.columns for col in original_cols) + + def test_expansion_with_custom_prefix(self, sample_validator, sample_dataframe): + """Test expansion with custom prefix""" + spark_validator = SparkValidator(sample_validator, column_prefix="val_") + df_validated = spark_validator.add_validation_column(sample_dataframe) + df_expanded = spark_validator.expand_validation_column(df_validated) + + assert "val_is_valid" in df_expanded.columns + assert "val_score" in df_expanded.columns + assert "val_pass_rate" in df_expanded.columns + + def test_expansion_without_validation_column( + self, sample_validator, sample_dataframe + ): + """Test expansion when validation column doesn't exist""" + spark_validator = SparkValidator(sample_validator) + + with pytest.raises( + ValueError, match="does not contain validation result column" + ): + spark_validator.expand_validation_column(sample_dataframe) + + +class TestValidationReport: + """Test validation report writing""" + + def test_write_parquet_report(self, sample_validator, sample_dataframe): + """Test writing validation report in Parquet format""" + spark_validator = SparkValidator(sample_validator) + + with tempfile.TemporaryDirectory() as tmpdir: + output_path = f"{tmpdir}/validation_report" + + spark_validator.write_validation_report( + sample_dataframe, output_path=output_path, format="parquet" + ) + + # Verify report was written + # Note: In real scenario, you'd read back and verify content + + def test_write_csv_report(self, sample_validator, sample_dataframe): + """Test writing validation report in CSV format""" + spark_validator = SparkValidator(sample_validator) + + with tempfile.TemporaryDirectory() as tmpdir: + output_path = f"{tmpdir}/validation_report" + + spark_validator.write_validation_report( + sample_dataframe, output_path=output_path, format="csv" + ) + + # Verify report was written + + +class TestErrorSampling: + """Test error sampling functionality""" + + def test_get_error_sample(self, sample_validator, sample_dataframe): + """Test getting error samples""" + spark_validator = SparkValidator(sample_validator) + error_samples = spark_validator.get_error_sample(sample_dataframe, limit=10) + + assert isinstance(error_samples, list) + assert len(error_samples) <= 10 + + # Check structure of error samples + if len(error_samples) > 0: + sample = error_samples[0] + assert "row" in sample + assert "errors" in sample + assert isinstance(sample["errors"], list) + + def test_error_sample_limit(self, sample_validator, sample_dataframe): + """Test error sample limit""" + spark_validator = SparkValidator(sample_validator) + + # Get small sample + small_sample = spark_validator.get_error_sample(sample_dataframe, limit=2) + assert len(small_sample) <= 2 + + # Get larger sample + large_sample = spark_validator.get_error_sample(sample_dataframe, limit=100) + assert len(large_sample) <= 100 + + def test_no_errors(self, spark, sample_validator): + """Test error sampling when there are no errors""" + data = [(1, "Alice", 25, "active"), (2, "Bob", 30, "inactive")] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + error_samples = spark_validator.get_error_sample(df, limit=10) + + assert len(error_samples) == 0 + + +class TestEdgeCases: + """Test edge cases and error handling""" + + def test_dataframe_with_null_values(self, spark, sample_validator): + """Test DataFrame with null values""" + data = [ + (1, "Alice", 25, "active"), + (2, None, 30, "inactive"), + (3, "Charlie", None, "active"), + ] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + summary = spark_validator.get_summary_statistics(df) + + # Should handle nulls gracefully + assert summary["total_rows"] == 3 + assert summary["invalid_rows"] > 0 # Nulls should cause failures + + def test_distributed_processing_consistency(self, spark, sample_validator): + """Test that distributed processing produces consistent results""" + # Create dataset that will be distributed across partitions + data = [(i, "Alice", 25, "active") for i in range(1, 101)] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + df = spark.createDataFrame(data, schema).repartition(4) + + spark_validator = SparkValidator(sample_validator) + + # Run validation multiple times + summary1 = spark_validator.get_summary_statistics(df) + summary2 = spark_validator.get_summary_statistics(df) + + # Results should be consistent + assert summary1 == summary2 + + +class TestIntegrationScenarios: + """Test complete integration scenarios""" + + def test_complete_workflow(self, sample_validator, sample_dataframe): + """Test complete validation workflow""" + spark_validator = SparkValidator(sample_validator) + + # Step 1: Get summary + summary = spark_validator.get_summary_statistics(sample_dataframe) + assert summary["total_rows"] == 5 + + # Step 2: Add validation column + df_validated = spark_validator.add_validation_column(sample_dataframe) + assert "dq_validation_result" in df_validated.columns + + # Step 3: Filter invalid rows + invalid_df = spark_validator.get_invalid_rows(sample_dataframe) + assert invalid_df.count() == summary["invalid_rows"] + + # Step 4: Expand columns + df_expanded = spark_validator.expand_validation_column(df_validated) + assert "dq_is_valid" in df_expanded.columns + + # Step 5: Get error samples + error_samples = spark_validator.get_error_sample(sample_dataframe, limit=10) + assert isinstance(error_samples, list) + + def test_lazy_evaluation(self, sample_validator, sample_dataframe): + """Test that operations use lazy evaluation""" + spark_validator = SparkValidator(sample_validator) + + # These operations should not trigger computation + df_validated = spark_validator.add_validation_column(sample_dataframe) + invalid_df = spark_validator.get_invalid_rows(sample_dataframe) + + # Only when we call an action (count, collect, etc.) should computation happen + count = invalid_df.count() + assert isinstance(count, int) + + +class TestValidationRowImpl: + """Test the validate_row_impl function execution and row validation logic""" + + def test_validation_result_content(self, sample_validator, sample_dataframe): + """Test that validation results contain correct data for each row""" + spark_validator = SparkValidator(sample_validator) + df_validated = spark_validator.add_validation_column(sample_dataframe) + + # Collect results to verify validation logic execution + results = df_validated.select("name", "age", "dq_validation_result").collect() + + # Verify we have results for all rows + assert len(results) == 5 + + # Check that validation results have expected structure + for row in results: + validation_result = row["dq_validation_result"] + + # Verify all fields are present + assert "is_valid" in validation_result + assert "score" in validation_result + assert "pass_rate" in validation_result + assert "total_checks" in validation_result + assert "passed_checks" in validation_result + assert "failed_checks" in validation_result + assert "error_count" in validation_result + assert "errors" in validation_result + + # Verify types + assert isinstance(validation_result["is_valid"], bool) + assert isinstance(validation_result["score"], str) # Score is a string like "3/3" + assert isinstance(validation_result["pass_rate"], (int, float)) + assert isinstance(validation_result["total_checks"], int) + assert isinstance(validation_result["passed_checks"], int) + assert isinstance(validation_result["failed_checks"], int) + assert isinstance(validation_result["error_count"], int) + assert isinstance(validation_result["errors"], list) + + # Verify logical consistency + assert validation_result["total_checks"] == ( + validation_result["passed_checks"] + validation_result["failed_checks"] + ) + assert validation_result["error_count"] == len(validation_result["errors"]) + + def test_row_conversion_to_list(self, spark, sample_validator): + """Test that Spark Row is correctly converted to list for validation""" + # Create a simple dataframe with known values + data = [ + (1, "Alice", 25, "active"), + (2, "Bob", 30, "inactive"), + ] + + schema = StructType([ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ]) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + df_validated = spark_validator.add_validation_column(df) + + # Collect and verify that validation was performed on list-converted rows + results = df_validated.collect() + + for row in results: + validation_result = row["dq_validation_result"] + # If validation ran successfully, it means list(row) conversion worked + assert validation_result is not None + assert "is_valid" in validation_result + + # Verify that checks were actually executed (not just returning empty results) + assert validation_result["total_checks"] > 0 + + def test_validation_with_errors(self, spark, sample_validator): + """Test that validation errors are properly captured in JSON format""" + # Create data that will fail validation + data = [ + (1, "A", 15, "invalid_status"), # Short name, young age, invalid status + ] + + schema = StructType([ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ]) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + df_validated = spark_validator.add_validation_column(df) + + # Collect and verify error messages + result = df_validated.collect()[0] + validation_result = result["dq_validation_result"] + + # Should have validation errors + assert validation_result["is_valid"] == False + assert validation_result["error_count"] > 0 + assert len(validation_result["errors"]) > 0 + + # Verify errors are JSON strings + import json + for error_str in validation_result["errors"]: + assert isinstance(error_str, str) + # Should be valid JSON + error_dict = json.loads(error_str) + assert isinstance(error_dict, dict) + # Should have expected error fields + assert "column" in error_dict or "check" in error_dict + + +if __name__ == "__main__": + pytest.main([__file__, "-v"]) diff --git a/tests/src/dq_validator/test_valid_values_check.py b/tests/src/dq_validator/test_valid_values_check.py new file mode 100644 index 0000000..fa911a2 --- /dev/null +++ b/tests/src/dq_validator/test_valid_values_check.py @@ -0,0 +1,394 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for ValidValuesCheck +""" + +import pytest +from wxdi.dq_validator.checks.valid_values_check import ValidValuesCheck +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + + +class TestValidValuesCheckInitialization: + """Test ValidValuesCheck initialization and parameter validation""" + + def test_init_with_string_values(self): + """Test initialization with string values""" + check = ValidValuesCheck(['active', 'inactive', 'pending']) + assert check.valid_values == ['active', 'inactive', 'pending'] + assert check.case_sensitive == False # Default + + def test_init_with_numeric_values(self): + """Test initialization with numeric values""" + check = ValidValuesCheck([1, 2, 3, 4, 5]) + assert check.valid_values == [1, 2, 3, 4, 5] + + def test_init_case_sensitive_true(self): + """Test initialization with case_sensitive=True""" + check = ValidValuesCheck(['Active', 'Inactive'], case_sensitive=True) + assert check.case_sensitive == True + + def test_init_case_sensitive_false(self): + """Test initialization with case_sensitive=False""" + check = ValidValuesCheck(['active', 'inactive'], case_sensitive=False) + assert check.case_sensitive == False + + def test_init_empty_list_raises_error(self): + """Test that empty list raises ValueError""" + with pytest.raises(ValueError) as exc_info: + ValidValuesCheck([]) + assert "valid_values cannot be empty" in str(exc_info.value) + + def test_init_not_list_raises_error(self): + """Test that non-list raises TypeError""" + with pytest.raises(TypeError) as exc_info: + ValidValuesCheck('not_a_list') # type: ignore[arg-type] + assert "valid_values must be a list" in str(exc_info.value) + + def test_init_with_mixed_types(self): + """Test initialization with mixed type values""" + check = ValidValuesCheck([1, 'text', True]) + assert check.valid_values == [1, 'text', True] + + def test_get_check_name(self): + """Test get_check_name returns correct name""" + check = ValidValuesCheck(['a', 'b']) + assert check.get_check_name() == "valid_values_check" + + def test_get_dimension(self): + """Test get_dimension returns correct dimension""" + check = ValidValuesCheck(['a', 'b']) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + def test_set_dimension(self): + """Test set_dimension changes the dimension""" + check = ValidValuesCheck(['a', 'b']) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + check.set_dimension(DataQualityDimension.CONSISTENCY) + assert check.get_dimension() == DataQualityDimension.CONSISTENCY + + +class TestValidValuesCheckCaseSensitive: + """Test ValidValuesCheck with case_sensitive=True""" + + def test_exact_match_passes(self): + """Test exact match passes with case_sensitive=True""" + check = ValidValuesCheck(['active', 'inactive', 'pending'], case_sensitive=True) + context = {'column_name': 'status'} + result = check.validate('active', context) + assert result is None + + def test_case_mismatch_fails(self): + """Test case mismatch fails with case_sensitive=True""" + check = ValidValuesCheck(['active', 'inactive', 'pending'], case_sensitive=True) + context = {'column_name': 'status'} + result = check.validate('Active', context) + assert result is not None + assert result.column_name == 'status' + assert result.check_name == 'valid_values_check' + assert "has invalid value 'Active'" in result.message + + def test_uppercase_mismatch_fails(self): + """Test uppercase mismatch fails with case_sensitive=True""" + check = ValidValuesCheck(['active', 'inactive'], case_sensitive=True) + context = {'column_name': 'status'} + result = check.validate('ACTIVE', context) + assert result is not None + assert "has invalid value 'ACTIVE'" in result.message + + def test_invalid_value_fails(self): + """Test invalid value fails with case_sensitive=True""" + check = ValidValuesCheck(['active', 'inactive'], case_sensitive=True) + context = {'column_name': 'status'} + result = check.validate('archived', context) + assert result is not None + assert "has invalid value 'archived'" in result.message + + +class TestValidValuesCheckCaseInsensitive: + """Test ValidValuesCheck with case_sensitive=False (default)""" + + def test_exact_match_passes(self): + """Test exact match passes with case_sensitive=False""" + check = ValidValuesCheck(['active', 'inactive', 'pending'], case_sensitive=False) + context = {'column_name': 'status'} + result = check.validate('active', context) + assert result is None + + def test_uppercase_match_passes(self): + """Test uppercase match passes with case_sensitive=False""" + check = ValidValuesCheck(['active', 'inactive', 'pending'], case_sensitive=False) + context = {'column_name': 'status'} + result = check.validate('ACTIVE', context) + assert result is None + + def test_titlecase_match_passes(self): + """Test titlecase match passes with case_sensitive=False""" + check = ValidValuesCheck(['active', 'inactive', 'pending'], case_sensitive=False) + context = {'column_name': 'status'} + result = check.validate('Active', context) + assert result is None + + def test_mixed_case_match_passes(self): + """Test mixed case match passes with case_sensitive=False""" + check = ValidValuesCheck(['active', 'inactive'], case_sensitive=False) + context = {'column_name': 'status'} + result = check.validate('AcTiVe', context) + assert result is None + + def test_invalid_value_fails(self): + """Test invalid value fails with case_sensitive=False""" + check = ValidValuesCheck(['active', 'inactive'], case_sensitive=False) + context = {'column_name': 'status'} + result = check.validate('archived', context) + assert result is not None + assert "has invalid value 'archived' (case-insensitive)" in result.message + + def test_default_is_case_insensitive(self): + """Test default behavior is case-insensitive""" + check = ValidValuesCheck(['active', 'inactive']) + context = {'column_name': 'status'} + result = check.validate('ACTIVE', context) + assert result is None + + +class TestValidValuesCheckNumericValues: + """Test ValidValuesCheck with numeric values""" + + def test_valid_integer_passes(self): + """Test valid integer value passes""" + check = ValidValuesCheck([1, 2, 3, 4, 5]) + context = {'column_name': 'priority'} + result = check.validate(3, context) + assert result is None + + def test_invalid_integer_fails(self): + """Test invalid integer value fails""" + check = ValidValuesCheck([1, 2, 3, 4, 5]) + context = {'column_name': 'priority'} + result = check.validate(6, context) + assert result is not None + assert "has invalid value '6'" in result.message + + def test_valid_float_passes(self): + """Test valid float value passes""" + check = ValidValuesCheck([1.0, 2.5, 3.7]) + context = {'column_name': 'rating'} + result = check.validate(2.5, context) + assert result is None + + def test_type_mismatch_string_vs_int_fails(self): + """Test type mismatch between string and int fails""" + check = ValidValuesCheck([1, 2, 3]) + context = {'column_name': 'priority'} + result = check.validate('3', context) + assert result is not None + assert "has invalid value '3'" in result.message + + def test_type_mismatch_int_vs_float_passes(self): + """Test that int and float with same value are considered equal (Python behavior)""" + check = ValidValuesCheck([1, 2, 3]) + context = {'column_name': 'value'} + result = check.validate(1.0, context) + assert result is None # 1 == 1.0 in Python (numeric equality) + + +class TestValidValuesCheckBooleanValues: + """Test ValidValuesCheck with boolean values""" + + def test_valid_true_passes(self): + """Test valid True value passes""" + check = ValidValuesCheck([True, False]) + context = {'column_name': 'is_active'} + result = check.validate(True, context) + assert result is None + + def test_valid_false_passes(self): + """Test valid False value passes""" + check = ValidValuesCheck([True, False]) + context = {'column_name': 'is_active'} + result = check.validate(False, context) + assert result is None + + def test_boolean_case_insensitive_ignored(self): + """Test case_sensitive is ignored for non-string types""" + check = ValidValuesCheck([True], case_sensitive=False) + context = {'column_name': 'flag'} + result = check.validate(True, context) + assert result is None + + +class TestValidValuesCheckNoneHandling: + """Test ValidValuesCheck with None values""" + + def test_none_value_fails(self): + """Test None value returns error""" + check = ValidValuesCheck(['active', 'inactive']) + context = {'column_name': 'status'} + result = check.validate(None, context) + assert result is not None + assert result.column_name == 'status' + assert "is None" in result.message + assert "expected one of" in result.message + + +class TestValidValuesCheckEmptyString: + """Test ValidValuesCheck with empty strings""" + + def test_empty_string_in_valid_values_passes(self): + """Test empty string passes when in valid values""" + check = ValidValuesCheck(['', 'value1', 'value2']) + context = {'column_name': 'optional_field'} + result = check.validate('', context) + assert result is None + + def test_empty_string_not_in_valid_values_fails(self): + """Test empty string fails when not in valid values""" + check = ValidValuesCheck(['value1', 'value2']) + context = {'column_name': 'required_field'} + result = check.validate('', context) + assert result is not None + assert "has invalid value ''" in result.message + + +class TestValidValuesCheckMixedTypes: + """Test ValidValuesCheck with mixed type values""" + + def test_mixed_types_string_passes(self): + """Test string value passes in mixed type list""" + check = ValidValuesCheck([1, 'text', True]) + context = {'column_name': 'mixed_field'} + result = check.validate('text', context) + assert result is None + + def test_mixed_types_int_passes(self): + """Test int value passes in mixed type list""" + check = ValidValuesCheck([1, 'text', True]) + context = {'column_name': 'mixed_field'} + result = check.validate(1, context) + assert result is None + + def test_mixed_types_bool_passes(self): + """Test bool value passes in mixed type list""" + check = ValidValuesCheck([1, 'text', True]) + context = {'column_name': 'mixed_field'} + result = check.validate(True, context) + assert result is None + + def test_mixed_types_invalid_fails(self): + """Test invalid value fails in mixed type list""" + check = ValidValuesCheck([1, 'text', True]) + context = {'column_name': 'mixed_field'} + result = check.validate(2, context) + assert result is not None + + +class TestValidValuesCheckDuplicates: + """Test ValidValuesCheck with duplicate values""" + + def test_duplicates_in_list_allowed(self): + """Test duplicates in valid_values list are allowed""" + check = ValidValuesCheck(['active', 'active', 'inactive']) + context = {'column_name': 'status'} + result = check.validate('active', context) + assert result is None + + def test_duplicates_case_insensitive(self): + """Test duplicates with different cases in case-insensitive mode""" + check = ValidValuesCheck(['Active', 'active', 'ACTIVE'], case_sensitive=False) + context = {'column_name': 'status'} + result = check.validate('active', context) + assert result is None + + +class TestValidValuesCheckErrorMessages: + """Test ValidValuesCheck error message formatting""" + + def test_error_message_includes_column_name(self): + """Test error message includes column name""" + check = ValidValuesCheck(['a', 'b']) + context = {'column_name': 'test_column'} + result = check.validate('c', context) + assert result is not None + assert 'test_column' in result.message + + def test_error_message_includes_invalid_value(self): + """Test error message includes the invalid value""" + check = ValidValuesCheck(['a', 'b']) + context = {'column_name': 'field'} + result = check.validate('invalid', context) + assert result is not None + assert 'invalid' in result.message + + def test_error_message_includes_valid_values(self): + """Test error message includes list of valid values""" + check = ValidValuesCheck(['a', 'b', 'c']) + context = {'column_name': 'field'} + result = check.validate('d', context) + assert result is not None + assert "['a', 'b', 'c']" in result.message + + def test_error_message_case_insensitive_note(self): + """Test error message includes case-insensitive note for strings""" + check = ValidValuesCheck(['active'], case_sensitive=False) + context = {'column_name': 'status'} + result = check.validate('invalid', context) + assert result is not None + assert "(case-insensitive)" in result.message + + def test_error_message_no_case_note_for_case_sensitive(self): + """Test error message has no case note for case-sensitive""" + check = ValidValuesCheck(['active'], case_sensitive=True) + context = {'column_name': 'status'} + result = check.validate('invalid', context) + assert result is not None + assert "(case-insensitive)" not in result.message + + +class TestValidValuesCheckEdgeCases: + """Test ValidValuesCheck edge cases""" + + def test_single_valid_value(self): + """Test with single valid value""" + check = ValidValuesCheck(['only_value']) + context = {'column_name': 'field'} + result = check.validate('only_value', context) + assert result is None + + def test_large_valid_values_list(self): + """Test with large list of valid values""" + large_list = [f'value_{i}' for i in range(1000)] + check = ValidValuesCheck(large_list) + context = {'column_name': 'field'} + result = check.validate('value_500', context) + assert result is None + + def test_special_characters_in_values(self): + """Test with special characters in values""" + check = ValidValuesCheck(['@#$%', '!@#', 'a-b-c']) + context = {'column_name': 'field'} + result = check.validate('@#$%', context) + assert result is None + + def test_repr(self): + """Test __repr__ method""" + check = ValidValuesCheck(['a', 'b', 'c'], case_sensitive=True) + repr_str = repr(check) + assert "ValidValuesCheck" in repr_str + assert "3" in repr_str # Number of values + assert "True" in repr_str # case_sensitive + diff --git a/tests/src/dq_validator/test_version.py b/tests/src/dq_validator/test_version.py new file mode 100644 index 0000000..678e5a2 --- /dev/null +++ b/tests/src/dq_validator/test_version.py @@ -0,0 +1,35 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for __version__ +""" +import pytest +from packaging.version import parse, Version + +class TestVersions: + test_version = Version('0.5.0') + """ Tests versions embedded in various components """ + def test_wxdi_version(self): + import wxdi + assert TestVersions.test_version < parse(wxdi.__version__) + + def test_wxdi_dqvalidator_version(self): + import wxdi.dq_validator + assert TestVersions.test_version < parse(wxdi.dq_validator.__version__) + + def test_wxdi_dqvalidator_integrations_version(self): + import wxdi.dq_validator.integrations + assert TestVersions.test_version < parse(wxdi.dq_validator.integrations.__version__) \ No newline at end of file diff --git a/tests/src/integration/README.md b/tests/src/integration/README.md new file mode 100644 index 0000000..752c2c1 --- /dev/null +++ b/tests/src/integration/README.md @@ -0,0 +1,221 @@ + +# Integration Tests + +This directory contains integration tests for the data-intelligence-sdk modules that require external service connections. + +## Overview + +Integration tests verify that the SDK works correctly with actual IBM Cloud services. These tests are skipped by default when the required configuration files are not present. + +## Test Files + +### 1. `test_dph_v1.py` +Tests for IBM Data Product Hub API Service (DPH Services). + +**Requirements:** +- Configuration file: `dph_v1.env` +- Valid IBM Cloud API credentials +- Access to IBM Data Product Hub service + +**Tests:** 49 integration tests covering: +- Container initialization +- Data product CRUD operations +- Draft management +- Release management +- Contract terms and documents +- Domain management + +### 2. `test_odcs_generator_collibra.py` +Tests for ODCS generation from Collibra. + +**Requirements:** +- Collibra API credentials +- Sample Collibra data assets + +### 3. `test_odcs_generator_informatica.py` +Tests for ODCS generation from Informatica. + +**Requirements:** +- Informatica API credentials +- Sample Informatica data assets + +### 4. `test_data_product_recommender_integration.py` +Tests for data product recommendation engine. + +**Requirements:** +- Sample query log data files +- Database connection (if testing with live data) + +## Configuration Setup + +### DPH Services Configuration (`dph_v1.env`) + +The `test_dph_v1.py` integration tests require a configuration file named `dph_v1.env` in this directory. + +#### Step 1: Copy the Template + +A template file `dph_v1.env` has been created in this directory. Edit it with your credentials. + +#### Step 2: Get IBM Cloud API Key + +1. Log in to [IBM Cloud](https://cloud.ibm.com/) +2. Navigate to **Manage** > **Access (IAM)** > **API keys** +3. Click **Create** to create a new API key +4. Copy the API key (you won't be able to see it again) + +#### Step 3: Configure the File + +Edit `tests/integration/dph_v1.env` and replace the placeholder values: + +```bash +# Required: Service URL +DATA_PRODUCT_HUB_API_SERVICE_URL=https://api.dataplatform.cloud.ibm.com/ + +# Required: Authentication type +DATA_PRODUCT_HUB_API_SERVICE_AUTH_TYPE=iam + +# Required: Your IBM Cloud API Key +DATA_PRODUCT_HUB_API_SERVICE_APIKEY=your-actual-key-here +``` + +#### Step 4: Verify Permissions + +Ensure your API key has the following permissions: +- Access to IBM Data Product Hub service +- Ability to create/read/update/delete data products +- Access to the target catalog/container + +### Configuration File Format + +The IBM Cloud SDK uses environment variables or configuration files in the following format: + +``` +_URL= +_AUTH_TYPE= +_APIKEY= +``` + +Where `` is the uppercase version of the service name with underscores. + +For DPH Services, the service name is `data_product_hub_api_service`, so variables are prefixed with `DATA_PRODUCT_HUB_API_SERVICE_`. + +## Running Integration Tests + +### Run All Integration Tests + +```bash +# Activate virtual environment +source .venv/bin/activate + +# Run all integration tests +pytest tests/integration/ -v +``` + +### Run Specific Test File + +```bash +# Run only DPH integration tests +pytest tests/integration/test_dph_v1.py -v + +# Run only ODCS Collibra tests +pytest tests/integration/test_odcs_generator_collibra.py -v +``` + +### Run with Coverage + +```bash +pytest tests/integration/ --cov=src/wxdi --cov-report=html +``` + +## Test Behavior + +### Without Configuration +When configuration files are missing, tests are automatically skipped with the message: +``` +SKIPPED [49] External configuration not available, skipping... +``` + +### With Configuration +When valid configuration is provided, tests will: +1. Connect to the actual IBM Cloud service +2. Perform real API operations +3. Validate responses and behavior +4. Clean up resources (where applicable) + +## Security Notes + +⚠️ **IMPORTANT SECURITY CONSIDERATIONS:** + +1. **Never commit credentials**: The `.env` files contain sensitive API keys and should NEVER be committed to version control +2. **Already protected**: The `.gitignore` file already excludes `.env` files +3. **Secure sharing**: Share credentials only through secure channels (password managers, encrypted communication) +4. **Rotate keys**: Regularly rotate API keys, especially if they may have been exposed +5. **Minimal permissions**: Use API keys with the minimum required permissions + +## Troubleshooting + +### Tests are Skipped + +**Problem:** All integration tests show as "skipped" + +**Solution:** +- Verify the configuration file exists: `tests/integration/dph_v1.env` +- Check that the file contains valid credentials +- Ensure the file is in the correct location + +### Authentication Errors + +**Problem:** Tests fail with 401 Unauthorized or 403 Forbidden + +**Solution:** +- Verify your API key is correct and not expired +- Check that your API key has the required permissions +- Ensure the service URL is correct for your region + +### Connection Errors + +**Problem:** Tests fail with connection timeouts or network errors + +**Solution:** +- Verify the service URL is correct +- Check your network connection +- Ensure you can access IBM Cloud services from your network +- Check if there are any firewall restrictions + +### Service-Specific Errors + +**Problem:** Tests fail with service-specific error messages + +**Solution:** +- Check the IBM Data Product Hub service status +- Verify your account has access to the service +- Review the test output for specific error messages +- Consult IBM Cloud documentation for the specific service + +## Additional Resources + +- [IBM Cloud SDK Core Documentation](https://github.com/IBM/python-sdk-core) +- [IBM Cloud Authentication Guide](https://cloud.ibm.com/docs/account?topic=account-userapikey) +- [IBM Data Product Hub Documentation](https://cloud.ibm.com/docs/data-product-hub) +- [pytest Documentation](https://docs.pytest.org/) + +## Support + +For issues related to: +- **SDK functionality**: Open an issue in the data-intelligence-sdk repository +- **IBM Cloud services**: Contact IBM Cloud Support +- **Authentication**: Refer to IBM Cloud IAM documentation \ No newline at end of file diff --git a/tests/src/integration/__init__.py b/tests/src/integration/__init__.py new file mode 100644 index 0000000..2f5430c --- /dev/null +++ b/tests/src/integration/__init__.py @@ -0,0 +1,19 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Integration tests""" + +# This file is only here to get pylint to check the files in this directory diff --git a/tests/src/integration/initial_setup_service.py b/tests/src/integration/initial_setup_service.py new file mode 100644 index 0000000..ecdfddc --- /dev/null +++ b/tests/src/integration/initial_setup_service.py @@ -0,0 +1,137 @@ +# -*- coding: utf-8 -*- +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Integration Tests for DataProductHubApiServiceV1 +""" + +from enum import Enum +from typing import Dict, List +import json + +from ibm_cloud_sdk_core import BaseService, DetailedResponse +from ibm_cloud_sdk_core.authenticators.authenticator import Authenticator +from ibm_cloud_sdk_core.get_authenticator import get_authenticator_from_environment +from ibm_cloud_sdk_core.utils import convert_model + +from wxdi.dph_services.common import get_sdk_headers + +############################################################################## +# Temporary setup Service +############################################################################## + + +class InitialSetupServiceV1(BaseService): + """The Data Product Hub API Service V1 service.""" + + DEFAULT_SERVICE_URL = 'https://api.dataplatform.dev.cloud.ibm.com/v2' + DEFAULT_SERVICE_NAME = 'cams_api_service' + + @classmethod + def new_instance( + cls, + service_name: str = DEFAULT_SERVICE_NAME, + ) -> 'InitialSetupServiceV1': + """ + Return a new client for the Initial Data Product Hub API Service setup using the + specified parameters and external configuration. + """ + authenticator = get_authenticator_from_environment(service_name) + service = cls(authenticator) + service.configure_service(service_name) + return service + + def __init__( + self, + authenticator: Authenticator = None, + ) -> None: + """ + Construct a new client for the Data Product Hub API Service service. + + :param Authenticator authenticator: The authenticator specifies the authentication mechanism. + Get up to date information from https://github.com/IBM/python-sdk-core/blob/main/README.md + about initializing the authenticator of your choice. + """ + BaseService.__init__(self, service_url=self.DEFAULT_SERVICE_URL, authenticator=authenticator) + + def create_data_product_catalog( + self, + **kwargs, + ) -> DetailedResponse: + headers = {} + sdk_headers = get_sdk_headers( + service_name='initial_Setup_service', + service_version='V1', + operation_id='create_data_product_catalog', + ) + headers.update(sdk_headers) + + data = { + 'name': 'Default Data Product Hub', + 'uid': 'ibm-default-hub', + 'subtype': 'ibm_data_product_catalog', + 'generator': 'catalogadmin', + } + data = {k: v for (k, v) in data.items() if v is not None} + data = json.dumps(data) + headers['content-type'] = 'application/json' + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = 'application/json' + + url = '/catalogs' + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def delete_data_product_catalog( + self, + id: str, + **kwargs, + ) -> DetailedResponse: + if not id: + raise ValueError('id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='delete_data_product_catalog', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + + path_param_keys = ['id'] + path_param_values = self.encode_path_vars(id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/catalogs/{id}'.format(**path_param_dict) + request = self.prepare_request( + method='DELETE', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response diff --git a/tests/src/integration/test_data_product_recommender_integration.py b/tests/src/integration/test_data_product_recommender_integration.py new file mode 100644 index 0000000..00e6eff --- /dev/null +++ b/tests/src/integration/test_data_product_recommender_integration.py @@ -0,0 +1,373 @@ +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Integration tests for Data Product Recommender + +These tests verify end-to-end functionality using sample data files. +""" + +import pytest +import os +import tempfile +import shutil +from pathlib import Path + +from wxdi.data_product_recommender.platforms import ( + SnowflakeQueryParser, + DatabricksQueryParser, + BigQueryQueryParser, + WatsonxDataQueryParser +) +from wxdi.data_product_recommender.recommender import DataProductRecommender + + +class TestSnowflakeIntegration: + """Integration tests for Snowflake platform""" + + @pytest.fixture + def sample_csv_file(self): + """Path to sample Snowflake CSV file""" + return 'input_samples/synthetic_snowflake_business_logs_1000.csv' + + @pytest.fixture + def sample_json_file(self): + """Path to sample Snowflake JSON file""" + return 'input_samples/synthetic_snowflake_business_logs_1000.json' + + @pytest.fixture + def output_dir(self): + """Create temporary output directory""" + temp_dir = tempfile.mkdtemp() + yield temp_dir + shutil.rmtree(temp_dir) + + def test_end_to_end_csv_workflow(self, sample_csv_file, output_dir): + """Test complete workflow with CSV input""" + # Skip if sample file doesn't exist + if not os.path.exists(sample_csv_file): + pytest.skip(f"Sample file not found: {sample_csv_file}") + + # Initialize + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + # Load data + recommender.load_query_logs_from_csv_file(sample_csv_file) + assert recommender.query_logs is not None + assert len(recommender.query_logs) > 0 + + # Calculate metrics + recommender.calculate_metrics() + assert recommender.table_metrics is not None + assert len(recommender.table_metrics) > 0 + + # Generate recommendations + recommendations = recommender.recommend_data_products(num_recommendations=10) + assert 'individual_tables' in recommendations + assert len(recommendations['individual_tables']) > 0 + + # Export markdown + md_file = os.path.join(output_dir, 'test_recommendations.md') + recommender.export_recommendations_markdown(recommendations, md_file) + assert os.path.exists(md_file) + assert os.path.getsize(md_file) > 0 + + # Export JSON + json_file = os.path.join(output_dir, 'test_recommendations.json') + recommender.export_recommendations_json(recommendations, json_file) + assert os.path.exists(json_file) + assert os.path.getsize(json_file) > 0 + + def test_end_to_end_json_workflow(self, sample_json_file, output_dir): + """Test complete workflow with JSON input""" + # Skip if sample file doesn't exist + if not os.path.exists(sample_json_file): + pytest.skip(f"Sample file not found: {sample_json_file}") + + # Initialize + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + # Load data + recommender.load_query_logs_from_json_file(sample_json_file) + assert recommender.query_logs is not None + assert len(recommender.query_logs) > 0 + + # Calculate metrics + recommender.calculate_metrics() + assert recommender.table_metrics is not None + + # Generate recommendations + recommendations = recommender.recommend_data_products(num_recommendations=10) + assert 'individual_tables' in recommendations + + def test_recommendations_with_min_score(self, sample_csv_file): + """Test recommendations with minimum score threshold""" + if not os.path.exists(sample_csv_file): + pytest.skip(f"Sample file not found: {sample_csv_file}") + + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_csv_file(sample_csv_file) + recommender.calculate_metrics() + + # Get recommendations with high threshold + recommendations = recommender.recommend_data_products( + num_recommendations=20, + min_score=70.0 + ) + + # Verify all recommendations meet threshold + for table in recommendations['individual_tables']: + assert table['recommendation_score'] >= 70.0 + + +class TestDatabricksIntegration: + """Integration tests for Databricks platform""" + + @pytest.fixture + def sample_csv_file(self): + """Path to sample Databricks CSV file""" + return 'input_samples/synthetic_databricks_healthcare_logs_1000.csv' + + @pytest.fixture + def sample_json_file(self): + """Path to sample Databricks JSON file""" + return 'input_samples/synthetic_databricks_healthcare_logs_1000.json' + + def test_databricks_csv_workflow(self, sample_csv_file): + """Test Databricks workflow with CSV input""" + if not os.path.exists(sample_csv_file): + pytest.skip(f"Sample file not found: {sample_csv_file}") + + parser = DatabricksQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_csv_file(sample_csv_file) + assert recommender.query_logs is not None + + recommender.calculate_metrics() + assert recommender.table_metrics is not None + + recommendations = recommender.recommend_data_products(num_recommendations=10) + assert 'individual_tables' in recommendations + + def test_databricks_json_workflow(self, sample_json_file): + """Test Databricks workflow with JSON input""" + if not os.path.exists(sample_json_file): + pytest.skip(f"Sample file not found: {sample_json_file}") + + parser = DatabricksQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_json_file(sample_json_file) + assert recommender.query_logs is not None + + recommender.calculate_metrics() + recommendations = recommender.recommend_data_products(num_recommendations=10) + assert 'individual_tables' in recommendations + + +class TestBigQueryIntegration: + """Integration tests for BigQuery platform""" + + @pytest.fixture + def sample_csv_file(self): + """Path to sample BigQuery CSV file""" + return 'input_samples/synthetic_bigquery_retail_logs_1000.csv' + + @pytest.fixture + def sample_json_file(self): + """Path to sample BigQuery JSON file""" + return 'input_samples/synthetic_bigquery_retail_logs_1000.json' + + def test_bigquery_csv_workflow(self, sample_csv_file): + """Test BigQuery workflow with CSV input""" + if not os.path.exists(sample_csv_file): + pytest.skip(f"Sample file not found: {sample_csv_file}") + + parser = BigQueryQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_csv_file(sample_csv_file) + assert recommender.query_logs is not None + + recommender.calculate_metrics() + assert recommender.table_metrics is not None + + recommendations = recommender.recommend_data_products(num_recommendations=10) + assert 'individual_tables' in recommendations + + def test_bigquery_json_workflow(self, sample_json_file): + """Test BigQuery workflow with JSON input""" + if not os.path.exists(sample_json_file): + pytest.skip(f"Sample file not found: {sample_json_file}") + + parser = BigQueryQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_json_file(sample_json_file) + assert recommender.query_logs is not None + + recommender.calculate_metrics() + recommendations = recommender.recommend_data_products(num_recommendations=10) + assert 'individual_tables' in recommendations + + +class TestWatsonxDataIntegration: + """Integration tests for watsonx.data platform""" + + @pytest.fixture + def sample_csv_file(self): + """Path to sample watsonx.data CSV file""" + return 'input_samples/synthetic_watsonx_telecom_logs_1000.csv' + + @pytest.fixture + def sample_json_file(self): + """Path to sample watsonx.data JSON file""" + return 'input_samples/synthetic_watsonx_telecom_logs_1000.json' + + def test_watsonxdata_csv_workflow(self, sample_csv_file): + """Test watsonx.data workflow with CSV input""" + if not os.path.exists(sample_csv_file): + pytest.skip(f"Sample file not found: {sample_csv_file}") + + parser = WatsonxDataQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_csv_file(sample_csv_file) + assert recommender.query_logs is not None + + recommender.calculate_metrics() + assert recommender.table_metrics is not None + + recommendations = recommender.recommend_data_products(num_recommendations=10) + assert 'individual_tables' in recommendations + + def test_watsonxdata_json_workflow(self, sample_json_file): + """Test watsonx.data workflow with JSON input""" + if not os.path.exists(sample_json_file): + pytest.skip(f"Sample file not found: {sample_json_file}") + + parser = WatsonxDataQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_json_file(sample_json_file) + assert recommender.query_logs is not None + + recommender.calculate_metrics() + recommendations = recommender.recommend_data_products(num_recommendations=10) + assert 'individual_tables' in recommendations + + +class TestTableGrouping: + """Integration tests for table grouping functionality""" + + @pytest.fixture + def sample_file(self): + """Path to sample file with table relationships""" + return 'input_samples/synthetic_snowflake_business_logs_1000.csv' + + def test_table_grouping_detection(self, sample_file): + """Test that table groups are detected correctly""" + if not os.path.exists(sample_file): + pytest.skip(f"Sample file not found: {sample_file}") + + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_csv_file(sample_file) + recommender.calculate_metrics() + + recommendations = recommender.recommend_data_products( + num_recommendations=20, + min_frequency_threshold=0.05 # 5% threshold + ) + + # Check if table groups were identified + if 'table_groups' in recommendations and len(recommendations['table_groups']) > 0: + # Verify group structure + for group in recommendations['table_groups']: + assert 'tables' in group + assert 'group_score' in group # Changed from 'score' to 'group_score' + assert len(group['tables']) >= 2 # Groups should have at least 2 tables + + +class TestOutputFormats: + """Integration tests for output format generation""" + + @pytest.fixture + def sample_file(self): + return 'input_samples/synthetic_snowflake_business_logs_1000.csv' + + @pytest.fixture + def output_dir(self): + temp_dir = tempfile.mkdtemp() + yield temp_dir + shutil.rmtree(temp_dir) + + def test_markdown_output_structure(self, sample_file, output_dir): + """Test that markdown output has correct structure""" + if not os.path.exists(sample_file): + pytest.skip(f"Sample file not found: {sample_file}") + + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_csv_file(sample_file) + recommender.calculate_metrics() + recommendations = recommender.recommend_data_products(num_recommendations=10) + + md_file = os.path.join(output_dir, 'test_output.md') + recommender.export_recommendations_markdown(recommendations, md_file) + + # Read and verify markdown content + with open(md_file, 'r') as f: + content = f.read() + assert '# Data Product Recommendations' in content + assert '## Summary Statistics' in content + # Check that content has data product information (tables or metrics) + assert ('ANALYTICS.' in content or 'SALES.' in content or 'PRODUCT.' in content) + + def test_json_output_structure(self, sample_file, output_dir): + """Test that JSON output has correct structure""" + if not os.path.exists(sample_file): + pytest.skip(f"Sample file not found: {sample_file}") + + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_csv_file(sample_file) + recommender.calculate_metrics() + recommendations = recommender.recommend_data_products(num_recommendations=10) + + json_file = os.path.join(output_dir, 'test_output.json') + recommender.export_recommendations_json(recommendations, json_file) + + # Read and verify JSON structure + import json + with open(json_file, 'r') as f: + data = json.load(f) + assert 'recommendations' in data + assert 'metadata' in data + assert 'generated_at' in data['metadata'] + assert 'total_queries_analyzed' in data['metadata'] + + +if __name__ == '__main__': + pytest.main([__file__, '-v']) + +# Made with Bob diff --git a/tests/src/integration/test_odcs_generator_collibra.py b/tests/src/integration/test_odcs_generator_collibra.py new file mode 100644 index 0000000..c2efdef --- /dev/null +++ b/tests/src/integration/test_odcs_generator_collibra.py @@ -0,0 +1,611 @@ +# -*- coding: utf-8 -*- +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Integration Tests for Collibra ODCS Generator +""" + +import os +import pytest +import tempfile +import yaml +from unittest.mock import Mock, patch, MagicMock +from datetime import datetime +from wxdi.odcs_generator.generate_odcs_from_collibra import ( + CollibraClient, + ODCSGenerator, + parse_arguments, + validate_arguments, + determine_output_file, + write_yaml_file +) + + +class TestCollibraClient: + """Test CollibraClient class""" + + def test_client_initialization(self): + """Test client initialization with valid parameters""" + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + + assert client.base_url == "https://acme.collibra.com" + assert client.auth == ("testuser", "testpass") + assert client.session is not None + + def test_base_url_trailing_slash_removal(self): + """Test that trailing slash is removed from base URL""" + client = CollibraClient( + base_url="https://acme.collibra.com/", + username="testuser", + password="testpass" + ) + + assert client.base_url == "https://acme.collibra.com" + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset(self, mock_get): + """Test asset retrieval""" + mock_response = Mock() + mock_response.json.return_value = { + "id": "test-asset-id", + "displayName": "Test Table", + "type": {"name": "Table"} + } + mock_response.raise_for_status = Mock() + mock_get.return_value = mock_response + + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + + asset = client.get_asset("test-asset-id") + + assert asset["id"] == "test-asset-id" + assert asset["displayName"] == "Test Table" + mock_get.assert_called_once() + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset_attributes(self, mock_get): + """Test asset attributes retrieval""" + mock_response = Mock() + mock_response.json.return_value = { + "results": [ + { + "type": {"name": "Description"}, + "value": "Test description" + }, + { + "type": {"name": "Data Type"}, + "value": "VARCHAR" + } + ] + } + mock_response.raise_for_status = Mock() + mock_get.return_value = mock_response + + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + + attributes = client.get_asset_attributes("test-asset-id") + + assert len(attributes) == 2 + assert attributes[0]["type"]["name"] == "Description" + assert attributes[1]["type"]["name"] == "Data Type" + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset_relations_as_source(self, mock_get): + """Test asset relations retrieval where asset is source""" + mock_response = Mock() + mock_response.json.return_value = { + "results": [ + { + "source": {"id": "asset-1"}, + "target": {"id": "column-1"} + } + ] + } + mock_response.raise_for_status = Mock() + mock_get.return_value = mock_response + + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + + relations = client.get_asset_relations("asset-1", as_source=True) + + assert len(relations) == 1 + assert relations[0]["source"]["id"] == "asset-1" + # Verify correct parameter was used + call_args = mock_get.call_args + assert 'sourceId' in call_args[1]['params'] + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset_relations_as_target(self, mock_get): + """Test asset relations retrieval where asset is target""" + mock_response = Mock() + mock_response.json.return_value = { + "results": [ + { + "source": {"id": "column-1"}, + "target": {"id": "asset-1"} + } + ] + } + mock_response.raise_for_status = Mock() + mock_get.return_value = mock_response + + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + + relations = client.get_asset_relations("asset-1", as_source=False) + + assert len(relations) == 1 + assert relations[0]["target"]["id"] == "asset-1" + # Verify correct parameter was used + call_args = mock_get.call_args + assert 'targetId' in call_args[1]['params'] + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset_tags(self, mock_get): + """Test asset tags retrieval""" + mock_response = Mock() + mock_response.json.return_value = [ + {"name": "PII"}, + {"name": "Sensitive"}, + {"name": "Production"} + ] + mock_response.raise_for_status = Mock() + mock_get.return_value = mock_response + + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + + tags = client.get_asset_tags("test-asset-id") + + assert len(tags) == 3 + assert "PII" in tags + assert "Sensitive" in tags + assert "Production" in tags + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset_tags_error_handling(self, mock_get): + """Test that tag retrieval errors are handled gracefully""" + mock_get.side_effect = Exception("API Error") + + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + + tags = client.get_asset_tags("test-asset-id") + + assert tags == [] + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.post') + def test_get_asset_classifications(self, mock_post): + """Test asset classifications retrieval via GraphQL""" + mock_response = Mock() + mock_response.json.return_value = { + "data": { + "api": { + "asset": { + "id": "test-asset-id", + "classesForAsset": [ + { + "id": "class-1", + "label": "Confidential", + "status": "ACCEPTED" + }, + { + "id": "class-2", + "label": "Public", + "status": "REJECTED" + } + ] + } + } + } + } + mock_response.raise_for_status = Mock() + mock_post.return_value = mock_response + + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + + classifications = client.get_asset_classifications("test-asset-id") + + # Only ACCEPTED classifications should be returned + assert len(classifications) == 1 + assert "Confidential" in classifications + assert "Public" not in classifications + + +class TestODCSGenerator: + """Test ODCSGenerator class""" + + def test_generator_initialization(self): + """Test generator initialization""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + assert generator.client == mock_client + + def test_convert_timestamp(self): + """Test timestamp conversion""" + # Test with valid timestamp (milliseconds) + timestamp_ms = 1609459200000 # 2021-01-01 00:00:00 UTC + result = ODCSGenerator._convert_timestamp(timestamp_ms) + + assert "2021-01-01" in result + assert result.endswith('Z') + + # Test with zero timestamp + result = ODCSGenerator._convert_timestamp(0) + assert result.endswith('Z') + + def test_build_attribute_map(self): + """Test attribute map building""" + attributes = [ + { + "type": {"name": "Description"}, + "value": "Test description" + }, + { + "type": {"name": "Data Type"}, + "value": "VARCHAR" + }, + { + "type": {"name": "Empty"}, + "value": None + } + ] + + attr_map = ODCSGenerator._build_attribute_map(attributes) + + assert attr_map["Description"] == "Test description" + assert attr_map["Data Type"] == "VARCHAR" + assert "Empty" not in attr_map + + def test_logical_type_mapping(self): + """Test logical type mapping""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["text"] == "string" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["whole number"] == "integer" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["decimal number"] == "number" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["date time"] == "timestamp" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["true/false"] == "boolean" + + def test_create_server_definition(self): + """Test server definition creation""" + server = ODCSGenerator._create_server_definition() + + assert "id" in server + assert server["id"].startswith("server-") + assert server["server"] == "CONFIGURE_SERVER_HOSTNAME" + assert server["type"] == "DEFINE_SERVER_TYPE" + + def test_extract_custom_properties(self): + """Test custom properties extraction""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + attr_map = { + "Description": "Test description", # Should be excluded + "Owner": "John Doe", + "Created Date": "2024-01-01", + "Custom Field": "Custom Value" + } + + custom_props = generator._extract_custom_properties(attr_map) + + # Description should be excluded + assert not any(prop["property"] == "description" for prop in custom_props) + + # Other properties should be included (converted to lowercase with underscores) + assert any(prop["property"] == "owner" for prop in custom_props) + assert any(prop["property"] == "created_date" for prop in custom_props) + + @patch.object(CollibraClient, 'get_asset') + @patch.object(CollibraClient, 'get_asset_tags') + @patch.object(CollibraClient, 'get_asset_attributes') + @patch.object(CollibraClient, 'get_asset_relations') + def test_generate_odcs_structure(self, mock_relations, mock_attrs, mock_tags, mock_asset): + """Test ODCS structure generation""" + # Mock asset data + mock_asset.return_value = { + "id": "test-asset-id", + "displayName": "Test Table", + "type": {"name": "Table"}, + "domain": {"name": "Finance"}, + "createdOn": 1609459200000 + } + + mock_tags.return_value = ["PII", "Production"] + mock_attrs.return_value = [ + {"type": {"name": "Description"}, "value": "Test table description"} + ] + mock_relations.return_value = [] + + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + generator = ODCSGenerator(client) + + odcs = generator.generate_odcs("test-asset-id") + + # Verify ODCS structure + assert odcs["id"] == "test-asset-id" + assert odcs["kind"] == "DataContract" + assert odcs["apiVersion"] == "v3.1.0" + assert odcs["domain"] == "Finance" + assert odcs["status"] == "active" + assert "PII" in odcs["tags"] + assert "Production" in odcs["tags"] + + # Verify authoritative definitions + assert len(odcs["description"]["authoritativeDefinitions"]) == 1 + auth_def = odcs["description"]["authoritativeDefinitions"][0] + assert auth_def["type"] == "collibra-asset" + assert "test-asset-id" in auth_def["url"] + + # Verify schema + assert "schema" in odcs + assert isinstance(odcs["schema"], list) + + # Verify servers + assert "servers" in odcs + assert isinstance(odcs["servers"], list) + + +class TestArgumentParsing: + """Test command-line argument parsing""" + + def test_parse_arguments_with_asset_id(self): + """Test parsing with required asset ID""" + with patch('sys.argv', ['script.py', 'test-asset-id']): + args = parse_arguments() + assert args.asset_id == 'test-asset-id' + + def test_parse_arguments_with_all_options(self): + """Test parsing with all command-line options""" + with patch('sys.argv', [ + 'script.py', + 'test-asset-id', + '-o', 'output.yaml', + '--url', 'https://acme.collibra.com', + '-u', 'testuser', + '-p', 'testpass' + ]): + args = parse_arguments() + assert args.asset_id == 'test-asset-id' + assert args.output == 'output.yaml' + assert args.url == 'https://acme.collibra.com' + assert args.username == 'testuser' + assert args.password == 'testpass' + + @patch.dict(os.environ, { + 'COLLIBRA_URL': 'https://env.collibra.com', + 'COLLIBRA_USERNAME': 'envuser', + 'COLLIBRA_PASSWORD': 'envpass' + }) + def test_parse_arguments_from_environment(self): + """Test that environment variables are used as defaults""" + with patch('sys.argv', ['script.py', 'test-asset-id']): + args = parse_arguments() + assert args.url == 'https://env.collibra.com' + assert args.username == 'envuser' + assert args.password == 'envpass' + + +class TestHelperFunctions: + """Test helper functions""" + + def test_determine_output_file_with_custom_output(self): + """Test output file determination with custom output""" + args = Mock() + args.output = 'custom-output.yaml' + + odcs_data = {'name': 'Test Contract'} + + output_file = determine_output_file(args, odcs_data) + assert output_file == 'custom-output.yaml' + + def test_determine_output_file_from_asset_name(self): + """Test output file determination from asset name""" + args = Mock() + args.output = None + + odcs_data = {'name': 'Customer Transactions'} + + output_file = determine_output_file(args, odcs_data) + assert output_file == 'customer-transactions-odcs.yaml' + + def test_determine_output_file_default(self): + """Test output file determination with default""" + args = Mock() + args.output = None + + odcs_data = {} + + output_file = determine_output_file(args, odcs_data) + assert output_file == 'asset-odcs.yaml' + + def test_write_yaml_file(self): + """Test YAML file writing""" + odcs_data = { + "id": "test-id", + "kind": "DataContract", + "apiVersion": "v3.1.0", + "schema": [{"name": "test_table"}], + "servers": [{"id": "server-1", "server": "CONFIGURE_SERVER_HOSTNAME"}] + } + + with tempfile.NamedTemporaryFile(mode='w', suffix='.yaml', delete=False) as f: + temp_file = f.name + + try: + # Write YAML + write_yaml_file(temp_file, odcs_data) + + # Read and verify + with open(temp_file, 'r') as f: + content = f.read() + loaded_data = yaml.safe_load(content) + + assert loaded_data["id"] == "test-id" + assert loaded_data["kind"] == "DataContract" + assert "⚠️" in content # Check for warning comments + finally: + if os.path.exists(temp_file): + os.unlink(temp_file) + + +class TestIntegrationScenarios: + """Integration test scenarios""" + + @patch.dict(os.environ, { + 'COLLIBRA_URL': 'https://acme.collibra.com', + 'COLLIBRA_USERNAME': 'testuser', + 'COLLIBRA_PASSWORD': 'testpass' + }) + @patch.object(CollibraClient, 'get_asset') + @patch.object(CollibraClient, 'get_asset_tags') + @patch.object(CollibraClient, 'get_asset_attributes') + @patch.object(CollibraClient, 'get_asset_relations') + def test_end_to_end_odcs_generation(self, mock_relations, mock_attrs, mock_tags, mock_asset): + """Test end-to-end ODCS generation flow""" + # Mock asset data + mock_asset.return_value = { + "id": "test-id", + "displayName": "CUSTOMER_TABLE", + "type": {"name": "Table"}, + "domain": {"name": "Sales"}, + "createdOn": 1609459200000 + } + + mock_tags.return_value = ["Production"] + + mock_attrs.return_value = [ + {"type": {"name": "Description"}, "value": "Customer information table"}, + {"type": {"name": "Owner"}, "value": "Data Team"} + ] + + # Mock relations with columns + mock_relations.side_effect = [ + [], # as_source + [ # as_target + { + "source": { + "id": "col-1", + "displayName": "customer_id", + "type": {"name": "Column"} + }, + "target": {"id": "test-id"} + } + ] + ] + + # Create client and generator + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + generator = ODCSGenerator(client) + + # Generate ODCS + odcs = generator.generate_odcs("test-id") + + # Verify ODCS structure + assert odcs["id"] == "test-id" + assert odcs["kind"] == "DataContract" + assert odcs["domain"] == "Sales" + assert len(odcs["schema"]) == 1 + assert odcs["schema"][0]["name"] == "CUSTOMER_TABLE" + assert odcs["schema"][0]["description"] == "Customer information table" + + def test_error_handling_invalid_asset_id(self): + """Test error handling for invalid asset ID""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + with pytest.raises(ValueError, match="Asset ID is required"): + generator.generate_odcs("") + + @patch.object(CollibraClient, 'get_asset') + def test_error_handling_api_failure(self, mock_asset): + """Test error handling when API calls fail""" + mock_asset.side_effect = Exception("API Error") + + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpswd" + ) + generator = ODCSGenerator(client) + + # Should handle error gracefully + with pytest.raises(Exception): + generator.generate_odcs("test-id") + + +class TestDataTypeMapping: + """Test data type mapping functionality""" + + def test_logical_type_mapping_coverage(self): + """Test that common data types are mapped""" + mapping = ODCSGenerator.LOGICAL_TYPE_MAPPING + + assert mapping["text"] == "string" + assert mapping["whole number"] == "integer" + assert mapping["decimal number"] == "number" + assert mapping["date time"] == "timestamp" + assert mapping["true/false"] == "boolean" + assert mapping["geographical"] == "string" + + def test_numeric_types_list(self): + """Test numeric types list""" + numeric_types = ODCSGenerator.NUMERIC_TYPES + + assert "DECIMAL" in numeric_types + assert "NUMERIC" in numeric_types + assert "NUMBER" in numeric_types + + +if __name__ == '__main__': + pytest.main([__file__, '-v']) + diff --git a/tests/src/integration/test_odcs_generator_informatica.py b/tests/src/integration/test_odcs_generator_informatica.py new file mode 100644 index 0000000..a256e9e --- /dev/null +++ b/tests/src/integration/test_odcs_generator_informatica.py @@ -0,0 +1,389 @@ +# -*- coding: utf-8 -*- +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Integration Tests for Informatica ODCS Generator +""" + +import os +import pytest +import tempfile +import yaml +from unittest.mock import Mock, patch, MagicMock +from wxdi.odcs_generator.generate_odcs_from_informatica import ( + InformaticaClient, + parse_arguments, + validate_arguments, + generate_odcs_yaml, + build_physical_type, + build_column_property, + build_custom_properties, + extract_column_position, + RESOURCE_TYPE_MAPPING, + SYSTEM_ATTRIBUTES_MAPPING +) + + +class TestInformaticaClient: + """Test InformaticaClient class""" + + def test_client_initialization(self): + """Test client initialization with valid parameters""" + client = InformaticaClient( + base_url="https://cdgc.dm-us.informaticacloud.com", + username="testuser", + password="testpass" + ) + + assert client.base_url == "https://cdgc.dm-us.informaticacloud.com" + assert client.username == "testuser" + assert client.password == "testpass" + assert client.region == "dm-us" + assert client.identity_url == "https://dm-us.informaticacloud.com" + + def test_extract_region_from_url(self): + """Test region extraction from various URL formats""" + client = InformaticaClient( + base_url="https://cdgc.dm-us.informaticacloud.com", + username="test", + password="test" + ) + + # Test various URL formats + assert client._extract_region_from_url("https://cdgc.dm-us.informaticacloud.com") == "dm-us" + assert client._extract_region_from_url("https://cdgc.na1.informaticacloud.com") == "na1" + assert client._extract_region_from_url("https://cdgc.eu1.informaticacloud.com/") == "eu1" + + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.requests.post') + def test_get_session_id(self, mock_post): + """Test session ID retrieval""" + mock_response = Mock() + mock_response.json.return_value = {"sessionId": "test-session-id"} + mock_response.raise_for_status = Mock() + mock_post.return_value = mock_response + + client = InformaticaClient( + base_url="https://cdgc.dm-us.informaticacloud.com", + username="testuser", + password="testpass" + ) + + session_data = client.get_session_id() + + assert session_data["sessionId"] == "test-session-id" + mock_post.assert_called_once() + + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.requests.post') + def test_get_auth_token_caching(self, mock_post): + """Test that auth token is cached""" + # Mock session ID call + mock_session_response = Mock() + mock_session_response.json.return_value = {"sessionId": "test-session-id"} + mock_session_response.raise_for_status = Mock() + + # Mock JWT token call + mock_jwt_response = Mock() + mock_jwt_response.json.return_value = {"jwt_token": "test-jwt-token"} + mock_jwt_response.raise_for_status = Mock() + + mock_post.side_effect = [mock_session_response, mock_jwt_response] + + client = InformaticaClient( + base_url="https://cdgc.dm-us.informaticacloud.com", + username="testuser", + password="testpass" + ) + + # First call should make API requests + token1 = client.get_auth_token() + assert token1 == "test-jwt-token" + assert mock_post.call_count == 2 + + # Second call should use cached token + token2 = client.get_auth_token() + assert token2 == "test-jwt-token" + assert mock_post.call_count == 2 # No additional calls + + +class TestHelperFunctions: + """Test helper functions""" + + def test_extract_column_position(self): + """Test column position extraction""" + # Valid position + col_data = {"selfAttributes": {"core.Position": "5"}} + assert extract_column_position(col_data) == 5 + + # Missing position + col_data = {"selfAttributes": {}} + assert extract_column_position(col_data) == 999999 + + # Invalid position + col_data = {"selfAttributes": {"core.Position": "invalid"}} + assert extract_column_position(col_data) == 999999 + + def test_build_physical_type(self): + """Test physical type construction""" + # Type with length + assert build_physical_type("VARCHAR", "255", "") == "VARCHAR(255)" + + # Type with length and scale + assert build_physical_type("DECIMAL", "10", "2") == "DECIMAL(10,2)" + + # Type without length + assert build_physical_type("INTEGER", "", "") == "INTEGER" + + # Type with length but scale is 0 + assert build_physical_type("NUMBER", "18", "0") == "NUMBER(18)" + + def test_build_column_property(self): + """Test column property building""" + column_detail = { + "summary": { + "core.name": "customer_id", + "core.description": "Customer identifier" + }, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "VARCHAR", + "com.infa.odin.models.relational.DatatypeLength": "50", + "com.infa.odin.models.relational.Nullable": "false", + "com.infa.odin.models.relational.PrimaryKeyColumn": "true" + } + } + + prop = build_column_property(column_detail) + + assert prop["name"] == "customer_id" + assert prop["physicalType"] == "VARCHAR(50)" + assert prop["description"] == "Customer identifier" + assert prop["required"] is True + assert prop["primaryKey"] is True + + def test_build_custom_properties(self): + """Test custom properties building""" + table_attrs = { + "core.resourceName": "TestCatalog", + "com.infa.odin.models.relational.NumberOfRows": "1000", + "core.origin": "Production", + "com.infa.odin.models.relational.Owner": "dbo" + } + + custom_props = build_custom_properties(table_attrs) + + assert len(custom_props) == 4 + assert {"property": "Catalog Source Name", "value": "TestCatalog"} in custom_props + assert {"property": "Number of rows", "value": "1000"} in custom_props + assert {"property": "Origin", "value": "Production"} in custom_props + assert {"property": "Schema", "value": "dbo"} in custom_props + + +class TestODCSGeneration: + """Test ODCS YAML generation""" + + def test_generate_odcs_yaml_structure(self): + """Test ODCS YAML structure generation""" + asset_data = { + "core.identity": "test-asset-id", + "summary": { + "core.name": "CUSTOMER_TABLE", + "core.description": "Customer information table" + }, + "selfAttributes": { + "core.name": "customer_table", + "core.businessName": "customer_table", + "com.infa.odin.models.relational.Owner": "dbo", + "core.resourceType": "Snowflake", + "core.resourceName": "TestCatalog" + } + } + + column_details = [ + { + "summary": { + "core.name": "id", + "core.description": "Primary key" + }, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "INTEGER", + "com.infa.odin.models.relational.Nullable": "false", + "com.infa.odin.models.relational.PrimaryKeyColumn": "true", + "core.Position": "1" + } + } + ] + + base_url = "https://cdgc.dm-us.informaticacloud.com" + + odcs = generate_odcs_yaml(asset_data, column_details, base_url) + + # Verify structure + assert odcs["id"] == "test-asset-id" + assert odcs["kind"] == "DataContract" + assert odcs["apiVersion"] == "v3.1.0" + assert odcs["status"] == "active" + + # Verify schema + assert len(odcs["schema"]) == 1 + schema = odcs["schema"][0] + assert schema["name"] == "customer_table" + assert schema["physicalName"] == "dbo/customer_table" + assert schema["physicalType"] == "Table" + assert len(schema["properties"]) == 1 + + # Verify column + column = schema["properties"][0] + assert column["name"] == "id" + assert column["physicalType"] == "INTEGER" + assert column["required"] is True + assert column["primaryKey"] is True + + # Verify servers + assert len(odcs["servers"]) == 1 + server = odcs["servers"][0] + assert server["type"] == "snowflake" + assert server["schema"] == "dbo" + + # Verify authoritative definitions + assert len(odcs["description"]["authoritativeDefinitions"]) == 1 + auth_def = odcs["description"]["authoritativeDefinitions"][0] + assert auth_def["type"] == "informatica-asset" + assert "test-asset-id" in auth_def["url"] + + +class TestResourceTypeMapping: + """Test resource type mapping""" + + def test_resource_type_mapping_coverage(self): + """Test that common database types are mapped""" + assert RESOURCE_TYPE_MAPPING["Snowflake"] == "snowflake" + assert RESOURCE_TYPE_MAPPING["PostgreSQL"] == "postgresql" + assert RESOURCE_TYPE_MAPPING["Oracle"] == "oracle" + assert RESOURCE_TYPE_MAPPING["SqlServer"] == "sqlserver" + assert RESOURCE_TYPE_MAPPING["BigQuery"] == "bigquery" + assert RESOURCE_TYPE_MAPPING["Redshift"] == "redshift" + + +class TestArgumentParsing: + """Test command-line argument parsing""" + + def test_parse_arguments_with_asset_id(self): + """Test parsing with required asset ID""" + with patch('sys.argv', ['script.py', 'test-asset-id']): + args = parse_arguments() + assert args.asset_id == 'test-asset-id' + + def test_parse_arguments_with_all_options(self): + """Test parsing with all command-line options""" + with patch('sys.argv', [ + 'script.py', + 'test-asset-id', + '-o', 'output.yaml', + '--cdgc-url', 'https://cdgc.test.com', + '-u', 'testuser', + '-p', 'testpass' + ]): + args = parse_arguments() + assert args.asset_id == 'test-asset-id' + assert args.output == 'output.yaml' + assert args.cdgc_url == 'https://cdgc.test.com' + assert args.username == 'testuser' + assert args.password == 'testpass' + + +class TestIntegrationScenarios: + """Integration test scenarios""" + + @patch.dict(os.environ, { + 'INFORMATICA_CDGC_URL': 'https://cdgc.dm-us.informaticacloud.com', + 'INFORMATICA_USERNAME': 'testuser', + 'INFORMATICA_PASSWORD': 'testpawd' + }) + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.InformaticaClient') + def test_end_to_end_odcs_generation(self, mock_client_class): + """Test end-to-end ODCS generation flow""" + # Mock client instance + mock_client = Mock() + mock_client.base_url = "https://cdgc.dm-us.informaticacloud.com" + mock_client_class.return_value = mock_client + + # Mock asset data + mock_client.get_asset_details.return_value = { + "core.identity": "test-id", + "summary": {"core.name": "TEST_TABLE"}, + "selfAttributes": { + "core.businessName": "test_table", + "com.infa.odin.models.relational.Owner": "public", + "core.resourceType": "PostgreSQL" + }, + "hierarchy": [{"core.identity": "col-1"}] + } + + # Mock column data + mock_client.get_column_details.return_value = { + "summary": {"core.name": "id"}, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "INTEGER", + "core.Position": "1" + } + } + + # Generate ODCS + asset_data = mock_client.get_asset_details("test-id") + column_ids = [col['core.identity'] for col in asset_data.get('hierarchy', [])] + column_details = [mock_client.get_column_details(col_id) for col_id in column_ids] + + odcs = generate_odcs_yaml(asset_data, column_details, mock_client.base_url) + + # Verify ODCS structure + assert odcs["id"] == "test-id" + assert odcs["kind"] == "DataContract" + assert len(odcs["schema"]) == 1 + assert len(odcs["schema"][0]["properties"]) == 1 + assert odcs["servers"][0]["type"] == "postgresql" + + def test_yaml_file_generation(self): + """Test YAML file writing""" + odcs_data = { + "id": "test-id", + "kind": "DataContract", + "apiVersion": "v3.1.0", + "schema": [{"name": "test_table"}], + "servers": [{"id": "server-1", "server": "CONFIGURE_SERVER_HOSTNAME"}] + } + + with tempfile.NamedTemporaryFile(mode='w', suffix='.yaml', delete=False) as f: + temp_file = f.name + + try: + # Write YAML + yaml_content = yaml.dump(odcs_data, default_flow_style=False, sort_keys=False) + with open(temp_file, 'w') as f: + f.write(yaml_content) + + # Read and verify + with open(temp_file, 'r') as f: + loaded_data = yaml.safe_load(f) + + assert loaded_data["id"] == "test-id" + assert loaded_data["kind"] == "DataContract" + assert loaded_data["apiVersion"] == "v3.1.0" + finally: + if os.path.exists(temp_file): + os.unlink(temp_file) + + +if __name__ == '__main__': + pytest.main([__file__, '-v']) + diff --git a/tests/src/odcs_generator/__init__.py b/tests/src/odcs_generator/__init__.py new file mode 100644 index 0000000..3b445d4 --- /dev/null +++ b/tests/src/odcs_generator/__init__.py @@ -0,0 +1,19 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""ODCS Generator test package""" + +# Made with Bob diff --git a/tests/src/odcs_generator/test_odcs_generator_collibra.py b/tests/src/odcs_generator/test_odcs_generator_collibra.py new file mode 100644 index 0000000..79152a0 --- /dev/null +++ b/tests/src/odcs_generator/test_odcs_generator_collibra.py @@ -0,0 +1,739 @@ +# -*- coding: utf-8 -*- +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Unit Tests for Collibra ODCS Generator +Tests individual functions and methods in isolation +""" + +import pytest +import os +from unittest.mock import Mock, patch +from datetime import datetime +from wxdi.odcs_generator.generate_odcs_from_collibra import ( + CollibraClient, + ODCSGenerator, + determine_output_file +) + + +class TestODCSGeneratorStaticMethods: + """Unit tests for ODCSGenerator static methods""" + + def test_convert_timestamp_with_valid_milliseconds(self): + """Test timestamp conversion with valid milliseconds""" + # 2021-01-01 00:00:00 UTC in milliseconds + timestamp_ms = 1609459200000 + result = ODCSGenerator._convert_timestamp(timestamp_ms) + + assert "2021-01-01" in result + assert result.endswith('Z') + + def test_convert_timestamp_with_zero(self): + """Test timestamp conversion with zero""" + result = ODCSGenerator._convert_timestamp(0) + + # Should return current time + assert result.endswith('Z') + assert 'T' in result + + def test_convert_timestamp_with_none(self): + """Test timestamp conversion with None/falsy value""" + result = ODCSGenerator._convert_timestamp(None) + + # Should return current time + assert result.endswith('Z') + + def test_convert_timestamp_format(self): + """Test that timestamp format is ISO 8601""" + timestamp_ms = 1609459200000 + result = ODCSGenerator._convert_timestamp(timestamp_ms) + + # Should contain date and time separator + assert 'T' in result + # Should end with Z for UTC + assert result.endswith('Z') + + def test_build_attribute_map_empty(self): + """Test building attribute map with empty list""" + result = ODCSGenerator._build_attribute_map([]) + assert result == {} + + def test_build_attribute_map_single_attribute(self): + """Test building attribute map with single attribute""" + attributes = [ + { + "type": {"name": "Description"}, + "value": "Test description" + } + ] + + result = ODCSGenerator._build_attribute_map(attributes) + + assert result["Description"] == "Test description" + + def test_build_attribute_map_multiple_attributes(self): + """Test building attribute map with multiple attributes""" + attributes = [ + {"type": {"name": "Description"}, "value": "Desc1"}, + {"type": {"name": "Data Type"}, "value": "VARCHAR"}, + {"type": {"name": "Owner"}, "value": "John"} + ] + + result = ODCSGenerator._build_attribute_map(attributes) + + assert len(result) == 3 + assert result["Description"] == "Desc1" + assert result["Data Type"] == "VARCHAR" + assert result["Owner"] == "John" + + def test_build_attribute_map_missing_type_name(self): + """Test that attributes without type name are skipped""" + attributes = [ + {"type": {}, "value": "Value1"}, + {"type": {"name": "Valid"}, "value": "Value2"} + ] + + result = ODCSGenerator._build_attribute_map(attributes) + + assert len(result) == 1 + assert result["Valid"] == "Value2" + + def test_build_attribute_map_missing_value(self): + """Test that attributes without value are skipped""" + attributes = [ + {"type": {"name": "Empty"}, "value": None}, + {"type": {"name": "Valid"}, "value": "Value"} + ] + + result = ODCSGenerator._build_attribute_map(attributes) + + assert len(result) == 1 + assert result["Valid"] == "Value" + + def test_build_attribute_map_empty_string_value(self): + """Test that attributes with empty string value are skipped""" + attributes = [ + {"type": {"name": "Empty"}, "value": ""}, + {"type": {"name": "Valid"}, "value": "Value"} + ] + + result = ODCSGenerator._build_attribute_map(attributes) + + assert len(result) == 1 + assert result["Valid"] == "Value" + + def test_create_server_definition(self): + """Test server definition creation""" + server = ODCSGenerator._create_server_definition() + + assert "id" in server + assert server["id"].startswith("server-") + assert len(server["id"]) == 15 # "server-" + 8 hex chars + assert server["server"] == "CONFIGURE_SERVER_HOSTNAME" + assert server["type"] == "DEFINE_SERVER_TYPE" + + def test_create_server_definition_unique_ids(self): + """Test that server definitions have unique IDs""" + server1 = ODCSGenerator._create_server_definition() + server2 = ODCSGenerator._create_server_definition() + + assert server1["id"] != server2["id"] + + +class TestODCSGeneratorExtractCustomProperties: + """Unit tests for _extract_custom_properties method""" + + def test_extract_custom_properties_empty(self): + """Test with empty attribute map""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + result = generator._extract_custom_properties({}) + assert result == [] + + def test_extract_custom_properties_excludes_description(self): + """Test that Description is excluded""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + attr_map = { + "Description": "This should be excluded", + "Owner": "John Doe" + } + + result = generator._extract_custom_properties(attr_map) + + # Description should not be in custom properties + assert not any(prop["property"] == "description" for prop in result) + # Owner should be included (converted to lowercase) + assert any(prop["property"] == "owner" for prop in result) + + def test_extract_custom_properties_multiple(self): + """Test with multiple custom properties""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + attr_map = { + "Owner": "John Doe", + "Created Date": "2024-01-01", + "Status": "Active", + "Custom Field": "Custom Value" + } + + result = generator._extract_custom_properties(attr_map) + + assert len(result) == 4 + properties = {prop["property"]: prop["value"] for prop in result} + # Property names are converted to lowercase with underscores + assert properties["owner"] == "John Doe" + assert properties["created_date"] == "2024-01-01" + assert properties["status"] == "Active" + assert properties["custom_field"] == "Custom Value" + + def test_extract_custom_properties_format(self): + """Test that custom properties have correct format""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + attr_map = {"Owner": "John"} + + result = generator._extract_custom_properties(attr_map) + + assert len(result) == 1 + assert "property" in result[0] + assert "value" in result[0] + assert result[0]["property"] == "owner" # Converted to lowercase + assert result[0]["value"] == "John" + + +class TestODCSGeneratorLogicalTypeMapping: + """Unit tests for logical type mapping""" + + def test_text_to_string(self): + """Test text maps to string""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["text"] == "string" + + def test_whole_number_to_integer(self): + """Test whole number maps to integer""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["whole number"] == "integer" + + def test_decimal_number_to_number(self): + """Test decimal number maps to number""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["decimal number"] == "number" + + def test_date_time_to_timestamp(self): + """Test date time maps to timestamp""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["date time"] == "timestamp" + + def test_boolean_mapping(self): + """Test true/false maps to boolean""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["true/false"] == "boolean" + + def test_geographical_to_string(self): + """Test geographical maps to string""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["geographical"] == "string" + + def test_standard_types(self): + """Test standard type mappings""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["string"] == "string" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["integer"] == "integer" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["number"] == "number" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["date"] == "date" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["time"] == "time" + + def test_complex_types(self): + """Test complex type mappings""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["object"] == "object" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["array"] == "array" + + def test_na_mapping(self): + """Test n/a maps to None""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["n/a"] is None + + def test_unmapped_type(self): + """Test that unmapped types return KeyError""" + with pytest.raises(KeyError): + _ = ODCSGenerator.LOGICAL_TYPE_MAPPING["unmapped_type"] + + +class TestODCSGeneratorNumericTypes: + """Unit tests for numeric types list""" + + def test_decimal_in_numeric_types(self): + """Test DECIMAL is in numeric types""" + assert "DECIMAL" in ODCSGenerator.NUMERIC_TYPES + + def test_numeric_in_numeric_types(self): + """Test NUMERIC is in numeric types""" + assert "NUMERIC" in ODCSGenerator.NUMERIC_TYPES + + def test_number_in_numeric_types(self): + """Test NUMBER is in numeric types""" + assert "NUMBER" in ODCSGenerator.NUMERIC_TYPES + + def test_numeric_types_count(self): + """Test expected number of numeric types""" + assert len(ODCSGenerator.NUMERIC_TYPES) == 3 + + +class TestODCSGeneratorExcludedAttributes: + """Unit tests for excluded attributes""" + + def test_description_excluded(self): + """Test that Description is in excluded attributes""" + assert "Description" in ODCSGenerator.EXCLUDED_ATTRIBUTES + + def test_excluded_attributes_is_set(self): + """Test that EXCLUDED_ATTRIBUTES is a set""" + assert isinstance(ODCSGenerator.EXCLUDED_ATTRIBUTES, set) + + +class TestCollibraClientUnit: + """Unit tests for CollibraClient class""" + + def test_client_initialization(self): + """Test client initialization""" + client = CollibraClient( + base_url="https://test.collibra.com", + username="testuser", + password="testpass" + ) + + assert client.base_url == "https://test.collibra.com" + assert client.auth == ("testuser", "testpass") + assert client.session is not None + + def test_base_url_trailing_slash_removed(self): + """Test that trailing slash is removed from base URL""" + client = CollibraClient( + base_url="https://test.collibra.com/", + username="user", + password="pass" + ) + + assert client.base_url == "https://test.collibra.com" + assert not client.base_url.endswith('/') + + def test_headers_json_constant(self): + """Test HEADERS_JSON constant""" + assert CollibraClient.HEADERS_JSON == {"Accept": "application/json"} + + def test_headers_content_json_constant(self): + """Test HEADERS_CONTENT_JSON constant""" + assert CollibraClient.HEADERS_CONTENT_JSON == {"Content-Type": "application/json"} + + def test_default_limit_constant(self): + """Test DEFAULT_LIMIT constant""" + assert CollibraClient.DEFAULT_LIMIT == 1000 + assert isinstance(CollibraClient.DEFAULT_LIMIT, int) + + +class TestDetermineOutputFile: + """Unit tests for determine_output_file function""" + + def test_with_custom_output_specified(self): + """Test with custom output file""" + args = Mock() + args.output = "my-custom-file.yaml" + odcs_data = {"name": "Test Contract"} + + result = determine_output_file(args, odcs_data) + assert result == "my-custom-file.yaml" + + def test_from_contract_name(self): + """Test output file from contract name""" + args = Mock() + args.output = None + odcs_data = {"name": "Customer Transactions"} + + result = determine_output_file(args, odcs_data) + assert result == "customer-transactions-odcs.yaml" + + def test_name_with_spaces(self): + """Test name with spaces is converted to hyphens""" + args = Mock() + args.output = None + odcs_data = {"name": "My Test Contract"} + + result = determine_output_file(args, odcs_data) + assert result == "my-test-contract-odcs.yaml" + assert " " not in result + + def test_name_with_uppercase(self): + """Test name is converted to lowercase""" + args = Mock() + args.output = None + odcs_data = {"name": "CUSTOMER_TABLE"} + + result = determine_output_file(args, odcs_data) + assert result == "customer_table-odcs.yaml" + assert result.islower() or "_" in result + + def test_default_when_no_name(self): + """Test default output file when no name""" + args = Mock() + args.output = None + odcs_data = {} + + result = determine_output_file(args, odcs_data) + assert result == "asset-odcs.yaml" + + def test_name_with_special_characters(self): + """Test name with special characters""" + args = Mock() + args.output = None + odcs_data = {"name": "Test/Table\\Name"} + + result = determine_output_file(args, odcs_data) + # Should handle special characters gracefully + assert result.endswith("-odcs.yaml") + + def test_empty_name(self): + """Test with empty name string""" + args = Mock() + args.output = None + odcs_data = {"name": ""} + + result = determine_output_file(args, odcs_data) + # Should use default when name is empty + assert result.endswith("-odcs.yaml") + + +class TestODCSGeneratorInitialization: + """Unit tests for ODCSGenerator initialization""" + + def test_generator_requires_client(self): + """Test that generator requires a client""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + assert generator.client == mock_client + + def test_generator_stores_client_reference(self): + """Test that generator stores client reference""" + mock_client = Mock(spec=CollibraClient) + mock_client.base_url = "https://test.com" + + generator = ODCSGenerator(mock_client) + + assert generator.client.base_url == "https://test.com" + + +class TestODCSGeneratorValidation: + """Unit tests for ODCSGenerator validation""" + + def test_generate_odcs_requires_asset_id(self): + """Test that generate_odcs requires asset ID""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + with pytest.raises(ValueError, match="Asset ID is required"): + generator.generate_odcs("") + + def test_generate_odcs_rejects_none_asset_id(self): + """Test that generate_odcs rejects None asset ID""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + with pytest.raises((ValueError, TypeError)): + generator.generate_odcs(None) + + def test_generate_odcs_accepts_valid_asset_id(self): + """Test that generate_odcs accepts valid asset ID""" + mock_client = Mock(spec=CollibraClient) + mock_client.get_asset.return_value = { + "id": "test-id", + "displayName": "Test", + "type": {"name": "Table"}, + "domain": {"name": "Domain"}, + "createdOn": 0 + } + mock_client.get_asset_tags.return_value = [] + mock_client.get_asset_attributes.return_value = [] + mock_client.get_asset_relations.return_value = [] + mock_client.base_url = "https://test.com" + + generator = ODCSGenerator(mock_client) + + # Should not raise an error + result = generator.generate_odcs("valid-asset-id") + assert result is not None + + +class TestODCSStructureValidation: + """Unit tests for ODCS structure validation""" + + def test_odcs_has_required_fields(self): + """Test that generated ODCS has required fields""" + mock_client = Mock(spec=CollibraClient) + mock_client.get_asset.return_value = { + "id": "test-id", + "displayName": "Test Table", + "type": {"name": "Table"}, + "domain": {"name": "Finance"}, + "createdOn": 1609459200000 + } + mock_client.get_asset_tags.return_value = [] + mock_client.get_asset_attributes.return_value = [] + mock_client.get_asset_relations.return_value = [] + mock_client.base_url = "https://test.com" + + generator = ODCSGenerator(mock_client) + odcs = generator.generate_odcs("test-id") + + # Check required ODCS fields + required_fields = [ + 'id', 'kind', 'apiVersion', 'domain', 'dataProduct', + 'version', 'name', 'status', 'contractCreatedTs', + 'description', 'tags', 'schema', 'servers' + ] + + for field in required_fields: + assert field in odcs, f"Missing required field: {field}" + + def test_odcs_kind_is_data_contract(self): + """Test that ODCS kind is DataContract""" + mock_client = Mock(spec=CollibraClient) + mock_client.get_asset.return_value = { + "id": "test-id", + "displayName": "Test", + "type": {"name": "Table"}, + "domain": {"name": "Domain"}, + "createdOn": 0 + } + mock_client.get_asset_tags.return_value = [] + mock_client.get_asset_attributes.return_value = [] + mock_client.get_asset_relations.return_value = [] + mock_client.base_url = "https://test.com" + + generator = ODCSGenerator(mock_client) + odcs = generator.generate_odcs("test-id") + + assert odcs["kind"] == "DataContract" + + def test_odcs_api_version(self): + """Test that ODCS has correct API version""" + mock_client = Mock(spec=CollibraClient) + mock_client.get_asset.return_value = { + "id": "test-id", + "displayName": "Test", + "type": {"name": "Table"}, + "domain": {"name": "Domain"}, + "createdOn": 0 + } + mock_client.get_asset_tags.return_value = [] + mock_client.get_asset_attributes.return_value = [] + mock_client.get_asset_relations.return_value = [] + mock_client.base_url = "https://test.com" + + generator = ODCSGenerator(mock_client) + odcs = generator.generate_odcs("test-id") + + assert odcs["apiVersion"] == "v3.1.0" + + +class TestCollibraClientAPIMethods: + """Unit tests for CollibraClient API methods with mocked responses""" + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset_success(self, mock_get): + """Test successful asset retrieval""" + mock_response = Mock() + mock_response.json.return_value = { + "id": "asset-123", + "displayName": "Test Table", + "type": {"name": "Table"}, + "domain": {"name": "Finance"} + } + mock_response.raise_for_status = Mock() + mock_get.return_value = mock_response + + client = CollibraClient( + "https://test.collibra.com", + "testuser", + "testpass" + ) + + result = client.get_asset("asset-123") + + assert result["id"] == "asset-123" + assert result["displayName"] == "Test Table" + mock_get.assert_called_once() + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset_attributes_success(self, mock_get): + """Test successful asset attributes retrieval""" + mock_response = Mock() + mock_response.json.return_value = { + "results": [ + {"type": {"name": "Description"}, "value": "Test description"}, + {"type": {"name": "Owner"}, "value": "John Doe"} + ] + } + mock_response.raise_for_status = Mock() + mock_get.return_value = mock_response + + client = CollibraClient( + "https://test.collibra.com", + "testuser", + "testpass" + ) + + result = client.get_asset_attributes("asset-123") + + assert len(result) == 2 + assert result[0]["type"]["name"] == "Description" + mock_get.assert_called_once() + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset_tags_success(self, mock_get): + """Test successful asset tags retrieval""" + mock_response = Mock() + mock_response.json.return_value = [ + {"name": "PII"}, + {"name": "Sensitive"} + ] + mock_response.raise_for_status = Mock() + mock_get.return_value = mock_response + + client = CollibraClient( + "https://test.collibra.com", + "testuser", + "testpass" + ) + + result = client.get_asset_tags("asset-123") + + assert len(result) == 2 + assert result[0] == "PII" + mock_get.assert_called_once() + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset_relations_success(self, mock_get): + """Test successful asset relations retrieval""" + mock_response = Mock() + mock_response.json.return_value = { + "results": [ + { + "type": {"role": "target"}, + "target": { + "id": "col-1", + "name": "customer_id", + "type": {"name": "Column"} + } + } + ] + } + mock_response.raise_for_status = Mock() + mock_get.return_value = mock_response + + client = CollibraClient( + "https://test.collibra.com", + "testuser", + "testpass" + ) + + result = client.get_asset_relations("asset-123") + + assert len(result) == 1 + assert result[0]["target"]["name"] == "customer_id" + mock_get.assert_called_once() + +class TestParseAndValidateArgumentsCollibra: + """Unit tests for Collibra argument parsing and validation""" + + @patch('sys.argv', ['script.py', 'asset-123', '--url', 'https://test.collibra.com', '-u', 'user', '-p', 'pass']) + def test_parse_arguments_all_provided(self): + """Test parsing with all arguments provided""" + from wxdi.odcs_generator.generate_odcs_from_collibra import parse_arguments + + args = parse_arguments() + + assert args.asset_id == 'asset-123' + assert args.url == 'https://test.collibra.com' + assert args.username == 'user' + assert args.password == 'pass' + + @patch('sys.argv', ['script.py', 'asset-123', '-o', 'output.yaml']) + @patch.dict(os.environ, { + 'COLLIBRA_URL': 'https://env.collibra.com', + 'COLLIBRA_USERNAME': 'envuser', + 'COLLIBRA_PASSWORD': 'envpass' # pragma: allowlist secret + }) + def test_parse_arguments_from_env(self): + """Test parsing with environment variables""" + from wxdi.odcs_generator.generate_odcs_from_collibra import parse_arguments + + args = parse_arguments() + + assert args.asset_id == 'asset-123' + assert args.output == 'output.yaml' + assert args.url == 'https://env.collibra.com' + assert args.username == 'envuser' + assert args.password == 'envpass' + + def test_validate_arguments_missing_url(self): + """Test validation fails with missing URL""" + from wxdi.odcs_generator.generate_odcs_from_collibra import validate_arguments + + args = Mock() + args.url = None + args.username = 'user' + args.password = 'pass' + + with pytest.raises(SystemExit): + validate_arguments(args) + + def test_validate_arguments_missing_username(self): + """Test validation fails with missing username""" + from wxdi.odcs_generator.generate_odcs_from_collibra import validate_arguments + + args = Mock() + args.url = 'https://test.com' + args.username = None + args.password = 'pass' + + with pytest.raises(SystemExit): + validate_arguments(args) + + def test_validate_arguments_missing_password(self): + """Test validation fails with missing password""" + from wxdi.odcs_generator.generate_odcs_from_collibra import validate_arguments + + args = Mock() + args.url = 'https://test.com' + args.username = 'user' + args.password = None + + with pytest.raises(SystemExit): + validate_arguments(args) + + def test_validate_arguments_all_present(self): + """Test validation passes with all arguments""" + from wxdi.odcs_generator.generate_odcs_from_collibra import validate_arguments + + args = Mock() + args.url = 'https://test.com' + args.username = 'user' + args.password = 'pass' + + # Should not raise an exception + validate_arguments(args) + + +if __name__ == '__main__': + pytest.main([__file__, '-v']) + diff --git a/tests/src/odcs_generator/test_odcs_generator_informatica.py b/tests/src/odcs_generator/test_odcs_generator_informatica.py new file mode 100644 index 0000000..c55ce14 --- /dev/null +++ b/tests/src/odcs_generator/test_odcs_generator_informatica.py @@ -0,0 +1,1060 @@ +# -*- coding: utf-8 -*- +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Unit Tests for Informatica ODCS Generator +Tests individual functions and methods in isolation +""" + +import pytest +import os +from unittest.mock import Mock, patch +from wxdi.odcs_generator.generate_odcs_from_informatica import ( + InformaticaClient, + extract_column_position, + build_physical_type, + build_column_property, + build_custom_properties, + generate_odcs_yaml, + determine_output_file, + RESOURCE_TYPE_MAPPING, + SYSTEM_ATTRIBUTES_MAPPING +) + + +class TestExtractColumnPosition: + """Unit tests for extract_column_position function""" + + def test_valid_position_as_string(self): + """Test extraction with valid position as string""" + col_data = {"selfAttributes": {"core.Position": "5"}} + assert extract_column_position(col_data) == 5 + + def test_valid_position_as_int(self): + """Test extraction with valid position as integer""" + col_data = {"selfAttributes": {"core.Position": 10}} + assert extract_column_position(col_data) == 10 + + def test_missing_self_attributes(self): + """Test with missing selfAttributes""" + col_data = {} + assert extract_column_position(col_data) == 999999 + + def test_missing_position_key(self): + """Test with missing Position key""" + col_data = {"selfAttributes": {}} + assert extract_column_position(col_data) == 999999 + + def test_none_position_value(self): + """Test with None position value""" + col_data = {"selfAttributes": {"core.Position": None}} + assert extract_column_position(col_data) == 999999 + + def test_invalid_position_string(self): + """Test with invalid position string""" + col_data = {"selfAttributes": {"core.Position": "invalid"}} + assert extract_column_position(col_data) == 999999 + + def test_empty_position_string(self): + """Test with empty position string""" + col_data = {"selfAttributes": {"core.Position": ""}} + assert extract_column_position(col_data) == 999999 + + def test_negative_position(self): + """Test with negative position""" + col_data = {"selfAttributes": {"core.Position": "-1"}} + assert extract_column_position(col_data) == -1 + + +class TestBuildPhysicalType: + """Unit tests for build_physical_type function""" + + def test_type_without_length(self): + """Test type without length or scale""" + assert build_physical_type("INTEGER", "", "") == "INTEGER" + assert build_physical_type("TIMESTAMP", None, None) == "TIMESTAMP" + + def test_type_with_length_only(self): + """Test type with length but no scale""" + assert build_physical_type("VARCHAR", "255", "") == "VARCHAR(255)" + assert build_physical_type("CHAR", "10", None) == "CHAR(10)" + + def test_type_with_length_and_scale(self): + """Test type with both length and scale""" + assert build_physical_type("DECIMAL", "10", "2") == "DECIMAL(10,2)" + assert build_physical_type("NUMBER", "18", "4") == "NUMBER(18,4)" + + def test_type_with_length_and_zero_scale(self): + """Test type with length and scale of 0""" + assert build_physical_type("NUMBER", "18", "0") == "NUMBER(18)" + assert build_physical_type("DECIMAL", "10", "0") == "DECIMAL(10)" + + def test_type_with_empty_strings(self): + """Test with empty strings for length and scale""" + assert build_physical_type("VARCHAR", "", "") == "VARCHAR" + + def test_various_data_types(self): + """Test with various common data types""" + assert build_physical_type("BIGINT", "", "") == "BIGINT" + assert build_physical_type("TEXT", "", "") == "TEXT" + assert build_physical_type("BOOLEAN", "", "") == "BOOLEAN" + assert build_physical_type("DATE", "", "") == "DATE" + + +class TestBuildColumnProperty: + """Unit tests for build_column_property function""" + + def test_basic_column_property(self): + """Test building basic column property""" + column_detail = { + "summary": { + "core.name": "customer_id" + }, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "INTEGER" + } + } + + prop = build_column_property(column_detail) + + assert prop["name"] == "customer_id" + assert prop["physicalType"] == "INTEGER" + assert prop["required"] is False # Default when Nullable not specified (nullable=true by default) + + def test_column_with_description(self): + """Test column with description""" + column_detail = { + "summary": { + "core.name": "email", + "core.description": "Customer email address" + }, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "VARCHAR", + "com.infa.odin.models.relational.DatatypeLength": "255" + } + } + + prop = build_column_property(column_detail) + + assert prop["name"] == "email" + assert prop["physicalType"] == "VARCHAR(255)" + assert prop["description"] == "Customer email address" + + def test_nullable_column(self): + """Test nullable column""" + column_detail = { + "summary": {"core.name": "middle_name"}, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "VARCHAR", + "com.infa.odin.models.relational.DatatypeLength": "50", + "com.infa.odin.models.relational.Nullable": "true" + } + } + + prop = build_column_property(column_detail) + + assert prop["required"] is False + + def test_non_nullable_column(self): + """Test non-nullable column""" + column_detail = { + "summary": {"core.name": "id"}, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "INTEGER", + "com.infa.odin.models.relational.Nullable": "false" + } + } + + prop = build_column_property(column_detail) + + assert prop["required"] is True + + def test_primary_key_column(self): + """Test primary key column""" + column_detail = { + "summary": {"core.name": "id"}, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "INTEGER", + "com.infa.odin.models.relational.PrimaryKeyColumn": "true" + } + } + + prop = build_column_property(column_detail) + + assert prop.get("primaryKey") is True + + def test_non_primary_key_column(self): + """Test non-primary key column""" + column_detail = { + "summary": {"core.name": "name"}, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "VARCHAR", + "com.infa.odin.models.relational.PrimaryKeyColumn": "false" + } + } + + prop = build_column_property(column_detail) + + assert "primaryKey" not in prop or prop.get("primaryKey") is False + + def test_decimal_column_with_scale(self): + """Test decimal column with precision and scale""" + column_detail = { + "summary": {"core.name": "price"}, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "DECIMAL", + "com.infa.odin.models.relational.DatatypeLength": "10", + "com.infa.odin.models.relational.DatatypeScale": "2" + } + } + + prop = build_column_property(column_detail) + + assert prop["physicalType"] == "DECIMAL(10,2)" + + def test_column_without_description(self): + """Test that description is not added when empty""" + column_detail = { + "summary": {"core.name": "col1"}, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "INTEGER" + } + } + + prop = build_column_property(column_detail) + + assert "description" not in prop + + +class TestBuildCustomProperties: + """Unit tests for build_custom_properties function""" + + def test_empty_attributes(self): + """Test with empty attributes""" + assert build_custom_properties({}) == [] + + def test_single_attribute(self): + """Test with single attribute""" + attrs = {"core.resourceName": "TestCatalog"} + props = build_custom_properties(attrs) + + assert len(props) == 1 + assert props[0]["property"] == "Catalog Source Name" + assert props[0]["value"] == "TestCatalog" + + def test_multiple_attributes(self): + """Test with multiple attributes""" + attrs = { + "core.resourceName": "TestCatalog", + "com.infa.odin.models.relational.NumberOfRows": "1000", + "core.origin": "Production" + } + props = build_custom_properties(attrs) + + assert len(props) == 3 + property_names = [p["property"] for p in props] + assert "Catalog Source Name" in property_names + assert "Number of rows" in property_names + assert "Origin" in property_names + + def test_unmapped_attributes_ignored(self): + """Test that unmapped attributes are ignored""" + attrs = { + "core.resourceName": "TestCatalog", + "unmapped.attribute": "SomeValue" + } + props = build_custom_properties(attrs) + + assert len(props) == 1 + assert props[0]["property"] == "Catalog Source Name" + + def test_all_system_attributes(self): + """Test with all system attributes""" + attrs = { + "core.resourceName": "Catalog1", + "com.infa.odin.models.relational.NumberOfRows": "5000", + "core.origin": "Prod", + "com.infa.odin.models.relational.Owner": "dbo", + "core.sourceCreatedBy": "admin", + "core.sourceCreatedOn": "2024-01-01", + "core.sourceModifiedBy": "user1", + "core.sourceModifiedOn": "2024-02-01" + } + props = build_custom_properties(attrs) + + assert len(props) == 8 + + +class TestGenerateOdcsYaml: + """Unit tests for generate_odcs_yaml function""" + + def test_basic_odcs_structure(self): + """Test basic ODCS structure generation""" + asset_data = { + "core.identity": "test-id", + "summary": {"core.name": "TEST_TABLE"}, + "selfAttributes": { + "core.businessName": "test_table", + "com.infa.odin.models.relational.Owner": "public" + } + } + column_details = [] + base_url = "https://cdgc.test.com" + + odcs = generate_odcs_yaml(asset_data, column_details, base_url) + + assert odcs["id"] == "test-id" + assert odcs["kind"] == "DataContract" + assert odcs["apiVersion"] == "v3.1.0" + assert odcs["status"] == "active" + assert "contractCreatedTs" in odcs + + def test_schema_with_columns(self): + """Test schema generation with columns""" + asset_data = { + "core.identity": "test-id", + "summary": {"core.name": "CUSTOMER"}, + "selfAttributes": { + "core.name": "customer", + "com.infa.odin.models.relational.Owner": "dbo" + } + } + column_details = [ + { + "summary": {"core.name": "id"}, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "INTEGER", + "core.Position": "1" + } + } + ] + base_url = "https://cdgc.test.com" + + odcs = generate_odcs_yaml(asset_data, column_details, base_url) + + assert len(odcs["schema"]) == 1 + assert odcs["schema"][0]["name"] == "customer" + assert len(odcs["schema"][0]["properties"]) == 1 + assert odcs["schema"][0]["properties"][0]["name"] == "id" + + def test_physical_name_construction(self): + """Test physical name construction with schema""" + asset_data = { + "core.identity": "test-id", + "summary": {"core.name": "TABLE1"}, + "selfAttributes": { + "core.name": "table1", + "com.infa.odin.models.relational.Owner": "schema1" + } + } + column_details = [] + base_url = "https://cdgc.test.com" + + odcs = generate_odcs_yaml(asset_data, column_details, base_url) + + assert odcs["schema"][0]["physicalName"] == "schema1/table1" + + def test_server_type_mapping(self): + """Test server type mapping from resource type""" + asset_data = { + "core.identity": "test-id", + "summary": {"core.name": "TABLE1"}, + "selfAttributes": { + "core.businessName": "table1", + "core.resourceType": "Snowflake", + "com.infa.odin.models.relational.Owner": "public" + } + } + column_details = [] + base_url = "https://cdgc.test.com" + + odcs = generate_odcs_yaml(asset_data, column_details, base_url) + + assert odcs["servers"][0]["type"] == "snowflake" + + def test_authoritative_definition(self): + """Test authoritative definition URL""" + asset_data = { + "core.identity": "asset-123", + "summary": {"core.name": "TABLE1"}, + "selfAttributes": {"core.businessName": "table1"} + } + column_details = [] + base_url = "https://cdgc.test.com" + + odcs = generate_odcs_yaml(asset_data, column_details, base_url) + + auth_defs = odcs["description"]["authoritativeDefinitions"] + assert len(auth_defs) == 1 + assert auth_defs[0]["type"] == "informatica-asset" + assert "asset-123" in auth_defs[0]["url"] + + +class TestDetermineOutputFile: + """Unit tests for determine_output_file function""" + + def test_with_custom_output(self): + """Test with custom output file specified""" + args = Mock() + args.output = "custom-file.yaml" + odcs_data = {"name": "Test"} + + result = determine_output_file(args, odcs_data) + assert result == "custom-file.yaml" + + def test_from_asset_name(self): + """Test output file from asset name""" + args = Mock() + args.output = None + odcs_data = {"name": "Customer Transactions"} + + result = determine_output_file(args, odcs_data) + assert result == "customer-transactions-odcs.yaml" + + def test_default_name(self): + """Test default output file name""" + args = Mock() + args.output = None + odcs_data = {} + + result = determine_output_file(args, odcs_data) + assert result == "asset-odcs.yaml" + + def test_name_with_special_characters(self): + """Test name with special characters""" + args = Mock() + args.output = None + odcs_data = {"name": "Test Table (Production)"} + + result = determine_output_file(args, odcs_data) + assert result == "test-table-(production)-odcs.yaml" + + +class TestResourceTypeMapping: + """Unit tests for RESOURCE_TYPE_MAPPING constant""" + + def test_common_databases_mapped(self): + """Test that common database types are mapped""" + assert RESOURCE_TYPE_MAPPING["Snowflake"] == "snowflake" + assert RESOURCE_TYPE_MAPPING["PostgreSQL"] == "postgresql" + assert RESOURCE_TYPE_MAPPING["Oracle"] == "oracle" + assert RESOURCE_TYPE_MAPPING["SqlServer"] == "sqlserver" + + def test_cloud_databases_mapped(self): + """Test that cloud database types are mapped""" + assert RESOURCE_TYPE_MAPPING["BigQuery"] == "bigquery" + assert RESOURCE_TYPE_MAPPING["Redshift"] == "redshift" + assert RESOURCE_TYPE_MAPPING["Databricks"] == "databricks" + assert RESOURCE_TYPE_MAPPING["Synapse"] == "synapse" + + def test_mapping_count(self): + """Test that expected number of mappings exist""" + assert len(RESOURCE_TYPE_MAPPING) >= 13 + + +class TestSystemAttributesMapping: + """Unit tests for SYSTEM_ATTRIBUTES_MAPPING constant""" + + def test_core_attributes_mapped(self): + """Test that core attributes are mapped""" + assert "core.resourceName" in SYSTEM_ATTRIBUTES_MAPPING + assert "core.origin" in SYSTEM_ATTRIBUTES_MAPPING + + def test_relational_attributes_mapped(self): + """Test that relational model attributes are mapped""" + assert "com.infa.odin.models.relational.NumberOfRows" in SYSTEM_ATTRIBUTES_MAPPING + assert "com.infa.odin.models.relational.Owner" in SYSTEM_ATTRIBUTES_MAPPING + + def test_timestamp_attributes_mapped(self): + """Test that timestamp attributes are mapped""" + assert "core.sourceCreatedOn" in SYSTEM_ATTRIBUTES_MAPPING + assert "core.sourceModifiedOn" in SYSTEM_ATTRIBUTES_MAPPING + + def test_user_attributes_mapped(self): + """Test that user attributes are mapped""" + assert "core.sourceCreatedBy" in SYSTEM_ATTRIBUTES_MAPPING + assert "core.sourceModifiedBy" in SYSTEM_ATTRIBUTES_MAPPING + + +class TestInformaticaClientUnit: + """Unit tests for InformaticaClient class methods""" + + def test_extract_region_from_standard_url(self): + """Test region extraction from standard URL""" + client = InformaticaClient( + "https://cdgc.dm-us.informaticacloud.com", + "user", + "pass" + ) + assert client.region == "dm-us" + + def test_extract_region_from_various_formats(self): + """Test region extraction from various URL formats""" + test_cases = [ + ("https://cdgc.na1.informaticacloud.com", "na1"), + ("https://cdgc.eu1.informaticacloud.com/", "eu1"), + ("https://cdgc.ap1.informaticacloud.com", "ap1"), + ] + + for url, expected_region in test_cases: + client = InformaticaClient(url, "user", "pass") + assert client.region == expected_region + + def test_identity_url_construction(self): + """Test identity URL construction""" + client = InformaticaClient( + "https://cdgc.dm-us.informaticacloud.com", + "user", + "pass" + ) + assert client.identity_url == "https://dm-us.informaticacloud.com" + + def test_base_url_normalization(self): + """Test that base URL trailing slash is removed""" + client = InformaticaClient( + "https://cdgc.test.com/", + "user", + "pass" + ) + assert client.base_url == "https://cdgc.test.com" + + +class TestInformaticaClientAPIMethods: + """Unit tests for InformaticaClient API methods with mocked responses""" + + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.requests.post') + def test_get_session_id_success(self, mock_post): + """Test successful session ID retrieval""" + mock_response = Mock() + mock_response.json.return_value = {"sessionId": "test-session-123"} + mock_response.raise_for_status = Mock() + mock_post.return_value = mock_response + + client = InformaticaClient( + "https://cdgc.dm-us.informaticacloud.com", + "testuser", + "testpass" + ) + + result = client.get_session_id() + + assert result["sessionId"] == "test-session-123" + mock_post.assert_called_once() + + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.requests.post') + def test_get_auth_token_success(self, mock_post): + """Test successful auth token retrieval""" + # Mock session ID call + mock_session_response = Mock() + mock_session_response.json.return_value = {"sessionId": "session-123"} + mock_session_response.raise_for_status = Mock() + + # Mock JWT token call + mock_jwt_response = Mock() + mock_jwt_response.json.return_value = {"jwt_token": "jwt-token-456"} + mock_jwt_response.raise_for_status = Mock() + + mock_post.side_effect = [mock_session_response, mock_jwt_response] + + client = InformaticaClient( + "https://cdgc.dm-us.informaticacloud.com", + "testuser", + "testpass" + ) + + token = client.get_auth_token() + + assert token == "jwt-token-456" + assert mock_post.call_count == 2 + + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.requests.post') + def test_get_auth_token_caching(self, mock_post): + """Test that auth token is cached""" + mock_session_response = Mock() + mock_session_response.json.return_value = {"sessionId": "session-123"} + mock_session_response.raise_for_status = Mock() + + mock_jwt_response = Mock() + mock_jwt_response.json.return_value = {"jwt_token": "jwt-token-456"} + mock_jwt_response.raise_for_status = Mock() + + mock_post.side_effect = [mock_session_response, mock_jwt_response] + + client = InformaticaClient( + "https://cdgc.dm-us.informaticacloud.com", + "testuser", + "testpass" + ) + + # First call + token1 = client.get_auth_token() + # Second call should use cached token + token2 = client.get_auth_token() + + assert token1 == token2 + # Should only call API twice (session + jwt), not four times + assert mock_post.call_count == 2 + + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.requests.get') + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.requests.post') + def test_get_asset_details_success(self, mock_post, mock_get): + """Test successful asset details retrieval""" + # Mock auth calls + mock_session_response = Mock() + mock_session_response.json.return_value = {"sessionId": "session-123"} + mock_session_response.raise_for_status = Mock() + + mock_jwt_response = Mock() + mock_jwt_response.json.return_value = {"jwt_token": "jwt-token-456"} + mock_jwt_response.raise_for_status = Mock() + + mock_post.side_effect = [mock_session_response, mock_jwt_response] + + # Mock asset details call + mock_asset_response = Mock() + mock_asset_response.json.return_value = { + "core.identity": "asset-123", + "summary": {"core.name": "TEST_TABLE"} + } + mock_asset_response.raise_for_status = Mock() + mock_get.return_value = mock_asset_response + + client = InformaticaClient( + "https://cdgc.dm-us.informaticacloud.com", + "testuser", + "testpass" + ) + + result = client.get_asset_details("asset-123") + + assert result["core.identity"] == "asset-123" + assert result["summary"]["core.name"] == "TEST_TABLE" + mock_get.assert_called_once() + + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.requests.get') + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.requests.post') + def test_get_column_details_success(self, mock_post, mock_get): + """Test successful column details retrieval""" + # Mock auth calls + mock_session_response = Mock() + mock_session_response.json.return_value = {"sessionId": "session-123"} + mock_session_response.raise_for_status = Mock() + + mock_jwt_response = Mock() + mock_jwt_response.json.return_value = {"jwt_token": "jwt-token-456"} + mock_jwt_response.raise_for_status = Mock() + + mock_post.side_effect = [mock_session_response, mock_jwt_response] + + # Mock column details call + mock_column_response = Mock() + mock_column_response.json.return_value = { + "summary": {"core.name": "customer_id"}, + "selfAttributes": {"com.infa.odin.models.relational.Datatype": "INTEGER"} + } + mock_column_response.raise_for_status = Mock() + mock_get.return_value = mock_column_response + + client = InformaticaClient( + "https://cdgc.dm-us.informaticacloud.com", + "testuser", + "testpass" + ) + + result = client.get_column_details("column-456") + + assert result["summary"]["core.name"] == "customer_id" + mock_get.assert_called_once() + + +class TestParseAndValidateArguments: + """Unit tests for argument parsing and validation""" + + @patch('sys.argv', ['script.py', 'asset-123', '--cdgc-url', 'https://test.com', '-u', 'user', '-p', 'pass']) + def test_parse_arguments_all_provided(self): + """Test parsing with all arguments provided""" + from wxdi.odcs_generator.generate_odcs_from_informatica import parse_arguments + + args = parse_arguments() + + assert args.asset_id == 'asset-123' + assert args.cdgc_url == 'https://test.com' + assert args.username == 'user' + assert args.password == 'pass' + + @patch('sys.argv', ['script.py', 'asset-123', '-o', 'output.yaml']) + @patch.dict(os.environ, { + 'INFORMATICA_CDGC_URL': 'https://env.com', + 'INFORMATICA_USERNAME': 'envuser', + 'INFORMATICA_PASSWORD': 'envpass' # pragma: allowlist secret + }) + def test_parse_arguments_from_env(self): + """Test parsing with environment variables""" + from wxdi.odcs_generator.generate_odcs_from_informatica import parse_arguments + + args = parse_arguments() + + assert args.asset_id == 'asset-123' + assert args.output == 'output.yaml' + assert args.cdgc_url == 'https://env.com' + assert args.username == 'envuser' + assert args.password == 'envpass' + + def test_validate_arguments_missing_url(self): + """Test validation fails with missing URL""" + from wxdi.odcs_generator.generate_odcs_from_informatica import validate_arguments + + args = Mock() + args.cdgc_url = None + args.username = 'user' + args.password = 'pass' + + with pytest.raises(SystemExit): + validate_arguments(args) + + def test_validate_arguments_missing_username(self): + """Test validation fails with missing username""" + from wxdi.odcs_generator.generate_odcs_from_informatica import validate_arguments + + args = Mock() + args.cdgc_url = 'https://test.com' + args.username = None + args.password = 'pass' + + with pytest.raises(SystemExit): + validate_arguments(args) + + def test_validate_arguments_missing_password(self): + """Test validation fails with missing password""" + from wxdi.odcs_generator.generate_odcs_from_informatica import validate_arguments + + args = Mock() + args.cdgc_url = 'https://test.com' + args.username = 'user' + args.password = None + + with pytest.raises(SystemExit): + validate_arguments(args) + + def test_validate_arguments_all_present(self): + """Test validation passes with all arguments""" + from wxdi.odcs_generator.generate_odcs_from_informatica import validate_arguments + + args = Mock() + args.cdgc_url = 'https://test.com' + args.username = 'user' + args.password = 'pass' + + # Should not raise an exception + + +class TestGenerateODCSYAML: + """Test cases for generate_odcs_yaml function""" + + def test_generate_odcs_yaml_with_description(self): + """Test ODCS generation with table description""" + asset_data = { + 'core.identity': 'table-123', + 'summary': { + 'core.name': 'test_table', + 'core.description': 'Test table description' + }, + 'selfAttributes': { + 'core.businessName': 'TestTable', + 'com.infa.odin.models.relational.Owner': 'test_schema', + 'core.resourceType': 'Snowflake' + } + } + + column_details = [ + { + 'core.identity': 'col-1', + 'summary': {'core.name': 'id'}, + 'selfAttributes': { + 'com.infa.odin.models.relational.Datatype': 'INTEGER', + 'com.infa.odin.models.relational.Nullable': 'false', + 'com.infa.odin.models.relational.PrimaryKeyColumn': 'true' + } + } + ] + + result = generate_odcs_yaml(asset_data, column_details, 'https://test.com') + + assert result['id'] == 'table-123' + assert result['schema'][0]['description'] == 'Test table description' + assert result['schema'][0]['properties'][0]['primaryKey'] is True + assert result['servers'][0]['type'] == 'snowflake' + + def test_generate_odcs_yaml_without_description(self): + """Test ODCS generation without table description""" + asset_data = { + 'core.identity': 'table-456', + 'summary': {'core.name': 'test_table'}, + 'selfAttributes': { + 'core.businessName': 'TestTable', + 'com.infa.odin.models.relational.Owner': 'test_schema' + } + } + + column_details = [] + + result = generate_odcs_yaml(asset_data, column_details, 'https://test.com') + + assert 'description' not in result['schema'][0] + + +class TestHelperFunctions: + """Test cases for helper functions""" + + def test_add_server_warning_comments(self): + """Test _add_server_warning_comments function""" + from wxdi.odcs_generator.generate_odcs_from_informatica import _add_server_warning_comments + + modified_lines = [] + _add_server_warning_comments(modified_lines, ' - id: server-123') + + assert len(modified_lines) == 5 + assert '⚠️ MANUAL CONFIGURATION REQUIRED' in modified_lines[1] + + def test_add_inline_comment_server(self): + """Test _add_inline_comment_if_needed for server field""" + from wxdi.odcs_generator.generate_odcs_from_informatica import _add_inline_comment_if_needed + + line = ' server: CONFIGURE_SERVER_HOSTNAME' + result = _add_inline_comment_if_needed(line) + assert '⚠️ UPDATE' in result + assert 'prod.snowflake.acme.com' in result + + def test_add_inline_comment_type(self): + """Test _add_inline_comment_if_needed for type field""" + from wxdi.odcs_generator.generate_odcs_from_informatica import _add_inline_comment_if_needed + + line = ' type: CONFIGURE_SERVER_TYPE' + result = _add_inline_comment_if_needed(line) + assert '⚠️ UPDATE' in result + assert 'snowflake' in result + + def test_add_inline_comment_schema(self): + """Test _add_inline_comment_if_needed for schema field""" + from wxdi.odcs_generator.generate_odcs_from_informatica import _add_inline_comment_if_needed + + line = ' schema: CONFIGURE_SCHEMA_NAME' + result = _add_inline_comment_if_needed(line) + assert '⚠️ UPDATE' in result + assert 'public' in result + + def test_add_inline_comment_no_change(self): + """Test _add_inline_comment_if_needed for normal line""" + from wxdi.odcs_generator.generate_odcs_from_informatica import _add_inline_comment_if_needed + + line = ' name: test' + result = _add_inline_comment_if_needed(line) + assert result == line + + def test_determine_output_file_with_arg(self): + """Test determine_output_file with output argument""" + args = Mock() + args.output = 'custom.yaml' + odcs_data = {'name': 'test'} + + result = determine_output_file(args, odcs_data) + assert result == 'custom.yaml' + + def test_determine_output_file_default(self): + """Test determine_output_file with default naming""" + args = Mock() + args.output = None + odcs_data = {'name': 'Test Table'} + + result = determine_output_file(args, odcs_data) + assert result == 'test-table-odcs.yaml' + + +class TestWriteYAMLFile: + """Test cases for write_yaml_file function""" + + @patch('builtins.open', create=True) + @patch('builtins.print') + def test_write_yaml_file(self, mock_print, mock_open_func): + """Test write_yaml_file function""" + from wxdi.odcs_generator.generate_odcs_from_informatica import write_yaml_file + + mock_file = Mock() + mock_open_func.return_value.__enter__.return_value = mock_file + + odcs_data = { + 'id': 'test-123', + 'servers': [ + { + 'id': 'server-1', + 'server': 'CONFIGURE_SERVER_HOSTNAME', + 'type': 'CONFIGURE_SERVER_TYPE', + 'schema': 'CONFIGURE_SCHEMA_NAME' + } + ] + } + + write_yaml_file('test.yaml', odcs_data) + + mock_open_func.assert_called_once_with('test.yaml', 'w') + mock_print.assert_called() + + +class TestMainFunction: + """Test cases for main function""" + + @patch('sys.argv', ['script.py', 'asset-123', '--cdgc-url', 'https://test.com', + '-u', 'user', '-p', 'pass']) + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.InformaticaClient') + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.write_yaml_file') + @patch('builtins.print') + def test_main_success(self, mock_print, mock_write, mock_client_class): + """Test main function success path""" + from wxdi.odcs_generator.generate_odcs_from_informatica import main + + # Mock client instance + mock_client = Mock() + mock_client.base_url = 'https://test.com' + mock_client.get_asset_details.return_value = { + 'core.identity': 'asset-123', + 'summary': {'core.name': 'test_table'}, + 'selfAttributes': {'core.businessName': 'TestTable'}, + 'hierarchy': [{'core.identity': 'col-1'}] + } + mock_client.get_column_details.return_value = { + 'core.identity': 'col-1', + 'summary': {'core.name': 'id'}, + 'selfAttributes': { + 'com.infa.odin.models.relational.Datatype': 'INTEGER', + 'com.infa.odin.models.relational.Position': '1' + } + } + mock_client_class.return_value = mock_client + + main() + + mock_client.get_asset_details.assert_called_once_with('asset-123') + mock_write.assert_called_once() + + @patch('sys.argv', ['script.py', 'asset-123', '--cdgc-url', 'https://test.com', + '-u', 'user', '-p', 'pass']) + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.InformaticaClient') + @patch('sys.exit') + def test_main_http_401_error(self, mock_exit, mock_client_class): + """Test main function with 401 HTTP error""" + from wxdi.odcs_generator.generate_odcs_from_informatica import main + import requests + + mock_client = Mock() + mock_response = Mock() + mock_response.status_code = 401 + http_error = requests.exceptions.HTTPError(response=mock_response) + mock_client.get_asset_details.side_effect = http_error + mock_client_class.return_value = mock_client + + main() + + mock_exit.assert_called_once_with(1) + + @patch('sys.argv', ['script.py', 'asset-123', '--cdgc-url', 'https://test.com', + '-u', 'user', '-p', 'pass']) + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.InformaticaClient') + @patch('sys.exit') + def test_main_http_404_error(self, mock_exit, mock_client_class): + """Test main function with 404 HTTP error""" + from wxdi.odcs_generator.generate_odcs_from_informatica import main + import requests + + mock_client = Mock() + mock_response = Mock() + mock_response.status_code = 404 + http_error = requests.exceptions.HTTPError(response=mock_response) + mock_client.get_asset_details.side_effect = http_error + mock_client_class.return_value = mock_client + + main() + + mock_exit.assert_called_once_with(1) + + @patch('sys.argv', ['script.py', 'asset-123', '--cdgc-url', 'https://test.com', + '-u', 'user', '-p', 'pass']) + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.InformaticaClient') + @patch('sys.exit') + def test_main_connection_error(self, mock_exit, mock_client_class): + """Test main function with connection error""" + from wxdi.odcs_generator.generate_odcs_from_informatica import main + import requests + + mock_client = Mock() + mock_client.get_asset_details.side_effect = requests.exceptions.ConnectionError() + mock_client_class.return_value = mock_client + + main() + + mock_exit.assert_called_once_with(1) + + @patch('sys.argv', ['script.py', 'asset-123', '--cdgc-url', 'https://test.com', + '-u', 'user', '-p', 'pass']) + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.InformaticaClient') + @patch('sys.exit') + def test_main_timeout_error(self, mock_exit, mock_client_class): + """Test main function with timeout error""" + from wxdi.odcs_generator.generate_odcs_from_informatica import main + import requests + + mock_client = Mock() + mock_client.get_asset_details.side_effect = requests.exceptions.Timeout() + mock_client_class.return_value = mock_client + + main() + + mock_exit.assert_called_once_with(1) + + @patch('sys.argv', ['script.py', 'asset-123', '--cdgc-url', 'https://test.com', + '-u', 'user', '-p', 'pass']) + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.InformaticaClient') + @patch('sys.exit') + def test_main_key_error(self, mock_exit, mock_client_class): + """Test main function with KeyError""" + from wxdi.odcs_generator.generate_odcs_from_informatica import main + + mock_client = Mock() + mock_client.get_asset_details.side_effect = KeyError('missing_field') + mock_client_class.return_value = mock_client + + main() + + mock_exit.assert_called_once_with(1) + + @patch('sys.argv', ['script.py', 'asset-123', '--cdgc-url', 'https://test.com', + '-u', 'user', '-p', 'pass']) + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.InformaticaClient') + @patch('sys.exit') + def test_main_unexpected_error(self, mock_exit, mock_client_class): + """Test main function with unexpected error""" + from wxdi.odcs_generator.generate_odcs_from_informatica import main + + mock_client = Mock() + mock_client.get_asset_details.side_effect = ValueError('unexpected error') + mock_client_class.return_value = mock_client + + main() + + mock_exit.assert_called_once_with(1) + + +if __name__ == '__main__': + pytest.main([__file__, '-v']) + From a306082e550e33d7762c8c44cee2d72b8cc6f66d Mon Sep 17 00:00:00 2001 From: Koichi Nishitani Date: Fri, 24 Apr 2026 00:57:43 +0900 Subject: [PATCH 02/33] chore: trigger versioning Signed-off-by: Koichi Nishitani --- .bumpversion.toml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.bumpversion.toml b/.bumpversion.toml index 86b135e..1d0c8e7 100644 --- a/.bumpversion.toml +++ b/.bumpversion.toml @@ -12,9 +12,10 @@ # limitations under the License. [tool.bumpversion] -current_version = "2.0.0-rc.1" +current_version = "1.0.0" commit = true message = "Update version {current_version} -> {new_version}" +ignore_missing_version = true parse = """(?x) (?P0|[1-9]\\d*)\\. (?P0|[1-9]\\d*)\\. From cb407e46613d034937050a9610f9b8ca35103e59 Mon Sep 17 00:00:00 2001 From: semantic-release-bot Date: Thu, 23 Apr 2026 16:11:42 +0000 Subject: [PATCH 03/33] Update version 1.0.0 -> 2.0.0 --- .bumpversion.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.bumpversion.toml b/.bumpversion.toml index 1d0c8e7..e50d6af 100644 --- a/.bumpversion.toml +++ b/.bumpversion.toml @@ -12,7 +12,7 @@ # limitations under the License. [tool.bumpversion] -current_version = "1.0.0" +current_version = "2.0.0" commit = true message = "Update version {current_version} -> {new_version}" ignore_missing_version = true From 98111961198d1818f778b85c95a228bbaee7f3eb Mon Sep 17 00:00:00 2001 From: semantic-release-bot Date: Thu, 23 Apr 2026 16:11:42 +0000 Subject: [PATCH 04/33] chore(release): 2.0.0 release notes # [2.0.0](https://github.com/IBM/data-intelligence-sdk/compare/v1.0.0...v2.0.0) (2026-04-23) * Upgrade to version 2.0.0 ([#13](https://github.com/IBM/data-intelligence-sdk/issues/13)) ([41f25ce](https://github.com/IBM/data-intelligence-sdk/commit/41f25cefc349a9d2add7d99bc94945d123104ca1)), closes [#7](https://github.com/IBM/data-intelligence-sdk/issues/7) ### BREAKING CHANGES * Bug fixes for Data Quality and addition of Data Product Hub modules Signed-off-by: Koichi Nishitani * fix relative path of Actions scripts Signed-off-by: Koichi Nishitani * fix script condition Signed-off-by: Koichi Nishitani * fix script condition again Signed-off-by: Koichi Nishitani * add missing Makefile and fix documentation Signed-off-by: Koichi Nishitani * add dq tests and upgrade python Signed-off-by: Koichi Nishitani * try to extend build timeout Signed-off-by: Koichi Nishitani * upgrade to python 3.10 as 3.9 is EOL Signed-off-by: Koichi Nishitani * modify constraint_model to use custom StrEnum in python 3.10 Signed-off-by: Koichi Nishitani * add pylint Signed-off-by: Koichi Nishitani * skip linting until ready Signed-off-by: Koichi Nishitani * make sure to build before checking Signed-off-by: Koichi Nishitani --- CHANGELOG.md | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index a42bb2b..2aff2e4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,59 @@ +# [2.0.0](https://github.com/IBM/data-intelligence-sdk/compare/v1.0.0...v2.0.0) (2026-04-23) + + +* Upgrade to version 2.0.0 ([#13](https://github.com/IBM/data-intelligence-sdk/issues/13)) ([41f25ce](https://github.com/IBM/data-intelligence-sdk/commit/41f25cefc349a9d2add7d99bc94945d123104ca1)), closes [#7](https://github.com/IBM/data-intelligence-sdk/issues/7) + + +### BREAKING CHANGES + +* Bug fixes for Data Quality and addition of Data Product Hub modules + +Signed-off-by: Koichi Nishitani + +* fix relative path of Actions scripts + +Signed-off-by: Koichi Nishitani + +* fix script condition + +Signed-off-by: Koichi Nishitani + +* fix script condition again + +Signed-off-by: Koichi Nishitani + +* add missing Makefile and fix documentation + +Signed-off-by: Koichi Nishitani + +* add dq tests and upgrade python + +Signed-off-by: Koichi Nishitani + +* try to extend build timeout + +Signed-off-by: Koichi Nishitani + +* upgrade to python 3.10 as 3.9 is EOL + +Signed-off-by: Koichi Nishitani + +* modify constraint_model to use custom StrEnum in python 3.10 + +Signed-off-by: Koichi Nishitani + +* add pylint + +Signed-off-by: Koichi Nishitani + +* skip linting until ready + +Signed-off-by: Koichi Nishitani + +* make sure to build before checking + +Signed-off-by: Koichi Nishitani + # [2.0.0-rc.1](https://github.com/IBM/data-intelligence-sdk/compare/v1.0.0...v2.0.0-rc.1) (2026-04-22) From cb7219e5550d8012e931a5c5e7680351a7256a13 Mon Sep 17 00:00:00 2001 From: Koichi Nishitani Date: Fri, 24 Apr 2026 11:27:38 +0900 Subject: [PATCH 05/33] chore: enable workflow_call documentation publish Signed-off-by: Koichi Nishitani --- .github/workflows/docs.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index a22c199..6cf2651 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -65,7 +65,7 @@ jobs: deploy: # Only deploy on push to main/master, not on PRs - if: github.event_name == 'push' && github.ref == 'refs/heads/main' + if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main' environment: name: github-pages From e5e721ae301aa051a5fa8a9dde99a253d7b1e1eb Mon Sep 17 00:00:00 2001 From: Greeshma Rajendran Date: Wed, 6 May 2026 17:28:28 +0530 Subject: [PATCH 06/33] [skip ci] dph documentation update, docs(api): updated DPH documentation (#14) Signed-off-by: Greeshma Rajendran --- docs/api/data_product_recommender/index.rst | 68 ++ docs/api/dph_services/core.rst | 57 ++ docs/api/dph_services/index.rst | 108 +++ docs/api/index.rst | 33 + docs/api/odcs_generator/index.rst | 53 ++ docs/chapters/05_dph_services/examples.rst | 514 +++++++++++++++ docs/chapters/05_dph_services/index.rst | 115 ++++ docs/chapters/05_dph_services/overview.rst | 266 ++++++++ docs/chapters/05_dph_services/usage_guide.rst | 614 ++++++++++++++++++ .../collibra_integration.rst | 113 ++++ docs/chapters/06_odcs_generator/examples.rst | 98 +++ docs/chapters/06_odcs_generator/index.rst | 189 ++++++ .../informatica_integration.rst | 93 +++ docs/chapters/06_odcs_generator/overview.rst | 345 ++++++++++ .../07_data_product_recommender/examples.rst | 100 +++ .../07_data_product_recommender/index.rst | 111 ++++ .../07_data_product_recommender/overview.rst | 65 ++ .../usage_guide.rst | 82 +++ .../index.rst | 216 +++--- docs/index.rst | 19 +- 20 files changed, 3149 insertions(+), 110 deletions(-) create mode 100644 docs/api/data_product_recommender/index.rst create mode 100644 docs/api/dph_services/core.rst create mode 100644 docs/api/dph_services/index.rst create mode 100644 docs/api/odcs_generator/index.rst create mode 100644 docs/chapters/05_dph_services/examples.rst create mode 100644 docs/chapters/05_dph_services/index.rst create mode 100644 docs/chapters/05_dph_services/overview.rst create mode 100644 docs/chapters/05_dph_services/usage_guide.rst create mode 100644 docs/chapters/06_odcs_generator/collibra_integration.rst create mode 100644 docs/chapters/06_odcs_generator/examples.rst create mode 100644 docs/chapters/06_odcs_generator/index.rst create mode 100644 docs/chapters/06_odcs_generator/informatica_integration.rst create mode 100644 docs/chapters/06_odcs_generator/overview.rst create mode 100644 docs/chapters/07_data_product_recommender/examples.rst create mode 100644 docs/chapters/07_data_product_recommender/index.rst create mode 100644 docs/chapters/07_data_product_recommender/overview.rst create mode 100644 docs/chapters/07_data_product_recommender/usage_guide.rst rename docs/chapters/{05_future_modules => 08_future_modules}/index.rst (96%) diff --git a/docs/api/data_product_recommender/index.rst b/docs/api/data_product_recommender/index.rst new file mode 100644 index 0000000..713e0f5 --- /dev/null +++ b/docs/api/data_product_recommender/index.rst @@ -0,0 +1,68 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _api_data_product_recommender: + +Data Product Recommender Reference +=================================== + +Class reference for the Data Product Recommender module. + +Core Classes +------------ + +.. currentmodule:: wxdi.data_product_recommender.recommender + +.. autoclass:: DataProductRecommender + :members: + :undoc-members: + :show-inheritance: + +Platform Parsers +---------------- + +.. currentmodule:: wxdi.data_product_recommender.platforms + +.. autoclass:: SnowflakeQueryParser + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: DatabricksQueryParser + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: BigQueryQueryParser + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: WatsonxDataQueryParser + :members: + :undoc-members: + :show-inheritance: + +Base Classes +------------ + +.. currentmodule:: wxdi.data_product_recommender.base + +.. autoclass:: QueryLogParser + :members: + :undoc-members: + :show-inheritance: + +.. Made with Bob diff --git a/docs/api/dph_services/core.rst b/docs/api/dph_services/core.rst new file mode 100644 index 0000000..2009b39 --- /dev/null +++ b/docs/api/dph_services/core.rst @@ -0,0 +1,57 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _api_dph_services_core: + +Core Classes +============ + +Main service class and data models for Data Product Hub Services. + +DphV1 Service +------------- + +.. currentmodule:: wxdi.dph_services + +.. autoclass:: DphV1 + :members: + :undoc-members: + :show-inheritance: + :inherited-members: + +Common Models +------------- + +.. autoclass:: wxdi.dph_services.common.DataProduct + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: wxdi.dph_services.common.DataProductDraft + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: wxdi.dph_services.common.ContractTerms + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: wxdi.dph_services.common.Domain + :members: + :undoc-members: + :show-inheritance: + +.. Made with Bob \ No newline at end of file diff --git a/docs/api/dph_services/index.rst b/docs/api/dph_services/index.rst new file mode 100644 index 0000000..d9adaa6 --- /dev/null +++ b/docs/api/dph_services/index.rst @@ -0,0 +1,108 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _api_dph_services: + +DPH Services API +================ + +API reference for the Data Product Hub Services module. + +.. toctree:: + :maxdepth: 2 + + core + +Main Service Class +------------------ + +.. currentmodule:: wxdi.dph_services + +.. autoclass:: DphV1 + :members: + :undoc-members: + :show-inheritance: + :special-members: __init__ + +Container Operations +-------------------- + +.. automethod:: DphV1.initialize +.. automethod:: DphV1.get_initialize_status +.. automethod:: DphV1.get_service_id_credentials +.. automethod:: DphV1.manage_api_keys + +Data Product Operations +----------------------- + +.. automethod:: DphV1.create_data_product +.. automethod:: DphV1.list_data_products +.. automethod:: DphV1.list_data_products_with_pager +.. automethod:: DphV1.get_data_product +.. automethod:: DphV1.update_data_product +.. automethod:: DphV1.delete_data_product + +Draft Operations +---------------- + +.. automethod:: DphV1.create_data_product_draft +.. automethod:: DphV1.list_data_product_drafts +.. automethod:: DphV1.get_data_product_draft +.. automethod:: DphV1.update_data_product_draft +.. automethod:: DphV1.delete_data_product_draft +.. automethod:: DphV1.publish_data_product_draft + +Release Operations +------------------ + +.. automethod:: DphV1.list_data_product_releases +.. automethod:: DphV1.get_data_product_release +.. automethod:: DphV1.update_data_product_release +.. automethod:: DphV1.retire_data_product_release + +Contract Terms Operations +------------------------- + +.. automethod:: DphV1.create_draft_contract_terms_document +.. automethod:: DphV1.get_data_product_draft_contract_terms +.. automethod:: DphV1.update_draft_contract_terms_document +.. automethod:: DphV1.delete_draft_contract_terms_document + +Domain Operations +----------------- + +.. automethod:: DphV1.list_data_product_domains +.. automethod:: DphV1.create_data_product_domain +.. automethod:: DphV1.create_data_product_subdomain +.. automethod:: DphV1.get_domain +.. automethod:: DphV1.update_data_product_domain +.. automethod:: DphV1.delete_domain + +Asset Visualization Operations +------------------------------- + +.. automethod:: DphV1.create_data_asset_visualization +.. automethod:: DphV1.reinitiate_data_asset_visualization + +Contract Template Operations +---------------------------- + +.. automethod:: DphV1.create_contract_template +.. automethod:: DphV1.list_data_product_contract_template +.. automethod:: DphV1.get_contract_template +.. automethod:: DphV1.update_data_product_contract_template +.. automethod:: DphV1.delete_data_product_contract_template + +.. Made with Bob \ No newline at end of file diff --git a/docs/api/index.rst b/docs/api/index.rst index bab9877..6455d0f 100644 --- a/docs/api/index.rst +++ b/docs/api/index.rst @@ -27,6 +27,9 @@ This API reference documentation is auto-generated from the source code docstrin common/index dq_validator/index + dph_services/index + odcs_generator/index + data_product_recommender/index Module Organization ------------------- @@ -50,6 +53,36 @@ In-memory data quality validation: * :ref:`REST API Providers` - IBM Cloud Pak for Data integration * :ref:`Result Consolidation` - Result aggregation and analysis +DPH Services Module +~~~~~~~~~~~~~~~~~~~ + +Data Product Hub API client: + +* :ref:`DphV1 Service` - Main service class for Data Product Hub operations +* Container, data product, draft, release, and domain management +* Contract terms and template operations +* Asset visualization + +ODCS Generator Module +~~~~~~~~~~~~~~~~~~~~~ + +Generate Open Data Contract Standard files: + +* :ref:`Collibra Integration` - CollibraClient and ODCSGenerator classes +* :ref:`Informatica Integration` - InformaticaClient and ODCSGenerator classes +* ODCS v3.1.0 compliant YAML generation +* Command-line and Python interfaces + +Data Product Recommender Module +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Query log analysis tool for data product recommendations: + +* :ref:`DataProductRecommender` - Core recommendation engine +* Platform-specific query log parsers (Snowflake, Databricks, BigQuery, watsonx.data) +* Scoring and ranking algorithms +* CLI and Python interfaces + Navigation Tips --------------- diff --git a/docs/api/odcs_generator/index.rst b/docs/api/odcs_generator/index.rst new file mode 100644 index 0000000..d7e3203 --- /dev/null +++ b/docs/api/odcs_generator/index.rst @@ -0,0 +1,53 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _api_odcs_generator: + +ODCS Generator Reference +======================== + +Class reference for the ODCS Generator module. + +Collibra Integration +-------------------- + +.. currentmodule:: wxdi.odcs_generator.generate_odcs_from_collibra + +.. autoclass:: CollibraClient + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: ODCSGenerator + :members: + :undoc-members: + :show-inheritance: + +Informatica Integration +----------------------- + +.. currentmodule:: wxdi.odcs_generator.generate_odcs_from_informatica + +.. autoclass:: InformaticaClient + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: ODCSGenerator + :members: + :undoc-members: + :show-inheritance: + +.. Made with Bob diff --git a/docs/chapters/05_dph_services/examples.rst b/docs/chapters/05_dph_services/examples.rst new file mode 100644 index 0000000..2d277c9 --- /dev/null +++ b/docs/chapters/05_dph_services/examples.rst @@ -0,0 +1,514 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _dph_services_examples: + +Examples +======== + +Complete examples demonstrating common use cases for the Data Product Hub Services module. + +Complete Workflow Example +-------------------------- + +This example demonstrates the complete lifecycle of a data product: + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + from ibm_cloud_sdk_core import ApiException + + # Initialize service + authenticator = IAMAuthenticator('your-api-key') + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + + # Step 1: Initialize container + print("Initializing container...") + init_response = dph_service.initialize( + include=['delivery_methods', 'data_product_samples', 'domains_multi_industry'] + ) + container_id = init_response.result['container']['id'] + print(f"Container initialized: {container_id}") + + # Step 2: Create a domain + print("\nCreating domain...") + domain = dph_service.create_data_product_domain( + name='Customer Analytics', + description='Customer behavior and analytics data products', + container={'id': container_id} + ) + domain_id = domain.result['id'] + print(f"Domain created: {domain_id}") + + # Step 3: Create data product with draft + print("\nCreating data product...") + data_product = dph_service.create_data_product( + drafts=[{ + 'version': '1.0.0', + 'name': 'Customer Purchase History', + 'description': 'Historical customer purchase data for analytics', + 'asset': { + 'id': 'asset-12345', + 'container': {'id': container_id} + }, + 'domain': { + 'id': domain_id, + 'name': 'Customer Analytics' + }, + 'parts_out': [{ + 'asset': { + 'id': 'asset-12345', + 'container': {'id': container_id} + }, + 'delivery_methods': [{ + 'id': 'delivery-method-001', + 'container': {'id': container_id} + }] + }] + }] + ) + + product_id = data_product.result['id'] + draft_id = data_product.result['drafts'][0]['id'] + print(f"Data product created: {product_id}") + print(f"Draft created: {draft_id}") + + # Step 4: Add contract terms + print("\nAdding contract terms...") + contract_terms = dph_service.get_data_product_draft_contract_terms( + data_product_id=product_id, + draft_id=draft_id + ) + + terms_id = contract_terms.result['id'] + + doc = dph_service.create_draft_contract_terms_document( + data_product_id=product_id, + draft_id=draft_id, + contract_terms_id=terms_id, + type='terms_and_conditions', + name='Data Usage Terms', + url='https://example.com/terms.pdf' + ) + print(f"Contract document added: {doc.result['id']}") + + # Step 5: Publish the draft + print("\nPublishing draft...") + release = dph_service.publish_data_product_draft( + data_product_id=product_id, + draft_id=draft_id + ) + release_id = release.result['id'] + print(f"Release published: {release_id}") + + # Step 6: Create a new version + print("\nCreating new version...") + new_draft = dph_service.create_data_product_draft( + data_product_id=product_id, + asset={'id': 'asset-12345', 'container': {'id': container_id}}, + version='1.1.0', + name='Customer Purchase History v1.1', + description='Enhanced with additional purchase metrics' + ) + new_draft_id = new_draft.result['id'] + print(f"New draft created: {new_draft_id}") + + print("\n✅ Complete workflow executed successfully!") + +Batch Operations Example +------------------------- + +Create multiple data products efficiently: + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + + # Initialize service + authenticator = IAMAuthenticator('your-api-key') + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + + # Define multiple data products + products_to_create = [ + { + 'name': 'Customer Demographics', + 'description': 'Customer demographic information', + 'asset_id': 'asset-001' + }, + { + 'name': 'Transaction History', + 'description': 'Historical transaction records', + 'asset_id': 'asset-002' + }, + { + 'name': 'Product Catalog', + 'description': 'Complete product catalog data', + 'asset_id': 'asset-003' + } + ] + + # Create all products + created_products = [] + + for product_info in products_to_create: + try: + product = dph_service.create_data_product( + drafts=[{ + 'version': '1.0.0', + 'name': product_info['name'], + 'description': product_info['description'], + 'asset': { + 'id': product_info['asset_id'], + 'container': {'id': 'container-123'} + } + }] + ) + created_products.append(product.result) + print(f"✅ Created: {product_info['name']}") + except Exception as e: + print(f"❌ Failed to create {product_info['name']}: {e}") + + print(f"\nTotal products created: {len(created_products)}") + +Pagination Example +------------------ + +Handle large datasets with pagination: + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + + # Initialize service + authenticator = IAMAuthenticator('your-api-key') + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + + # Method 1: Using pager (recommended) + print("Fetching all data products using pager...") + all_products = [] + pager = dph_service.list_data_products_with_pager(limit=50) + + for page in pager: + all_products.extend(page['data_products']) + print(f"Fetched {len(page['data_products'])} products...") + + print(f"Total products: {len(all_products)}") + + # Method 2: Manual pagination + print("\nManual pagination example...") + all_products_manual = [] + start = None + + while True: + response = dph_service.list_data_products( + limit=50, + start=start + ) + + products = response.result['data_products'] + all_products_manual.extend(products) + + # Check if there are more pages + if 'next' not in response.result or not response.result['next']: + break + + # Extract start token from next link + start = response.result['next'].get('start') + + print(f"Total products (manual): {len(all_products_manual)}") + +Error Handling Example +---------------------- + +Robust error handling for production use: + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + from ibm_cloud_sdk_core import ApiException + import time + + # Initialize service + authenticator = IAMAuthenticator('your-api-key') + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + + def create_data_product_with_retry(dph_service, drafts, max_retries=3): + """Create data product with retry logic""" + for attempt in range(max_retries): + try: + response = dph_service.create_data_product(drafts=drafts) + return response + except ApiException as e: + if e.code == 429: # Rate limit + wait_time = 2 ** attempt + print(f"Rate limited. Waiting {wait_time}s before retry...") + time.sleep(wait_time) + elif e.code >= 500: # Server error + if attempt < max_retries - 1: + print(f"Server error. Retrying... (attempt {attempt + 1})") + time.sleep(2 ** attempt) + else: + raise + elif e.code == 404: + print(f"Resource not found: {e.message}") + raise + elif e.code == 401: + print("Authentication failed. Check your credentials.") + raise + elif e.code == 403: + print("Insufficient permissions.") + raise + else: + print(f"API Error {e.code}: {e.message}") + raise + + raise Exception("Max retries exceeded") + + # Use the retry function + try: + drafts = [{ + 'version': '1.0.0', + 'name': 'Test Product', + 'description': 'Test description', + 'asset': {'id': 'asset-123', 'container': {'id': 'container-456'}} + }] + + product = create_data_product_with_retry(dph_service, drafts) + print(f"✅ Product created: {product.result['id']}") + except Exception as e: + print(f"❌ Failed to create product: {e}") + +Search and Filter Example +-------------------------- + +Find specific data products: + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + + # Initialize service + authenticator = IAMAuthenticator('your-api-key') + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + + # Get all products and filter + all_products = [] + pager = dph_service.list_data_products_with_pager(limit=100) + + for page in pager: + all_products.extend(page['data_products']) + + # Filter by name pattern + customer_products = [ + p for p in all_products + if 'customer' in p['name'].lower() + ] + print(f"Found {len(customer_products)} customer-related products") + + # Filter by domain + analytics_products = [ + p for p in all_products + if p.get('domain', {}).get('name') == 'Customer Analytics' + ] + print(f"Found {len(analytics_products)} analytics products") + + # Filter by version + v1_products = [ + p for p in all_products + if p['version'].startswith('1.') + ] + print(f"Found {len(v1_products)} v1.x products") + +Contract Template Example +-------------------------- + +Create and use contract templates: + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + + # Initialize service + authenticator = IAMAuthenticator('your-api-key') + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + + # Create a reusable contract template + template = dph_service.create_contract_template( + name='Standard Data Sharing Agreement', + description='Standard terms for internal data sharing', + contract_terms_documents=[ + { + 'type': 'terms_and_conditions', + 'name': 'Terms and Conditions', + 'url': 'https://example.com/standard-terms.pdf' + }, + { + 'type': 'sla', + 'name': 'Service Level Agreement', + 'url': 'https://example.com/sla.pdf' + } + ] + ) + + template_id = template.result['id'] + print(f"Template created: {template_id}") + + # Use template when creating data products + data_product = dph_service.create_data_product( + drafts=[{ + 'version': '1.0.0', + 'name': 'Sales Data', + 'description': 'Monthly sales data', + 'asset': {'id': 'asset-123', 'container': {'id': 'container-456'}}, + 'contract_terms': { + 'template_id': template_id + } + }] + ) + + print(f"Data product created with template: {data_product.result['id']}") + +Domain Hierarchy Example +------------------------ + +Create and manage domain hierarchies: + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + + # Initialize service + authenticator = IAMAuthenticator('your-api-key') + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + + # Create parent domain + parent_domain = dph_service.create_data_product_domain( + name='Customer Data', + description='All customer-related data products', + container={'id': 'container-123'} + ) + parent_id = parent_domain.result['id'] + print(f"Parent domain created: {parent_id}") + + # Create subdomains + subdomains = [ + {'name': 'Demographics', 'description': 'Customer demographic data'}, + {'name': 'Behavior', 'description': 'Customer behavior analytics'}, + {'name': 'Transactions', 'description': 'Customer transaction history'} + ] + + for subdomain_info in subdomains: + subdomain = dph_service.create_data_product_subdomain( + domain_id=parent_id, + name=subdomain_info['name'], + description=subdomain_info['description'] + ) + print(f"Subdomain created: {subdomain_info['name']}") + + # List all domains with hierarchy + domains = dph_service.list_data_product_domains(limit=100) + + for domain in domains.result['domains']: + print(f"\n{domain['name']}") + if 'subdomains' in domain: + for subdomain in domain['subdomains']: + print(f" └─ {subdomain['name']}") + +Monitoring and Reporting Example +--------------------------------- + +Generate reports on data product usage: + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + from collections import defaultdict + from datetime import datetime + + # Initialize service + authenticator = IAMAuthenticator('your-api-key') + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + + # Collect all data products + all_products = [] + pager = dph_service.list_data_products_with_pager(limit=100) + + for page in pager: + all_products.extend(page['data_products']) + + # Generate statistics + stats = { + 'total_products': len(all_products), + 'by_domain': defaultdict(int), + 'by_version': defaultdict(int), + 'by_state': defaultdict(int) + } + + for product in all_products: + # Count by domain + domain_name = product.get('domain', {}).get('name', 'Unknown') + stats['by_domain'][domain_name] += 1 + + # Count by version + version = product.get('version', 'Unknown') + major_version = version.split('.')[0] if '.' in version else version + stats['by_version'][f"v{major_version}.x"] += 1 + + # Count by state + state = product.get('state', 'Unknown') + stats['by_state'][state] += 1 + + # Print report + print("=" * 50) + print("DATA PRODUCT REPORT") + print("=" * 50) + print(f"\nTotal Data Products: {stats['total_products']}") + + print("\nBy Domain:") + for domain, count in sorted(stats['by_domain'].items()): + print(f" {domain}: {count}") + + print("\nBy Version:") + for version, count in sorted(stats['by_version'].items()): + print(f" {version}: {count}") + + print("\nBy State:") + for state, count in sorted(stats['by_state'].items()): + print(f" {state}: {count}") + +See Also +-------- + +- :ref:`dph_services_usage` - Detailed usage guide +- :ref:`api_dph_services` - API reference +- :ref:`dph_services_overview` - Architecture overview + +.. Made with Bob diff --git a/docs/chapters/05_dph_services/index.rst b/docs/chapters/05_dph_services/index.rst new file mode 100644 index 0000000..59a1455 --- /dev/null +++ b/docs/chapters/05_dph_services/index.rst @@ -0,0 +1,115 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _dph_services: + +Data Product Hub Services +========================== + +Python client library for IBM Data Product Hub API, providing programmatic access to data product management, container operations, contract terms, and asset visualization. + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + overview + usage_guide + examples + +Overview +-------- + +The ``dph_services`` module provides a complete Python SDK for interacting with IBM Data Product Hub services. It enables developers to programmatically manage the entire data product lifecycle, from initialization to publication and retirement. + +Key Features +------------ + +**Container Management** + Initialize and configure data product containers with delivery methods, samples, and domain structures. + +**Data Product Lifecycle** + Create, update, publish, and retire data products with full version control and draft management. + +**Contract Terms** + Manage contract terms, documents, and templates for data product agreements. + +**Asset Visualization** + Create and manage data asset visualizations for better data discovery. + +**Domain Organization** + Organize data products into domains and subdomains for better categorization. + +**Release Management** + Handle data product releases with versioning and retirement capabilities. + +Quick Start +----------- + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + + # Initialize authenticator + authenticator = IAMAuthenticator('your-api-key') + + # Create service instance + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + + # Initialize container + response = dph_service.initialize( + include=['delivery_methods', 'data_product_samples', 'domains_multi_industry'] + ) + + # Create a data product + data_product = dph_service.create_data_product( + drafts=[{ + 'version': '1.0.0', + 'name': 'Customer Analytics Data Product', + 'description': 'Comprehensive customer analytics dataset', + 'asset': { + 'id': 'asset-123', + 'container': {'id': 'container-456'} + } + }] + ) + +Use Cases +--------- + +**Data Product Onboarding** + Automate the creation and configuration of new data products in your data marketplace. + +**Lifecycle Automation** + Build workflows that automatically promote drafts to releases based on quality checks. + +**Contract Management** + Programmatically manage data sharing agreements and terms of use. + +**Catalog Integration** + Integrate with data catalogs to automatically create data products from existing assets. + +**Governance Workflows** + Implement approval workflows and governance policies for data product publication. + +Next Steps +---------- + +- :ref:`dph_services_usage` - Detailed usage guide with examples +- :ref:`dph_services_examples` - Practical code examples +- :ref:`api_dph_services` - Complete API reference + +.. Made with Bob diff --git a/docs/chapters/05_dph_services/overview.rst b/docs/chapters/05_dph_services/overview.rst new file mode 100644 index 0000000..ba2042e --- /dev/null +++ b/docs/chapters/05_dph_services/overview.rst @@ -0,0 +1,266 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _dph_services_overview: + +DPH Services Overview +===================== + +The Data Product Hub Services module provides a comprehensive Python SDK for managing data products in IBM Data Product Hub. + +Architecture +------------ + +The module is built on top of the IBM Cloud SDK Core and provides: + +- **Type-safe API**: Full type hints for better IDE support +- **Error handling**: Comprehensive exception handling with detailed error messages +- **Pagination support**: Built-in pagination for large result sets +- **Authentication**: Seamless integration with IBM Cloud authentication + +Core Components +--------------- + +Container Management +~~~~~~~~~~~~~~~~~~~~ + +Containers are the foundation of Data Product Hub, providing: + +- Delivery method configurations +- Sample data products +- Domain structures +- Service credentials + +Data Products +~~~~~~~~~~~~~ + +Data products represent packaged data assets with: + +- Metadata and descriptions +- Version control +- Asset references +- Domain associations +- Contract terms + +Drafts and Releases +~~~~~~~~~~~~~~~~~~~ + +**Drafts**: Work-in-progress versions that can be edited and updated + +**Releases**: Published versions that are immutable and available for consumption + +Contract Terms +~~~~~~~~~~~~~~ + +Legal and business terms governing data product usage: + +- Terms and conditions documents +- Service level agreements +- Usage restrictions +- Compliance requirements + +Domains +~~~~~~~ + +Organizational structure for categorizing data products: + +- Top-level domains (e.g., Finance, Marketing) +- Subdomains for finer categorization +- Multi-industry domain support + +Data Flow +--------- + +1. **Initialize Container**: Set up the data product hub environment +2. **Create Draft**: Define a new data product version +3. **Add Contract Terms**: Attach legal and business terms +4. **Publish Draft**: Convert draft to a release +5. **Manage Lifecycle**: Update, version, or retire releases + +Authentication +-------------- + +The module supports multiple authentication methods: + +**IAM Authentication** (Recommended) + +.. code-block:: python + + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + + authenticator = IAMAuthenticator('your-api-key') + dph_service = DphV1(authenticator=authenticator) + +**Bearer Token Authentication** + +.. code-block:: python + + from ibm_cloud_sdk_core.authenticators import BearerTokenAuthenticator + + authenticator = BearerTokenAuthenticator('your-bearer-token') + dph_service = DphV1(authenticator=authenticator) + +**Cloud Pak for Data Authentication** + +.. code-block:: python + + from ibm_cloud_sdk_core.authenticators import CloudPakForDataAuthenticator + + authenticator = CloudPakForDataAuthenticator( + username='your-username', + password='your-password', + url='https://your-cpd-instance.com' + ) + dph_service = DphV1(authenticator=authenticator) + +Error Handling +-------------- + +The SDK uses standard IBM Cloud SDK exceptions: + +.. code-block:: python + + from ibm_cloud_sdk_core import ApiException + + try: + response = dph_service.get_data_product(data_product_id='invalid-id') + except ApiException as e: + print(f"Error Code: {e.code}") + print(f"Error Message: {e.message}") + print(f"HTTP Status: {e.http_response.status_code}") + +Common error codes: + +- **400**: Bad Request - Invalid parameters +- **401**: Unauthorized - Authentication failed +- **403**: Forbidden - Insufficient permissions +- **404**: Not Found - Resource doesn't exist +- **409**: Conflict - Resource already exists +- **500**: Internal Server Error - Service error + +Best Practices +-------------- + +**Use Pagination for Large Datasets** + +.. code-block:: python + + # Use pager for automatic pagination + pager = dph_service.list_data_products_with_pager(limit=50) + for page in pager: + for product in page['data_products']: + process_product(product) + +**Implement Retry Logic** + +.. code-block:: python + + import time + from ibm_cloud_sdk_core import ApiException + + def create_with_retry(dph_service, drafts, max_retries=3): + for attempt in range(max_retries): + try: + return dph_service.create_data_product(drafts=drafts) + except ApiException as e: + if e.code == 429 or e.code >= 500: # Rate limit or server error + if attempt < max_retries - 1: + time.sleep(2 ** attempt) # Exponential backoff + continue + raise + +**Validate Before Publishing** + +.. code-block:: python + + def validate_draft(draft): + """Validate draft before publishing""" + required_fields = ['name', 'version', 'asset', 'domain'] + for field in required_fields: + if field not in draft or not draft[field]: + raise ValueError(f"Missing required field: {field}") + return True + +**Use JSON Patch for Updates** + +.. code-block:: python + + # Efficient updates using JSON Patch + patch_operations = [ + {'op': 'replace', 'path': '/description', 'value': 'Updated description'}, + {'op': 'add', 'path': '/tags/-', 'value': 'new-tag'} + ] + + dph_service.update_data_product( + data_product_id=product_id, + json_patch_instructions=patch_operations + ) + +Performance Considerations +-------------------------- + +**Batch Operations** + +When creating multiple data products, consider batching to reduce API calls: + +.. code-block:: python + + # Create multiple drafts in a single data product + dph_service.create_data_product( + drafts=[draft1, draft2, draft3] + ) + +**Caching** + +Cache frequently accessed data to reduce API calls: + +.. code-block:: python + + from functools import lru_cache + + @lru_cache(maxsize=100) + def get_domain_cached(domain_id): + return dph_service.get_domain(domain_id=domain_id) + +**Parallel Processing** + +Use concurrent requests for independent operations: + +.. code-block:: python + + from concurrent.futures import ThreadPoolExecutor + + def get_product(product_id): + return dph_service.get_data_product(data_product_id=product_id) + + with ThreadPoolExecutor(max_workers=5) as executor: + products = list(executor.map(get_product, product_ids)) + +Requirements +------------ + +- Python 3.8 or higher +- ibm-cloud-sdk-core >= 3.16.7 +- requests >= 2.32.4 +- python-dateutil >= 2.5.3 + +See Also +-------- + +- :ref:`dph_services_usage` - Detailed usage guide +- :ref:`dph_services_examples` - Code examples +- :ref:`api_dph_services` - API reference + +.. Made with Bob diff --git a/docs/chapters/05_dph_services/usage_guide.rst b/docs/chapters/05_dph_services/usage_guide.rst new file mode 100644 index 0000000..af9b340 --- /dev/null +++ b/docs/chapters/05_dph_services/usage_guide.rst @@ -0,0 +1,614 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _dph_services_usage: + +Usage Guide +=========== + +This guide provides detailed instructions for using the Data Product Hub Services module. + +Installation +------------ + +Install the data-intelligence-sdk package: + +.. code-block:: bash + + pip install -e . + +Or install from PyPI (when available): + +.. code-block:: bash + + pip install data-intelligence-sdk + +Setup and Configuration +----------------------- + +Initialize the Service +~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + + # Create authenticator + authenticator = IAMAuthenticator('your-api-key') + + # Initialize service + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + +Environment Variables +~~~~~~~~~~~~~~~~~~~~~ + +You can also configure using environment variables: + +.. code-block:: bash + + export DPH_APIKEY=your-api-key + export DPH_URL=https://your-dph-instance.com + +.. code-block:: python + + from wxdi.dph_services import DphV1 + + # Automatically uses environment variables + dph_service = DphV1.new_instance() + +Container Operations +-------------------- + +Initialize Container +~~~~~~~~~~~~~~~~~~~~ + +Initialize a new container with default settings: + +.. code-block:: python + + response = dph_service.initialize( + include=[ + 'delivery_methods', + 'data_product_samples', + 'domains_multi_industry' + ] + ) + + print(f"Container ID: {response.result['container']['id']}") + +Check Initialization Status +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + status = dph_service.get_initialize_status() + + if status.result['status'] == 'SUCCEEDED': + print("Container is ready") + elif status.result['status'] == 'IN_PROGRESS': + print("Initialization in progress") + else: + print(f"Status: {status.result['status']}") + +Get Service Credentials +~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + credentials = dph_service.get_service_id_credentials() + print(f"Service ID: {credentials.result['service_id']}") + +Data Product Management +----------------------- + +Create a Data Product +~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + data_product = dph_service.create_data_product( + drafts=[{ + 'version': '1.0.0', + 'name': 'Customer Analytics Dataset', + 'description': 'Comprehensive customer behavior analytics', + 'asset': { + 'id': 'asset-123', + 'container': {'id': 'container-456'} + }, + 'domain': { + 'id': 'domain-789', + 'name': 'Customer Analytics' + }, + 'parts_out': [{ + 'asset': { + 'id': 'asset-123', + 'container': {'id': 'container-456'} + }, + 'delivery_methods': [{ + 'id': 'method-001', + 'container': {'id': 'container-456'} + }] + }] + }] + ) + + product_id = data_product.result['id'] + print(f"Created data product: {product_id}") + +List Data Products +~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + # List with pagination + response = dph_service.list_data_products(limit=50) + + for product in response.result['data_products']: + print(f"- {product['name']} (v{product['version']})") + + # Use pager for all results + all_products = [] + pager = dph_service.list_data_products_with_pager(limit=50) + + for page in pager: + all_products.extend(page['data_products']) + +Get Data Product Details +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + product = dph_service.get_data_product(data_product_id=product_id) + + print(f"Name: {product.result['name']}") + print(f"Version: {product.result['version']}") + print(f"Description: {product.result['description']}") + print(f"Status: {product.result['state']}") + +Update Data Product +~~~~~~~~~~~~~~~~~~~ + +Use JSON Patch operations for updates: + +.. code-block:: python + + updated = dph_service.update_data_product( + data_product_id=product_id, + json_patch_instructions=[ + { + 'op': 'replace', + 'path': '/description', + 'value': 'Updated comprehensive customer analytics' + }, + { + 'op': 'add', + 'path': '/tags/-', + 'value': 'analytics' + } + ] + ) + +Delete Data Product +~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + dph_service.delete_data_product(data_product_id=product_id) + print("Data product deleted") + +Draft Management +---------------- + +Create a Draft +~~~~~~~~~~~~~~ + +.. code-block:: python + + draft = dph_service.create_data_product_draft( + data_product_id=product_id, + asset={'id': 'asset-123', 'container': {'id': 'container-456'}}, + version='1.1.0', + name='Customer Analytics Dataset v1.1', + description='Enhanced version with additional metrics' + ) + + draft_id = draft.result['id'] + +List Drafts +~~~~~~~~~~~ + +.. code-block:: python + + drafts = dph_service.list_data_product_drafts( + data_product_id=product_id, + limit=50 + ) + + for draft in drafts.result['drafts']: + print(f"- Draft {draft['version']}: {draft['state']}") + +Update Draft +~~~~~~~~~~~~ + +.. code-block:: python + + updated_draft = dph_service.update_data_product_draft( + data_product_id=product_id, + draft_id=draft_id, + json_patch_instructions=[ + { + 'op': 'replace', + 'path': '/description', + 'value': 'Updated draft description' + } + ] + ) + +Publish Draft +~~~~~~~~~~~~~ + +.. code-block:: python + + # Publish draft to create a release + release = dph_service.publish_data_product_draft( + data_product_id=product_id, + draft_id=draft_id + ) + + print(f"Published release: {release.result['id']}") + +Delete Draft +~~~~~~~~~~~~ + +.. code-block:: python + + dph_service.delete_data_product_draft( + data_product_id=product_id, + draft_id=draft_id + ) + +Contract Terms Management +------------------------- + +Create Contract Terms Document +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + doc = dph_service.create_draft_contract_terms_document( + data_product_id=product_id, + draft_id=draft_id, + contract_terms_id=terms_id, + type='terms_and_conditions', + name='Terms and Conditions', + url='https://example.com/terms.pdf', + attachment={ + 'id': 'attachment-123' + } + ) + +Get Contract Terms +~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + terms = dph_service.get_data_product_draft_contract_terms( + data_product_id=product_id, + draft_id=draft_id + ) + + for doc in terms.result['documents']: + print(f"- {doc['name']}: {doc['type']}") + +Update Contract Terms Document +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + updated_doc = dph_service.update_draft_contract_terms_document( + data_product_id=product_id, + draft_id=draft_id, + contract_terms_id=terms_id, + document_id=doc_id, + json_patch_instructions=[ + { + 'op': 'replace', + 'path': '/url', + 'value': 'https://example.com/updated-terms.pdf' + } + ] + ) + +Delete Contract Terms Document +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + dph_service.delete_draft_contract_terms_document( + data_product_id=product_id, + draft_id=draft_id, + contract_terms_id=terms_id, + document_id=doc_id + ) + +Release Management +------------------ + +List Releases +~~~~~~~~~~~~~ + +.. code-block:: python + + releases = dph_service.list_data_product_releases( + data_product_id=product_id, + state=['available', 'retired'] + ) + + for release in releases.result['releases']: + print(f"- v{release['version']}: {release['state']}") + +Get Release Details +~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + release = dph_service.get_data_product_release( + data_product_id=product_id, + release_id=release_id + ) + + print(f"Version: {release.result['version']}") + print(f"State: {release.result['state']}") + print(f"Published: {release.result['created_at']}") + +Update Release +~~~~~~~~~~~~~~ + +.. code-block:: python + + updated_release = dph_service.update_data_product_release( + data_product_id=product_id, + release_id=release_id, + json_patch_instructions=[ + { + 'op': 'replace', + 'path': '/description', + 'value': 'Updated release description' + } + ] + ) + +Retire Release +~~~~~~~~~~~~~~ + +.. code-block:: python + + retired = dph_service.retire_data_product_release( + data_product_id=product_id, + release_id=release_id + ) + + print(f"Release retired: {retired.result['state']}") + +Domain Management +----------------- + +List Domains +~~~~~~~~~~~~ + +.. code-block:: python + + domains = dph_service.list_data_product_domains(limit=50) + + for domain in domains.result['domains']: + print(f"- {domain['name']}: {domain['description']}") + +Create Domain +~~~~~~~~~~~~~ + +.. code-block:: python + + domain = dph_service.create_data_product_domain( + name='Customer Analytics', + description='Customer-related data products and analytics', + container={'id': 'container-123'} + ) + + domain_id = domain.result['id'] + +Create Subdomain +~~~~~~~~~~~~~~~~ + +.. code-block:: python + + subdomain = dph_service.create_data_product_subdomain( + domain_id=domain_id, + name='Customer Segmentation', + description='Customer segmentation and clustering datasets' + ) + +Get Domain +~~~~~~~~~~ + +.. code-block:: python + + domain = dph_service.get_domain(domain_id=domain_id) + + print(f"Name: {domain.result['name']}") + print(f"Subdomains: {len(domain.result.get('subdomains', []))}") + +Update Domain +~~~~~~~~~~~~~ + +.. code-block:: python + + updated_domain = dph_service.update_data_product_domain( + domain_id=domain_id, + json_patch_instructions=[ + { + 'op': 'replace', + 'path': '/description', + 'value': 'Updated domain description' + } + ] + ) + +Delete Domain +~~~~~~~~~~~~~ + +.. code-block:: python + + dph_service.delete_domain(domain_id=domain_id) + +Asset Visualization +------------------- + +Create Visualization +~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + visualization = dph_service.create_data_asset_visualization( + container={'id': 'container-123'}, + assets=[ + {'id': 'asset-1', 'container': {'id': 'container-123'}}, + {'id': 'asset-2', 'container': {'id': 'container-123'}}, + {'id': 'asset-3', 'container': {'id': 'container-123'}} + ] + ) + + print(f"Visualization created: {visualization.result['id']}") + +Reinitiate Visualization +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + reinitiated = dph_service.reinitiate_data_asset_visualization( + container={'id': 'container-123'}, + assets=[ + {'id': 'asset-1', 'container': {'id': 'container-123'}}, + {'id': 'asset-4', 'container': {'id': 'container-123'}} + ] + ) + +Contract Templates +------------------ + +Create Template +~~~~~~~~~~~~~~~ + +.. code-block:: python + + template = dph_service.create_contract_template( + name='Standard Data Sharing Agreement', + description='Standard terms for data product sharing', + contract_terms_documents=[{ + 'type': 'terms_and_conditions', + 'name': 'Standard Terms', + 'url': 'https://example.com/standard-terms.pdf' + }] + ) + + template_id = template.result['id'] + +List Templates +~~~~~~~~~~~~~~ + +.. code-block:: python + + templates = dph_service.list_data_product_contract_template(limit=50) + + for template in templates.result['contract_templates']: + print(f"- {template['name']}") + +Get Template +~~~~~~~~~~~~ + +.. code-block:: python + + template = dph_service.get_contract_template( + contract_template_id=template_id + ) + +Update Template +~~~~~~~~~~~~~~~ + +.. code-block:: python + + updated_template = dph_service.update_data_product_contract_template( + contract_template_id=template_id, + json_patch_instructions=[ + { + 'op': 'replace', + 'path': '/description', + 'value': 'Updated template description' + } + ] + ) + +Delete Template +~~~~~~~~~~~~~~~ + +.. code-block:: python + + dph_service.delete_data_product_contract_template( + contract_template_id=template_id + ) + +Advanced Topics +--------------- + +Custom Headers +~~~~~~~~~~~~~~ + +Add custom headers to requests: + +.. code-block:: python + + response = dph_service.get_data_product( + data_product_id=product_id, + headers={ + 'Custom-Header': 'value', + 'X-Request-ID': 'unique-id' + } + ) + +Timeout Configuration +~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + dph_service.set_http_config({ + 'timeout': 60 # 60 seconds + }) + +Disable SSL Verification (Development Only) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + dph_service.set_disable_ssl_verification(True) + +See Also +-------- + +- :ref:`dph_services_examples` - Complete code examples +- :ref:`api_dph_services` - API reference +- :ref:`dph_services_overview` - Architecture overview + +.. Made with Bob diff --git a/docs/chapters/06_odcs_generator/collibra_integration.rst b/docs/chapters/06_odcs_generator/collibra_integration.rst new file mode 100644 index 0000000..d5fe3fa --- /dev/null +++ b/docs/chapters/06_odcs_generator/collibra_integration.rst @@ -0,0 +1,113 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _odcs_generator_collibra: + +Collibra Integration +==================== + +Generate ODCS files from Collibra data catalog assets. + +Overview +-------- + +The Collibra integration extracts metadata from Collibra assets and generates ODCS v3.1.0 compliant YAML files. + +Features +-------- + +- ✅ Automatic metadata extraction via REST API +- ✅ Column discovery through asset relations +- ✅ Data type mapping (logical and physical) +- ✅ Classification support via GraphQL API +- ✅ Tag integration at asset and column levels +- ✅ Custom attribute preservation + +Installation +------------ + +.. code-block:: bash + + pip install -e . + +Configuration +------------- + +Environment Variables +~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: bash + + export COLLIBRA_URL="https://your-instance.collibra.com" + export COLLIBRA_USERNAME="your_username" + export COLLIBRA_PASSWORD="your_password" + +Required Permissions +~~~~~~~~~~~~~~~~~~~~ + +- Read access to assets +- Read access to attributes +- Read access to relations +- Access to GraphQL API +- Read access to tags + +Usage +----- + +Command Line +~~~~~~~~~~~~ + +.. code-block:: bash + + python -m wxdi.odcs_generator.generate_odcs_from_collibra + +With options: + +.. code-block:: bash + + python -m wxdi.odcs_generator.generate_odcs_from_collibra \ + --output my-contract.yaml \ + --url https://collibra.com \ + --username myuser \ + --password mypass + +Python API +~~~~~~~~~~ + +.. code-block:: python + + from wxdi.odcs_generator.generate_odcs_from_collibra import CollibraClient, ODCSGenerator + + # Initialize client + client = CollibraClient( + base_url="https://your-instance.collibra.com", + username="your_username", + password="your_password" + ) + + # Create generator + generator = ODCSGenerator(client) + + # Generate ODCS + odcs_data = generator.generate_odcs("asset-id") + + # Save to file + generator.save_to_yaml(odcs_data, "output.yaml") + +See Also +-------- + +- :ref:`odcs_generator_examples` - Complete examples +- :ref:`api_odcs_generator` - API reference diff --git a/docs/chapters/06_odcs_generator/examples.rst b/docs/chapters/06_odcs_generator/examples.rst new file mode 100644 index 0000000..37c6f58 --- /dev/null +++ b/docs/chapters/06_odcs_generator/examples.rst @@ -0,0 +1,98 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _odcs_generator_examples: + +Examples +======== + +Complete examples for ODCS Generator module. + +Collibra Example +---------------- + +.. code-block:: python + + from wxdi.odcs_generator.generate_odcs_from_collibra import CollibraClient, ODCSGenerator + + # Initialize client + client = CollibraClient( + base_url="https://your-instance.collibra.com", + username="your_username", + password="your_password" + ) + + # Create generator + generator = ODCSGenerator(client) + + # Generate ODCS + odcs_data = generator.generate_odcs("019a57f9-62d2-7aa0-9f22-4fa2cea1180b") + + # Customize + odcs_data['dataProduct'] = 'Customer Data Product' + odcs_data['version'] = '2.0.0' + + # Save to file + generator.save_to_yaml(odcs_data, "customer-data-odcs.yaml") + +Informatica Example +------------------- + +.. code-block:: python + + from wxdi.odcs_generator.generate_odcs_from_informatica import InformaticaClient, ODCSGenerator + + # Initialize client + client = InformaticaClient( + base_url="https://your-informatica-instance.com", + username="your_username", + password="your_password" + ) + + # Create generator + generator = ODCSGenerator(client) + + # Generate ODCS + odcs_data = generator.generate_odcs("asset-id-123") + + # Save to file + generator.save_to_yaml(odcs_data, "output.yaml") + +Batch Processing +---------------- + +.. code-block:: python + + from wxdi.odcs_generator.generate_odcs_from_collibra import CollibraClient, ODCSGenerator + + client = CollibraClient(base_url, username, password) + generator = ODCSGenerator(client) + + asset_ids = ['id1', 'id2', 'id3'] + + for asset_id in asset_ids: + try: + odcs_data = generator.generate_odcs(asset_id) + generator.save_to_yaml(odcs_data, f"{asset_id}-odcs.yaml") + print(f"✅ Generated ODCS for {asset_id}") + except Exception as e: + print(f"❌ Failed for {asset_id}: {e}") + +See Also +-------- + +- :ref:`odcs_generator_collibra` - Collibra integration +- :ref:`odcs_generator_informatica` - Informatica integration +- :ref:`api_odcs_generator` - API reference diff --git a/docs/chapters/06_odcs_generator/index.rst b/docs/chapters/06_odcs_generator/index.rst new file mode 100644 index 0000000..31b1ce5 --- /dev/null +++ b/docs/chapters/06_odcs_generator/index.rst @@ -0,0 +1,189 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _odcs_generator: + +ODCS Generator +============== + +Tools to automatically generate ODCS (Open Data Contract Standard) v3.1.0 compliant YAML files from data catalog metadata. + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + overview + collibra_integration + informatica_integration + examples + +Overview +-------- + +The ``odcs_generator`` module provides automated generation of Open Data Contract Standard (ODCS) files from enterprise data catalogs. It extracts metadata from catalog systems and transforms it into standardized data contracts. + +Key Features +------------ + +**Multi-Catalog Support** + Generate ODCS files from Collibra and Informatica CDGC data catalogs. + +**Automatic Metadata Extraction** + Fetch asset details, attributes, relations, and classifications automatically. + +**Column Discovery** + Automatically discover and document table columns through catalog relations. + +**Data Type Mapping** + Intelligent mapping of catalog data types to ODCS standard types. + +**Classification Support** + Extract and include data classifications and sensitivity labels. + +**ODCS v3.1.0 Compliance** + Generate fully compliant ODCS YAML files ready for use. + +Quick Start +----------- + +**Collibra Integration** + +.. code-block:: python + + from wxdi.odcs_generator.generate_odcs_from_collibra import CollibraClient, ODCSGenerator + + # Initialize client + client = CollibraClient( + base_url="https://your-instance.collibra.com", + username="your_username", + password="your_password" + ) + + # Create generator + generator = ODCSGenerator(client) + + # Generate ODCS + odcs_data = generator.generate_odcs("asset-id") + + # Save to file + generator.save_to_yaml(odcs_data, "output.yaml") + +**Informatica Integration** + +.. code-block:: python + + from wxdi.odcs_generator.generate_odcs_from_informatica import InformaticaClient, ODCSGenerator + + # Initialize client + client = InformaticaClient( + base_url="https://your-informatica-instance.com", + username="your_username", + password="your_password" + ) + + # Create generator + generator = ODCSGenerator(client) + + # Generate ODCS + odcs_data = generator.generate_odcs("asset-id") + +Use Cases +--------- + +**Data Contract Automation** + Automatically generate data contracts from existing catalog metadata. + +**Catalog Migration** + Export catalog metadata to standardized ODCS format for migration. + +**Documentation Generation** + Create comprehensive data documentation from catalog assets. + +**Compliance Reporting** + Generate standardized contracts for compliance and governance. + +**Data Product Onboarding** + Accelerate data product creation with automated contract generation. + +Supported Catalogs +------------------ + +**Collibra** + - Asset metadata extraction + - Column discovery via relations + - Data classifications via GraphQL + - Tag integration + - Custom attributes + +**Informatica CDGC** + - Asset metadata extraction + - Column schema discovery + - System attributes + - Technical metadata + - Business glossary terms + +What is ODCS? +------------- + +The Open Data Contract Standard (ODCS) is an open-source specification for defining data contracts. It provides: + +- **Standardized Format**: Common structure for data contracts across organizations +- **Schema Definition**: Detailed column-level metadata and constraints +- **Quality Rules**: Data quality expectations and validation rules +- **Service Level Agreements**: Performance and availability commitments +- **Governance**: Data ownership, stewardship, and compliance information + +ODCS v3.1.0 Structure +~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: yaml + + id: unique-contract-id + kind: DataContract + apiVersion: v3.1.0 + domain: domain-name + dataProduct: product-name + version: 1.0.0 + name: contract-name + status: active + description: + authoritativeDefinitions: + - type: source-system + url: source-url + schema: + - id: table-id + name: table-name + columns: + - id: column-id + name: column-name + logicalType: string + physicalType: VARCHAR(255) + description: column description + classification: PII + quality: + - id: rule-id + name: rule-name + type: completeness + column: column-name + +Next Steps +---------- + +- :ref:`odcs_generator_collibra` - Collibra integration guide +- :ref:`odcs_generator_informatica` - Informatica integration guide +- :ref:`odcs_generator_examples` - Complete code examples +- :ref:`api_odcs_generator` - API reference + +.. Made with Bob diff --git a/docs/chapters/06_odcs_generator/informatica_integration.rst b/docs/chapters/06_odcs_generator/informatica_integration.rst new file mode 100644 index 0000000..735691e --- /dev/null +++ b/docs/chapters/06_odcs_generator/informatica_integration.rst @@ -0,0 +1,93 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _odcs_generator_informatica: + +Informatica Integration +======================== + +Generate ODCS files from Informatica CDGC (Cloud Data Governance and Catalog) assets. + +Overview +-------- + +The Informatica integration extracts metadata from Informatica CDGC and generates ODCS v3.1.0 compliant YAML files. + +Features +-------- + +- ✅ Asset metadata extraction via REST API +- ✅ Column schema discovery +- ✅ System attribute handling +- ✅ Technical metadata extraction +- ✅ Business glossary term integration + +Installation +------------ + +.. code-block:: bash + + pip install -e . + +Configuration +------------- + +Environment Variables +~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: bash + + export INFORMATICA_URL="https://your-informatica-instance.com" + export INFORMATICA_USERNAME="your_username" + export INFORMATICA_PASSWORD="your_password" + +Usage +----- + +Command Line +~~~~~~~~~~~~ + +.. code-block:: bash + + python -m wxdi.odcs_generator.generate_odcs_from_informatica + +Python API +~~~~~~~~~~ + +.. code-block:: python + + from wxdi.odcs_generator.generate_odcs_from_informatica import InformaticaClient, ODCSGenerator + + # Initialize client + client = InformaticaClient( + base_url="https://your-informatica-instance.com", + username="your_username", + password="your_password" + ) + + # Create generator + generator = ODCSGenerator(client) + + # Generate ODCS + odcs_data = generator.generate_odcs("asset-id") + + # Save to file + generator.save_to_yaml(odcs_data, "output.yaml") + +See Also +-------- + +- :ref:`odcs_generator_examples` - Complete examples +- :ref:`api_odcs_generator` - API reference diff --git a/docs/chapters/06_odcs_generator/overview.rst b/docs/chapters/06_odcs_generator/overview.rst new file mode 100644 index 0000000..e75c68d --- /dev/null +++ b/docs/chapters/06_odcs_generator/overview.rst @@ -0,0 +1,345 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _odcs_generator_overview: + +ODCS Generator Overview +======================= + +The ODCS Generator module automates the creation of Open Data Contract Standard (ODCS) v3.1.0 compliant YAML files from enterprise data catalog metadata. + +Architecture +------------ + +The module uses a modular architecture with catalog-specific clients and a common generator: + +.. code-block:: text + + ┌─────────────────────────────────────────┐ + │ ODCS Generator Module │ + ├─────────────────────────────────────────┤ + │ │ + │ ┌──────────────┐ ┌───────────────┐ │ + │ │ Collibra │ │ Informatica │ │ + │ │ Client │ │ Client │ │ + │ └──────┬───────┘ └───────┬───────┘ │ + │ │ │ │ + │ └───────┬───────────┘ │ + │ │ │ + │ ┌───────▼────────┐ │ + │ │ ODCS Generator │ │ + │ └───────┬────────┘ │ + │ │ │ + │ ┌───────▼────────┐ │ + │ │ YAML Output │ │ + │ └────────────────┘ │ + └─────────────────────────────────────────┘ + +Core Components +--------------- + +Catalog Clients +~~~~~~~~~~~~~~~ + +**CollibraClient** + - REST API integration + - GraphQL API for classifications + - Asset, attribute, and relation extraction + - Tag and classification support + +**InformaticaClient** + - REST API integration + - Asset metadata extraction + - Column schema discovery + - System attribute handling + +ODCS Generator +~~~~~~~~~~~~~~ + +The generator transforms catalog metadata into ODCS format: + +1. **Metadata Extraction**: Fetch asset details from catalog +2. **Column Discovery**: Identify and extract column information +3. **Type Mapping**: Convert catalog types to ODCS types +4. **Classification Mapping**: Extract data classifications +5. **YAML Generation**: Create compliant ODCS YAML file + +Data Type Mapping +----------------- + +Logical Type Mapping +~~~~~~~~~~~~~~~~~~~~ + +Catalog types are mapped to ODCS logical types: + +.. list-table:: + :header-rows: 1 + :widths: 30 30 40 + + * - Catalog Type + - ODCS Logical Type + - Description + * - text, string, varchar + - string + - Text data + * - whole number, int, integer + - integer + - Whole numbers + * - decimal number, float, double + - number + - Decimal numbers + * - date time, timestamp + - timestamp + - Date and time + * - true/false, boolean + - boolean + - Boolean values + * - geographical, geo + - string + - Geographic data + +Physical Type Mapping +~~~~~~~~~~~~~~~~~~~~~ + +Physical types preserve database-specific information: + +- ``VARCHAR(255)`` - Variable character with length +- ``DECIMAL(10,2)`` - Decimal with precision and scale +- ``NUMBER(18,4)`` - Numeric with precision and scale +- ``TIMESTAMP(6)`` - Timestamp with precision + +Classification Support +---------------------- + +The generator extracts and maps data classifications: + +**Collibra Classifications** + - Extracted via GraphQL API + - Mapped to ODCS classification field + - Supports custom classification schemes + +**Informatica Classifications** + - Extracted from asset attributes + - Mapped to ODCS tags and classifications + - Supports data sensitivity labels + +Common Classifications +~~~~~~~~~~~~~~~~~~~~~~ + +- **PII** - Personally Identifiable Information +- **PHI** - Protected Health Information +- **Confidential** - Confidential business data +- **Public** - Publicly available data +- **Internal** - Internal use only + +ODCS Structure +-------------- + +Generated ODCS files follow this structure: + +Contract Metadata +~~~~~~~~~~~~~~~~~ + +.. code-block:: yaml + + id: unique-contract-id + kind: DataContract + apiVersion: v3.1.0 + domain: domain-name + dataProduct: product-name + version: 1.0.0 + name: contract-name + status: active + contractCreatedTs: 2026-04-16T06:00:00Z + +Description Section +~~~~~~~~~~~~~~~~~~~ + +.. code-block:: yaml + + description: + purpose: Purpose of the data + authoritativeDefinitions: + - type: collibra-asset + url: https://collibra.com/asset/123 + limitations: Usage limitations + usage: Intended usage + +Schema Section +~~~~~~~~~~~~~~ + +.. code-block:: yaml + + schema: + - id: table-id + name: table_name + physicalName: PHYSICAL_TABLE_NAME + physicalType: table + description: Table description + tags: + - customer-data + - analytics + columns: + - id: column-id + name: column_name + logicalType: string + physicalType: VARCHAR(255) + description: Column description + isNullable: false + isPrimaryKey: false + classification: PII + tags: + - sensitive + +Quality Section +~~~~~~~~~~~~~~~ + +.. code-block:: yaml + + quality: + - id: rule-001 + name: completeness-check + type: completeness + column: customer_id + dimension: completeness + threshold: 0.95 + +Service Level Agreement +~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: yaml + + sla: + interval: daily + uptime: 99.9% + responseTime: 100ms + +Best Practices +-------------- + +**Validate Catalog Connectivity** + +.. code-block:: python + + try: + client = CollibraClient(base_url, username, password) + # Test connection + asset = client.get_asset("test-id") + except Exception as e: + print(f"Connection failed: {e}") + +**Handle Missing Metadata** + +.. code-block:: python + + # Provide defaults for missing fields + odcs_data = generator.generate_odcs( + asset_id, + defaults={ + 'dataProduct': 'Default Product', + 'version': '1.0.0', + 'status': 'draft' + } + ) + +**Batch Processing** + +.. code-block:: python + + asset_ids = ['id1', 'id2', 'id3'] + + for asset_id in asset_ids: + try: + odcs_data = generator.generate_odcs(asset_id) + generator.save_to_yaml(odcs_data, f"{asset_id}-odcs.yaml") + except Exception as e: + print(f"Failed for {asset_id}: {e}") + +**Customize Output** + +.. code-block:: python + + # Generate ODCS + odcs_data = generator.generate_odcs(asset_id) + + # Customize before saving + odcs_data['dataProduct'] = 'My Data Product' + odcs_data['version'] = '2.0.0' + odcs_data['quality'] = [ + { + 'id': 'custom-rule', + 'name': 'Custom Quality Rule', + 'type': 'accuracy' + } + ] + + # Save customized ODCS + generator.save_to_yaml(odcs_data, 'custom-odcs.yaml') + +Error Handling +-------------- + +Common errors and solutions: + +**Authentication Errors** + +.. code-block:: python + + from requests.exceptions import HTTPError + + try: + client = CollibraClient(url, username, password) + except HTTPError as e: + if e.response.status_code == 401: + print("Invalid credentials") + elif e.response.status_code == 403: + print("Insufficient permissions") + +**Asset Not Found** + +.. code-block:: python + + try: + odcs_data = generator.generate_odcs(asset_id) + except ValueError as e: + print(f"Asset not found: {e}") + +**Missing Columns** + +.. code-block:: python + + odcs_data = generator.generate_odcs(asset_id) + + if not odcs_data.get('schema', [{}])[0].get('columns'): + print("Warning: No columns found for asset") + +Requirements +------------ + +- Python 3.8 or higher +- requests >= 2.32.4 +- pyyaml >= 5.4.0 +- urllib3 >= 2.6.3 +- python-dateutil >= 2.5.3 + +See Also +-------- + +- :ref:`odcs_generator_collibra` - Collibra integration +- :ref:`odcs_generator_informatica` - Informatica integration +- :ref:`odcs_generator_examples` - Code examples +- :ref:`api_odcs_generator` - API reference + +.. Made with Bob diff --git a/docs/chapters/07_data_product_recommender/examples.rst b/docs/chapters/07_data_product_recommender/examples.rst new file mode 100644 index 0000000..ae39ed9 --- /dev/null +++ b/docs/chapters/07_data_product_recommender/examples.rst @@ -0,0 +1,100 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _data_product_recommender_examples: + +Examples +======== + +Basic Example +------------- + +.. code-block:: python + + from wxdi.data_product_recommender.platforms import SnowflakeQueryParser + from wxdi.data_product_recommender.recommender import DataProductRecommender + + # Initialize with Snowflake parser + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + # Load and analyze query logs + recommender.load_query_logs_from_csv_file('query_logs.csv') + recommender.calculate_metrics() + + # Get top 20 recommendations + recommendations = recommender.recommend_data_products(num_recommendations=20) + + # Export to Markdown + recommender.export_recommendations_markdown(recommendations, 'output/recommendations.md') + + # Export to JSON + recommender.export_recommendations_json(recommendations, 'output/recommendations.json') + +Multi-Platform Example +---------------------- + +.. code-block:: python + + from wxdi.data_product_recommender.platforms import ( + SnowflakeQueryParser, + DatabricksQueryParser, + BigQueryQueryParser + ) + from wxdi.data_product_recommender.recommender import DataProductRecommender + + # Snowflake + snowflake_parser = SnowflakeQueryParser() + snowflake_recommender = DataProductRecommender(snowflake_parser) + snowflake_recommender.load_query_logs_from_csv_file('snowflake_logs.csv') + + # Databricks + databricks_parser = DatabricksQueryParser() + databricks_recommender = DataProductRecommender(databricks_parser) + databricks_recommender.load_query_logs_from_csv_file('databricks_logs.csv') + + # BigQuery + bigquery_parser = BigQueryQueryParser() + bigquery_recommender = DataProductRecommender(bigquery_parser) + bigquery_recommender.load_query_logs_from_csv_file('bigquery_logs.csv') + +Custom Scoring Weights +---------------------- + +.. code-block:: python + + from wxdi.data_product_recommender.platforms import SnowflakeQueryParser + from wxdi.data_product_recommender.recommender import DataProductRecommender + + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + # Customize scoring weights + recommender.weights = { + 'query_count': 0.5, # Emphasize query volume + 'user_diversity': 0.3, # Moderate user diversity + 'recency': 0.1, # Less emphasis on recency + 'consistency': 0.1 # Less emphasis on consistency + } + + recommender.load_query_logs_from_csv_file('query_logs.csv') + recommender.calculate_metrics() + recommendations = recommender.recommend_data_products(num_recommendations=20) + +See Also +-------- + +- :ref:`data_product_recommender_usage` - Usage guide +- :ref:`api_data_product_recommender` - API reference diff --git a/docs/chapters/07_data_product_recommender/index.rst b/docs/chapters/07_data_product_recommender/index.rst new file mode 100644 index 0000000..44eb21f --- /dev/null +++ b/docs/chapters/07_data_product_recommender/index.rst @@ -0,0 +1,111 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _data_product_recommender: + +Data Product Recommender +========================= + +Analyze database query logs to identify high-value tables and logical groupings for data product prioritization. + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + overview + usage_guide + examples + +Overview +-------- + +The ``data_product_recommender`` module analyzes query log files to identify which tables should be prioritized as data products in a data marketplace. + +Key Features +------------ + +**Multi-Platform Support** + Supports Snowflake, Databricks, BigQuery, and watsonx.data query log formats. + +**File-Based Input** + Works with CSV and JSON query log files (no direct database connection required). + +**Intelligent Scoring** + Combines query frequency, user diversity, recency, and consistency metrics. + +**Table Grouping** + Identifies tables frequently used together for logical data product groupings. + +**Multiple Output Formats** + Generates both Markdown (human-readable) and JSON (agent-consumable) reports. + +**CLI and Python API** + Use from command line or integrate into applications. + +Quick Start +----------- + +Command Line +~~~~~~~~~~~~ + +.. code-block:: bash + + python -m wxdi.data_product_recommender.cli \ + --platform snowflake \ + --input-file query_logs.csv \ + --output output \ + --num-recommendations 20 + +Python API +~~~~~~~~~~ + +.. code-block:: python + + from wxdi.data_product_recommender.platforms import SnowflakeQueryParser + from wxdi.data_product_recommender.recommender import DataProductRecommender + + # Initialize + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + # Load and analyze + recommender.load_query_logs_from_csv_file('query_logs.csv') + recommender.calculate_metrics() + recommendations = recommender.recommend_data_products(num_recommendations=20) + + # Export + recommender.export_recommendations_markdown(recommendations, 'output/recommendations.md') + +Use Cases +--------- + +**Accelerate Data Product Onboarding** + Leverage existing usage patterns rather than starting from scratch. + +**Identify High-Value Assets** + Find tables with demonstrated business value through real usage. + +**Discover Logical Groupings** + Identify tables commonly used together for cohesive data products. + +**Prioritize Catalog Promotion** + Focus efforts on tables with highest user demand and diversity. + +Next Steps +---------- + +- :ref:`data_product_recommender_usage` - Detailed usage guide +- :ref:`data_product_recommender_examples` - Code examples +- :ref:`api_data_product_recommender` - API reference diff --git a/docs/chapters/07_data_product_recommender/overview.rst b/docs/chapters/07_data_product_recommender/overview.rst new file mode 100644 index 0000000..7b46a50 --- /dev/null +++ b/docs/chapters/07_data_product_recommender/overview.rst @@ -0,0 +1,65 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _data_product_recommender_overview: + +Overview +======== + +The Data Product Recommender analyzes query logs to identify high-value tables for data product creation. + +Scoring Methodology +------------------- + +Individual Table Scoring (0-100) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- **37.5%** Query Count - Volume of usage +- **37.5%** User Diversity - Breadth of usage across teams +- **15%** Recency - Recent activity +- **10%** Consistency - Regular usage patterns + +Table Group Scoring (0-100) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- **30%** Cohesion - How tightly tables are connected +- **20%** Usage - Relative usage compared to other groups +- **15%** User Reach - Percentage of users querying the group +- **20%** Recency - Recent activity across tables +- **10%** Consistency - Regular usage patterns +- **5%** Size - Number of tables in the group + +Star Rating Scale +~~~~~~~~~~~~~~~~~ + +- ⭐⭐⭐⭐⭐ **Excellent (80-100)**: Implement immediately +- ⭐⭐⭐⭐ **Good (60-79)**: Medium priority +- ⭐⭐⭐ **Fair (40-59)**: Consider splitting or implement later +- ⭐⭐ **Weak (20-39)**: Reconsider grouping +- ⭐ **Poor (0-19)**: Do not implement + +Platform Support +---------------- + +- ✅ **Snowflake** - Export from SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY +- ✅ **Databricks** - Export from system.query.history +- ✅ **BigQuery** - Export from INFORMATION_SCHEMA.JOBS_BY_PROJECT +- ✅ **watsonx.data** - Export from system.runtime.queries + +See Also +-------- + +- :ref:`data_product_recommender_usage` - Usage guide +- :ref:`data_product_recommender_examples` - Examples diff --git a/docs/chapters/07_data_product_recommender/usage_guide.rst b/docs/chapters/07_data_product_recommender/usage_guide.rst new file mode 100644 index 0000000..e638e48 --- /dev/null +++ b/docs/chapters/07_data_product_recommender/usage_guide.rst @@ -0,0 +1,82 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _data_product_recommender_usage: + +Usage Guide +=========== + +Installation +------------ + +.. code-block:: bash + + pip install -e . + +CLI Usage +--------- + +.. code-block:: bash + + python -m wxdi.data_product_recommender.cli \ + --platform snowflake \ + --input-file query_logs.csv \ + --output output \ + --num-recommendations 20 \ + --min-score 60.0 + +Options +~~~~~~~ + +- ``--platform`` - Database platform (snowflake, databricks, bigquery, watsonxdata) +- ``--input-file`` - Path to CSV or JSON query log file +- ``--output`` - Output directory (default: output) +- ``--output-format`` - Output format: markdown or json (default: markdown) +- ``--num-recommendations`` - Number of recommendations (default: 20) +- ``--min-score`` - Minimum score threshold 0-100 + +Python API +---------- + +.. code-block:: python + + from wxdi.data_product_recommender.platforms import SnowflakeQueryParser + from wxdi.data_product_recommender.recommender import DataProductRecommender + + # Initialize + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + # Load query logs + recommender.load_query_logs_from_csv_file('query_logs.csv') + + # Calculate metrics + recommender.calculate_metrics() + + # Get recommendations + recommendations = recommender.recommend_data_products( + num_recommendations=20, + min_score=60.0 + ) + + # Export results + recommender.export_recommendations_markdown(recommendations, 'output/recommendations.md') + recommender.export_recommendations_json(recommendations, 'output/recommendations.json') + +See Also +-------- + +- :ref:`data_product_recommender_examples` - Complete examples +- :ref:`api_data_product_recommender` - API reference diff --git a/docs/chapters/05_future_modules/index.rst b/docs/chapters/08_future_modules/index.rst similarity index 96% rename from docs/chapters/05_future_modules/index.rst rename to docs/chapters/08_future_modules/index.rst index 9eb46fb..38136f0 100644 --- a/docs/chapters/05_future_modules/index.rst +++ b/docs/chapters/08_future_modules/index.rst @@ -1,108 +1,108 @@ -.. - Copyright 2026 IBM Corporation - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - -.. _future_modules: - -Future Modules -============== - -The ``IBM watsonx.data intelligence SDK`` is designed with a modular architecture that allows different teams to contribute specialized functionality while sharing common components like authentication. - -Architecture for Extensibility -------------------------------- - -The SDK's modular design enables: - -* **Independent Development**: Teams can develop modules independently -* **Shared Infrastructure**: All modules use common authentication and configuration -* **Consistent API**: Modules follow the same design patterns -* **Easy Integration**: New modules integrate seamlessly with existing ones - -Adding New Modules ------------------- - -Teams adding new modules should: - -1. **Use Common Authentication**: Leverage the ``common.auth`` module for authentication -2. **Follow Naming Conventions**: Use clear, descriptive module names -3. **Provide Documentation**: Include comprehensive documentation following this structure -4. **Include Examples**: Provide working code examples -5. **Add Tests**: Include unit and integration tests - -Documentation Structure for New Modules ----------------------------------------- - -When adding a new module, create documentation following this pattern: - -.. code-block:: text - - docs/chapters/0X_module_name/ - ├── index.rst # Module overview - ├── core_concepts.rst # Key concepts - ├── usage.rst # Usage guide - ├── examples.rst # Code examples - └── api_reference.rst # API documentation - -API Reference Structure -~~~~~~~~~~~~~~~~~~~~~~~ - -Add API reference documentation: - -.. code-block:: text - - docs/api/module_name/ - ├── index.rst # API overview - ├── classes.rst # Main classes - └── utilities.rst # Utility functions - -Planned Modules ---------------- - -While specific modules are still being defined, potential areas include: - -* Data profiling and statistics -* Data lineage tracking -* Data catalog integration -* Additional data quality features -* Custom analytics capabilities - -Contact -------- - -If your team is planning to add a module to the SDK: - -* Review the existing module structure (``dq_validator``) -* Follow the authentication patterns in ``common.auth`` -* Coordinate with the SDK maintainers -* Submit documentation along with your module - -For questions or to propose a new module: - -* Email: Data_Intelligence_SDK@wwpdl.vnet.ibm.com -* GitHub: Open an issue or discussion - -Contributing ------------- - -See the CONTRIBUTING.md file in the repository for detailed guidelines on: - -* Code style and standards -* Testing requirements -* Documentation requirements -* Pull request process - -We look forward to growing the SDK with contributions from teams across IBM! - -.. Made with Bob +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _future_modules: + +Future Modules +============== + +The ``IBM watsonx.data intelligence SDK`` is designed with a modular architecture that allows different teams to contribute specialized functionality while sharing common components like authentication. + +Architecture for Extensibility +------------------------------- + +The SDK's modular design enables: + +* **Independent Development**: Teams can develop modules independently +* **Shared Infrastructure**: All modules use common authentication and configuration +* **Consistent API**: Modules follow the same design patterns +* **Easy Integration**: New modules integrate seamlessly with existing ones + +Adding New Modules +------------------ + +Teams adding new modules should: + +1. **Use Common Authentication**: Leverage the ``common.auth`` module for authentication +2. **Follow Naming Conventions**: Use clear, descriptive module names +3. **Provide Documentation**: Include comprehensive documentation following this structure +4. **Include Examples**: Provide working code examples +5. **Add Tests**: Include unit and integration tests + +Documentation Structure for New Modules +---------------------------------------- + +When adding a new module, create documentation following this pattern: + +.. code-block:: text + + docs/chapters/0X_module_name/ + ├── index.rst # Module overview + ├── core_concepts.rst # Key concepts + ├── usage.rst # Usage guide + ├── examples.rst # Code examples + └── api_reference.rst # API documentation + +API Reference Structure +~~~~~~~~~~~~~~~~~~~~~~~ + +Add API reference documentation: + +.. code-block:: text + + docs/api/module_name/ + ├── index.rst # API overview + ├── classes.rst # Main classes + └── utilities.rst # Utility functions + +Planned Modules +--------------- + +While specific modules are still being defined, potential areas include: + +* Data profiling and statistics +* Data lineage tracking +* Data catalog integration +* Additional data quality features +* Custom analytics capabilities + +Contact +------- + +If your team is planning to add a module to the SDK: + +* Review the existing module structure (``dq_validator``) +* Follow the authentication patterns in ``common.auth`` +* Coordinate with the SDK maintainers +* Submit documentation along with your module + +For questions or to propose a new module: + +* Email: Data_Intelligence_SDK@wwpdl.vnet.ibm.com +* GitHub: Open an issue or discussion + +Contributing +------------ + +See the CONTRIBUTING.md file in the repository for detailed guidelines on: + +* Code style and standards +* Testing requirements +* Documentation requirements +* Pull request process + +We look forward to growing the SDK with contributions from teams across IBM! + +.. Made with Bob diff --git a/docs/index.rst b/docs/index.rst index 151d37b..428f467 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -16,12 +16,15 @@ IBM watsonx.data intelligence SDK for Python ============================================ -The ``IBM watsonx.data intelligence SDK`` for Python is a comprehensive toolkit for data intelligence operations, providing modular components for data quality validation, authentication, and more. +The ``IBM watsonx.data intelligence SDK`` for Python is a comprehensive toolkit for data intelligence operations, providing modular components for data quality validation, data product management, ODCS generation, and intelligent recommendations. This SDK is designed with a modular architecture, allowing different teams to contribute specialized functionality while sharing common components like authentication. Currently, the SDK includes: * **Common Modules**: Shared authentication and configuration for all SDK modules * **DQ Validator**: In-memory data quality validation for streaming data, Pandas DataFrames, and PySpark DataFrames +* **DPH Services**: Python client for IBM Data Product Hub API +* **ODCS Generator**: Generate Open Data Contract Standard files from data catalogs +* **Data Product Recommender**: Analyze query logs to identify high-value data products The ``IBM watsonx.data intelligence SDK`` is supported on Python 3.8+. @@ -31,6 +34,15 @@ Key Features **Data Quality Validation** Comprehensive validation framework with 9 check types, support for array-based records and DataFrames, and integration with IBM Cloud Pak for Data. +**Data Product Hub Integration** + Complete Python SDK for managing data products, drafts, releases, contract terms, and domains. + +**ODCS Generation** + Automated generation of ODCS v3.1.0 compliant YAML files from Collibra and Informatica catalogs. + +**Intelligent Recommendations** + Query log analysis to identify high-value tables and logical groupings for data product prioritization. + **Multi-Environment Authentication** Unified authentication supporting IBM Cloud, AWS Cloud, Government Cloud, and on-premises deployments. @@ -48,7 +60,10 @@ Key Features chapters/02_overview/index chapters/03_common_modules/index chapters/04_dq_validator/index - chapters/05_future_modules/index + chapters/05_dph_services/index + chapters/06_odcs_generator/index + chapters/07_data_product_recommender/index + chapters/08_future_modules/index api/index .. Made with Bob From eb971151b39c40362550f9171a94caa638aaeac7 Mon Sep 17 00:00:00 2001 From: Greeshma Rajendran Date: Fri, 8 May 2026 18:50:49 +0530 Subject: [PATCH 07/33] docs(api): Fixed documentation warnings Signed-off-by: Greeshma Rajendran --- docs/api/dph_services/core.rst | 23 ---------- docs/api/dph_services/index.rst | 73 ++----------------------------- docs/api/odcs_generator/index.rst | 5 --- src/wxdi/dph_services/dph_v1.py | 4 +- 4 files changed, 5 insertions(+), 100 deletions(-) diff --git a/docs/api/dph_services/core.rst b/docs/api/dph_services/core.rst index 2009b39..aa92c32 100644 --- a/docs/api/dph_services/core.rst +++ b/docs/api/dph_services/core.rst @@ -31,27 +31,4 @@ DphV1 Service :show-inheritance: :inherited-members: -Common Models -------------- - -.. autoclass:: wxdi.dph_services.common.DataProduct - :members: - :undoc-members: - :show-inheritance: - -.. autoclass:: wxdi.dph_services.common.DataProductDraft - :members: - :undoc-members: - :show-inheritance: - -.. autoclass:: wxdi.dph_services.common.ContractTerms - :members: - :undoc-members: - :show-inheritance: - -.. autoclass:: wxdi.dph_services.common.Domain - :members: - :undoc-members: - :show-inheritance: - .. Made with Bob \ No newline at end of file diff --git a/docs/api/dph_services/index.rst b/docs/api/dph_services/index.rst index d9adaa6..c42ff65 100644 --- a/docs/api/dph_services/index.rst +++ b/docs/api/dph_services/index.rst @@ -28,6 +28,9 @@ API reference for the Data Product Hub Services module. Main Service Class ------------------ +The DphV1 class provides access to all Data Product Hub Services operations. +For detailed API reference including all methods, see :ref:`api_dph_services_core`. + .. currentmodule:: wxdi.dph_services .. autoclass:: DphV1 @@ -35,74 +38,6 @@ Main Service Class :undoc-members: :show-inheritance: :special-members: __init__ - -Container Operations --------------------- - -.. automethod:: DphV1.initialize -.. automethod:: DphV1.get_initialize_status -.. automethod:: DphV1.get_service_id_credentials -.. automethod:: DphV1.manage_api_keys - -Data Product Operations ------------------------ - -.. automethod:: DphV1.create_data_product -.. automethod:: DphV1.list_data_products -.. automethod:: DphV1.list_data_products_with_pager -.. automethod:: DphV1.get_data_product -.. automethod:: DphV1.update_data_product -.. automethod:: DphV1.delete_data_product - -Draft Operations ----------------- - -.. automethod:: DphV1.create_data_product_draft -.. automethod:: DphV1.list_data_product_drafts -.. automethod:: DphV1.get_data_product_draft -.. automethod:: DphV1.update_data_product_draft -.. automethod:: DphV1.delete_data_product_draft -.. automethod:: DphV1.publish_data_product_draft - -Release Operations ------------------- - -.. automethod:: DphV1.list_data_product_releases -.. automethod:: DphV1.get_data_product_release -.. automethod:: DphV1.update_data_product_release -.. automethod:: DphV1.retire_data_product_release - -Contract Terms Operations -------------------------- - -.. automethod:: DphV1.create_draft_contract_terms_document -.. automethod:: DphV1.get_data_product_draft_contract_terms -.. automethod:: DphV1.update_draft_contract_terms_document -.. automethod:: DphV1.delete_draft_contract_terms_document - -Domain Operations ------------------ - -.. automethod:: DphV1.list_data_product_domains -.. automethod:: DphV1.create_data_product_domain -.. automethod:: DphV1.create_data_product_subdomain -.. automethod:: DphV1.get_domain -.. automethod:: DphV1.update_data_product_domain -.. automethod:: DphV1.delete_domain - -Asset Visualization Operations -------------------------------- - -.. automethod:: DphV1.create_data_asset_visualization -.. automethod:: DphV1.reinitiate_data_asset_visualization - -Contract Template Operations ----------------------------- - -.. automethod:: DphV1.create_contract_template -.. automethod:: DphV1.list_data_product_contract_template -.. automethod:: DphV1.get_contract_template -.. automethod:: DphV1.update_data_product_contract_template -.. automethod:: DphV1.delete_data_product_contract_template + :no-index: .. Made with Bob \ No newline at end of file diff --git a/docs/api/odcs_generator/index.rst b/docs/api/odcs_generator/index.rst index d7e3203..2b45be6 100644 --- a/docs/api/odcs_generator/index.rst +++ b/docs/api/odcs_generator/index.rst @@ -45,9 +45,4 @@ Informatica Integration :undoc-members: :show-inheritance: -.. autoclass:: ODCSGenerator - :members: - :undoc-members: - :show-inheritance: - .. Made with Bob diff --git a/src/wxdi/dph_services/dph_v1.py b/src/wxdi/dph_services/dph_v1.py index a05cec9..d7908ff 100644 --- a/src/wxdi/dph_services/dph_v1.py +++ b/src/wxdi/dph_services/dph_v1.py @@ -1392,9 +1392,7 @@ def replace_data_product_draft_contract_terms( contract. :param List[ContractTemplateSLA] sla: (optional) Service Level Agreement details. - :param List[ContractTemplateSupportAndCommunication] - support_and_communication: (optional) Support and communication details for - the contract. + :param List[ContractTemplateSupportAndCommunication] support_and_communication: (optional) Support and communication details for the contract. :param List[ContractTemplateCustomProperty] custom_properties: (optional) Custom properties that are not part of the standard contract. :param ContractTest contract_test: (optional) Contains the contract test From 3b2da9ff64f5f4b09b5f9ea6cb64008eef62435b Mon Sep 17 00:00:00 2001 From: Greeshma Rajendran Date: Fri, 8 May 2026 18:59:53 +0530 Subject: [PATCH 08/33] chore: updated secrets Signed-off-by: Greeshma Rajendran --- .secrets.baseline | 62 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 61 insertions(+), 1 deletion(-) diff --git a/.secrets.baseline b/.secrets.baseline index e1737e6..9098ff2 100644 --- a/.secrets.baseline +++ b/.secrets.baseline @@ -3,7 +3,7 @@ "files": "^.secrets.baseline$", "lines": null }, - "generated_at": "2026-04-09T04:39:13Z", + "generated_at": "2026-05-08T13:27:00Z", "plugins_used": [ { "name": "AWSKeyDetector" @@ -301,6 +301,66 @@ "verified_result": null } ], + "docs/chapters/05_dph_services/overview.rst": [ + { + "hashed_secret": "45d676e7c6ab44cf4b8fa366ef2d8fccd3e6d6e6", + "is_secret": false, + "is_verified": false, + "line_number": 123, + "type": "Secret Keyword", + "verified_result": null + } + ], + "docs/chapters/05_dph_services/usage_guide.rst": [ + { + "hashed_secret": "45d676e7c6ab44cf4b8fa366ef2d8fccd3e6d6e6", + "is_secret": false, + "is_verified": false, + "line_number": 63, + "type": "Secret Keyword", + "verified_result": null + } + ], + "docs/chapters/06_odcs_generator/collibra_integration.rst": [ + { + "hashed_secret": "ddc861617551c5e789c290865300f615e6f51cc7", + "is_secret": false, + "is_verified": false, + "line_number": 97, + "type": "Secret Keyword", + "verified_result": null + } + ], + "docs/chapters/06_odcs_generator/examples.rst": [ + { + "hashed_secret": "ddc861617551c5e789c290865300f615e6f51cc7", + "is_secret": false, + "is_verified": false, + "line_number": 61, + "type": "Secret Keyword", + "verified_result": null + } + ], + "docs/chapters/06_odcs_generator/index.rst": [ + { + "hashed_secret": "ddc861617551c5e789c290865300f615e6f51cc7", + "is_secret": false, + "is_verified": false, + "line_number": 93, + "type": "Secret Keyword", + "verified_result": null + } + ], + "docs/chapters/06_odcs_generator/informatica_integration.rst": [ + { + "hashed_secret": "0eb9a6a3306220b901c7b4920cd9896899f219be", + "is_secret": false, + "is_verified": false, + "line_number": 77, + "type": "Secret Keyword", + "verified_result": null + } + ], "examples/auth_provider_usage.py": [ { "hashed_secret": "df5cc5832dc34a455c18662ac84587ea19cf2435", From f6995e0245f4b958d8c9086cf5a4a6dc387057e5 Mon Sep 17 00:00:00 2001 From: Greeshma Rajendran Date: Mon, 11 May 2026 11:26:33 +0530 Subject: [PATCH 09/33] chore: updated secrets Signed-off-by: Greeshma Rajendran --- .secrets.baseline | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.secrets.baseline b/.secrets.baseline index 9098ff2..182558b 100644 --- a/.secrets.baseline +++ b/.secrets.baseline @@ -3,7 +3,7 @@ "files": "^.secrets.baseline$", "lines": null }, - "generated_at": "2026-05-08T13:27:00Z", + "generated_at": "2026-05-08T13:46:12Z", "plugins_used": [ { "name": "AWSKeyDetector" From fb9f088d3423703dd880c4d874da195c67928756 Mon Sep 17 00:00:00 2001 From: Greeshma Rajendran Date: Mon, 11 May 2026 14:37:13 +0530 Subject: [PATCH 10/33] chore: updated secrets Signed-off-by: Greeshma Rajendran --- .secrets.baseline | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/.secrets.baseline b/.secrets.baseline index 182558b..6a81058 100644 --- a/.secrets.baseline +++ b/.secrets.baseline @@ -3,7 +3,7 @@ "files": "^.secrets.baseline$", "lines": null }, - "generated_at": "2026-05-08T13:46:12Z", + "generated_at": "2026-05-11T09:05:52Z", "plugins_used": [ { "name": "AWSKeyDetector" @@ -313,7 +313,7 @@ ], "docs/chapters/05_dph_services/usage_guide.rst": [ { - "hashed_secret": "45d676e7c6ab44cf4b8fa366ef2d8fccd3e6d6e6", + "hashed_secret": "11fa7c37d697f30e6aee828b4426a10f83ab2380", "is_secret": false, "is_verified": false, "line_number": 63, @@ -323,7 +323,7 @@ ], "docs/chapters/06_odcs_generator/collibra_integration.rst": [ { - "hashed_secret": "ddc861617551c5e789c290865300f615e6f51cc7", + "hashed_secret": "564e340cd48437d2dfe876ee154cc99dc4d0d137", "is_secret": false, "is_verified": false, "line_number": 97, @@ -333,7 +333,7 @@ ], "docs/chapters/06_odcs_generator/examples.rst": [ { - "hashed_secret": "ddc861617551c5e789c290865300f615e6f51cc7", + "hashed_secret": "564e340cd48437d2dfe876ee154cc99dc4d0d137", "is_secret": false, "is_verified": false, "line_number": 61, @@ -343,7 +343,7 @@ ], "docs/chapters/06_odcs_generator/index.rst": [ { - "hashed_secret": "ddc861617551c5e789c290865300f615e6f51cc7", + "hashed_secret": "564e340cd48437d2dfe876ee154cc99dc4d0d137", "is_secret": false, "is_verified": false, "line_number": 93, @@ -353,7 +353,7 @@ ], "docs/chapters/06_odcs_generator/informatica_integration.rst": [ { - "hashed_secret": "0eb9a6a3306220b901c7b4920cd9896899f219be", + "hashed_secret": "564e340cd48437d2dfe876ee154cc99dc4d0d137", "is_secret": false, "is_verified": false, "line_number": 77, From a48601fd8b89c93412809f2fa269f8c006db57a1 Mon Sep 17 00:00:00 2001 From: Greeshma Rajendran Date: Mon, 11 May 2026 16:10:57 +0530 Subject: [PATCH 11/33] removing changes Signed-off-by: Greeshma Rajendran --- .bumpversion.toml | 3 +- .github/workflows/docs.yml | 2 +- CHANGELOG.md | 56 -------------------------------------- 3 files changed, 2 insertions(+), 59 deletions(-) diff --git a/.bumpversion.toml b/.bumpversion.toml index e50d6af..86b135e 100644 --- a/.bumpversion.toml +++ b/.bumpversion.toml @@ -12,10 +12,9 @@ # limitations under the License. [tool.bumpversion] -current_version = "2.0.0" +current_version = "2.0.0-rc.1" commit = true message = "Update version {current_version} -> {new_version}" -ignore_missing_version = true parse = """(?x) (?P0|[1-9]\\d*)\\. (?P0|[1-9]\\d*)\\. diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 6cf2651..a22c199 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -65,7 +65,7 @@ jobs: deploy: # Only deploy on push to main/master, not on PRs - if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main' + if: github.event_name == 'push' && github.ref == 'refs/heads/main' environment: name: github-pages diff --git a/CHANGELOG.md b/CHANGELOG.md index 2aff2e4..a42bb2b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,59 +1,3 @@ -# [2.0.0](https://github.com/IBM/data-intelligence-sdk/compare/v1.0.0...v2.0.0) (2026-04-23) - - -* Upgrade to version 2.0.0 ([#13](https://github.com/IBM/data-intelligence-sdk/issues/13)) ([41f25ce](https://github.com/IBM/data-intelligence-sdk/commit/41f25cefc349a9d2add7d99bc94945d123104ca1)), closes [#7](https://github.com/IBM/data-intelligence-sdk/issues/7) - - -### BREAKING CHANGES - -* Bug fixes for Data Quality and addition of Data Product Hub modules - -Signed-off-by: Koichi Nishitani - -* fix relative path of Actions scripts - -Signed-off-by: Koichi Nishitani - -* fix script condition - -Signed-off-by: Koichi Nishitani - -* fix script condition again - -Signed-off-by: Koichi Nishitani - -* add missing Makefile and fix documentation - -Signed-off-by: Koichi Nishitani - -* add dq tests and upgrade python - -Signed-off-by: Koichi Nishitani - -* try to extend build timeout - -Signed-off-by: Koichi Nishitani - -* upgrade to python 3.10 as 3.9 is EOL - -Signed-off-by: Koichi Nishitani - -* modify constraint_model to use custom StrEnum in python 3.10 - -Signed-off-by: Koichi Nishitani - -* add pylint - -Signed-off-by: Koichi Nishitani - -* skip linting until ready - -Signed-off-by: Koichi Nishitani - -* make sure to build before checking - -Signed-off-by: Koichi Nishitani - # [2.0.0-rc.1](https://github.com/IBM/data-intelligence-sdk/compare/v1.0.0...v2.0.0-rc.1) (2026-04-22) From e232d4af1839f67a986bb725085c2781384571b5 Mon Sep 17 00:00:00 2001 From: Greeshma Rajendran Date: Mon, 11 May 2026 16:18:20 +0530 Subject: [PATCH 12/33] chore : adding DCO Signed-off-by: Greeshma Rajendran --- docs/chapters/05_dph_services/usage_guide.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/chapters/05_dph_services/usage_guide.rst b/docs/chapters/05_dph_services/usage_guide.rst index af9b340..6e4fbc0 100644 --- a/docs/chapters/05_dph_services/usage_guide.rst +++ b/docs/chapters/05_dph_services/usage_guide.rst @@ -611,4 +611,3 @@ See Also - :ref:`api_dph_services` - API reference - :ref:`dph_services_overview` - Architecture overview -.. Made with Bob From b1917e047db0347a3d783bfc9d19be13b03dd6c5 Mon Sep 17 00:00:00 2001 From: semantic-release-bot Date: Tue, 17 Mar 2026 12:56:52 +0000 Subject: [PATCH 13/33] chore(release): 1.0.0 release notes # [1.0.0](https://github.com/IBM/data-intelligence-sdk/compare/v0.5.3...v1.0.0) (2026-03-17) ### Performance Improvements * Empty commit ([5741e42](https://github.com/IBM/data-intelligence-sdk/commit/5741e42b1dad1d61836cfd4e7790f60fd5774ca9)) * fix version string ([#2](https://github.com/IBM/data-intelligence-sdk/issues/2)) ([d672e40](https://github.com/IBM/data-intelligence-sdk/commit/d672e40768419b84a5951af9328da9017c55d6d8)) ### BREAKING CHANGES * Initial release bump version 1.0.0 Signed-off-by: Koichi Nishitani Signed-off-by: Greeshma Rajendran --- CHANGELOG.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..aa2844f --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,14 @@ +# [1.0.0](https://github.com/IBM/data-intelligence-sdk/compare/v0.5.3...v1.0.0) (2026-03-17) + + +### Performance Improvements + +* Empty commit ([5741e42](https://github.com/IBM/data-intelligence-sdk/commit/5741e42b1dad1d61836cfd4e7790f60fd5774ca9)) +* fix version string ([#2](https://github.com/IBM/data-intelligence-sdk/issues/2)) ([d672e40](https://github.com/IBM/data-intelligence-sdk/commit/d672e40768419b84a5951af9328da9017c55d6d8)) + + +### BREAKING CHANGES + +* Initial release bump version 1.0.0 + +Signed-off-by: Koichi Nishitani From d459a008ce9ad6480e52cde6606b327a940ef4d7 Mon Sep 17 00:00:00 2001 From: Koichi Nishitani Date: Wed, 25 Mar 2026 22:35:41 +0900 Subject: [PATCH 14/33] [skip ci] documentation update (#4) * [skip ci] documentation update Update the documentation Signed-off-by: Koichi Nishitani * add back newline Signed-off-by: Koichi Nishitani * fix newline again Signed-off-by: Koichi Nishitani --------- Signed-off-by: Koichi Nishitani Signed-off-by: Greeshma Rajendran --- .bumpversion.toml | 12 ++- README.md | 18 ++-- docs/chapters/01_welcome/installation.rst | 6 +- docs/chapters/01_welcome/prereqs.rst | 2 +- docs/chapters/01_welcome/versioning.rst | 2 +- docs/chapters/02_overview/known_issues.rst | 2 +- docs/chapters/02_overview/release_notes.rst | 99 ++------------------- docs/chapters/05_future_modules/index.rst | 2 +- docs/conf.py | 4 +- examples/checks_usage.py | 21 ++--- requirements.txt | 2 +- setup.py | 6 +- 12 files changed, 48 insertions(+), 128 deletions(-) diff --git a/.bumpversion.toml b/.bumpversion.toml index 9af5f2c..9c72852 100644 --- a/.bumpversion.toml +++ b/.bumpversion.toml @@ -29,4 +29,14 @@ replace = "version='{new_version}'" [[tool.bumpversion.files]] filename = "README.md" search = "{current_version}" -replace = "{new_version}" \ No newline at end of file +replace = "{new_version}" + +[[tool.bumpversion.files]] +filename = "docs/conf.py" +search = "{current_version}" +replace = "{new_version}" + +[[tool.bumpversion.files]] +filename = "docs/chapters/01_welcome/installation.rst" +search = "{current_version}" +replace = "{new_version}" diff --git a/README.md b/README.md index 461a060..5d9a43c 100644 --- a/README.md +++ b/README.md @@ -753,11 +753,12 @@ issues.update_tested_records(issue_id="issue-123", tested_records=1000) # Set ignored status issues.set_issue_ignored(issue_id="issue-123", ignored=True) -# Update multiple metrics at once -issues.update_issue_metrics( +# Update issue metrics +issues.update_issue_values( issue_id="issue-123", - occurrences=767, - tested_records=1000 + occurrences=10, + tested_records=100, + project_id="project-123" ) ``` @@ -820,6 +821,7 @@ checks = ChecksProvider(config) # Create a new check check_id = checks.create_check( + name="Format Check", native_id="asset-id/column-name", check_type="format", dimension_id="dimension-id", @@ -828,9 +830,9 @@ check_id = checks.create_check( # Get existing checks checks_list = checks.get_checks( - native_id="asset-id/column-name", - project_id="project-id", - include_children=True + asset_id="asset-id", + check_type="format", + project_id="project-id" ) ``` @@ -1144,6 +1146,6 @@ For issues, questions, or contributions, please open an issue on GitHub. - pytest >= 7.0.0 - pytest-cov >= 4.0.0 - pytest-mock >= 3.7.0 -- black >= 23.0.0 +- black >= 26.3.1 - mypy >= 1.0.0 - flake8 >= 6.0.0 diff --git a/docs/chapters/01_welcome/installation.rst b/docs/chapters/01_welcome/installation.rst index d9d6fa4..2e55d91 100644 --- a/docs/chapters/01_welcome/installation.rst +++ b/docs/chapters/01_welcome/installation.rst @@ -100,7 +100,7 @@ To verify that the SDK is installed correctly: >>> import wxdi.dq_validator >>> from wxdi.common.auth import AuthProvider >>> print(dq_validator.__version__) - 0.5.0 + 1.0.0 Versioning ---------- @@ -116,7 +116,7 @@ Version numbers follow the format ``MAJOR.MINOR.PATCH``: Current Version ~~~~~~~~~~~~~~~ -The current version of the SDK is **0.5.0**. +The current version of the SDK is **1.0.0**. Checking Your Version ~~~~~~~~~~~~~~~~~~~~~ @@ -133,7 +133,7 @@ Or programmatically: >>> import wxdi.dq_validator >>> print(dq_validator.__version__) - 0.5.0 + 1.0.0 Upgrading --------- diff --git a/docs/chapters/01_welcome/prereqs.rst b/docs/chapters/01_welcome/prereqs.rst index 199c2e6..2379fdd 100644 --- a/docs/chapters/01_welcome/prereqs.rst +++ b/docs/chapters/01_welcome/prereqs.rst @@ -69,7 +69,7 @@ If you plan to contribute to the SDK or run tests: .. code-block:: console $ pip install pytest>=7.0.0 pytest-cov>=4.0.0 pytest-mock>=3.7.0 - $ pip install black>=23.0.0 mypy>=1.0.0 flake8>=6.0.0 + $ pip install black>=26.3.1 mypy>=1.0.0 flake8>=6.0.0 Environment Setup ----------------- diff --git a/docs/chapters/01_welcome/versioning.rst b/docs/chapters/01_welcome/versioning.rst index 81ebae1..12ddb39 100644 --- a/docs/chapters/01_welcome/versioning.rst +++ b/docs/chapters/01_welcome/versioning.rst @@ -102,7 +102,7 @@ Or in Python: >>> import wxdi.dq_validator >>> print(dq_validator.__version__) - 0.5.0 + 1.0.0 Support Policy -------------- diff --git a/docs/chapters/02_overview/known_issues.rst b/docs/chapters/02_overview/known_issues.rst index 63590de..0238282 100644 --- a/docs/chapters/02_overview/known_issues.rst +++ b/docs/chapters/02_overview/known_issues.rst @@ -23,7 +23,7 @@ This page documents known issues, limitations, and workarounds for the ``IBM wat Current Known Issues -------------------- -Version 0.5.0 +Version 1.0.0 ~~~~~~~~~~~~~ **Performance** diff --git a/docs/chapters/02_overview/release_notes.rst b/docs/chapters/02_overview/release_notes.rst index 54f1493..38d5441 100644 --- a/docs/chapters/02_overview/release_notes.rst +++ b/docs/chapters/02_overview/release_notes.rst @@ -20,7 +20,7 @@ Release Notes This page documents the release history and changes for the ``IBM watsonx.data intelligence SDK``. -Version 0.5.0 (Current) +Version 1.0.0 (Current) ----------------------- *Release Date: March 2026* @@ -87,87 +87,12 @@ See :ref:`Known Issues` for current limitations. Previous Versions ----------------- -Version 0.4.0 (Beta) -~~~~~~~~~~~~~~~~~~~~ - -*Release Date: February 2026* - -* Beta release for internal testing -* Core validation engine implementation -* Basic DataFrame integration -* Initial REST API providers - -Version 0.3.0 (Alpha) -~~~~~~~~~~~~~~~~~~~~~ - -*Release Date: January 2026* - -* Alpha release for early adopters -* Prototype validation checks -* Authentication framework - -Version 0.2.0 (Alpha) -~~~~~~~~~~~~~~~~~~~~~ - -*Release Date: December 2025* - -* Initial alpha release -* Basic validation framework -* Proof of concept - -Version 0.1.0 (Pre-Alpha) -~~~~~~~~~~~~~~~~~~~~~~~~~ - -*Release Date: November 2025* - -* Initial development version -* Project structure and architecture +There are no previous versions. Upgrade Guide ------------- -From 0.4.x to 0.5.0 -~~~~~~~~~~~~~~~~~~~ - -**Breaking Changes** - -* Authentication configuration now uses Pydantic models instead of dictionaries -* ValidationResult structure has been enhanced with additional fields -* Some provider method signatures have changed for consistency - -**Migration Steps** - -1. Update authentication configuration: - - .. code-block:: python - - # Old (0.4.x) - config = { - 'base_url': 'https://api.example.com', - 'username': 'user', - 'api_key': 'key' - } - - # New (0.5.0) - from wxdi.common.auth import AuthConfig - config = AuthConfig( - base_url='https://api.example.com', - username='user', - api_key='key' - ) - -2. Update ValidationResult usage: - - .. code-block:: python - - # Old (0.4.x) - score = result.score - - # New (0.5.0) - score = result.validation_score - pass_rate = result.pass_rate # New field - -3. Review provider method signatures in the API documentation +There are no upgrades. Deprecation Notices ------------------- @@ -177,21 +102,7 @@ No features are currently deprecated. Future deprecations will be announced here Future Releases --------------- -Planned for Version 0.6.0 -~~~~~~~~~~~~~~~~~~~~~~~~~ - -* Enhanced error messages with suggestions -* Performance optimizations for large datasets -* Additional validation check types -* Improved documentation with more examples - -Planned for Version 1.0.0 -~~~~~~~~~~~~~~~~~~~~~~~~~ - -* Stable API with backward compatibility guarantees -* Additional modules from partner teams -* Enhanced integration capabilities -* Production-ready performance and reliability +We have plans to extend functionality Stay Updated ------------ @@ -206,6 +117,6 @@ Feedback We welcome your feedback! Please report issues or suggest features through: * GitHub Issues: https://github.com/IBM/data-intelligence-sdk/issues -* Email: data-intelligence-sdk@ibm.com +* Email: Data_Intelligence_SDK@wwpdl.vnet.ibm.com .. Made with Bob diff --git a/docs/chapters/05_future_modules/index.rst b/docs/chapters/05_future_modules/index.rst index afa1a2c..9eb46fb 100644 --- a/docs/chapters/05_future_modules/index.rst +++ b/docs/chapters/05_future_modules/index.rst @@ -90,7 +90,7 @@ If your team is planning to add a module to the SDK: For questions or to propose a new module: -* Email: data-intelligence-sdk@ibm.com +* Email: Data_Intelligence_SDK@wwpdl.vnet.ibm.com * GitHub: Open an issue or discussion Contributing diff --git a/docs/conf.py b/docs/conf.py index 7fb99a5..363a656 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -38,9 +38,9 @@ # the built documents. # # The short X.Y version. -version = "0.5.0" +version = "1.0.0" # The full version, including alpha/beta/rc tags. -release = "0.5.0" +release = "1.0.0" # -- General configuration --------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration diff --git a/examples/checks_usage.py b/examples/checks_usage.py index 4ceb765..3048cc8 100644 --- a/examples/checks_usage.py +++ b/examples/checks_usage.py @@ -21,7 +21,7 @@ create_check(): Create a new data quality check - Requires: name, dimension_id, native_id, and either project_id OR catalog_id - Optional: check_type (defaults to the check name if not provided) - - Returns: The created check_id + - Returns: The check ID (string) get_checks(): Retrieve checks for a specific asset filtered by check type - Requires: asset_id, check_type, and either project_id OR catalog_id @@ -69,7 +69,7 @@ def main(): try: # Note: Update these values with your actual data # When check_type is not provided, it defaults to the check name - new_check_id = check_provider.create_check( + check_id = check_provider.create_check( name="uniqueness_check", dimension_id="371114cd-5516-4691-8b2e-1e66edf66486", # Use appropriate dimension ID native_id="your-asset-id/your-check-id-1", # Format: / @@ -77,9 +77,7 @@ def main(): # check_type not specified - will default to "uniqueness_check" ) print(f"\n✓ Successfully created check") - print(f" New Check ID: {new_check_id}") - print(f" Check Name: uniqueness_check") - print(f" Check Type: uniqueness_check (defaulted to name)") + print(f" New Check ID: {check_id}") except ValueError as e: print(f"\n✗ Error creating check: {e}") @@ -92,7 +90,7 @@ def main(): print("=" * 70) try: - new_check_id = check_provider.create_check( + check_id = check_provider.create_check( name="Example Comparison Check", dimension_id="ec453723-669c-48bb-82c1-11b69b3b8c93", # Validity dimension native_id="your-asset-id/your-check-id-2", @@ -100,8 +98,7 @@ def main(): project_id=project_id ) print(f"\n✓ Successfully created comparison check") - print(f" New Check ID: {new_check_id}") - print(f" Check Type: comparison") + print(f" New Check ID: {check_id}") except ValueError as e: print(f"\n✗ Error creating check: {e}") @@ -114,7 +111,7 @@ def main(): print("=" * 70) try: - new_check_id = check_provider.create_check( + check_id = check_provider.create_check( name="Catalog Check Example", dimension_id="371114cd-5516-4691-8b2e-1e66edf66486", native_id="catalog-asset-id/catalog-check-id", @@ -122,8 +119,7 @@ def main(): catalog_id=catalog_id ) print(f"\n✓ Successfully created check using catalog_id") - print(f" New Check ID: {new_check_id}") - print(f" Check Type: data_rule") + print(f" New Check ID: {check_id}") except ValueError as e: print(f"\n✗ Error creating check: {e}") @@ -275,12 +271,13 @@ def main(): print("Examples Complete!") print("=" * 70) print("\nKey Points:") - print(" • create_check() creates a new data quality check") + print(" • create_check() creates a new data quality check and returns the check ID") print(" • get_checks() retrieves checks for an asset filtered by type") print(" • Either project_id OR catalog_id must be provided (not both)") print(" • check_type defaults to the check name if not specified in create_check()") print(" • native_id format: /") print(" • get_checks() returns a list of check objects") + print(" • Extract check ID from returned dict using check_body.get('id') for each check in the list of retrieved checks") if __name__ == "__main__": diff --git a/requirements.txt b/requirements.txt index b4ab34b..1b9de16 100644 --- a/requirements.txt +++ b/requirements.txt @@ -22,7 +22,7 @@ regex>=2023.0.0 pytest>=7.0.0 pytest-cov>=4.0.0 pytest-mock>=3.7.0 -black>=23.0.0 +# black is defined in setup.py extras_require['dev'] to avoid BOM conflicts mypy>=1.0.0 flake8>=6.0.0 diff --git a/setup.py b/setup.py index 1417e74..03e0d0c 100644 --- a/setup.py +++ b/setup.py @@ -27,7 +27,7 @@ setup( name="data-intelligence-sdk", - version='1.0.0', + version='0.5.3', author="IBM", author_email="Data_Intelligence_SDK@wwpdl.vnet.ibm.com", description="A Python SDK for performing data quality validations on streaming data records and DataFrames", @@ -64,7 +64,7 @@ "pytest>=7.0.0", "pytest-cov>=4.0.0", "pytest-mock>=3.7.0", - "black>=23.0.0", + "black>=26.3.1", "mypy>=1.0.0", "flake8>=6.0.0", ], @@ -84,7 +84,7 @@ "pytest>=7.0.0", "pytest-cov>=4.0.0", "pytest-mock>=3.7.0", - "black>=23.0.0", + "black>=26.3.1", "mypy>=1.0.0", "flake8>=6.0.0", ], From 06c2ded8ca6d2e19bb2cbab951c084a4ad50e98c Mon Sep 17 00:00:00 2001 From: Koichi Nishitani Date: Thu, 26 Mar 2026 08:43:55 +0900 Subject: [PATCH 15/33] [skip ci] Enable publish on workflow dispatch (#5) Enable publish on workflow dispatch Signed-off-by: Koichi Nishitani Signed-off-by: Greeshma Rajendran --- .github/workflows/docs.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index ac6ce66..2032a4e 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -67,7 +67,7 @@ jobs: deploy: # Only deploy on push to main/master, not on PRs - if: github.event_name == 'push' && github.ref == 'refs/heads/main' + if: contains(fromJSON('["push", "workflow_dispatch"]'), github.event_name) && github.ref == 'refs/heads/main' environment: name: github-pages From 8594bd8d18040923474e16b0e18f7ee97cb5e8d8 Mon Sep 17 00:00:00 2001 From: Koichi Nishitani Date: Tue, 21 Apr 2026 13:36:13 +0900 Subject: [PATCH 16/33] Perf: Sync from enterprise 4e91aee (via .ignore) on 2026-04-17 (#7) * Perf: Sync from enterprise 4e91aee (via .ignore) on 2026-04-17 BREAKING CHANGE: Bug fixes for Data Quality and addition of Data Product Hub modules Signed-off-by: Koichi Nishitani * fix relative path of Actions scripts Signed-off-by: Koichi Nishitani * fix script condition Signed-off-by: Koichi Nishitani * fix script condition again Signed-off-by: Koichi Nishitani * add missing Makefile and fix documentation Signed-off-by: Koichi Nishitani * add dq tests and upgrade python Signed-off-by: Koichi Nishitani * try to extend build timeout Signed-off-by: Koichi Nishitani * upgrade to python 3.10 as 3.9 is EOL Signed-off-by: Koichi Nishitani * modify constraint_model to use custom StrEnum in python 3.10 Signed-off-by: Koichi Nishitani * add pylint Signed-off-by: Koichi Nishitani * skip linting until ready Signed-off-by: Koichi Nishitani * make sure to build before checking Signed-off-by: Koichi Nishitani --------- Signed-off-by: Koichi Nishitani Signed-off-by: Greeshma Rajendran --- .github/workflows/build.yaml | 87 +- .github/workflows/deploy-package.yaml | 104 + .github/workflows/docs.yml | 8 +- .github/workflows/verify.yaml | 67 + .pylintrc | 498 + .releaserc | 2 +- .secrets.baseline | 188 +- Makefile.build | 58 + README.md | 91 +- docs/README.md | 2 +- docs/chapters/02_overview/faq.rst | 2 +- docs/chapters/02_overview/known_issues.rst | 2 +- examples/__init__.py | 19 + examples/assets_usage.py | 72 +- examples/auth_provider_usage.py | 90 +- examples/basic_usage.py | 54 +- examples/checks_usage.py | 97 +- examples/consolidation_usage.py | 112 +- examples/data_product_recommender_example.py | 136 + examples/dimensions_usage.py | 56 +- examples/dq_workflow_usage.py | 80 +- examples/end_to_end_example.py | 404 + examples/glossary_usage.py | 86 +- examples/issues_usage.py | 204 +- examples/odcs_generator_example.py | 217 + .../odcs_generator_informatica_example.py | 415 + examples/pandas_dataframe_usage.py | 56 +- examples/spark_dataframe_usage.py | 88 +- examples/test_dph_v1_examples.py | 1600 ++ requirements-dev.txt | 17 + requirements.txt | 12 +- setup.py | 23 +- src/wxdi/__init__.py | 9 +- src/wxdi/data_product_recommender/README.md | 218 + src/wxdi/data_product_recommender/__init__.py | 24 + src/wxdi/data_product_recommender/base.py | 37 + src/wxdi/data_product_recommender/cli.py | 152 + .../data_product_recommender/platforms.py | 167 + .../data_product_recommender/recommender.py | 1292 ++ src/wxdi/dph_services/README.md | 458 + src/wxdi/dph_services/__init__.py | 22 + src/wxdi/dph_services/common.py | 75 + src/wxdi/dph_services/common_constants.py | 48 + src/wxdi/dph_services/dph_v1.py | 12411 ++++++++++++++ src/wxdi/dph_services/version.py | 20 + src/wxdi/dq_validator/issue_reporting.py | 581 +- src/wxdi/dq_validator/provider/checks.py | 117 +- .../dq_validator/provider/constraint_model.py | 8 +- src/wxdi/dq_validator/provider/issues.py | 314 +- .../dq_validator/provider/response_model.py | 5 +- .../README-GENERATE-ODCS-SCRIPT.md | 1095 ++ src/wxdi/odcs_generator/__init__.py | 55 + .../generate_odcs_from_collibra.py | 765 + .../generate_odcs_from_informatica.py | 500 + tests/data/data_asset_response.json | 739 + tests/data/term_draft.json | 54 + tests/data/term_latest_version.json | 31 + tests/data/term_response.json | 117 + tests/src/__init__.py | 19 + tests/src/auth/__init__.py | 18 + tests/src/auth/test_auth_config.py | 275 + tests/src/auth/test_auth_provider.py | 309 + .../src/auth/test_gov_cloud_authenticator.py | 259 + .../src/auth/test_gov_cloud_token_manager.py | 403 + .../src/data_product_recommender/__init__.py | 19 + .../test_data_product_recommender.py | 613 + .../test_data_product_recommender_cli.py | 306 + .../test_data_product_recommender_conftest.py | 144 + .../test_data_product_recommender_parsers.py | 236 + tests/src/dph_services/__init__.py | 19 + tests/src/dph_services/test_common.py | 50 + tests/src/dph_services/test_dph_v1.py | 14205 ++++++++++++++++ tests/src/dq_validator/__init__.py | 20 + .../src/dq_validator/provider/test_assets.py | 445 + tests/src/dq_validator/provider/test_cams.py | 513 + .../src/dq_validator/provider/test_checks.py | 796 + .../src/dq_validator/provider/test_config.py | 284 + .../dq_validator/provider/test_dimensions.py | 227 + .../dq_validator/provider/test_dq_search.py | 279 + .../dq_validator/provider/test_glossary.py | 697 + .../src/dq_validator/provider/test_issues.py | 1129 ++ .../provider/test_thread_safety.py | 129 + tests/src/dq_validator/test_case_check.py | 258 + .../src/dq_validator/test_comparison_check.py | 491 + .../dq_validator/test_completeness_check.py | 131 + .../src/dq_validator/test_data_asset_model.py | 187 + tests/src/dq_validator/test_datatype_check.py | 424 + tests/src/dq_validator/test_format_check.py | 357 + tests/src/dq_validator/test_integration.py | 466 + .../src/dq_validator/test_issue_reporting.py | 1609 ++ tests/src/dq_validator/test_length_check.py | 307 + .../src/dq_validator/test_pandas_validator.py | 535 + tests/src/dq_validator/test_range_check.py | 380 + tests/src/dq_validator/test_regex_check.py | 280 + .../dq_validator/test_result_consolidator.py | 471 + tests/src/dq_validator/test_rule_loader.py | 559 + .../src/dq_validator/test_spark_validator.py | 805 + .../dq_validator/test_valid_values_check.py | 394 + tests/src/dq_validator/test_version.py | 35 + tests/src/integration/README.md | 221 + tests/src/integration/__init__.py | 19 + .../src/integration/initial_setup_service.py | 137 + ...st_data_product_recommender_integration.py | 373 + .../test_odcs_generator_collibra.py | 611 + .../test_odcs_generator_informatica.py | 389 + tests/src/odcs_generator/__init__.py | 19 + .../test_odcs_generator_collibra.py | 739 + .../test_odcs_generator_informatica.py | 1060 ++ 108 files changed, 56119 insertions(+), 793 deletions(-) create mode 100644 .github/workflows/deploy-package.yaml create mode 100644 .github/workflows/verify.yaml create mode 100644 .pylintrc create mode 100644 Makefile.build create mode 100644 examples/__init__.py create mode 100644 examples/data_product_recommender_example.py create mode 100644 examples/end_to_end_example.py create mode 100644 examples/odcs_generator_example.py create mode 100644 examples/odcs_generator_informatica_example.py create mode 100644 examples/test_dph_v1_examples.py create mode 100644 requirements-dev.txt create mode 100644 src/wxdi/data_product_recommender/README.md create mode 100644 src/wxdi/data_product_recommender/__init__.py create mode 100644 src/wxdi/data_product_recommender/base.py create mode 100644 src/wxdi/data_product_recommender/cli.py create mode 100644 src/wxdi/data_product_recommender/platforms.py create mode 100644 src/wxdi/data_product_recommender/recommender.py create mode 100644 src/wxdi/dph_services/README.md create mode 100644 src/wxdi/dph_services/__init__.py create mode 100644 src/wxdi/dph_services/common.py create mode 100644 src/wxdi/dph_services/common_constants.py create mode 100644 src/wxdi/dph_services/dph_v1.py create mode 100644 src/wxdi/dph_services/version.py create mode 100644 src/wxdi/odcs_generator/README-GENERATE-ODCS-SCRIPT.md create mode 100644 src/wxdi/odcs_generator/__init__.py create mode 100644 src/wxdi/odcs_generator/generate_odcs_from_collibra.py create mode 100644 src/wxdi/odcs_generator/generate_odcs_from_informatica.py create mode 100644 tests/data/data_asset_response.json create mode 100644 tests/data/term_draft.json create mode 100644 tests/data/term_latest_version.json create mode 100644 tests/data/term_response.json create mode 100644 tests/src/__init__.py create mode 100644 tests/src/auth/__init__.py create mode 100644 tests/src/auth/test_auth_config.py create mode 100644 tests/src/auth/test_auth_provider.py create mode 100644 tests/src/auth/test_gov_cloud_authenticator.py create mode 100644 tests/src/auth/test_gov_cloud_token_manager.py create mode 100644 tests/src/data_product_recommender/__init__.py create mode 100644 tests/src/data_product_recommender/test_data_product_recommender.py create mode 100644 tests/src/data_product_recommender/test_data_product_recommender_cli.py create mode 100644 tests/src/data_product_recommender/test_data_product_recommender_conftest.py create mode 100644 tests/src/data_product_recommender/test_data_product_recommender_parsers.py create mode 100644 tests/src/dph_services/__init__.py create mode 100644 tests/src/dph_services/test_common.py create mode 100644 tests/src/dph_services/test_dph_v1.py create mode 100644 tests/src/dq_validator/__init__.py create mode 100644 tests/src/dq_validator/provider/test_assets.py create mode 100644 tests/src/dq_validator/provider/test_cams.py create mode 100644 tests/src/dq_validator/provider/test_checks.py create mode 100644 tests/src/dq_validator/provider/test_config.py create mode 100644 tests/src/dq_validator/provider/test_dimensions.py create mode 100644 tests/src/dq_validator/provider/test_dq_search.py create mode 100644 tests/src/dq_validator/provider/test_glossary.py create mode 100644 tests/src/dq_validator/provider/test_issues.py create mode 100644 tests/src/dq_validator/provider/test_thread_safety.py create mode 100644 tests/src/dq_validator/test_case_check.py create mode 100644 tests/src/dq_validator/test_comparison_check.py create mode 100644 tests/src/dq_validator/test_completeness_check.py create mode 100644 tests/src/dq_validator/test_data_asset_model.py create mode 100644 tests/src/dq_validator/test_datatype_check.py create mode 100644 tests/src/dq_validator/test_format_check.py create mode 100644 tests/src/dq_validator/test_integration.py create mode 100644 tests/src/dq_validator/test_issue_reporting.py create mode 100644 tests/src/dq_validator/test_length_check.py create mode 100644 tests/src/dq_validator/test_pandas_validator.py create mode 100644 tests/src/dq_validator/test_range_check.py create mode 100644 tests/src/dq_validator/test_regex_check.py create mode 100644 tests/src/dq_validator/test_result_consolidator.py create mode 100644 tests/src/dq_validator/test_rule_loader.py create mode 100644 tests/src/dq_validator/test_spark_validator.py create mode 100644 tests/src/dq_validator/test_valid_values_check.py create mode 100644 tests/src/dq_validator/test_version.py create mode 100644 tests/src/integration/README.md create mode 100644 tests/src/integration/__init__.py create mode 100644 tests/src/integration/initial_setup_service.py create mode 100644 tests/src/integration/test_data_product_recommender_integration.py create mode 100644 tests/src/integration/test_odcs_generator_collibra.py create mode 100644 tests/src/integration/test_odcs_generator_informatica.py create mode 100644 tests/src/odcs_generator/__init__.py create mode 100644 tests/src/odcs_generator/test_odcs_generator_collibra.py create mode 100644 tests/src/odcs_generator/test_odcs_generator_informatica.py diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 17c7e1c..1096af0 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -13,67 +13,20 @@ name: build on: - push: + pull_request: + types: ['opened', 'synchronize', 'reopened', 'closed'] branches: - main - tags: ["*"] # Required to trigger PyPI job - pull_request: - branches: ["**"] + - sandbox workflow_dispatch: jobs: - detect-secrets: - if: "!contains(github.event.head_commit.message, '[skip ci]')" - name: detect-secrets - runs-on: ubuntu-latest - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Setup Python - uses: actions/setup-python@v5 - with: - python-version: "3.14" - - - name: Install detect-secrets - run: | - pip install --upgrade "git+https://github.com/ibm/detect-secrets.git@master#egg=detect-secrets" - - - name: Run detect-secrets - run: | - detect-secrets scan --update .secrets.baseline - detect-secrets -v audit --report --fail-on-unaudited --fail-on-live --fail-on-audited-real .secrets.baseline - - build: - if: "!contains(github.event.head_commit.message, '[skip ci]')" - name: build (python ${{ matrix.python-version }}) - needs: detect-secrets - runs-on: ubuntu-latest - strategy: - matrix: - python-version: ["3.9", "3.10", "3.11"] - - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v5 - with: - python-version: ${{ matrix.python-version }} - - - name: Install Python dependencies - run: | - python -m pip install --upgrade pip build twine - - - name: Build & Verify - run: | - python -m build - python -m twine check dist/* + build-and-verify: + uses: ./.github/workflows/verify.yaml publish-release: - if: "github.ref_name == 'main' && github.event_name != 'pull_request'" - needs: build + if: contains(fromJSON('["main", "sandbox"]'), github.ref_name) && github.event.pull_request.action == 'closed' && github.event.pull_request.merged + needs: build-and-verify name: semantic-release runs-on: ubuntu-latest steps: @@ -82,6 +35,7 @@ jobs: with: persist-credentials: false fetch-depth: 0 + ref: github.ref - name: Setup Node.js uses: actions/setup-node@v4 @@ -102,28 +56,3 @@ jobs: env: GITHUB_TOKEN: ${{ secrets.GH_TOKEN }} run: npm run semantic-release - - deploy-pypi: - if: startsWith(github.ref, 'refs/tags/') - name: deploy-pypi - runs-on: ubuntu-latest - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Set up Python - uses: actions/setup-python@v5 - with: - python-version: "3.14" - - - name: Install Build Dependencies - run: | - python -m pip install --upgrade pip setuptools wheel build twine - - - name: Build and Publish to PyPI - env: - TWINE_USERNAME: __token__ - TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }} - run: | - python -m build - twine upload dist/* diff --git a/.github/workflows/deploy-package.yaml b/.github/workflows/deploy-package.yaml new file mode 100644 index 0000000..7c0be1a --- /dev/null +++ b/.github/workflows/deploy-package.yaml @@ -0,0 +1,104 @@ +# Copyright 2026 IBM Corporation +# Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +name: Deploy package + +on: + push: + tags: ['**'] + workflow_dispatch: + +jobs: + build: + uses: ./.github/workflows/verify.yaml + + deploy-pypi: + if: startsWith(github.ref, 'refs/tags/') && !contains(github.ref, '-rc.') + name: deploy-pypi + runs-on: ubuntu-latest + needs: build + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.14" + + - name: Install Build Dependencies + run: | + python -m pip install --upgrade pip setuptools wheel build twine + + - name: Build and Publish to PyPI + env: + TWINE_USERNAME: __token__ + TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }} + run: | + python -m build + twine upload dist/* + + deploy-testpypi: + if: startsWith(github.ref, 'refs/tags/') && contains(github.ref, '-rc.') + name: deploy-testpypi + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v4 + with: + ref: 'refs/heads/sandbox' + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.14" + + - name: Install Build Dependencies + run: | + python -m pip install --upgrade pip setuptools wheel build twine + + - name: Build and Publish to PyPI + env: + TWINE_USERNAME: __token__ + TWINE_PASSWORD: ${{ secrets.TESTPYPI_TOKEN }} + run: | + python -m build + twine upload --repository testpypi dist/* + + build-documentation: + if: startsWith(github.ref, 'refs/tags/') && !contains(github.ref, '-rc.') + name: Build and deploy documentation + uses: ./.github/workflows/docs.yml + + update-sandbox: + if: startsWith(github.ref, 'refs/tags/') && !contains(github.ref, '-rc.') + name: Update sandbox branch + needs: build-documentation + runs-on: ubuntu-latest + steps: + - name: Checkout main + uses: actions/checkout@v4 + with: + persist-credentials: false + fetch-depth: 0 + ref: 'refs/heads/main' + + - name: Also get the sandbox branch + run: | + git remote set-branches --add origin sandbox + git fetch + - name: Checkout main + run: git checkout sandbox + - name: Merge main into sandbox + run: git merge --signoff main + - name: Push changes to sandbox + run: git push -v origin diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 2032a4e..4444513 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -13,14 +13,12 @@ name: Build and Deploy Documentation on: - push: - branches: - - main - tags: ['*'] pull_request: branches: - main + - sandbox workflow_dispatch: + workflow_call: permissions: contents: read @@ -67,7 +65,7 @@ jobs: deploy: # Only deploy on push to main/master, not on PRs - if: contains(fromJSON('["push", "workflow_dispatch"]'), github.event_name) && github.ref == 'refs/heads/main' + if: github.event_name == 'push' && github.ref == 'refs/heads/main' environment: name: github-pages diff --git a/.github/workflows/verify.yaml b/.github/workflows/verify.yaml new file mode 100644 index 0000000..8348e6d --- /dev/null +++ b/.github/workflows/verify.yaml @@ -0,0 +1,67 @@ +# Copyright 2026 IBM Corporation +# Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +name: verify + +on: + workflow_call: + +jobs: + detect-secrets: + if: "!contains(github.event.head_commit.message, '[skip ci]')" + name: detect-secrets + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: "3.14" + + - name: Install detect-secrets + run: | + pip install --upgrade "git+https://github.com/ibm/detect-secrets.git@master#egg=detect-secrets" + + - name: Run detect-secrets + run: | + detect-secrets scan --update .secrets.baseline + detect-secrets -v audit --report --fail-on-unaudited --fail-on-live --fail-on-audited-real .secrets.baseline + + build: + if: "!contains(github.event.head_commit.message, '[skip ci]')" + name: build (python ${{ matrix.python-version }}) + needs: detect-secrets + runs-on: ubuntu-latest + strategy: + matrix: + python-version: ["3.10", "3.11", "3.12"] + + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Set up Python ${{ matrix.python-version }} + uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + + - name: Install Python dependencies + run: | + python -m pip install --upgrade pip build twine + + - name: Build & Verify + run: | + make -f Makefile.build ci + python -m build + python -m twine check dist/* diff --git a/.pylintrc b/.pylintrc new file mode 100644 index 0000000..3edec5a --- /dev/null +++ b/.pylintrc @@ -0,0 +1,498 @@ +[MASTER] + +# Use multiple processes to speed up Pylint. Specifying 0 will auto-detect the +# number of processors available to use. +jobs=1 + +# Control the amount of potential inferred values when inferring a single +# object. This can help the performance when dealing with large functions or +# complex, nested conditions. +limit-inference-results=100 + +# List of plugins (as comma separated values of python module names) to load, +# usually to register additional checkers. +#load-plugins= + +# Pickle collected data for later comparisons. +persistent=yes + +# Specify a configuration file. +#rcfile= + +# When enabled, pylint would attempt to guess common misconfiguration and emit +# user-friendly hints instead of false-positive error messages. +suggestion-mode=yes + +# Allow loading of arbitrary C extensions. Extensions are imported into the +# active Python interpreter and may run arbitrary code. +#unsafe-load-any-extension=no + + +[MESSAGES CONTROL] + +# Only show warnings with the listed confidence levels. Leave empty to show +# all. Valid levels: HIGH, INFERENCE, INFERENCE_FAILURE, UNDEFINED. +confidence=HIGH + +# Disable the message, report, category or checker with the given id(s). You +# can either give multiple identifiers separated by comma (,) or put this +# option multiple times (only on the command line, not in the configuration +# file where it should appear only once). You can also use "--disable=all" to +# disable everything first and then reenable specific checks. For example, if +# you want to run only the similarities checker, you can use "--disable=all +# --enable=similarities". If you want to run only the classes checker, but have +# no Warning level messages displayed, use "--disable=all --enable=classes +# --disable=W". +disable=too-many-arguments, + too-many-public-methods, + too-few-public-methods, + too-many-instance-attributes, + too-many-locals, + too-many-branches, + too-many-lines, + line-too-long, + similarities, + import-error, + raw-checker-failed, + bad-inline-option, + locally-disabled, + file-ignored, + suppressed-message, + useless-suppression, + deprecated-pragma, + use-symbolic-message-instead, + invalid-name, + global-statement, + use-implicit-booleaness-not-comparison + +# Enable the message, report, category or checker with the given id(s). You can +# either give multiple identifier separated by comma (,) or put this option +# multiple time (only on the command line, not in the configuration file where +# it should appear only once). See also the "--disable" option for examples. +#enable=c-extension-no-member + + +[REPORTS] + +# Python expression which should return a score less than or equal to 10. You +# have access to the variables 'error', 'warning', 'refactor', and 'convention' +# which contain the number of messages in each category, as well as 'statement' +# which is the total number of statements analyzed. This score is used by the +# global evaluation report (RP0004). +evaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10) + +# Template used to display messages. This is a python new-style format string +# used to format the message information. See doc for all details. +#msg-template= + +# Set the output format. Available formats are text, parseable, colorized, json +# and msvs (visual studio). You can also give a reporter class, e.g. +# mypackage.mymodule.MyReporterClass. +output-format=text + +# Tells whether to display a full report or only the messages. +reports=no + +# Activate the evaluation score. +score=yes + + +[REFACTORING] + +# Maximum number of nested blocks for function / method body +max-nested-blocks=5 + +# Complete name of functions that never returns. When checking for +# inconsistent-return-statements if a never returning function is called then +# it will be considered as an explicit return statement and no message will be +# printed. +never-returning-functions=sys.exit + + +[LOGGING] + +# Format style used to check logging format string. `old` means using % +# formatting, `new` is for `{}` formatting,and `fstr` is for f-strings. +logging-format-style=old + +# Logging modules to check that the string format arguments are in logging +# function parameter format. +logging-modules=logging + + +[FORMAT] + +# Expected format of line ending, e.g. empty (any line ending), LF or CRLF. +expected-line-ending-format= + +# Regexp for a line that is allowed to be longer than the limit. +ignore-long-lines=^\s*(# )??$ + +# Number of spaces of indent required inside a hanging or continued line. +indent-after-paren=4 + +# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1 +# tab). +indent-string=' ' + +# Maximum number of characters on a single line. +max-line-length=120 + +# Maximum number of lines in a module. +max-module-lines=3000 + +# List of optional constructs for which whitespace checking is disabled. `dict- +# separator` is used to allow tabulation in dicts, etc.: {1 : 1,\n222: 2}. +# `trailing-comma` allows a space between comma and closing bracket: (a, ). +# `empty-line` allows space-only lines. +no-space-check=trailing-comma, + dict-separator + +# Allow the body of a class to be on the same line as the declaration if body +# contains single statement. +single-line-class-stmt=no + +# Allow the body of an if to be on the same line as the test if there is no +# else. +single-line-if-stmt=no + + +[SPELLING] + +# Limits count of emitted suggestions for spelling mistakes. +max-spelling-suggestions=4 + +# Spelling dictionary name. Available dictionaries: none. To make it work, +# install the python-enchant package. +spelling-dict= + +# List of comma separated words that should not be checked. +spelling-ignore-words= + +# A path to a file that contains the private dictionary; one word per line. +spelling-private-dict-file= + +# Tells whether to store unknown words to the private dictionary (see the +# --spelling-private-dict-file option) instead of raising a message. +spelling-store-unknown-words=no + + +[BASIC] + +# Naming style matching correct argument names. +argument-naming-style=snake_case + +# Regular expression matching correct argument names. Overrides argument- +# naming-style. +#argument-rgx= + +# Naming style matching correct attribute names. +attr-naming-style=snake_case + +# Regular expression matching correct attribute names. Overrides attr-naming- +# style. +#attr-rgx= + +# Naming style matching correct class attribute names. +class-attribute-naming-style=any + +# Regular expression matching correct class attribute names. Overrides class- +# attribute-naming-style. +#class-attribute-rgx= + +# Naming style matching correct class names. +class-naming-style=PascalCase + +# Regular expression matching correct class names. Overrides class-naming- +# style. +#class-rgx= + +# Naming style matching correct constant names. +const-naming-style=UPPER_CASE + +# Regular expression matching correct constant names. Overrides const-naming- +# style. +#const-rgx= + +# Minimum line length for functions/classes that require docstrings, shorter +# ones are exempt. +docstring-min-length=-1 + +# Naming style matching correct function names. +function-naming-style=snake_case + +# Regular expression matching correct function names. Overrides function- +# naming-style. +#function-rgx= + +# Good variable names which should always be accepted, separated by a comma. +good-names=i, + j, + k, + ex, + Run, + _ + +# Include a hint for the correct naming format with invalid-name. +include-naming-hint=no + +# Naming style matching correct inline iteration names. +inlinevar-naming-style=any + +# Regular expression matching correct inline iteration names. Overrides +# inlinevar-naming-style. +#inlinevar-rgx= + +# Naming style matching correct method names. +method-naming-style=snake_case + +# Regular expression matching correct method names. Overrides method-naming- +# style. +#method-rgx= + +# Naming style matching correct module names. +module-naming-style=snake_case + +# Regular expression matching correct module names. Overrides module-naming- +# style. +#module-rgx= + +# Colon-delimited sets of names that determine each other's naming style when +# the name regexes allow several styles. +name-group= + +# Regular expression which should only match function or class names that do +# not require a docstring. +no-docstring-rgx=^_ + +# List of decorators that produce properties, such as abc.abstractproperty. Add +# to this list to register other decorators that produce valid properties. +# These decorators are taken in consideration only for invalid-name. +property-classes=abc.abstractproperty + +# Naming style matching correct variable names. +variable-naming-style=snake_case + +# Regular expression matching correct variable names. Overrides variable- +# naming-style. +#variable-rgx= + + +[STRING] + +# This flag controls whether the implicit-str-concat-in-sequence should +# generate a warning on implicit string concatenation in sequences defined over +# several lines. +check-str-concat-over-line-jumps=no + + +[TYPECHECK] + +# List of decorators that produce context managers, such as +# contextlib.contextmanager. Add to this list to register other decorators that +# produce valid context managers. +contextmanager-decorators=contextlib.contextmanager + +# List of members which are set dynamically and missed by pylint inference +# system, and so shouldn't trigger E1101 when accessed. Python regular +# expressions are accepted. +generated-members= + +# Tells whether missing members accessed in mixin class should be ignored. A +# mixin class is detected if its name ends with "mixin" (case insensitive). +ignore-mixin-members=yes + +# Tells whether to warn about missing members when the owner of the attribute +# is inferred to be None. +ignore-none=yes + +# This flag controls whether pylint should warn about no-member and similar +# checks whenever an opaque object is returned when inferring. The inference +# can return multiple potential results while evaluating a Python object, but +# some branches might not be evaluated, which results in partial inference. In +# that case, it might be useful to still emit no-member and other checks for +# the rest of the inferred objects. +ignore-on-opaque-inference=yes + +# List of class names for which member attributes should not be checked (useful +# for classes with dynamically set attributes). This supports the use of +# qualified names. +ignored-classes=optparse.Values,thread._local,_thread._local + +# List of module names for which member attributes should not be checked +# (useful for modules/projects where namespaces are manipulated during runtime +# and thus existing member attributes cannot be deduced by static analysis). It +# supports qualified module names, as well as Unix pattern matching. +ignored-modules= + +# Show a hint with possible names when a member name was not found. The aspect +# of finding the hint is based on edit distance. +missing-member-hint=yes + +# The minimum edit distance a name should have in order to be considered a +# similar match for a missing member name. +missing-member-hint-distance=1 + +# The total number of similar names that should be taken in consideration when +# showing a hint for a missing member. +missing-member-max-choices=1 + +# List of decorators that change the signature of a decorated function. +signature-mutators= + + +[SIMILARITIES] + +# Ignore comments when computing similarities. +ignore-comments=yes + +# Ignore docstrings when computing similarities. +ignore-docstrings=yes + +# Ignore imports when computing similarities. +ignore-imports=no + +# Minimum lines number of a similarity. +min-similarity-lines=4 + + +[MISCELLANEOUS] + +# List of note tags to take in consideration, separated by a comma. +notes=FIXME, + XXX, + TODO + + +[VARIABLES] + +# List of additional names supposed to be defined in builtins. Remember that +# you should avoid defining new builtins when possible. +additional-builtins= + +# Tells whether unused global variables should be treated as a violation. +allow-global-unused-variables=yes + +# List of strings which can identify a callback function by name. A callback +# name must start or end with one of those strings. +callbacks=cb_, + _cb + +# A regular expression matching the name of dummy variables (i.e. expected to +# not be used). +dummy-variables-rgx=_+$|(_[a-zA-Z0-9_]*[a-zA-Z0-9]+?$)|dummy|^ignored_|^unused_ + +# Argument names that match this expression will be ignored. Default to name +# with leading underscore. +ignored-argument-names=_.*|^ignored_|^unused_ + +# Tells whether we should check for unused import in __init__ files. +init-import=no + +# List of qualified module names which can have objects that can redefine +# builtins. +redefining-builtins-modules=six.moves,past.builtins,future.builtins,builtins,io + + +[DESIGN] + +# Maximum number of arguments for function / method. +max-args=5 + +# Maximum number of attributes for a class (see R0902). +max-attributes=7 + +# Maximum number of boolean expressions in an if statement (see R0916). +max-bool-expr=5 + +# Maximum number of branch for function / method body. +max-branches=12 + +# Maximum number of locals for function / method body. +max-locals=15 + +# Maximum number of parents for a class (see R0901). +max-parents=7 + +# Maximum number of public methods for a class (see R0904). +max-public-methods=20 + +# Maximum number of return / yield for function / method body. +max-returns=6 + +# Maximum number of statements in function / method body. +max-statements=50 + +# Minimum number of public methods for a class (see R0903). +min-public-methods=2 + + +[IMPORTS] + +# List of modules that can be imported at any level, not just the top level +# one. +allow-any-import-level= + +# Allow wildcard imports from modules that define __all__. +allow-wildcard-with-all=no + +# Analyse import fallback blocks. This can be used to support both Python 2 and +# 3 compatible code, which means that the block might have code that exists +# only in one or another interpreter, leading to false positives when analysed. +analyse-fallback-blocks=no + +# Deprecated modules which should not be used, separated by a comma. +deprecated-modules=optparse,tkinter.tix + +# Create a graph of external dependencies in the given file (report RP0402 must +# not be disabled). +ext-import-graph= + +# Create a graph of every (i.e. internal and external) dependencies in the +# given file (report RP0402 must not be disabled). +import-graph= + +# Create a graph of internal dependencies in the given file (report RP0402 must +# not be disabled). +int-import-graph= + +# Force import order to recognize a module as part of the standard +# compatibility libraries. +known-standard-library= + +# Force import order to recognize a module as part of a third party library. +known-third-party=enchant + +# Couples of modules and preferred modules, separated by a comma. +preferred-modules= + + +[CLASSES] + +# List of method names used to declare (i.e. assign) instance attributes. +defining-attr-methods=__init__, + __new__, + setUp, + __post_init__ + +# List of member names, which should be excluded from the protected access +# warning. +exclude-protected=_asdict, + _fields, + _replace, + _source, + _make + +# List of valid names for the first argument in a class method. +valid-classmethod-first-arg=cls + +# List of valid names for the first argument in a metaclass class method. +valid-metaclass-classmethod-first-arg=cls + + +[EXCEPTIONS] + +# Exceptions that will emit a warning when being caught. Defaults to +# "BaseException, Exception". +overgeneral-exceptions=builtins.BaseException, + builtins.Exception \ No newline at end of file diff --git a/.releaserc b/.releaserc index 8d0e9f8..ebc4e19 100644 --- a/.releaserc +++ b/.releaserc @@ -1,6 +1,6 @@ { "debug": true, - "branches": [ "main" ], + "branches": [ "main", {"name": "sandbox", "prerelease": "rc"} ], "plugins": [ "@semantic-release/commit-analyzer", "@semantic-release/release-notes-generator", diff --git a/.secrets.baseline b/.secrets.baseline index 68bc281..e1737e6 100644 --- a/.secrets.baseline +++ b/.secrets.baseline @@ -3,7 +3,7 @@ "files": "^.secrets.baseline$", "lines": null }, - "generated_at": "2026-03-06T04:33:02Z", + "generated_at": "2026-04-09T04:39:13Z", "plugins_used": [ { "name": "AWSKeyDetector" @@ -82,7 +82,7 @@ "hashed_secret": "11fa7c37d697f30e6aee828b4426a10f83ab2380", "is_secret": false, "is_verified": false, - "line_number": 350, + "line_number": 354, "type": "Secret Keyword", "verified_result": null } @@ -136,7 +136,7 @@ "hashed_secret": "aecdccc1cf64595b34e0cc152d238daabb32183a", "is_secret": false, "is_verified": false, - "line_number": 845, + "line_number": 915, "type": "Secret Keyword", "verified_result": null }, @@ -144,7 +144,7 @@ "hashed_secret": "f38bc57e171a3c9c5e4f405adb19a64667f875f5", "is_secret": false, "is_verified": false, - "line_number": 875, + "line_number": 945, "type": "Secret Keyword", "verified_result": null }, @@ -152,7 +152,7 @@ "hashed_secret": "e30ca465df4f3e851bad07da49e40817b45bae75", "is_secret": false, "is_verified": false, - "line_number": 888, + "line_number": 958, "type": "Secret Keyword", "verified_result": null }, @@ -160,7 +160,7 @@ "hashed_secret": "45d676e7c6ab44cf4b8fa366ef2d8fccd3e6d6e6", "is_secret": false, "is_verified": false, - "line_number": 902, + "line_number": 972, "type": "Secret Keyword", "verified_result": null }, @@ -168,7 +168,7 @@ "hashed_secret": "11fa7c37d697f30e6aee828b4426a10f83ab2380", "is_secret": false, "is_verified": false, - "line_number": 918, + "line_number": 988, "type": "Secret Keyword", "verified_result": null } @@ -178,7 +178,7 @@ "hashed_secret": "aecdccc1cf64595b34e0cc152d238daabb32183a", "is_secret": false, "is_verified": false, - "line_number": 783, + "line_number": 887, "type": "Secret Keyword", "verified_result": null }, @@ -186,7 +186,7 @@ "hashed_secret": "f38bc57e171a3c9c5e4f405adb19a64667f875f5", "is_secret": false, "is_verified": false, - "line_number": 813, + "line_number": 917, "type": "Secret Keyword", "verified_result": null }, @@ -194,7 +194,7 @@ "hashed_secret": "e30ca465df4f3e851bad07da49e40817b45bae75", "is_secret": false, "is_verified": false, - "line_number": 826, + "line_number": 930, "type": "Secret Keyword", "verified_result": null }, @@ -202,7 +202,7 @@ "hashed_secret": "45d676e7c6ab44cf4b8fa366ef2d8fccd3e6d6e6", "is_secret": false, "is_verified": false, - "line_number": 840, + "line_number": 944, "type": "Secret Keyword", "verified_result": null }, @@ -210,7 +210,7 @@ "hashed_secret": "11fa7c37d697f30e6aee828b4426a10f83ab2380", "is_secret": false, "is_verified": false, - "line_number": 856, + "line_number": 960, "type": "Secret Keyword", "verified_result": null } @@ -220,17 +220,7 @@ "hashed_secret": "11fa7c37d697f30e6aee828b4426a10f83ab2380", "is_secret": false, "is_verified": false, - "line_number": 83, - "type": "Secret Keyword", - "verified_result": null - } - ], - "docs/chapters/02_overview/release_notes.rst": [ - { - "hashed_secret": "a62f2225bf70bfaccbc7f1ef2a397836717377de", - "is_secret": false, - "is_verified": false, - "line_number": 141, + "line_number": 98, "type": "Secret Keyword", "verified_result": null } @@ -240,7 +230,7 @@ "hashed_secret": "1042188d51afe0c0b267d5c98e5ac2f2c741b28f", "is_secret": false, "is_verified": false, - "line_number": 74, + "line_number": 89, "type": "Secret Keyword", "verified_result": null }, @@ -248,7 +238,7 @@ "hashed_secret": "da6d6e9daf684cee4efd410e4d165ec5a2ee39a2", "is_secret": false, "is_verified": false, - "line_number": 113, + "line_number": 128, "type": "Secret Keyword", "verified_result": null }, @@ -256,7 +246,7 @@ "hashed_secret": "c58be5891091085cf51bf3e4d19317dce52767ae", "is_secret": false, "is_verified": false, - "line_number": 145, + "line_number": 160, "type": "Secret Keyword", "verified_result": null }, @@ -264,7 +254,7 @@ "hashed_secret": "ac52c3fa11d37a567f21c61397670e9dff7b8a52", "is_secret": false, "is_verified": false, - "line_number": 177, + "line_number": 192, "type": "Secret Keyword", "verified_result": null }, @@ -272,7 +262,7 @@ "hashed_secret": "45d676e7c6ab44cf4b8fa366ef2d8fccd3e6d6e6", "is_secret": false, "is_verified": false, - "line_number": 194, + "line_number": 209, "type": "Secret Keyword", "verified_result": null }, @@ -280,7 +270,7 @@ "hashed_secret": "a033a528b603fed46f861d4b3542c417b99d41c8", "is_secret": false, "is_verified": false, - "line_number": 228, + "line_number": 243, "type": "Secret Keyword", "verified_result": null }, @@ -288,7 +278,7 @@ "hashed_secret": "a62f2225bf70bfaccbc7f1ef2a397836717377de", "is_secret": false, "is_verified": false, - "line_number": 417, + "line_number": 432, "type": "Secret Keyword", "verified_result": null }, @@ -296,7 +286,7 @@ "hashed_secret": "11fa7c37d697f30e6aee828b4426a10f83ab2380", "is_secret": false, "is_verified": false, - "line_number": 454, + "line_number": 469, "type": "Secret Keyword", "verified_result": null } @@ -306,7 +296,7 @@ "hashed_secret": "11fa7c37d697f30e6aee828b4426a10f83ab2380", "is_secret": false, "is_verified": false, - "line_number": 33, + "line_number": 48, "type": "Secret Keyword", "verified_result": null } @@ -369,6 +359,16 @@ "verified_result": null } ], + "examples/end_to_end_example.py": [ + { + "hashed_secret": "df5cc5832dc34a455c18662ac84587ea19cf2435", + "is_secret": false, + "is_verified": false, + "line_number": 63, + "type": "Secret Keyword", + "verified_result": null + } + ], "examples/glossary_usage.py": [ { "hashed_secret": "aecdccc1cf64595b34e0cc152d238daabb32183a", @@ -379,6 +379,16 @@ "verified_result": null } ], + "examples/odcs_generator_informatica_example.py": [ + { + "hashed_secret": "0eb9a6a3306220b901c7b4920cd9896899f219be", + "is_secret": false, + "is_verified": false, + "line_number": 54, + "type": "Secret Keyword", + "verified_result": null + } + ], "src/wxdi/common/auth/__init__.py": [ { "hashed_secret": "11fa7c37d697f30e6aee828b4426a10f83ab2380", @@ -419,7 +429,33 @@ "verified_result": null } ], - "tests/auth/test_auth_config.py": [ + "src/wxdi/odcs_generator/README-GENERATE-ODCS-SCRIPT.md": [ + { + "hashed_secret": "ddc861617551c5e789c290865300f615e6f51cc7", + "is_secret": false, + "is_verified": false, + "line_number": 160, + "type": "Secret Keyword", + "verified_result": null + }, + { + "hashed_secret": "91dfd9ddb4198affc5c194cd8ce6d338fde470e2", + "is_secret": false, + "is_verified": false, + "line_number": 592, + "type": "Secret Keyword", + "verified_result": null + }, + { + "hashed_secret": "c359253705924b5bda99d65a476c651b43d5a9ab", + "is_secret": false, + "is_verified": false, + "line_number": 656, + "type": "Secret Keyword", + "verified_result": null + } + ], + "tests/src/auth/test_auth_config.py": [ { "hashed_secret": "2e7a7ee14caebf378fc32d6cf6f557f347c96773", "is_secret": false, @@ -437,7 +473,7 @@ "verified_result": null } ], - "tests/auth/test_auth_provider.py": [ + "tests/src/auth/test_auth_provider.py": [ { "hashed_secret": "3acfb2c2b433c0ea7ff107e33df91b18e52f960f", "is_secret": false, @@ -487,7 +523,7 @@ "verified_result": null } ], - "tests/auth/test_gov_cloud_authenticator.py": [ + "tests/src/auth/test_gov_cloud_authenticator.py": [ { "hashed_secret": "2e7a7ee14caebf378fc32d6cf6f557f347c96773", "is_secret": false, @@ -513,7 +549,7 @@ "verified_result": null } ], - "tests/auth/test_gov_cloud_token_manager.py": [ + "tests/src/auth/test_gov_cloud_token_manager.py": [ { "hashed_secret": "9a7ac473b40a179da10d64f3ed185894d54f0973", "is_secret": false, @@ -534,7 +570,7 @@ "hashed_secret": "2e7a7ee14caebf378fc32d6cf6f557f347c96773", "is_secret": false, "is_verified": false, - "line_number": 336, + "line_number": 318, "type": "Secret Keyword", "verified_result": null }, @@ -542,7 +578,7 @@ "hashed_secret": "3acfb2c2b433c0ea7ff107e33df91b18e52f960f", "is_secret": false, "is_verified": false, - "line_number": 387, + "line_number": 370, "type": "Secret Keyword", "verified_result": null }, @@ -550,12 +586,12 @@ "hashed_secret": "042857cf4ffc7022a50002083e0a34208b9333a8", "is_secret": false, "is_verified": false, - "line_number": 408, + "line_number": 391, "type": "Secret Keyword", "verified_result": null } ], - "tests/provider/test_config.py": [ + "tests/src/dq_validator/provider/test_config.py": [ { "hashed_secret": "74ba31d41223751c75cc0a453dd7df04889bdc72", "is_secret": false, @@ -572,6 +608,78 @@ "type": "Secret Keyword", "verified_result": null } + ], + "tests/src/integration/README.md": [ + { + "hashed_secret": "ba707e7fc663168cdca48990656bcce07d058474", + "is_secret": false, + "is_verified": false, + "line_number": 92, + "type": "Secret Keyword", + "verified_result": null + }, + { + "hashed_secret": "c5afd23baeb1464149e957bb12f02d39543b0c26", + "is_secret": false, + "is_verified": false, + "line_number": 109, + "type": "Secret Keyword", + "verified_result": null + } + ], + "tests/src/integration/test_odcs_generator_collibra.py": [ + { + "hashed_secret": "fca268ae2442d5cabc3e12d87b349adf8bf7d76c", + "is_secret": false, + "is_verified": false, + "line_number": 423, + "type": "Secret Keyword", + "verified_result": null + }, + { + "hashed_secret": "206c80413b9a96c1312cc346b7d2517b84463edd", + "is_secret": false, + "is_verified": false, + "line_number": 546, + "type": "Secret Keyword", + "verified_result": null + }, + { + "hashed_secret": "46e1833f63ae8d02127adb8316b25bea3e2051f4", + "is_secret": false, + "is_verified": false, + "line_number": 577, + "type": "Secret Keyword", + "verified_result": null + } + ], + "tests/src/integration/test_odcs_generator_informatica.py": [ + { + "hashed_secret": "206c80413b9a96c1312cc346b7d2517b84463edd", + "is_secret": false, + "is_verified": false, + "line_number": 106, + "type": "Secret Keyword", + "verified_result": null + }, + { + "hashed_secret": "7560d06f2f0b04f5b40643b505a1c8a4048a20da", + "is_secret": false, + "is_verified": false, + "line_number": 311, + "type": "Secret Keyword", + "verified_result": null + } + ], + "tests/src/odcs_generator/test_odcs_generator_collibra.py": [ + { + "hashed_secret": "206c80413b9a96c1312cc346b7d2517b84463edd", + "is_secret": false, + "is_verified": false, + "line_number": 312, + "type": "Secret Keyword", + "verified_result": null + } ] }, "version": "0.13.1+ibm.64.dss", diff --git a/Makefile.build b/Makefile.build new file mode 100644 index 0000000..d85e603 --- /dev/null +++ b/Makefile.build @@ -0,0 +1,58 @@ +# Copyright 2026 IBM Corporation +# Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# See the LICENSE file in the project root for license information. + +PYTHON=python +LINT_DIRS=src/wxdi/dph_services src/wxdi/data_product_recommender src/wxdi/odcs_generator tests/src/data_product_recommender tests/src/dph_services tests/src/odcs_generator tests/src/integration examples + +setup: deps dev_deps install_project + +all: upgrade_pip setup test-unit lint + +ci: setup test-unit # lint # enable when ready + +upgrade_pip: + ${PYTHON} -m pip install --upgrade pip + +deps: + ${PYTHON} -m pip install -r requirements.txt + +dev_deps: + ${PYTHON} -m pip install -r requirements-dev.txt + +install_project: + ${PYTHON} -m pip install -e . + +test: test-unit test-int + +test-unit: + ${PYTHON} -m pytest --cov=src/wxdi/dph_services --cov=src/wxdi/data_product_recommender --cov=src/wxdi/odcs_generator \ + --cov-report=xml:coverage.xml --cov-report=html --cov-report=term tests/src/dph_services tests/src/data_product_recommender \ + tests/src/odcs_generator tests/src/dq_validator tests/src/auth + +test-int: + ${PYTHON} -m pytest tests/src/integration + +coverage: + ${PYTHON} -m pytest --cov=src/wxdi/dph_services --cov-report=xml:coverage.xml --cov-report=html --cov-report=term tests/src/dph_services tests/src/integration + +test-examples: + ${PYTHON} -m pytest examples + +lint: + ${PYTHON} -m pylint ${LINT_DIRS} + black --check ${LINT_DIRS} + +lint-fix: + black ${LINT_DIRS} \ No newline at end of file diff --git a/README.md b/README.md index 5d9a43c..ae0c783 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,11 @@ --> # IBM watsonx.data intelligence SDK Version 1.0.0 -A comprehensive Python SDK for performing data quality validations on streaming data records (arrays), Pandas DataFrames, and PySpark DataFrames with complete REST API integration for IBM Cloud Pak for Data. +A comprehensive Python SDK for data intelligence operations including: +- **Data Quality Validation**: Validate streaming data records, Pandas DataFrames, and PySpark DataFrames +- **Data Product Hub Services**: Complete Python client for IBM Data Product Hub API +- **ODCS Generation**: Generate Open Data Contract Standard (ODCS) files from Collibra and Informatica +- **Data Product Recommendations**: Analyze query logs to identify high-value data products ## Features @@ -48,7 +52,7 @@ A comprehensive Python SDK for performing data quality validations on streaming ### REST API Integration - **GlossaryProvider**: Fetch glossary terms and data quality constraints from IBM Cloud Pak for Data - **CamsProvider**: Fetch data assets from CAMS (Catalog Asset Management System) -- **IssuesProvider**: Manage data quality issues (occurrences, tested records, ignored status) +- **IssuesProvider**: Manage data quality issues (create single/bulk occurrences, tested records, ignored status, update metrics, ignored status) - **DQSearchProvider**: Search for DQ checks and assets by native ID - **Thread-Safe**: Concurrent access support with thread-local sessions @@ -57,6 +61,14 @@ A comprehensive Python SDK for performing data quality validations on streaming - **Automatic Protocol Handling**: Environment-specific authentication methods - **Type-Safe Configuration**: Full type hints and validation - **SSL Control**: Configurable SSL verification for on-premises +### Additional Modules + +For detailed documentation on additional modules, see their respective READMEs: + +- **[Data Product Hub Services](src/wxdi/dph_services/README.md)**: Complete Python client for IBM Data Product Hub API +- **[ODCS Generator](src/wxdi/odcs_generator/README-GENERATE-ODCS-SCRIPT.md)**: Generate ODCS v3.1.0 compliant YAML from Collibra and Informatica +- **[Data Product Recommender](src/wxdi/data_product_recommender/README.md)**: Analyze query logs to identify high-value data products + ## Installation @@ -263,23 +275,37 @@ df_expanded.select('name', 'dq_is_valid', 'dq_score', 'dq_pass_rate').show() # Write validation report spark_validator.write_validation_report(df, output_path='validation_report', format='parquet') -``` -## Core Concepts +### Data Product Hub Services -### AssetMetadata +```python +from wxdi.dph_services import DphV1 +from ibm_cloud_sdk_core.authenticators import IAMAuthenticator -Defines the structure of your data asset (table) with column information: +# Initialize the service +authenticator = IAMAuthenticator('your-api-key') +dph_service = DphV1(authenticator=authenticator) +dph_service.set_service_url('https://your-dph-instance.com') -```python -metadata = AssetMetadata( - table_name='my_table', - columns=[ - ColumnMetadata('id', DataType.INTEGER), - ColumnMetadata('name', DataType.STRING, length=100), - ColumnMetadata('amount', DataType.DECIMAL, precision=10, scale=2), - ] +# Initialize a container +container_response = dph_service.initialize( + include=['delivery_methods', 'data_product_samples', 'domains_multi_industry'] +) + +# Create a data product +data_product = dph_service.create_data_product( + drafts=[{ + 'version': '1.0.0', + 'name': 'My Data Product', + 'description': 'A sample data product', + 'asset': { + 'id': 'asset-123', + 'container': {'id': 'container-456'} + } + }] ) + +print(f"Created data product: {data_product.result['id']}") ``` ### ValidationRule @@ -737,21 +763,33 @@ for column in asset.column_info: ### IssuesProvider -Manage data quality issues (occurrences, tested records, ignored status). +Manage data quality issues (create single/bulk create single/bulk occurrences, tested records, ignored status, update metrics, ignored status). ```python from wxdi.dq_validator.provider import IssuesProvider issues = IssuesProvider(config) -# Update issue occurrences -issues.update_issue_occurrences(issue_id="issue-123", occurrences=767) - -# Update tested records -issues.update_tested_records(issue_id="issue-123", tested_records=1000) +# Create single issue +issue_id = issues.create_issue( + dq_check_id="check-123", + reported_for_id="asset-456", + number_of_occurrences=10, + number_of_tested_records=100, + project_id="project-123" +) -# Set ignored status -issues.set_issue_ignored(issue_id="issue-123", ignored=True) +# Create multiple issues in bulk +bulk_payload = { + "issues": [...], + "assets": [...], + "existing_checks": [...] +} +response = issues.create_issues_bulk( + payload=bulk_payload, + project_id="project-123", + incremental_reporting=False +) # Update issue metrics issues.update_issue_values( @@ -819,18 +857,19 @@ from wxdi.dq_validator.provider import ChecksProvider checks = ChecksProvider(config) -# Create a new check +# Create a new check (returns check ID as string) check_id = checks.create_check( name="Format Check", native_id="asset-id/column-name", check_type="format", dimension_id="dimension-id", - project_id="project-id" + project_id="project-id", + parent_id=None # Optional: for parent-child hierarchy ) # Get existing checks checks_list = checks.get_checks( - asset_id="asset-id", + dq_asset_id="asset-id", check_type="format", project_id="project-id" ) @@ -1148,4 +1187,4 @@ For issues, questions, or contributions, please open an issue on GitHub. - pytest-mock >= 3.7.0 - black >= 26.3.1 - mypy >= 1.0.0 -- flake8 >= 6.0.0 + diff --git a/docs/README.md b/docs/README.md index 377f7de..3fe648e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -273,4 +273,4 @@ When contributing documentation: For documentation issues or questions: - Open an issue on GitHub -- Contact: data-intelligence-sdk@ibm.com \ No newline at end of file +- Contact: Data_Intelligence_SDK@wwpdl.vnet.ibm.com \ No newline at end of file diff --git a/docs/chapters/02_overview/faq.rst b/docs/chapters/02_overview/faq.rst index 23f51b5..ae8df01 100644 --- a/docs/chapters/02_overview/faq.rst +++ b/docs/chapters/02_overview/faq.rst @@ -329,6 +329,6 @@ If your question isn't answered here: * Check the :ref:`API Reference` * Review the code examples * Open an issue on GitHub -* Contact: data-intelligence-sdk@ibm.com +* Contact: Data_Intelligence_SDK@wwpdl.vnet.ibm.com .. Made with Bob diff --git a/docs/chapters/02_overview/known_issues.rst b/docs/chapters/02_overview/known_issues.rst index 0238282..afcbc97 100644 --- a/docs/chapters/02_overview/known_issues.rst +++ b/docs/chapters/02_overview/known_issues.rst @@ -275,7 +275,7 @@ If you're experiencing issues: * Check the :ref:`API Reference` for detailed documentation * Search GitHub issues for similar problems * Open a new issue with detailed information -* Contact: data-intelligence-sdk@ibm.com +* Contact: Data_Intelligence_SDK@wwpdl.vnet.ibm.com We appreciate your patience as we continue to improve the SDK! diff --git a/examples/__init__.py b/examples/__init__.py new file mode 100644 index 0000000..a5165bf --- /dev/null +++ b/examples/__init__.py @@ -0,0 +1,19 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Examples""" + +# This file is only here to get pylint to check the files in this directory diff --git a/examples/assets_usage.py b/examples/assets_usage.py index 510a2b2..a1c04aa 100644 --- a/examples/assets_usage.py +++ b/examples/assets_usage.py @@ -33,58 +33,58 @@ def main(): """Main function demonstrating AssetProvider usage.""" - + print("=" * 70) print("AssetProvider - Usage Examples") print("=" * 70) - + # Step 1: Configure the provider with your instance URL and authentication token config = ProviderConfig( url="https://your-instance.cloud.ibm.com", auth_token="Bearer your-auth-token-here" ) - + print("\nConfiguration:") print(f" URL: {config.url}") print(f" Auth Token: {config.auth_token[:50]}...") - + # Step 2: Create a DQAssetsProvider instance asset_provider = DQAssetsProvider(config) print("\n✓ DQAssetsProvider initialized") - + # Define IDs that will be used throughout the examples project_id = "your-project-id-here" # Replace with your actual project ID catalog_id = "your-catalog-id-here" # Alternative to project_id - + # ========================================================================= # Example 1: Get data assets with children (using project_id) # ========================================================================= print("\n" + "=" * 70) print("Example 1: Get Data Assets with Children (using project_id)") print("=" * 70) - + try: print(f"\nParameters:") print(f" project_id: {project_id}") print(f" include_children: True") print(f" asset_type: data_asset") - + assets_response = asset_provider.get_assets( project_id=project_id, include_children=True, asset_type="data_asset", limit=5 ) - + print(f"\n✓ Successfully retrieved data assets") - + assets = assets_response.get("assets", []) total_count = assets_response.get("total_count", 0) - + print(f"\nSummary:") print(f" Total Count: {total_count}") print(f" Assets Returned: {len(assets)}") - + if assets: asset = assets[0] print(f"\nFirst asset:") @@ -93,72 +93,72 @@ def main(): print(f" Type: {asset.get('type', 'N/A')}") children = asset.get('children', []) print(f" Children Count: {len(children)}") - + except ValueError as e: print(f"\n✗ Error: {e}") - + # ========================================================================= # Example 2: Get column assets without children (using project_id) # ========================================================================= print("\n" + "=" * 70) print("Example 2: Get Column Assets without Children (using project_id)") print("=" * 70) - + try: print(f"\nParameters:") print(f" project_id: {project_id}") print(f" include_children: False") print(f" asset_type: column") - + assets_response = asset_provider.get_assets( project_id=project_id, include_children=False, asset_type="column", limit=5 ) - + print(f"\n✓ Successfully retrieved column assets") - + assets = assets_response.get("assets", []) print(f"\nAssets Returned: {len(assets)}") - + if assets: print(f"\nFirst 2 columns:") for idx, asset in enumerate(assets[:2], 1): print(f" {idx}. {asset.get('name', 'N/A')} (ID: {asset.get('id')})") - + except ValueError as e: print(f"\n✗ Error: {e}") - + # ========================================================================= # Example 3: Get data assets using catalog_id # ========================================================================= print("\n" + "=" * 70) print("Example 3: Get Data Assets using catalog_id") print("=" * 70) - + try: print(f"\nParameters:") print(f" catalog_id: {catalog_id}") print(f" include_children: True") print(f" asset_type: data_asset") - + assets_response = asset_provider.get_assets( catalog_id=catalog_id, include_children=True, asset_type="data_asset", limit=5 ) - + print(f"\n✓ Successfully retrieved data assets using catalog_id") - + assets = assets_response.get("assets", []) total_count = assets_response.get("total_count", 0) - + print(f"\nSummary:") print(f" Total Count: {total_count}") print(f" Assets Returned: {len(assets)}") - + if assets: asset = assets[0] print(f"\nFirst asset:") @@ -166,41 +166,41 @@ def main(): print(f" Name: {asset.get('name', 'N/A')}") children = asset.get('children', []) print(f" Children Count: {len(children)}") - + except ValueError as e: print(f"\n✗ Error: {e}") - + # ========================================================================= # Example 4: Get column assets using catalog_id # ========================================================================= print("\n" + "=" * 70) print("Example 4: Get Column Assets using catalog_id") print("=" * 70) - + try: print(f"\nParameters:") print(f" catalog_id: {catalog_id}") print(f" asset_type: column") - + assets_response = asset_provider.get_assets( catalog_id=catalog_id, asset_type="column", limit=5 ) - + print(f"\n✓ Successfully retrieved column assets using catalog_id") - + assets = assets_response.get("assets", []) print(f"\nAssets Returned: {len(assets)}") - + if assets: print(f"\nFirst 2 columns:") for idx, asset in enumerate(assets[:2], 1): print(f" {idx}. {asset.get('name', 'N/A')} (ID: {asset.get('id')})") - + except ValueError as e: print(f"\n✗ Error: {e}") - + print("\n" + "=" * 70) print("Examples Complete!") print("=" * 70) diff --git a/examples/auth_provider_usage.py b/examples/auth_provider_usage.py index 5700088..3508984 100644 --- a/examples/auth_provider_usage.py +++ b/examples/auth_provider_usage.py @@ -65,7 +65,7 @@ def example_ibm_cloud(): """ Example: IBM Cloud authentication using IAMAuthenticator - + Requirements: - api_key: Your IBM Cloud API key - url: Not required for production. Provide only for non-production environments. @@ -73,7 +73,7 @@ def example_ibm_cloud(): print("=" * 60) print("IBM CLOUD AUTHENTICATION") print("=" * 60) - + # Create environment configuration # Required: environment_type, api_key # url: Not required for production @@ -85,28 +85,28 @@ def example_ibm_cloud(): # url='https://host_name', # This is not required for production env. For non-default custom add this property. # disable_ssl_verification=True # Optional, default is True ) - + print(f"Environment Type: {config.environment_type.value}") print(f"URL: {config.url}") - + # Create auth provider auth_provider = AuthProvider(config) print(f"Authenticator: {type(auth_provider.authenticator).__name__}") - + # Get token (uncomment when you have valid credentials) # try: # token = auth_provider.get_token() # print(f"Token obtained successfully: {token[:20]}...") # except Exception as e: # print(f"Error getting token: {e}") - + print() def example_aws_cloud(): """ Example: AWS Cloud authentication using MCSPV2Authenticator - + Requirements: - api_key: Your AWS API key - account_id: Your AWS account ID @@ -115,7 +115,7 @@ def example_aws_cloud(): print("=" * 60) print("AWS CLOUD AUTHENTICATION") print("=" * 60) - + # Create environment configuration # Required: environment_type, api_key, account_id # url: Not required for production @@ -128,29 +128,29 @@ def example_aws_cloud(): # url='https://host_name', # This is not required for production env. For non-default custom add this property. # disable_ssl_verification=True # Optional, default is True ) - + print(f"Environment Type: {config.environment_type.value}") print(f"URL: {config.url}") print(f"Account ID: {config.account_id}") - + # Create auth provider auth_provider = AuthProvider(config) print(f"Authenticator: {type(auth_provider.authenticator).__name__}") - + # Get token (uncomment when you have valid credentials) # try: # token = auth_provider.get_token() # print(f"Token obtained successfully: {token[:20]}...") # except Exception as e: # print(f"Error getting token: {e}") - + print() def example_gov_cloud(): """ Example: Government Cloud authentication using GovCloudAuthenticator - + Requirements: - api_key: Your Government Cloud API key - url: Not required for production. Provide only for non-production environments. @@ -158,7 +158,7 @@ def example_gov_cloud(): print("=" * 60) print("GOVERNMENT CLOUD AUTHENTICATION") print("=" * 60) - + # Create environment configuration # Required: environment_type, api_key # url: Not required for production environments @@ -170,39 +170,39 @@ def example_gov_cloud(): # url='https://host_name', # This is not required for production env. For non-default custom add this property. # disable_ssl_verification=True # Optional, default is True ) - + print(f"Environment Type: {config.environment_type.value}") print(f"URL: {config.url}") - + # Create auth provider auth_provider = AuthProvider(config) print(f"Authenticator: {type(auth_provider.authenticator).__name__}") - + # Get token (uncomment when you have valid credentials) # try: # token = auth_provider.get_token() # print(f"Token obtained successfully: {token[:20]}...") # except Exception as e: # print(f"Error getting token: {e}") - + print() def example_on_prem_with_apikey(): """ Example: On-Premises authentication using CloudPakForDataAuthenticator with API key - + Requirements: - url: Your on-premises CP4D URL (required) - username: Your username - api_key: Your API key - + Note: When using api_key, username is required but password should NOT be provided. """ print("=" * 60) print("ON-PREMISES AUTHENTICATION (with API Key)") print("=" * 60) - + # Create environment configuration # Required: environment_type, url, username, api_key (or password) # Optional: disable_ssl_verification (default: True) @@ -214,41 +214,41 @@ def example_on_prem_with_apikey(): api_key='your-cp4d-api-key-here' # disable_ssl_verification=True # Optional, default is True ) - + print(f"Environment Type: {config.environment_type.value}") print(f"URL: {config.url}") print(f"Username: {config.username}") print(f"Authentication Method: API Key") - + # Create auth provider auth_provider = AuthProvider(config) print(f"Authenticator: {type(auth_provider.authenticator).__name__}") - + # Get token (uncomment when you have valid credentials) # try: # token = auth_provider.get_token() # print(f"Token obtained successfully: {token[:20]}...") # except Exception as e: # print(f"Error getting token: {e}") - + print() def example_on_prem_with_password(): """ Example: On-Premises authentication using CloudPakForDataAuthenticator with password - + Requirements: - url: Your on-premises CP4D URL (required) - username: Your username - password: Your password - + Note: When using password, api_key should NOT be provided. """ print("=" * 60) print("ON-PREMISES AUTHENTICATION (with Username/Password)") print("=" * 60) - + # Create environment configuration # Required: environment_type, url, username, password (or api_key) # Optional: disable_ssl_verification (default: True) @@ -259,37 +259,37 @@ def example_on_prem_with_password(): password='your-password', # disable_ssl_verification=True # Optional, default is True ) - + print(f"Environment Type: {config.environment_type.value}") print(f"URL: {config.url}") print(f"Username: {config.username}") print(f"Authentication Method: Username/Password") - + # Create auth provider auth_provider = AuthProvider(config) print(f"Authenticator: {type(auth_provider.authenticator).__name__}") - + # Get token (uncomment when you have valid credentials) # try: # token = auth_provider.get_token() # print(f"Token obtained successfully: {token[:20]}...") # except Exception as e: # print(f"Error getting token: {e}") - + print() def example_custom_url(): """ Example: Using custom URL for non-production environments - + Provide url only for non-production environments (staging, preprod, dev). For production, omit url and the default production URL will be used automatically. """ print("=" * 60) print("NON-production CUSTOM URL EXAMPLE") print("=" * 60) - + # Provide url only for non-production environments # For production, omit url - the default production URL is used automatically # Trailing slashes are automatically stripped @@ -299,14 +299,14 @@ def example_custom_url(): api_key='your-api-key-here', # disable_ssl_verification=True # Optional, default is True ) - + print(f"Environment Type: {config.environment_type.value}") print(f"Custom URL: {config.url}") - + # Create auth provider auth_provider = AuthProvider(config) print(f"Authenticator: {type(auth_provider.authenticator).__name__}") - + # Get token (uncomment when you have valid credentials) # try: # token = auth_provider.get_token() @@ -320,13 +320,13 @@ def example_custom_url(): def example_error_handling(): """ Example: Error handling and validation - + The AuthConfig validates required fields based on environment type. """ print("=" * 60) print("ERROR HANDLING EXAMPLES") print("=" * 60) - + # Example 1: Missing API key for IBM_CLOUD print("\n1. Missing API key for IBM_CLOUD:") try: @@ -336,7 +336,7 @@ def example_error_handling(): ) except ValueError as e: print(f" ✓ Caught error: {e}") - + # Example 2: Missing account_id for AWS_CLOUD print("\n2. Missing account_id for AWS_CLOUD:") try: @@ -347,7 +347,7 @@ def example_error_handling(): ) except ValueError as e: print(f" ✓ Caught error: {e}") - + # Example 3: Missing username for ON_PREM print("\n3. Missing username for ON_PREM:") try: @@ -359,7 +359,7 @@ def example_error_handling(): ) except ValueError as e: print(f" ✓ Caught error: {e}") - + # Example 4: Missing URL for ON_PREM print("\n4. Missing URL for ON_PREM:") try: @@ -371,7 +371,7 @@ def example_error_handling(): ) except ValueError as e: print(f" ✓ Caught error: {e}") - + print() @@ -379,7 +379,7 @@ def example_error_handling(): print("\n" + "=" * 60) print("AUTHPROVIDER USAGE EXAMPLES") print("=" * 60 + "\n") - + # Run all examples example_ibm_cloud() example_aws_cloud() @@ -388,7 +388,7 @@ def example_error_handling(): example_on_prem_with_password() example_custom_url() example_error_handling() - + print("=" * 60) print("All examples completed!") print("=" * 60) \ No newline at end of file diff --git a/examples/basic_usage.py b/examples/basic_usage.py index ad7f4c6..8b06bff 100644 --- a/examples/basic_usage.py +++ b/examples/basic_usage.py @@ -34,7 +34,7 @@ def main(): print("=" * 60) print("IBM watsonx.data intelligence SDK - Basic Usage Example") print("=" * 60) - + metadata = AssetMetadata( table_name='employee_data', columns=[ @@ -52,25 +52,25 @@ def main(): ColumnMetadata('employee_code', DataType.STRING, length=10) ] ) - + print(f"\nAsset: {metadata.table_name}") print(f"Columns: {len(metadata.columns)}") - + # Step 2: Create validator with rules validator = Validator(metadata) - + # Add length check for name (works with any type) validator.add_rule( ValidationRule('name') .add_check(LengthCheck(min_length=2, max_length=100)) ) - + # Add length check for emp_id (integer converted to string) validator.add_rule( ValidationRule('emp_id') .add_check(LengthCheck(min_length=4, max_length=6)) ) - + # Add valid values check for department (case-insensitive) validator.add_rule( ValidationRule('department') @@ -79,7 +79,7 @@ def main(): case_sensitive=False )) ) - + # Add comparison checks using enum validator.add_rule( ValidationRule('age') @@ -92,7 +92,7 @@ def main(): target_value=65 )) ) - + validator.add_rule( ValidationRule('salary') .add_check(ComparisonCheck( @@ -100,43 +100,43 @@ def main(): target_column='min_salary' )) ) - + # Add CaseCheck for email (must be lowercase) validator.add_rule( ValidationRule('email') .add_check(CaseCheck(case_type=ColumnCaseEnum.LOWER_CASE)) ) - + # Add CompletenessCheck for name (required field) validator.add_rule( ValidationRule('name') .add_check(CompletenessCheck(missing_values_allowed=False)) ) - + # Add RangeCheck for bonus (0 to 50000) validator.add_rule( ValidationRule('bonus') .add_check(RangeCheck(min_value=0, max_value=50000)) ) - + # Add RegexCheck for phone (format: XXX-XXX-XXXX) validator.add_rule( ValidationRule('phone') .add_check(RegexCheck(pattern=r'^\d{3}-\d{3}-\d{4}$')) ) - + # Add CaseCheck for status (must be NameCase) validator.add_rule( ValidationRule('status') .add_check(CaseCheck(case_type=ColumnCaseEnum.NAME_CASE)) ) - + # Add DataTypeCheck for age (must be INTEGER) validator.add_rule( ValidationRule('age') .add_check(DataTypeCheck(expected_type=DType(dtype=DataTypeEnum.INT32))) ) - + # Add FormatCheck for hire_date (must be valid date format) # NEW: Using DateTimeFormats constants for readable format names validator.add_rule( @@ -150,7 +150,7 @@ def main(): } )) ) - + print(f"\nValidator configured with {len(validator.rules)} rules") print("Checks included:") print(" - LengthCheck (name, emp_id)") @@ -162,12 +162,12 @@ def main(): print(" - RegexCheck (phone)") print(" - DataTypeCheck (age)") print(" - FormatCheck (hire_date)") - + # Step 3: Validate records print("\n" + "=" * 60) print("Validating Records") print("=" * 60) - + records = [ # [emp_id, name, email, age, department, salary, min_salary, phone, status, bonus, hire_date, employee_code] [1001, 'John Doe', 'john@company.com', 30, 'engineering', 75000.00, 60000.00, '555-123-4567', 'Active', 5000.00, '2020-01-15', 'EMP001'], @@ -177,39 +177,39 @@ def main(): [1005, 'Charlie Brown', 'charlie@company.com', 40, 'Finance', 55000.00, 60000.00, '555-567-8901', 'Active', 4000.00, '2023-06-10', 'EMP005'], [1006, None, 'test@company.com', 28, 'Engineering', 70000.00, 65000.00, '555-678-9012', 'Active', 3500.00, '07/20/2024', 'EMP006'], # Name is None (completeness check) ] - + results = validator.validate_batch(records) - + # Step 4: Display results for idx, result in enumerate(results): status_symbol = '[PASS]' if result.is_valid else '[FAIL]' - + print(f"\nRecord {idx + 1}: {status_symbol} (Score: {result.score}, Pass Rate: {result.pass_rate:.1f}%)") - + if not result.is_valid: for error in result.errors: print(f" - {error.column_name}: {error.message}") - + # Step 5: Summary statistics print("\n" + "=" * 60) print("Summary") print("=" * 60) - + total_records = len(results) valid_records = sum(1 for r in results if r.is_valid) invalid_records = total_records - valid_records overall_pass_rate = (valid_records / total_records) * 100 - + print(f"Total Records: {total_records}") print(f"Valid Records: {valid_records}") print(f"Invalid Records: {invalid_records}") print(f"Overall Pass Rate: {overall_pass_rate:.1f}%") - + # Step 6: Detailed validation result for first failed record print("\n" + "=" * 60) print("Detailed Result Example (First Failed Record)") print("=" * 60) - + failed_result = next((r for r in results if not r.is_valid), None) if failed_result: print(f"\nRecord Index: {failed_result.record_index}") diff --git a/examples/checks_usage.py b/examples/checks_usage.py index 3048cc8..63f3e75 100644 --- a/examples/checks_usage.py +++ b/examples/checks_usage.py @@ -24,7 +24,7 @@ - Returns: The check ID (string) get_checks(): Retrieve checks for a specific asset filtered by check type - - Requires: asset_id, check_type, and either project_id OR catalog_id + - Requires: dq_asset_id, check_type, and either project_id OR catalog_id - Optional: include_children (defaults to True) - Returns: List of check objects matching the criteria @@ -36,36 +36,36 @@ def main(): """Main function demonstrating CheckProvider usage.""" - + print("=" * 70) print("CheckProvider - Usage Examples") print("=" * 70) - + # Step 1: Configure the provider with your instance URL and authentication token config = ProviderConfig( url="https://your-instance.com", auth_token="Bearer your-auth-token-here" ) - + print("\nConfiguration:") print(f" URL: {config.url}") print(f" Auth Token: {config.auth_token[:50]}...") - + # Step 2: Create a ChecksProvider instance check_provider = ChecksProvider(config) print("\n✓ ChecksProvider initialized") - + # Define IDs that will be used throughout the examples project_id = "your-project-id-here" catalog_id = "your-catalog-id-here" # Alternative to project_id - + # ========================================================================= # Example 1: Create a new check without specifying check_type (using project_id) # ========================================================================= print("\n" + "=" * 70) print("Example 1: Create Check without check_type (defaults to name)") print("=" * 70) - + try: # Note: Update these values with your actual data # When check_type is not provided, it defaults to the check name @@ -78,38 +78,43 @@ def main(): ) print(f"\n✓ Successfully created check") print(f" New Check ID: {check_id}") - + except ValueError as e: print(f"\n✗ Error creating check: {e}") - + # ========================================================================= - # Example 2: Create a comparison check (using project_id) + # Example 2: Create a comparison check with parent_check_id (using project_id) # ========================================================================= print("\n" + "=" * 70) - print("Example 2: Create a Comparison Check (using project_id)") + print("Example 2: Create a Comparison Check with parent_check_id (using project_id)") print("=" * 70) - + try: + # First, let's assume we have a parent check ID from Example 1 + parent_check_id = "848aaddc-7401-4a43-ad2b-96a0946d4674" # Replace with actual parent ID + check_id = check_provider.create_check( name="Example Comparison Check", dimension_id="ec453723-669c-48bb-82c1-11b69b3b8c93", # Validity dimension native_id="your-asset-id/your-check-id-2", check_type="comparison", - project_id=project_id + project_id=project_id, + parent_check_id=parent_check_id # Optional: Link to parent check ) - print(f"\n✓ Successfully created comparison check") + print(f"\n✓ Successfully created comparison check with parent") print(f" New Check ID: {check_id}") - + print(f" Parent Check ID: {parent_check_id}") + except ValueError as e: print(f"\n✗ Error creating check: {e}") - + # ========================================================================= # Example 3: Create a check using catalog_id instead of project_id # ========================================================================= print("\n" + "=" * 70) print("Example 3: Create Check using catalog_id") print("=" * 70) - + try: check_id = check_provider.create_check( name="Catalog Check Example", @@ -120,17 +125,17 @@ def main(): ) print(f"\n✓ Successfully created check using catalog_id") print(f" New Check ID: {check_id}") - + except ValueError as e: print(f"\n✗ Error creating check: {e}") - + # ========================================================================= # Example 4: Create multiple checks with different types # ========================================================================= print("\n" + "=" * 70) print("Example 4: Create Multiple Checks with Different Types") print("=" * 70) - + check_configs = [ { "name": "Uniqueness Check", @@ -142,25 +147,31 @@ def main(): "name": "Completeness Check", "dimension_id": "ec453723-669c-48bb-82c1-11b69b3b8c93", "native_id": "asset-1/completeness-check", - "check_type": "data_rule" + "check_type": "data_rule", + "parent_check_id": "parent-check-id-123" # Optional: Add parent for hierarchical checks }, { "name": "Format Validation Check", "dimension_id": "371114cd-5516-4691-8b2e-1e66edf66486", "native_id": "asset-1/format-check", - "check_type": "comparison" + "check_type": "comparison", + "parent_check_id": "parent-check-id-456" # Optional: Add parent for hierarchical checks } ] - + created_checks = [] for config in check_configs: try: + # Get parent_check_id if it exists in config, otherwise None + parent_check_id = config.get("parent_check_id") + check_id = check_provider.create_check( name=config["name"], dimension_id=config["dimension_id"], native_id=config["native_id"], check_type=config["check_type"], - project_id=project_id + project_id=project_id, + parent_check_id=parent_check_id ) created_checks.append({ "id": check_id, @@ -172,16 +183,16 @@ def main(): print(f" Type: {config['check_type']}") except ValueError as e: print(f"\n✗ Error creating {config['name']}: {e}") - + print(f"\n\nSummary: Successfully created {len(created_checks)} checks") - + # ========================================================================= # Example 5: Error handling - Missing required parameters # ========================================================================= print("\n" + "=" * 70) print("Example 5: Error Handling - Missing project_id and catalog_id") print("=" * 70) - + try: # This should fail because neither project_id nor catalog_id is provided check_provider.create_check( @@ -192,14 +203,14 @@ def main(): ) except ValueError as e: print(f"\n✓ Expected error caught: {e}") - + # ========================================================================= # Example 6: Error handling - Both project_id and catalog_id provided # ========================================================================= print("\n" + "=" * 70) print("Example 6: Error Handling - Both project_id and catalog_id provided") print("=" * 70) - + try: # This should fail because both project_id and catalog_id are provided check_provider.create_check( @@ -211,30 +222,30 @@ def main(): ) except ValueError as e: print(f"\n✓ Expected error caught: {e}") - + # ========================================================================= # Example 7: Get checks for a specific asset filtered by check type # ========================================================================= print("\n" + "=" * 70) print("Example 7: Get Checks for an Asset (filtered by check_type)") print("=" * 70) - + try: # Retrieve all checks for a specific column asset filtered by check type column_asset_id = "your-column-asset-id-here" check_type = "case" # e.g., "case", "completeness", "comparison", etc. - + checks = check_provider.get_checks( - asset_id=column_asset_id, + dq_asset_id=column_asset_id, check_type=check_type, project_id=project_id ) - + print(f"\n✓ Successfully retrieved checks") print(f" Asset ID: {column_asset_id}") print(f" Check Type Filter: {check_type}") print(f" Number of checks found: {len(checks)}") - + # Display details of each check for i, check in enumerate(checks, 1): print(f"\n Check {i}:") @@ -242,31 +253,31 @@ def main(): print(f" Name: {check.get('name')}") print(f" Type: {check.get('type')}") print(f" Native ID: {check.get('native_id')}") - + except ValueError as e: print(f"\n✗ Error retrieving checks: {e}") - + # ========================================================================= # Example 8: Get checks with include_children parameter # ========================================================================= print("\n" + "=" * 70) print("Example 8: Get Checks with include_children=False") print("=" * 70) - + try: checks = check_provider.get_checks( - asset_id="your-asset-id", + dq_asset_id="your-asset-id", check_type="completeness", project_id=project_id, include_children=False # Don't include child checks ) - + print(f"\n✓ Successfully retrieved checks (without children)") print(f" Number of checks found: {len(checks)}") - + except ValueError as e: print(f"\n✗ Error retrieving checks: {e}") - + print("\n" + "=" * 70) print("Examples Complete!") print("=" * 70) diff --git a/examples/consolidation_usage.py b/examples/consolidation_usage.py index 20745eb..0926d1b 100644 --- a/examples/consolidation_usage.py +++ b/examples/consolidation_usage.py @@ -36,7 +36,7 @@ def main(): print("=" * 70) print("IBM watsonx.data intelligence SDK - Consolidation Usage Example") print("=" * 70) - + metadata = AssetMetadata( table_name='employee_data', columns=[ @@ -54,24 +54,24 @@ def main(): ColumnMetadata('employee_code', DataType.STRING, length=10) ] ) - + print(f"\nAsset: {metadata.table_name}") print(f"Columns: {len(metadata.columns)}") - + # Step 2: Create validator with rules (same as basic_usage.py) validator = Validator(metadata) - + validator.add_rule( ValidationRule('name') .add_check(LengthCheck(min_length=2, max_length=100)) .add_check(CompletenessCheck(missing_values_allowed=False)) ) - + validator.add_rule( ValidationRule('emp_id') .add_check(LengthCheck(min_length=4, max_length=6)) ) - + validator.add_rule( ValidationRule('department') .add_check(ValidValuesCheck( @@ -79,7 +79,7 @@ def main(): case_sensitive=False )) ) - + validator.add_rule( ValidationRule('age') .add_check(ComparisonCheck( @@ -92,7 +92,7 @@ def main(): )) .add_check(DataTypeCheck(expected_type=DType(dtype=DataTypeEnum.INT32))) ) - + validator.add_rule( ValidationRule('salary') .add_check(ComparisonCheck( @@ -100,27 +100,27 @@ def main(): target_column='min_salary' )) ) - + validator.add_rule( ValidationRule('email') .add_check(CaseCheck(case_type=ColumnCaseEnum.LOWER_CASE)) ) - + validator.add_rule( ValidationRule('bonus') .add_check(RangeCheck(min_value=0, max_value=50000)) ) - + validator.add_rule( ValidationRule('phone') .add_check(RegexCheck(pattern=r'^\d{3}-\d{3}-\d{4}$')) ) - + validator.add_rule( ValidationRule('status') .add_check(CaseCheck(case_type=ColumnCaseEnum.NAME_CASE)) ) - + validator.add_rule( ValidationRule('hire_date') .add_check(FormatCheck( @@ -132,14 +132,14 @@ def main(): } )) ) - + print(f"\nValidator configured with {len(validator.rules)} rules") - + # Step 3: Validate records print("\n" + "=" * 70) print("Validating Records") print("=" * 70) - + records = [ [1001, 'John Doe', 'john@company.com', 30, 'engineering', 75000.00, 60000.00, '555-123-4567', 'Active', 5000.00, '2020-01-15', 'EMP001'], [12, 'Jane Smith', 'jane@company.com', 25, 'SALES', 65000.00, 55000.00, '555-234-5678', 'Active', 3000.00, '01/15/2021', 'EMP002'], @@ -148,23 +148,23 @@ def main(): [1005, 'Charlie Brown', 'charlie@company.com', 40, 'Finance', 55000.00, 60000.00, '555-567-8901', 'Active', 4000.00, '2023-06-10', 'EMP005'], [1006, None, 'test@company.com', 28, 'Engineering', 70000.00, 65000.00, '555-678-9012', 'Active', 3500.00, '07/20/2024', 'EMP006'], ] - + results = validator.validate_batch(records) - + # Display basic results for idx, result in enumerate(results): status_symbol = '[PASS]' if result.is_valid else '[FAIL]' print(f"Record {idx + 1}: {status_symbol} (Score: {result.score})") - + # Step 4: NEW - Use ValidationResultConsolidated for detailed statistics print("\n" + "=" * 70) print("Consolidated Statistics (NEW FEATURE)") print("=" * 70) - + # Create consolidator with error storage enabled consolidator = ValidationResultConsolidated(validator=validator, store_errors=True) consolidator.add_results(results) - + # Overall statistics print("\n[Overall Statistics]") overall = consolidator.get_overall_statistics() @@ -173,43 +173,43 @@ def main(): print(f" Invalid Records: {overall['invalid_records']}") print(f" Pass Rate: {overall['pass_rate']:.1f}%") print(f" Total Errors: {overall['total_errors']}") - + # Statistics by column print("\n[Statistics by Column]") print(f" {'Column':<20} {'Passed':<10} {'Failed':<10} {'Total':<10}") print(f" {'-'*20} {'-'*10} {'-'*10} {'-'*10}") - + for column in sorted(consolidator.get_columns()): stats = consolidator.get_column_statistics(column) print(f" {column:<20} {stats['passed']:<10} {stats['failed']:<10} {stats['total']:<10}") - + # Statistics by check type print("\n[Statistics by Check Type]") print(f" {'Check Type':<30} {'Passed':<10} {'Failed':<10} {'Total':<10}") print(f" {'-'*30} {'-'*10} {'-'*10} {'-'*10}") - + for check in sorted(consolidator.get_checks()): stats = consolidator.get_check_statistics(check) print(f" {check:<30} {stats['passed']:<10} {stats['failed']:<10} {stats['total']:<10}") - + # Combined statistics (column + check) print("\n[Combined Statistics (Column + Check Type)]") combined = consolidator.get_combined_statistics() - + for column in sorted(combined.keys()): print(f"\n {column}:") for check, stats in sorted(combined[column].items()): print(f" {check:<28} Failed: {stats['failed']} Passed: {stats['passed']}") - + # Error details for specific columns print("\n" + "=" * 70) print("Error Details by Column") print("=" * 70) - + # Show errors for columns with failures - columns_with_errors = [col for col in consolidator.get_columns() + columns_with_errors = [col for col in consolidator.get_columns() if consolidator.get_column_statistics(col)['failed'] > 0] - + for column in sorted(columns_with_errors)[:3]: # Show first 3 columns with errors errors = consolidator.get_errors_by_column(column) print(f"\n[X] {column} ({len(errors)} error(s)):") @@ -218,27 +218,27 @@ def main(): print(f" Value: {error['value']}") if error['expected']: print(f" Expected: {error['expected']}") - + # Error details by check type print("\n" + "=" * 70) print("Error Details by Check Type") print("=" * 70) - + # Show errors for specific check types check_types_with_errors = [check for check in consolidator.get_checks() if consolidator.get_check_statistics(check)['failed'] > 0] - + for check in sorted(check_types_with_errors)[:3]: # Show first 3 check types errors = consolidator.get_errors_by_check(check) print(f"\n[Check: {check}] ({len(errors)} error(s)):") for error in errors[:2]: # Show first 2 errors per check print(f" Column '{error['column']}' at record {error['record_index']}: {error['message']}") - + # Specific column + check combination print("\n" + "=" * 70) print("Specific Column + Check Combination") print("=" * 70) - + # Example: Get all case_check errors for email column if 'email' in consolidator.get_columns() and 'case_check' in consolidator.get_checks(): email_case_errors = consolidator.get_errors_by_column_and_check('email', 'case_check') @@ -246,42 +246,42 @@ def main(): for error in email_case_errors: print(f" Record {error['record_index']}: {error['message']}") print(f" Value: '{error['value']}'") - + # Issues by Data Quality Dimension print("\n" + "=" * 70) print("Issues by Data Quality Dimension") print("=" * 70) - + # 1. Get issues for all dimensions print("\n[All Dimensions]") print(f" {'Dimension':<20} {'Issues':<10}") print(f" {'-'*20} {'-'*10}") - + all_dimension_issues = consolidator.get_all_dimension_issues() for dimension_name, issue_count in sorted(all_dimension_issues.items()): print(f" {dimension_name:<20} {issue_count:<10}") - + # 2. Get issues for only the VALIDITY dimension print("\n[VALIDITY Dimension Only]") validity_issues = consolidator.get_issues_by_dimension(DataQualityDimension.VALIDITY) print(f" VALIDITY: {validity_issues} issue(s)") - + # Step 5: Report Issues to CPD (Optional) print("\n" + "=" * 70) print("Reporting Issues to CPD (Optional)") print("=" * 70) - + # Uncomment and configure the following section to report issues to CAMS from wxdi.dq_validator.provider import ProviderConfig from wxdi.dq_validator.issue_reporting import IssueReporter - + # Configure provider config = ProviderConfig( url="https://your-cpd-instance.com", auth_token="Bearer your-token-here", project_id="your-project-id-here" ) - + # Initialize IssueReporter reporter = IssueReporter(config) print("\n[IssueReporter Initialized]") @@ -289,14 +289,14 @@ def main(): print(" ✓ IssuesProvider ready") print(" ✓ DimensionProvider ready") print(" ✓ AssetProvider ready") - + # Report issues cams_asset_id = "your-asset-id-here" - + print(f"\n[Configuration]") print(f" Asset ID: {cams_asset_id}") print(f" Project ID: {config.project_id}") - + try: reporter.report_issues( stats=combined, @@ -304,50 +304,50 @@ def main(): validator=validator ) print("\n[SUCCESS] Issues reported to CPD successfully!") - + except Exception as e: print(f"\n[ERROR] Failed to report issues: {str(e)}") - + print("\n[Note] Issue reporting to CPD is optional and requires:") print(" 1. Valid CPD instance URL") print(" 2. Authentication token") print(" 3. CAMS asset ID") print(" 4. Project ID or Catalog ID") print(" Uncomment the code above and configure to enable.") - + # Export to dictionary print("\n" + "=" * 70) print("Export Consolidated Data") print("=" * 70) - + consolidated_dict = consolidator.to_dict() print(f"\n[Consolidated data structure]") print(f" - Overall statistics: {len(consolidated_dict['overall'])} metrics") print(f" - Columns tracked: {len(consolidated_dict['columns'])}") print(f" - Check types tracked: {len(consolidated_dict['checks'])}") print(f" - Total errors stored: {consolidated_dict['error_count']}") - + # Memory-efficient mode demonstration print("\n" + "=" * 70) print("Memory-Efficient Mode (No Error Storage)") print("=" * 70) - + # Create consolidator without error storage consolidator_lite = ValidationResultConsolidated(validator=validator, store_errors=False) consolidator_lite.add_results(results) - + print("\n[Memory-efficient consolidator]") print(f" Total Records: {consolidator_lite.total_records}") print(f" Valid Records: {consolidator_lite.valid_records}") print(f" Statistics available: YES") print(f" Error details available: NO (memory efficient)") - + # Try to access error details (will raise error) try: consolidator_lite.get_errors_by_column('email') except RuntimeError as e: print(f"\n Expected error when accessing error details: {str(e)[:50]}...") - + print("\n" + "=" * 70) print("[SUCCESS] Consolidation example completed!") print("=" * 70) diff --git a/examples/data_product_recommender_example.py b/examples/data_product_recommender_example.py new file mode 100644 index 0000000..13a1523 --- /dev/null +++ b/examples/data_product_recommender_example.py @@ -0,0 +1,136 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Example usage of Data Product Recommender + +This example demonstrates how to use the Data Product Recommender to analyze +query logs and generate recommendations for data products. +""" + +from wxdi.data_product_recommender.platforms import SnowflakeQueryParser +from wxdi.data_product_recommender.recommender import DataProductRecommender + + +def main(): + """Example: Analyze Snowflake query logs from a CSV file""" + + # Initialize platform-specific parser + print("Initializing Snowflake query parser...") + parser = SnowflakeQueryParser() + + # Initialize recommender + recommender = DataProductRecommender(parser) + + # Load query logs from CSV file + # Replace with your actual query log file path + csv_file_path = 'path/to/your/query_logs.csv' + + print(f"\nLoading query logs from: {csv_file_path}") + try: + recommender.load_query_logs_from_csv_file(csv_file_path) + except FileNotFoundError: + print(f"Error: File not found: {csv_file_path}") + print("Please update the csv_file_path variable with your actual query log file.") + return + + # Calculate metrics + print("\nCalculating table metrics...") + recommender.calculate_metrics() + + # Generate recommendations + print("\nGenerating recommendations...") + recommendations = recommender.recommend_data_products( + num_recommendations=20, + min_score=50.0 # Only include tables with score >= 50 + ) + + # Export recommendations as Markdown (human-readable) + output_file_md = 'output/recommendations_example.md' + print(f"\nExporting recommendations to: {output_file_md}") + recommender.export_recommendations_markdown(recommendations, output_file_md) + + # Export recommendations as JSON (agent-consumable) + output_file_json = 'output/recommendations_example.json' + print(f"Exporting recommendations to: {output_file_json}") + recommender.export_recommendations_json(recommendations, output_file_json) + + # Print summary + print("\n✓ Analysis complete!") + print(f" - Queries analyzed: {len(recommender.query_logs):,}") + print(f" - Tables identified: {len(recommender.table_metrics)}") + print(f" - Recommendations: {len(recommendations['individual_tables'])}") + if 'table_groups' in recommendations: + print(f" - Table groups: {len(recommendations['table_groups'])}") + print(f" - Markdown output: {output_file_md}") + print(f" - JSON output: {output_file_json}") + + +def example_with_json_input(): + """Example: Analyze query logs from a JSON file""" + + from wxdi.data_product_recommender.platforms import DatabricksQueryParser + + # Initialize for Databricks + parser = DatabricksQueryParser() + recommender = DataProductRecommender(parser) + + # Load from JSON file + json_file_path = 'path/to/your/query_logs.json' + recommender.load_query_logs_from_json_file(json_file_path) + + # Calculate and generate recommendations + recommender.calculate_metrics() + recommendations = recommender.recommend_data_products(num_recommendations=15) + + # Export + recommender.export_recommendations_markdown(recommendations, 'output/databricks_recommendations.md') + + +def example_with_custom_scoring(): + """Example: Use custom scoring weights""" + + from wxdi.data_product_recommender.platforms import BigQueryQueryParser + + parser = BigQueryQueryParser() + recommender = DataProductRecommender(parser) + + # Load data + recommender.load_query_logs_from_csv_file('path/to/query_logs.csv') + recommender.calculate_metrics() + + # Score tables with custom weights + # Emphasize user diversity over query frequency + scored_tables = recommender.score_tables( + query_weight=0.25, # 25% weight on query frequency + user_weight=0.50, # 50% weight on user diversity + recency_weight=0.15, # 15% weight on recency + consistency_weight=0.10 # 10% weight on consistency + ) + + print("\nTop 10 tables by custom scoring:") + print(scored_tables[['table', 'recommendation_score', 'query_count', 'unique_users']].head(10)) + + +if __name__ == '__main__': + # Run the main example + main() + + # Uncomment to run other examples: + # example_with_json_input() + # example_with_custom_scoring() + +# Made with Bob diff --git a/examples/dimensions_usage.py b/examples/dimensions_usage.py index c4d9e4c..4b5a008 100644 --- a/examples/dimensions_usage.py +++ b/examples/dimensions_usage.py @@ -29,103 +29,103 @@ def main(): """Main function demonstrating DimensionsProvider usage.""" - + print("=" * 70) print("DimensionsProvider - Usage Examples") print("=" * 70) - + # Step 1: Configure the provider with your instance URL and authentication token config = ProviderConfig( url="https://your-instance.cloud.ibm.com", auth_token="Bearer your-auth-token-here" ) - + print("\nConfiguration:") print(f" URL: {config.url}") print(f" Auth Token: {config.auth_token[:50]}...") - + # Step 2: Create a DimensionsProvider instance dimension_provider = DimensionsProvider(config) print("\n✓ DimensionsProvider initialized") - + # ========================================================================= # Example 1: Get Completeness dimension # ========================================================================= print("\n" + "=" * 70) print("Example 1: Get Completeness Dimension") print("=" * 70) - + try: dimension_name = "Completeness" dimension_id = dimension_provider.search_dimension(dimension_name) - + print(f"\n✓ Successfully retrieved dimension: {dimension_name}") print(f" Dimension ID: {dimension_id}") - + except ValueError as e: print(f"\n✗ Error getting dimension: {e}") - + # ========================================================================= # Example 2: Get Accuracy dimension # ========================================================================= print("\n" + "=" * 70) print("Example 2: Get Accuracy Dimension") print("=" * 70) - + try: dimension_name = "Accuracy" dimension_id = dimension_provider.search_dimension(dimension_name) - + print(f"\n✓ Successfully retrieved dimension: {dimension_name}") print(f" Dimension ID: {dimension_id}") - + except ValueError as e: print(f"\n✗ Error getting dimension: {e}") - + # ========================================================================= # Example 3: Get Consistency dimension # ========================================================================= print("\n" + "=" * 70) print("Example 3: Get Consistency Dimension") print("=" * 70) - + try: dimension_name = "Consistency" dimension_id = dimension_provider.search_dimension(dimension_name) - + print(f"\n✓ Successfully retrieved dimension: {dimension_name}") print(f" Dimension ID: {dimension_id}") - + except ValueError as e: print(f"\n✗ Error getting dimension: {e}") - + # ========================================================================= # Example 4: Test case-insensitive matching # ========================================================================= print("\n" + "=" * 70) print("Example 4: Case-Insensitive Matching (lowercase 'completeness')") print("=" * 70) - + try: dimension_name = "completeness" # lowercase dimension_id = dimension_provider.search_dimension(dimension_name) - + print(f"\n✓ Successfully retrieved dimension with lowercase name") print(f" Dimension Name: {dimension_name}") print(f" Dimension ID: {dimension_id}") - + except ValueError as e: print(f"\n✗ Error getting dimension: {e}") - + # ========================================================================= # Example 5: Get multiple dimensions in a loop # ========================================================================= print("\n" + "=" * 70) print("Example 5: Get Multiple Dimensions") print("=" * 70) - + dimension_names = ["Completeness", "Accuracy", "Consistency"] - + print("\nRetrieving multiple dimensions:") for idx, dim_name in enumerate(dimension_names, 1): try: @@ -133,25 +133,25 @@ def main(): print(f"{idx}. ✓ {dim_name}: {dim_id}") except ValueError as e: print(f"{idx}. ✗ {dim_name}: Failed - {e}") - + # ========================================================================= # Example 6: Try to get a non-existent dimension (error handling) # ========================================================================= print("\n" + "=" * 70) print("Example 6: Error Handling - Non-existent Dimension") print("=" * 70) - + try: dimension_name = "NonExistentDimension" dimension_id = dimension_provider.search_dimension(dimension_name) - + print(f"\n✓ Retrieved dimension: {dimension_name}") print(f" Dimension ID: {dimension_id}") - + except ValueError as e: print(f"\n✗ Expected error for non-existent dimension:") print(f" Error: {e}") - + print("\n" + "=" * 70) print("Examples Complete!") print("=" * 70) diff --git a/examples/dq_workflow_usage.py b/examples/dq_workflow_usage.py index 85cd414..443ba30 100644 --- a/examples/dq_workflow_usage.py +++ b/examples/dq_workflow_usage.py @@ -39,51 +39,51 @@ def main(): catalog_id = None # Alternative: use catalog_id instead of project_id column_name = "RTN" # Column to check check_type = "format" # Type of check to find - + config = ProviderConfig( url="https://cpd-ikc.apps.dqdev.ibm.com", auth_token="Bearer your-token-here", project_id=project_id ) - + # Initialize providers cams_provider = CamsProvider(config) dq_search = DQSearchProvider(config) issues_provider = IssuesProvider(config) - + # Step 1: Fetch CAMS asset by data_asset_id and project_id print("=" * 70) print("STEP 1: Fetch CAMS Asset") print("=" * 70) - + try: cams_asset = cams_provider.get_asset_by_id( asset_id=data_asset_id, options={"hide_deprecated_response_fields": "false"} ) - + print(f"✓ Fetched CAMS Asset:") print(f" Asset ID: {cams_asset.metadata.asset_id}") print(f" Name: {cams_asset.metadata.name}") print(f" Asset Type: {cams_asset.metadata.asset_type}") - + except ValueError as e: print(f"✗ Error fetching CAMS asset: {e}") return - + # Step 2: Read the CAMS object to get check_id from column_info # Iterate through column_checks and find the first check with matching type print("\n" + "=" * 70) print("STEP 2: Extract Check ID from CAMS Asset") print("=" * 70) - + check_id = None - + try: # Navigate to column_info for the specified column if hasattr(cams_asset.entity, 'column_info') and column_name in cams_asset.entity.column_info: column_info = cams_asset.entity.column_info[column_name] - + # Iterate through column_checks to find the check with matching type if hasattr(column_info, 'column_checks') and column_info.column_checks: for check in column_info.column_checks: @@ -93,7 +93,7 @@ def main(): print(f" Check ID: {check_id}") print(f" Check Type: {check.metadata.type}") break - + if not check_id: print(f"✗ No check with type '{check_type}' found for column '{column_name}'") return @@ -103,19 +103,19 @@ def main(): else: print(f"✗ Column '{column_name}' not found in asset") return - + except Exception as e: print(f"✗ Error extracting check ID: {e}") return - + # Step 3: Search for DQ check by native_id # The native_id format is: / print("\n" + "=" * 70) print("STEP 3: Search for DQ Check") print("=" * 70) - + check_native_id = f"{data_asset_id}/{check_id}" - + try: check_result = dq_search.search_dq_check( native_id=check_native_id, @@ -123,28 +123,28 @@ def main(): project_id=project_id, include_children=True ) - + print(f"✓ Found DQ Check:") print(f" ID: {check_result['id']}") print(f" Name: {check_result['name']}") print(f" Type: {check_result['type']}") print(f" Native ID: {check_result['native_id']}") - + dq_check_id = check_result['id'] - + except ValueError as e: print(f"✗ Error searching for check: {e}") return - + # Step 4: Search for DQ asset by native_id # The native_id format is: / print("\n" + "=" * 70) print("STEP 4: Search for DQ Asset") print("=" * 70) - + asset_native_id = f"{data_asset_id}/{column_name}" asset_type = "column" - + try: asset_result = dq_search.search_dq_asset( native_id=asset_native_id, @@ -153,24 +153,24 @@ def main(): include_children=True, get_actual_asset=False ) - + print(f"✓ Found DQ Asset:") print(f" ID: {asset_result['id']}") print(f" Name: {asset_result['name']}") print(f" Type: {asset_result['type']}") print(f" Native ID: {asset_result['native_id']}") - + dq_asset_id = asset_result['id'] - + except ValueError as e: print(f"✗ Error searching for asset: {e}") return - + # Step 5: Search issue for a specific asset and check print("\n" + "=" * 70) print("STEP 5: Search Issue for Asset and Check") print("=" * 70) - + try: # Use either project_id or catalog_id issue_id = issues_provider.get_issue_id( @@ -178,19 +178,19 @@ def main(): check_id=dq_check_id, project_id=project_id # Or use catalog_id=catalog_id ) - + print(f"✓ Found Issue:") print(f" Issue ID: {issue_id}") - + except ValueError as e: print(f"✗ Error searching for issue: {e}") return - + # Step 6: Update (patch) the issue by issue_id print("\n" + "=" * 70) print("STEP 6: Update Issue (Patch)") print("=" * 70) - + try: # Update both occurrences and tested records in a single call new_occurrences = 100 @@ -205,16 +205,16 @@ def main(): ) print(f"✓ Updated occurrences: {update_result.get('number_of_occurrences', 'N/A')}") print(f"✓ Updated tested records: {update_result.get('number_of_tested_records', 'N/A')}") - + # The percent_occurrences is typically calculated automatically by the system # based on number_of_occurrences and number_of_tested_records if 'percent_occurrences' in update_result: print(f"✓ Calculated percent occurrences: {update_result['percent_occurrences']}%") - + except ValueError as e: print(f"✗ Error updating issue: {e}") return - + print("\n" + "=" * 70) print("WORKFLOW COMPLETED SUCCESSFULLY") print("=" * 70) @@ -228,16 +228,16 @@ def example_with_add_operations(): url="https://cpd-ikc.apps.dqdev.ibm.com", auth_token="Bearer your-token-here" ) - + issues_provider = IssuesProvider(config) issue_id = "b8f4252b-cd35-4668-9b35-4635bfc6e2e0" - + print("=" * 70) print("EXAMPLE: Add Operations") print("=" * 70) - + project_id = "your-project-id-here" - + try: # Add 10 more occurrences and 50 more tested records to the existing counts print("\nAdding 10 occurrences and 50 tested records to existing counts...") @@ -250,7 +250,7 @@ def example_with_add_operations(): ) print(f"✓ New occurrences count: {result.get('number_of_occurrences', 'N/A')}") print(f"✓ New tested records count: {result.get('number_of_tested_records', 'N/A')}") - + except ValueError as e: print(f"✗ Error: {e}") @@ -269,10 +269,10 @@ def example_with_add_operations(): print("6. Update (patch) the issue by issue_id") print("\nNote: All methods support either project_id OR catalog_id (but not both)") print("\n" + "=" * 70 + "\n") - + # Run the main workflow main() - + # Uncomment to run additional examples: # print("\n\n") # example_with_add_operations() diff --git a/examples/end_to_end_example.py b/examples/end_to_end_example.py new file mode 100644 index 0000000..574a2c7 --- /dev/null +++ b/examples/end_to_end_example.py @@ -0,0 +1,404 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +End-to-End Data Quality Validation Example + +This comprehensive example demonstrates the complete workflow: +1. Generate authentication token for IBM Cloud environment +2. Load asset metadata from CAMS API using DQAssetsProvider +3. Load data from a CSV file using Pandas DataFrame +4. Add additional validation rules (e.g., range check) +5. Run validation on the DataFrame +6. Print and analyze the results + +Prerequisites: +- IBM Cloud API key +- Project ID or Catalog ID +- Asset ID from CAMS +- CSV file matching the asset metadata +- pandas library installed +""" + +import pandas as pd +from wxdi.common.auth import AuthConfig, EnvironmentType, AuthProvider +from wxdi.dq_validator.provider import ProviderConfig +from wxdi.dq_validator.provider.cams import CamsProvider +from wxdi.dq_validator.rule_loader import RuleLoader +from wxdi.dq_validator import ValidationRule, RangeCheck, ComparisonCheck, ComparisonOperator +from wxdi.dq_validator.integrations import PandasValidator + + +def main(): + """ + Main function demonstrating end-to-end data quality validation workflow. + """ + print("=" * 80) + print("END-TO-END DATA QUALITY VALIDATION EXAMPLE") + print("=" * 80) + + # ========================================================================= + # Step 1: Generate Authentication Token for IBM Cloud + # ========================================================================= + print("\n" + "=" * 80) + print("STEP 1: Generate Authentication Token") + print("=" * 80) + + # Configure authentication for IBM Cloud environment + # Replace with your actual IBM Cloud API key + auth_config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='your-ibm-cloud-api-key-here', + url='https://iam.test.cloud.ibm.com', # Optional, default URL is used if not provided + + # url is optional for production IBM Cloud (default URL is used) + # disable_ssl_verification=True # Optional, default is True + ) + + print(f"\nEnvironment Type: {auth_config.environment_type.value}") + print(f"Authentication URL: {auth_config.url}") + + # Create auth provider and get token + auth_provider = AuthProvider(auth_config) + print(f"Authenticator Type: {type(auth_provider.authenticator).__name__}") + + # Get the authentication token + # Uncomment the following lines when you have valid credentials + # try: + # auth_token = auth_provider.get_token() + # print(f"✓ Token generated successfully: {auth_token[:20]}...") + # except Exception as e: + # print(f"✗ Error generating token: {e}") + # return + + # For demonstration purposes, using a placeholder token + auth_token = "Bearer your-auth-token-here" + print(f"✓ Using token: {auth_token[:30]}...") + + # ========================================================================= + # Step 2: Load Asset Metadata from CAMS API + # ========================================================================= + print("\n" + "=" * 80) + print("STEP 2: Load Asset Metadata from CAMS") + print("=" * 80) + + # Configure the provider with your instance URL and authentication + base_url = "https://your-instance.cloud.ibm.com" + project_id = "your-project-id-here" # Or use catalog_id instead + asset_id = "your-asset-id-here" + + print(f"\nConfiguration:") + print(f" Base URL: {base_url}") + print(f" Project ID: {project_id}") + print(f" Asset ID: {asset_id}") + + # Create provider configuration + provider_config = ProviderConfig( + url=base_url, + auth_token=auth_token, + project_id=project_id + ) + + # Create CAMS provider + cams_provider = CamsProvider(provider_config) + print(f"\n✓ CAMS Provider initialized") + + # Load asset metadata and validation rules from CAMS + # Uncomment the following lines when you have valid credentials + # try: + # data_asset = cams_provider.get_asset_by_id(asset_id) + # print(f"✓ Asset loaded: {data_asset.metadata.name}") + # print(f" Asset Type: {data_asset.metadata.asset_type}") + # print(f" Columns: {len(data_asset.entity.data_asset.columns)}") + # + # # Use RuleLoader to extract metadata and validation rules + # rule_loader = RuleLoader(base_url, auth_token) + # validator = rule_loader.load_from_data_asset(data_asset) + # print(f"✓ Validator created with {len(validator.rules)} rules from CAMS") + # except Exception as e: + # print(f"✗ Error loading asset: {e}") + # return + + # For demonstration, create sample metadata manually + from wxdi.dq_validator import AssetMetadata, ColumnMetadata, DataType, Validator + + metadata = AssetMetadata( + table_name='customer_data', + columns=[ + ColumnMetadata('customer_id', DataType.INTEGER), + ColumnMetadata('name', DataType.STRING, length=100), + ColumnMetadata('email', DataType.STRING, length=255), + ColumnMetadata('age', DataType.INTEGER), + ColumnMetadata('account_balance', DataType.DECIMAL, precision=10, scale=2), + ColumnMetadata('registration_date', DataType.DATE), + ColumnMetadata('status', DataType.STRING, length=20), + ] + ) + + print(f"\n✓ Using sample metadata:") + print(f" Table: {metadata.table_name}") + print(f" Columns: {len(metadata.columns)}") + for col in metadata.columns: + print(f" - {col.name} ({col.data_type.value})") + + # Create validator with metadata + validator = Validator(metadata) + print(f"\n✓ Validator initialized") + + # ========================================================================= + # Step 3: Load Data from CSV File using Pandas + # ========================================================================= + print("\n" + "=" * 80) + print("STEP 3: Load Data from CSV File") + print("=" * 80) + + # For demonstration, create sample data that matches the metadata + # In production, you would load from an actual CSV file: + # df = pd.read_csv('customer_data.csv') + + sample_data = { + 'customer_id': [1001, 1002, 1003, 1004, 1005, 1006], + 'name': ['John Doe', 'Jane Smith', 'Bob Johnson', 'Alice Williams', 'Charlie Brown', 'Diana Prince'], + 'email': ['john@example.com', 'jane@example.com', 'bob@invalid', 'alice@example.com', 'charlie@example.com', 'diana@example.com'], + 'age': [25, 30, 17, 45, 35, 28], # Note: 17 is below typical minimum age + 'account_balance': [1500.50, 2500.75, 500.00, 5000.00, 3500.25, 1200.00], + 'registration_date': ['2023-01-15', '2023-02-20', '2023-03-10', '2023-04-05', '2023-05-12', '2023-06-18'], + 'status': ['active', 'active', 'pending', 'active', 'inactive', 'active'] + } + + df = pd.DataFrame(sample_data) + + print(f"\n✓ Data loaded from CSV (sample data)") + print(f" Rows: {len(df)}") + print(f" Columns: {list(df.columns)}") + print(f"\nFirst 3 rows:") + print(df.head(3).to_string(index=False)) + + # ========================================================================= + # Step 4: Add Additional Validation Rules + # ========================================================================= + print("\n" + "=" * 80) + print("STEP 4: Add Additional Validation Rules") + print("=" * 80) + + # Add range check for age (must be between 18 and 100) + age_rule = ValidationRule('age') + age_rule.add_check(RangeCheck(min_value=18, max_value=100)) + validator.add_rule(age_rule) + print(f"\n✓ Added range check for 'age' column (18-100)") + + # Add range check for account_balance (must be >= 1000) + balance_rule = ValidationRule('account_balance') + balance_rule.add_check(ComparisonCheck( + operator=ComparisonOperator.GREATER_THAN_OR_EQUAL, + target_value=1000.00 + )) + validator.add_rule(balance_rule) + print(f"✓ Added comparison check for 'account_balance' column (>= 1000)") + + # Add valid values check for status + from wxdi.dq_validator import ValidValuesCheck + status_rule = ValidationRule('status') + status_rule.add_check(ValidValuesCheck( + valid_values=['active', 'inactive', 'suspended'], + case_sensitive=False + )) + validator.add_rule(status_rule) + print(f"✓ Added valid values check for 'status' column") + + print(f"\n✓ Total validation rules: {len(validator.rules)}") + for rule in validator.rules: + print(f" - {rule.column_name}: {len(rule.checks)} check(s)") + + # ========================================================================= + # Step 5: Run Validation on DataFrame + # ========================================================================= + print("\n" + "=" * 80) + print("STEP 5: Run Validation") + print("=" * 80) + + # Create Pandas validator + pandas_validator = PandasValidator(validator, chunk_size=1000) + print(f"\n✓ Pandas Validator created (chunk_size=1000)") + + # Get summary statistics + print(f"\nRunning validation...") + summary = pandas_validator.get_summary_statistics(df) + + print(f"\n✓ Validation completed!") + + # ========================================================================= + # Step 6: Print and Analyze Results + # ========================================================================= + print("\n" + "=" * 80) + print("STEP 6: Validation Results") + print("=" * 80) + + # Print summary statistics + print(f"\n{'SUMMARY STATISTICS':^80}") + print("-" * 80) + print(f"Total Rows: {summary['total_rows']:>10}") + print(f"Valid Rows: {summary['valid_rows']:>10}") + print(f"Invalid Rows: {summary['invalid_rows']:>10}") + print(f"Pass Rate: {summary['pass_rate']:>9.2f}%") + print("-" * 80) + print(f"Total Checks: {summary['total_checks']:>10}") + print(f"Passed Checks: {summary['passed_checks']:>10}") + print(f"Failed Checks: {summary['failed_checks']:>10}") + print("-" * 80) + + # Add validation column to DataFrame + df_validated = pandas_validator.add_validation_column(df) + + # Print detailed results for each row + print(f"\n{'DETAILED VALIDATION RESULTS':^80}") + print("-" * 80) + + for idx, row in df_validated.iterrows(): + validation = row['dq_validation_result'] + is_valid = bool(validation['is_valid']) + status_icon = "✓" if is_valid else "✗" + status_text = "PASS" if is_valid else "FAIL" + row_num = idx + 1 if isinstance(idx, int) else idx # type: ignore[operator] + + print(f"\nRow {row_num}: {status_icon} {status_text}") + print(f" Customer: {row['name']} (ID: {row['customer_id']})") + print(f" Age: {row['age']}, Balance: ${row['account_balance']:.2f}, Status: {row['status']}") + print(f" Validation Score: {validation['score']} ({validation['pass_rate']:.1f}%)") + + if not is_valid: + print(f" Errors: {validation['error_count']}") + errors = validation['errors'] + if errors is not None and len(errors) > 0: + for error in validation['errors'][:3]: # Show first 3 errors + print(f" - {error['column']}: {error['message']}") + + # Get and display invalid rows + print(f"\n{'INVALID ROWS DETAILS':^80}") + print("-" * 80) + + invalid_df = pandas_validator.get_invalid_rows(df) + + if len(invalid_df) > 0: + print(f"\nFound {len(invalid_df)} invalid row(s):\n") + + for idx, row in invalid_df.iterrows(): + validation = row['dq_validation_result'] + row_num = idx + 1 if isinstance(idx, int) else idx # type: ignore[operator] + print(f"Row {row_num}:") + print(f" Customer ID: {row['customer_id']}") + print(f" Name: {row['name']}") + print(f" Age: {row['age']}") + print(f" Balance: ${row['account_balance']:.2f}") + print(f" Status: {row['status']}") + print(f" Validation Score: {validation['score']} ({validation['pass_rate']:.1f}%)") + print(f" Failed Checks: {validation['error_count']}") + + errors = validation['errors'] + if errors is not None and len(errors) > 0: + print(f" Errors:") + for error in validation['errors']: + print(f" - Column '{error['column']}': {error['message']}") + if error.get('value') is not None: + print(f" Value: {error['value']}") + if error.get('expected') is not None: + print(f" Expected: {error['expected']}") + print() + else: + print("\n✓ All rows passed validation!") + + # Expand validation columns for analysis + print(f"\n{'EXPANDED VALIDATION COLUMNS':^80}") + print("-" * 80) + + df_expanded = pandas_validator.expand_validation_column(df_validated) + + # Show key validation columns + validation_cols = ['customer_id', 'name', 'dq_is_valid', 'dq_score', 'dq_pass_rate', 'dq_error_count'] + print(f"\nValidation Summary by Row:") + print(df_expanded[validation_cols].to_string(index=False)) + + # Save results to files + print(f"\n{'SAVING RESULTS':^80}") + print("-" * 80) + + # Save invalid rows + if len(invalid_df) > 0: + invalid_df.to_csv('invalid_customers.csv', index=False) + print(f"\n✓ Saved {len(invalid_df)} invalid rows to: invalid_customers.csv") + + # Save full validation results + df_expanded.to_csv('validation_results.csv', index=False) + print(f"✓ Saved full validation results to: validation_results.csv") + + # Save summary report + with open('validation_summary.txt', 'w') as f: + f.write("=" * 80 + "\n") + f.write("DATA QUALITY VALIDATION SUMMARY REPORT\n") + f.write("=" * 80 + "\n\n") + f.write(f"Asset: {metadata.table_name}\n") + f.write(f"Total Rows: {summary['total_rows']}\n") + f.write(f"Valid Rows: {summary['valid_rows']}\n") + f.write(f"Invalid Rows: {summary['invalid_rows']}\n") + f.write(f"Pass Rate: {summary['pass_rate']:.2f}%\n\n") + f.write(f"Total Checks: {summary['total_checks']}\n") + f.write(f"Passed Checks: {summary['passed_checks']}\n") + f.write(f"Failed Checks: {summary['failed_checks']}\n\n") + f.write("Validation Rules Applied:\n") + for rule in validator.rules: + f.write(f" - {rule.column_name}: {len(rule.checks)} check(s)\n") + + print(f"✓ Saved summary report to: validation_summary.txt") + + # ========================================================================= + # Completion + # ========================================================================= + print("\n" + "=" * 80) + print("END-TO-END VALIDATION COMPLETE!") + print("=" * 80) + + print(f"\nKey Takeaways:") + print(f" • Authenticated with IBM Cloud using API key") + print(f" • Loaded asset metadata from CAMS (or created sample)") + print(f" • Loaded {len(df)} rows from CSV file") + print(f" • Applied {len(validator.rules)} validation rules") + print(f" • Validated data with {summary['pass_rate']:.1f}% pass rate") + print(f" • Identified {summary['invalid_rows']} invalid row(s)") + print(f" • Saved results to CSV files for further analysis") + + print(f"\nNext Steps:") + print(f" 1. Review invalid_customers.csv for data quality issues") + print(f" 2. Analyze validation_results.csv for detailed insights") + print(f" 3. Update source data or adjust validation rules as needed") + print(f" 4. Re-run validation to verify improvements") + + print("\n" + "=" * 80) + + +if __name__ == '__main__': + try: + main() + except ImportError as e: + print(f"Error: {e}") + print("\nTo run this example, install required dependencies:") + print(" pip install pandas") + print("Or install with all dependencies:") + print(" pip install wxdi[pandas]") + except Exception as e: + print(f"\nUnexpected error: {e}") + import traceback + traceback.print_exc() + +# Made with Bob diff --git a/examples/glossary_usage.py b/examples/glossary_usage.py index 492f115..e51a8e3 100644 --- a/examples/glossary_usage.py +++ b/examples/glossary_usage.py @@ -42,73 +42,73 @@ def main(): print("=" * 70) print("GlossaryProvider - Usage Examples") print("=" * 70) - + # ========================================================================= # Configuration Option 1: Using static auth token # ========================================================================= print("\n" + "-" * 70) print("Configuration Option 1: Using Static Auth Token") print("-" * 70) - + config_with_token = ProviderConfig( url="https://your-instance.cloud.ibm.com", auth_token="Bearer your-auth-token-here" ) - + print("\nConfiguration:") print(f" URL: {config_with_token.url}") print(f" Auth Token: {config_with_token.auth_token[:50]}...") - + # ========================================================================= # Configuration Option 2: Using AuthConfig (recommended) # ========================================================================= print("\n" + "-" * 70) print("Configuration Option 2: Using AuthConfig (Recommended)") print("-" * 70) - + # Example with IBM Cloud auth_config = AuthConfig( environment_type=EnvironmentType.IBM_CLOUD, api_key="your-api-key-here" ) - + config_with_auth = ProviderConfig( url="https://your-instance.cloud.ibm.com", auth_config=auth_config ) - + print("\nConfiguration:") print(f" URL: {config_with_auth.url}") print(f" Environment: {auth_config.environment_type.value}") print( " Auth Provider: Configured") - + # For the examples below, we'll use config_with_token # In production, use config_with_auth for automatic token management config = config_with_token - + # Step 2: Create a GlossaryProvider instance glossary_provider = GlossaryProvider(config) print("\n✓ GlossaryProvider initialized") - + # Define IDs that will be used throughout the examples artifact_id = "your-artifact-id-here" # Replace with your actual artifact ID version_id = "your-version-id-here" # Replace with your actual version ID - + # ========================================================================= # Example 1: Get published glossary term by artifact ID # ========================================================================= print("\n" + "=" * 70) print("Example 1: Get Published Glossary Term by Artifact ID") print("=" * 70) - + try: print(PARAMETERS_HEADER) print(f" artifact_id: {artifact_id}") - + term = glossary_provider.get_published_artifact_by_id(artifact_id) - + print( "\n✓ Successfully retrieved glossary term") - + print(TERM_DETAILS_HEADER) print(f" Name: {term.metadata.name}") print(f" Artifact Type: {term.metadata.artifact_type}") @@ -117,73 +117,73 @@ def main(): print(f" Version ID: {term.metadata.version_id}") print(f" Created At: {term.metadata.created_at}") print(f" Modified At: {term.metadata.modified_at}") - + if term.metadata.tags: print(f" Tags: {', '.join(term.metadata.tags)}") - + # Display entity information if available if hasattr(term.entity, 'extended_attribute_groups') and term.entity.extended_attribute_groups: dq_constraints = term.entity.extended_attribute_groups.dq_constraints print(f" Data Quality Constraints: {len(dq_constraints)}") - + except ValueError as e: print(f"\n✗ Error: {e}") except Exception as e: print(f"\n✗ Unexpected error: {e}") - + # ========================================================================= # Example 2: Get published term with additional options # ========================================================================= print("\n" + "=" * 70) print("Example 2: Get Published Term with Additional Options") print("=" * 70) - + try: print(PARAMETERS_HEADER) print(f" artifact_id: {artifact_id}") - + # Options can include query parameters like 'include', 'limit', etc. options = { - "include": "relationships" + "include_relationship": "all" } print(f" options: {options}") - + term = glossary_provider.get_published_artifact_by_id( artifact_id, options=options ) - + print( "\n✓ Successfully retrieved glossary term with options") - + print(TERM_DETAILS_HEADER) print(f" Name: {term.metadata.name}") print(f" State: {term.metadata.state}") print(f" Revision: {term.metadata.revision}") - + except ValueError as e: print(f"\n✗ Error: {e}") except Exception as e: print(f"\n✗ Unexpected error: {e}") - + # ========================================================================= # Example 3: Get specific version of a glossary term # ========================================================================= print("\n" + "=" * 70) print("Example 3: Get Specific Version of a Glossary Term") print("=" * 70) - + try: print(PARAMETERS_HEADER) print(f" artifact_id: {artifact_id}") print(f" version_id: {version_id}") - + term = glossary_provider.get_term_by_version_id( artifact_id, version_id ) - + print( "\n✓ Successfully retrieved glossary term version") - + print(TERM_DETAILS_HEADER) print(f" Name: {term.metadata.name}") print(f" Version ID: {term.metadata.version_id}") @@ -191,58 +191,58 @@ def main(): print(f" Effective Start Date: {term.metadata.effective_start_date}") print(f" Created By: {term.metadata.created_by}") print(f" Modified By: {term.metadata.modified_by}") - + if term.metadata.draft_ancestor_id: print(f" Draft Ancestor ID: {term.metadata.draft_ancestor_id}") - + except ValueError as e: print(f"\n✗ Error: {e}") except Exception as e: print(f"\n✗ Unexpected error: {e}") - + # ========================================================================= # Example 4: Get term version with options # ========================================================================= print("\n" + "=" * 70) print("Example 4: Get Term Version with Options") print("=" * 70) - + try: print(PARAMETERS_HEADER) print(f" artifact_id: {artifact_id}") print(f" version_id: {version_id}") - + options = { - "include": "all" + "all_parents": "true" } print(f" options: {options}") - + term = glossary_provider.get_term_by_version_id( artifact_id, version_id, options=options ) - + print( "\n✓ Successfully retrieved glossary term version with options") - + print(TERM_DETAILS_HEADER) print(f" Name: {term.metadata.name}") print(f" Version ID: {term.metadata.version_id}") print(f" State: {term.metadata.state}") print(f" Global ID: {term.metadata.global_id}") print(f" Source Repository ID: {term.metadata.source_repository_id}") - + # Display steward information if available if term.metadata.steward_ids: print(f" Steward IDs: {', '.join(term.metadata.steward_ids)}") if term.metadata.steward_group_ids: print(f" Steward Group IDs: {', '.join(term.metadata.steward_group_ids)}") - + except ValueError as e: print(f"\n✗ Error: {e}") except Exception as e: print(f"\n✗ Unexpected error: {e}") - + print("\n" + "=" * 70) print("Examples Complete!") print("=" * 70) diff --git a/examples/issues_usage.py b/examples/issues_usage.py index e86e587..9d4d321 100644 --- a/examples/issues_usage.py +++ b/examples/issues_usage.py @@ -16,7 +16,7 @@ """ Example usage of IssuesProvider for managing data quality issues. -This example demonstrates four main operations: +This example demonstrates five main operations: 1. get_issues(): Retrieve a specific issue for a DQ asset, check type, and check_id - Requires: dq_asset_id, check_type, check_id, and either project_id OR catalog_id @@ -28,17 +28,22 @@ - Optional: operation ("add" or "replace", default: "add") 3. update_issue_metrics(): Update issue metrics using CAMS asset and check IDs OR check_native_id - - Option A: Provide cams_asset_id and cams_check_id - - Option B: Provide check_native_id (format: "/") + - Option A: Provide asset_id and check_id + - Option B: Provide check_native_id (format: "/") - Also requires: occurrences, tested_records, column_name, check_type, and either project_id OR catalog_id - Optional: asset_type (default: "column"), operation (default: "add") - This method automatically searches for the DQ asset, check, and issue before updating 4. create_issue(): Create a new data quality issue for a check - - Requires: check_id, reported_for_id, number_of_occurrences, number_of_tested_records, and either project_id OR catalog_id + - Requires: dq_check_id, reported_for_id, number_of_occurrences, number_of_tested_records, and either project_id OR catalog_id - Optional: status (default: "actual"), ignored (default: False) - Returns: The created issue_id +5. create_issues_bulk(): Create multiple issues, assets, and checks in a single API call + - Requires: payload (dict with issues, assets, existing_checks), and either project_id OR catalog_id + - Optional: incremental_reporting (default: False), refresh_assets (default: False) + - Returns: The full API response with created issues + Both update methods require occurrences and tested_records as mandatory parameters. The operation parameter (default: "add") applies to both metrics. Either project_id or catalog_id must be provided (but not both). @@ -49,27 +54,27 @@ def main(): """Main function demonstrating IssuesProvider usage.""" - + # Step 1: Configure the provider with your instance URL and authentication token config = ProviderConfig( url="https://your-instance.cloud.ibm.com", auth_token="Bearer your-auth-token-here" ) - + # Step 2: Create an IssuesProvider instance issues_provider = IssuesProvider(config) - + # Define IDs that will be used throughout the examples issue_id = "your-issue-id-here" project_id = "your-project-id-here" catalog_id = "your-catalog-id-here" # Alternative to project_id - + # Step 3: Get issues for a specific DQ asset, check type, and check_id print("\n--- Getting issues for a DQ asset with check_id filter ---") dq_asset_id = "08b139ca-35a6-4b61-b87b-aa832870d89c" check_type = "format" check_id = "065c2b72-4600-4d15-8c48-298a2abf66cd" # The check ID to filter by - + try: # Get issues using catalog_id and check_id issue_result = issues_provider.get_issues( @@ -93,7 +98,7 @@ def main(): print(f"No issue found matching check_id {check_id}") except ValueError as e: print(f"Error getting issues: {e}") - + # Example: Get issues using project_id instead try: issue_result = issues_provider.get_issues( @@ -110,7 +115,7 @@ def main(): print(f"No matching issue found") except ValueError as e: print(f"Error getting issues with project_id: {e}") - + # Step 4: Update both occurrences and tested records using issue ID directly # Note: Both occurrences and tested_records are mandatory # Either project_id OR catalog_id must be provided (but not both) @@ -127,7 +132,7 @@ def main(): print(f"Response: {result}") except ValueError as e: print(f"Error updating issue: {e}") - + # Step 4: Use catalog_id instead of project_id try: result = issues_provider.update_issue_values( @@ -140,7 +145,7 @@ def main(): print(f"Response: {result}") except ValueError as e: print(f"Error updating issue with catalog_id: {e}") - + # Step 5: Use replace operation instead of add try: result = issues_provider.update_issue_values( @@ -154,7 +159,7 @@ def main(): print(f"Response: {result}") except ValueError as e: print(f"Error replacing metrics: {e}") - + # Example: Using replace operation with different values try: # Replace both metrics with specific values @@ -169,14 +174,14 @@ def main(): print(f"Response: {result}") except ValueError as e: print(f"Error resetting metrics: {e}") - + # Example: Update multiple issues with both metrics issues_to_update = [ ("issue-123", 10, 100), ("issue-456", 25, 250), ("issue-789", 50, 500), ] - + print("\n--- Updating multiple issues with both metrics ---") for issue_id, occurrences, records in issues_to_update: try: @@ -189,14 +194,14 @@ def main(): print(f"✓ Updated {issue_id}: +{occurrences} occurrences, +{records} tested records") except ValueError as e: print(f"✗ Failed to update {issue_id}: {e}") - + # Step 6: Update issue metrics using CAMS IDs print("\n--- Updating issue metrics using CAMS IDs ---") try: # Update issue metrics by providing CAMS asset and check IDs with project_id result = issues_provider.update_issue_metrics( - cams_asset_id="b2debda2-6ab9-4a39-8c23-17954e004dcf", - cams_check_id="7377e2cd-ac0e-4833-8760-fd0e8cb682aa", + asset_id="b2debda2-6ab9-4a39-8c23-17954e004dcf", + check_id="7377e2cd-ac0e-4833-8760-fd0e8cb682aa", occurrences=10, tested_records=100, column_name="RTN", # Required for column type assets @@ -208,12 +213,12 @@ def main(): print(f"Response: {result}") except ValueError as e: print(f"Error updating issue with CAMS IDs: {e}") - + # Example: Using catalog_id instead of project_id try: result = issues_provider.update_issue_metrics( - cams_asset_id="b2debda2-6ab9-4a39-8c23-17954e004dcf", - cams_check_id="7377e2cd-ac0e-4833-8760-fd0e8cb682aa", + asset_id="b2debda2-6ab9-4a39-8c23-17954e004dcf", + check_id="7377e2cd-ac0e-4833-8760-fd0e8cb682aa", occurrences=10, tested_records=100, column_name="RTN", @@ -225,7 +230,7 @@ def main(): print(f"Response: {result}") except ValueError as e: print(f"Error updating issue with CAMS IDs and catalog_id: {e}") - + # Example: Using check_native_id instead of cams_asset_id and cams_check_id print("\n--- Updating issue metrics using check_native_id ---") try: @@ -244,7 +249,7 @@ def main(): print(f"Response: {result}") except ValueError as e: print(f"Error updating issue with check_native_id: {e}") - + # Example: Using check_native_id for comparison check with target column try: # For comparison checks with target columns, the native_id includes the operator and target column @@ -262,12 +267,12 @@ def main(): print(f"Response: {result}") except ValueError as e: print(f"Error updating comparison check with check_native_id: {e}") - + # Example: Update multiple issues using CAMS IDs cams_updates = [ { - "cams_asset_id": "asset-1", - "cams_check_id": "check-1", + "asset_id": "asset-1", + "check_id": "check-1", "occurrences": 5, "tested_records": 50, "column_name": "test_column", @@ -276,8 +281,8 @@ def main(): "asset_type": "column" }, { - "cams_asset_id": "asset-2", - "cams_check_id": "check-2", + "asset_id": "asset-2", + "check_id": "check-2", "occurrences": 15, "tested_records": 150, "column_name": "another_column", @@ -286,15 +291,15 @@ def main(): "asset_type": "table" }, ] - + print("\n--- Updating multiple issues using CAMS IDs ---") for update_params in cams_updates: try: result = issues_provider.update_issue_metrics(**update_params) - print(f"✓ Updated issue for asset {update_params['cams_asset_id']}") + print(f"✓ Updated issue for asset {update_params['asset_id']}") except ValueError as e: - print(f"✗ Failed to update issue for asset {update_params['cams_asset_id']}: {e}") - + print(f"✗ Failed to update issue for asset {update_params['asset_id']}: {e}") + # Step 7: Create a new issue for a check (using project_id) print("\n--- Creating a new issue for a check (using project_id) ---") try: @@ -303,7 +308,7 @@ def main(): # reported_for_id: The ID of the DQ asset being reported on issue_check_id = "c3c97a92-8a45-4456-be6e-ab0d13b65a33" reported_for_id = "3f73a9d8-1664-482b-829f-9c879a4dd5d6" - + print(f"\nCreating issue with:") print(f" Check ID: {issue_check_id}") print(f" Reported For ID: {reported_for_id}") @@ -312,9 +317,9 @@ def main(): print(f" Status: actual") print(f" Ignored: False") print(f" Project ID: {project_id}") - + new_issue_id = issues_provider.create_issue( - check_id=issue_check_id, + dq_check_id=issue_check_id, reported_for_id=reported_for_id, number_of_occurrences=25, number_of_tested_records=1000, @@ -322,18 +327,18 @@ def main(): ignored=False, project_id=project_id ) - + print(f"\n✓ Successfully created issue!") print(f" New Issue ID: {new_issue_id}") - + except ValueError as e: print(f"\n✗ Error creating issue: {e}") - + # Step 8: Create issue with different parameters (using project_id) print("\n--- Creating issue with ignored=True (using project_id) ---") try: new_issue_id = issues_provider.create_issue( - check_id="c3c97a92-8a45-4456-be6e-ab0d13b65a33", + dq_check_id="c3c97a92-8a45-4456-be6e-ab0d13b65a33", reported_for_id="3f73a9d8-1664-482b-829f-9c879a4dd5d6", number_of_occurrences=100, number_of_tested_records=5000, @@ -341,21 +346,21 @@ def main(): ignored=True, # This issue is ignored project_id=project_id ) - + print(f"\n✓ Successfully created issue with ignored=True") print(f" New Issue ID: {new_issue_id}") print(f" Occurrences: 100") print(f" Tested Records: 5000") print(f" Ignored: True") - + except ValueError as e: print(f"\n✗ Error creating issue: {e}") - + # Step 9: Create issue using catalog_id print("\n--- Creating issue using catalog_id ---") try: new_issue_id = issues_provider.create_issue( - check_id="c3c97a92-8a45-4456-be6e-ab0d13b65a33", + dq_check_id="c3c97a92-8a45-4456-be6e-ab0d13b65a33", reported_for_id="3f73a9d8-1664-482b-829f-9c879a4dd5d6", number_of_occurrences=50, number_of_tested_records=2000, @@ -363,10 +368,117 @@ def main(): ignored=False, catalog_id=catalog_id ) - + print(f"\n✓ Successfully created issue using catalog_id") print(f" New Issue ID: {new_issue_id}") - + + except ValueError as e: + print(f"\n✗ Error: {e}") + + # Step 10: Create multiple issues in bulk (using project_id) + print("\n" + "=" * 70) + print("Creating Multiple Issues in Bulk") + print("=" * 70) + + # Construct the bulk payload with issues, assets, and existing_checks + bulk_payload = { + "issues": [ + { + "check": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/Validity", + "type": "format" + }, + "reported_for": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f", + "type": "data_asset" + }, + "number_of_occurrences": 200, + "number_of_tested_records": 1000, + "status": "aggregation", + "ignored": False + }, + { + "check": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/sample3", + "type": "format" + }, + "reported_for": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/NAME", + "type": "column" + }, + "number_of_occurrences": 200, + "number_of_tested_records": 1000, + "status": "actual", + "ignored": False + } + ], + "assets": [ + { + "name": "ACCOUNT_HOLDERS.csv", + "type": "data_asset", + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f", + "weight": 1 + }, + { + "name": "NAME", + "type": "column", + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/NAME", + "parent": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f", + "type": "data_asset" + }, + "weight": 1 + } + ], + "existing_checks": [ + { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/Validity", + "type": "format" + }, + { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/sample3", + "type": "format" + } + ] + } + + try: + print(f"\nCreating bulk issues with:") + print(f" Number of issues: {len(bulk_payload['issues'])}") + print(f" Number of assets: {len(bulk_payload['assets'])}") + print(f" Number of checks: {len(bulk_payload['existing_checks'])}") + print(f" Project ID: {project_id}") + print(f" Incremental Reporting: False") + print(f" Refresh Assets: False") + + response = issues_provider.create_issues_bulk( + payload=bulk_payload, + project_id=project_id, + incremental_reporting=False, + refresh_assets=False + ) + + print(f"\n✓ Successfully created issues in bulk!") + print(f" Response keys: {list(response.keys())}") + + except ValueError as e: + print(f"\n✗ Error creating bulk issues: {e}") + + # Step 11: Create bulk issues with incremental reporting (using catalog_id) + print("\n--- Creating bulk issues with incremental reporting (using catalog_id) ---") + try: + # Same payload structure but with incremental_reporting=True + response = issues_provider.create_issues_bulk( + payload=bulk_payload, + catalog_id=catalog_id, + incremental_reporting=True, # Adds archived counts to new issues + refresh_assets=False + ) + + print(f"\n✓ Successfully created bulk issues with incremental reporting!") + print(f" Using catalog_id: {catalog_id}") + print(f" Incremental reporting enabled") + except ValueError as e: print(f"\n✗ Error: {e}") diff --git a/examples/odcs_generator_example.py b/examples/odcs_generator_example.py new file mode 100644 index 0000000..1165ed2 --- /dev/null +++ b/examples/odcs_generator_example.py @@ -0,0 +1,217 @@ +#!/usr/bin/env python3 +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Example usage of ODCS Generator to create ODCS YAML files from Collibra assets. + +This example demonstrates: +1. Connecting to Collibra using credentials +2. Generating ODCS from a Collibra asset +3. Saving the ODCS to a YAML file +4. Using the generator programmatically +""" + +import os +import sys +import yaml +from wxdi.odcs_generator import CollibraClient, ODCSGenerator + + +def example_basic_usage(): + """Basic example: Generate ODCS from a Collibra asset""" + + # Get credentials from environment variables + collibra_url = os.getenv('COLLIBRA_URL') + username = os.getenv('COLLIBRA_USERNAME') + password = os.getenv('COLLIBRA_PASSWORD') + + if not all([collibra_url, username, password]): + print("Error: Please set COLLIBRA_URL, COLLIBRA_USERNAME, and COLLIBRA_PASSWORD environment variables") + sys.exit(1) + + # Replace with your actual asset ID + asset_id = "019a57f9-62d2-7aa0-9f22-4fa2cea1180b" + + print(f"Connecting to Collibra at {collibra_url}...") + + # Initialize Collibra client + client = CollibraClient( + base_url=collibra_url, + username=username, + password=password + ) + + # Create ODCS generator + generator = ODCSGenerator(client) + + print(f"Generating ODCS for asset: {asset_id}") + + # Generate ODCS from asset + odcs_data = generator.generate_odcs(asset_id) + + # Save to YAML file + output_file = f"{odcs_data.get('name', 'asset').lower().replace(' ', '-')}-odcs.yaml" + + print(f"Writing ODCS to {output_file}...") + with open(output_file, 'w') as f: + yaml.dump(odcs_data, f, default_flow_style=False, sort_keys=False, allow_unicode=True) + + print(f"✓ Successfully generated ODCS file: {output_file}") + + return odcs_data + + +def example_custom_processing(): + """Advanced example: Generate ODCS and perform custom processing""" + + # Get credentials from environment variables + collibra_url = os.getenv('COLLIBRA_URL') + username = os.getenv('COLLIBRA_USERNAME') + password = os.getenv('COLLIBRA_PASSWORD') + + if not all([collibra_url, username, password]): + print("Error: Please set COLLIBRA_URL, COLLIBRA_USERNAME, and COLLIBRA_PASSWORD environment variables") + sys.exit(1) + + # Replace with your actual asset ID + asset_id = "019a57f9-62d2-7aa0-9f22-4fa2cea1180b" + + # Initialize client and generator + client = CollibraClient(base_url=collibra_url, username=username, password=password) + generator = ODCSGenerator(client) + + # Generate ODCS + odcs_data = generator.generate_odcs(asset_id) + + # Custom processing: Update contract metadata + odcs_data['dataProduct'] = 'My Custom Data Product' + odcs_data['version'] = '2.0.0' + odcs_data['name'] = 'Custom Contract Name' + + # Custom processing: Add quality rules + if 'schema' in odcs_data and len(odcs_data['schema']) > 0: + schema = odcs_data['schema'][0] + schema['quality'] = [ + { + 'type': 'completeness', + 'dimension': 'completeness', + 'mustBe': 100, + 'mustNotBe': None + } + ] + + # Custom processing: Update server configuration + if 'servers' in odcs_data and len(odcs_data['servers']) > 0: + server = odcs_data['servers'][0] + server['server'] = 'prod.snowflake.mycompany.com' + server['type'] = 'snowflake' + server['account'] = 'my_account' + server['database'] = 'my_database' + server['schema'] = 'my_schema' + + # Save customized ODCS + output_file = 'custom-odcs.yaml' + with open(output_file, 'w') as f: + yaml.dump(odcs_data, f, default_flow_style=False, sort_keys=False, allow_unicode=True) + + print(f"✓ Successfully generated customized ODCS file: {output_file}") + + return odcs_data + + +def example_batch_processing(): + """Example: Generate ODCS for multiple assets""" + + # Get credentials from environment variables + collibra_url = os.getenv('COLLIBRA_URL') + username = os.getenv('COLLIBRA_USERNAME') + password = os.getenv('COLLIBRA_PASSWORD') + + if not all([collibra_url, username, password]): + print("Error: Please set COLLIBRA_URL, COLLIBRA_USERNAME, and COLLIBRA_PASSWORD environment variables") + sys.exit(1) + + # List of asset IDs to process + asset_ids = [ + "019a57f9-62d2-7aa0-9f22-4fa2cea1180b", + # Add more asset IDs here + ] + + # Initialize client and generator once + client = CollibraClient(base_url=collibra_url, username=username, password=password) + generator = ODCSGenerator(client) + + results = [] + + for asset_id in asset_ids: + try: + print(f"\nProcessing asset: {asset_id}") + odcs_data = generator.generate_odcs(asset_id) + + # Save to file + output_file = f"{odcs_data.get('name', asset_id).lower().replace(' ', '-')}-odcs.yaml" + with open(output_file, 'w') as f: + yaml.dump(odcs_data, f, default_flow_style=False, sort_keys=False, allow_unicode=True) + + results.append({'asset_id': asset_id, 'status': 'success', 'file': output_file}) + print(f"✓ Generated: {output_file}") + + except Exception as e: + results.append({'asset_id': asset_id, 'status': 'failed', 'error': str(e)}) + print(f"✗ Failed: {str(e)}") + + # Print summary + print("\n=== Batch Processing Summary ===") + success_count = sum(1 for r in results if r['status'] == 'success') + failed_count = sum(1 for r in results if r['status'] == 'failed') + print(f"Total: {len(results)}, Success: {success_count}, Failed: {failed_count}") + + return results + + +if __name__ == '__main__': + print("=== ODCS Generator Examples ===\n") + + # Run basic example + print("1. Basic Usage Example") + print("-" * 50) + try: + example_basic_usage() + except Exception as e: + print(f"Error: {str(e)}") + + print("\n" + "=" * 50 + "\n") + + # Uncomment to run other examples: + + # print("2. Custom Processing Example") + # print("-" * 50) + # try: + # example_custom_processing() + # except Exception as e: + # print(f"Error: {str(e)}") + + # print("\n" + "=" * 50 + "\n") + + # print("3. Batch Processing Example") + # print("-" * 50) + # try: + # example_batch_processing() + # except Exception as e: + # print(f"Error: {str(e)}") + +# Made with Bob diff --git a/examples/odcs_generator_informatica_example.py b/examples/odcs_generator_informatica_example.py new file mode 100644 index 0000000..73b2905 --- /dev/null +++ b/examples/odcs_generator_informatica_example.py @@ -0,0 +1,415 @@ +#!/usr/bin/env python3 +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Example usage of ODCS Generator to create ODCS YAML files from Informatica CDGC assets. + +This example demonstrates: +1. Connecting to Informatica CDGC using credentials +2. Generating ODCS from an Informatica asset +3. Saving the ODCS to a YAML file +4. Using the generator programmatically +5. Batch processing multiple assets +""" + +import os +import sys +import yaml +from concurrent.futures import ThreadPoolExecutor, as_completed +from wxdi.odcs_generator.generate_odcs_from_informatica import ( + InformaticaClient, + generate_odcs_yaml, + write_yaml_file, + extract_column_position +) + + +def example_basic_usage(): + """Basic example: Generate ODCS from an Informatica asset""" + + # Get credentials from environment variables + cdgc_url = os.getenv('INFORMATICA_CDGC_URL') + username = os.getenv('INFORMATICA_USERNAME') + password = os.getenv('INFORMATICA_PASSWORD') + + if not all([cdgc_url, username, password]): + print("Error: Please set INFORMATICA_CDGC_URL, INFORMATICA_USERNAME, and INFORMATICA_PASSWORD environment variables") + print("\nExample:") + print(" export INFORMATICA_CDGC_URL='https://cdgc.dm-us.informaticacloud.com'") + print(" export INFORMATICA_USERNAME='your_username'") + print(" export INFORMATICA_PASSWORD='your_pwd'") + sys.exit(1) + + # Replace with your actual asset ID + asset_id = "1b5fc805-252d-4ba2-bd90-e943103e411b" + + print(f"Connecting to Informatica CDGC at {cdgc_url}...") + + # Initialize Informatica client + client = InformaticaClient(cdgc_url, username, password) + + print(f"Fetching asset details for {asset_id}...") + + # Fetch asset data + asset_data = client.get_asset_details(asset_id) + + # Get column IDs from hierarchy + column_ids = [col['core.identity'] for col in asset_data.get('hierarchy', [])] + + print(f"Found {len(column_ids)} columns. Fetching column details...") + + # Fetch column details concurrently + column_details = [] + with ThreadPoolExecutor(max_workers=10) as executor: + future_to_col_id = { + executor.submit(client.get_column_details, col_id): col_id + for col_id in column_ids + } + + for future in as_completed(future_to_col_id): + try: + col_data = future.result() + column_details.append(col_data) + except Exception as e: + col_id = future_to_col_id[future] + print(f"Warning: Failed to fetch column {col_id}: {e}") + + # Sort columns by position + column_details.sort(key=extract_column_position) + + print("Generating ODCS YAML...") + + # Generate ODCS + odcs_data = generate_odcs_yaml(asset_data, column_details, client.base_url) + + # Determine output filename + asset_name = odcs_data.get('name', 'asset').lower().replace(' ', '-') + output_file = f"{asset_name}-odcs.yaml" + + # Write to file + write_yaml_file(output_file, odcs_data) + + print(f"✓ Successfully generated ODCS file: {output_file}") + + return odcs_data + + +def example_custom_processing(): + """Example: Generate ODCS with custom processing""" + + cdgc_url = os.getenv('INFORMATICA_CDGC_URL') + username = os.getenv('INFORMATICA_USERNAME') + password = os.getenv('INFORMATICA_PASSWORD') + + if not all([cdgc_url, username, password]): + print("Error: Environment variables not set") + sys.exit(1) + + asset_id = "1b5fc805-252d-4ba2-bd90-e943103e411b" + + print("Example: Custom ODCS Processing") + print("=" * 50) + + # Initialize client + client = InformaticaClient(cdgc_url, username, password) + + # Fetch data + asset_data = client.get_asset_details(asset_id) + column_ids = [col['core.identity'] for col in asset_data.get('hierarchy', [])] + + # Fetch columns + column_details = [] + with ThreadPoolExecutor(max_workers=10) as executor: + futures = [executor.submit(client.get_column_details, col_id) for col_id in column_ids] + for future in as_completed(futures): + try: + column_details.append(future.result()) + except Exception as e: + print(f"Warning: {e}") + + column_details.sort(key=extract_column_position) + + # Generate base ODCS + odcs_data = generate_odcs_yaml(asset_data, column_details, client.base_url) + + # Customize the ODCS + print("\nCustomizing ODCS metadata...") + + # Update contract metadata + odcs_data['domain'] = 'Finance' + odcs_data['dataProduct'] = 'Customer Analytics' + odcs_data['version'] = '2.0.0' + odcs_data['name'] = 'Customer Transaction Contract' + + # Add quality rules + odcs_data['quality'] = [ + { + 'type': 'completeness', + 'description': 'All required fields must be populated', + 'dimension': 'completeness', + 'weight': 'high' + }, + { + 'type': 'uniqueness', + 'description': 'Primary key must be unique', + 'dimension': 'uniqueness', + 'weight': 'critical' + } + ] + + # Update server configuration with actual values + if odcs_data.get('servers'): + odcs_data['servers'][0]['server'] = 'prod.snowflake.acme.com' + odcs_data['servers'][0]['database'] = 'ANALYTICS_DB' + odcs_data['servers'][0]['account'] = 'acme_prod' + + # Add stakeholders + odcs_data['stakeholders'] = [ + { + 'username': 'data.owner@acme.com', + 'role': 'Data Owner', + 'dateIn': '2024-01-01', + 'dateOut': None, + 'replaced': False + }, + { + 'username': 'data.steward@acme.com', + 'role': 'Data Steward', + 'dateIn': '2024-01-01', + 'dateOut': None, + 'replaced': False + } + ] + + # Save customized ODCS + output_file = "customized-customer-contract-odcs.yaml" + write_yaml_file(output_file, odcs_data) + + print(f"✓ Customized ODCS saved to: {output_file}") + + return odcs_data + + +def example_batch_processing(): + """Example: Process multiple assets in batch""" + + cdgc_url = os.getenv('INFORMATICA_CDGC_URL') + username = os.getenv('INFORMATICA_USERNAME') + password = os.getenv('INFORMATICA_PASSWORD') + + if not all([cdgc_url, username, password]): + print("Error: Environment variables not set") + sys.exit(1) + + # List of asset IDs to process + asset_ids = [ + "1b5fc805-252d-4ba2-bd90-e943103e411b", + "2c6gd916-363e-5cb1-af01-5gb3dfb2291c", + "3d7he027-474f-6dc2-bg12-6hc4egc3302d" + ] + + print("Example: Batch Processing Multiple Assets") + print("=" * 50) + + # Initialize client + client = InformaticaClient(cdgc_url, username, password) + + results = { + 'success': [], + 'failed': [] + } + + for i, asset_id in enumerate(asset_ids, 1): + print(f"\nProcessing asset {i}/{len(asset_ids)}: {asset_id}") + + try: + # Fetch asset data + asset_data = client.get_asset_details(asset_id) + column_ids = [col['core.identity'] for col in asset_data.get('hierarchy', [])] + + # Fetch columns + column_details = [] + with ThreadPoolExecutor(max_workers=10) as executor: + futures = [executor.submit(client.get_column_details, col_id) for col_id in column_ids] + for future in as_completed(futures): + try: + column_details.append(future.result()) + except Exception: + pass + + column_details.sort(key=extract_column_position) + + # Generate ODCS + odcs_data = generate_odcs_yaml(asset_data, column_details, client.base_url) + + # Save to file + asset_name = odcs_data.get('name', f'asset-{i}').lower().replace(' ', '-') + output_file = f"batch-{asset_name}-odcs.yaml" + write_yaml_file(output_file, odcs_data) + + results['success'].append({ + 'asset_id': asset_id, + 'output_file': output_file + }) + + print(f" ✓ Success: {output_file}") + + except Exception as e: + results['failed'].append({ + 'asset_id': asset_id, + 'error': str(e) + }) + print(f" ✗ Failed: {e}") + + # Print summary + print("\n" + "=" * 50) + print("Batch Processing Summary") + print("=" * 50) + print(f"Total assets: {len(asset_ids)}") + print(f"Successful: {len(results['success'])}") + print(f"Failed: {len(results['failed'])}") + + if results['failed']: + print("\nFailed assets:") + for item in results['failed']: + print(f" - {item['asset_id']}: {item['error']}") + + return results + + +def example_programmatic_usage(): + """Example: Using the module programmatically in your code""" + + print("Example: Programmatic Usage") + print("=" * 50) + + # Configuration + config = { + 'cdgc_url': os.getenv('INFORMATICA_CDGC_URL'), + 'username': os.getenv('INFORMATICA_USERNAME'), + 'password': os.getenv('INFORMATICA_PASSWORD'), + 'asset_id': '1b5fc805-252d-4ba2-bd90-e943103e411b' + } + + if not all([config['cdgc_url'], config['username'], config['password']]): + print("Error: Environment variables not set") + return None + + try: + # Step 1: Initialize client + print("Step 1: Initializing Informatica client...") + client = InformaticaClient( + config['cdgc_url'], + config['username'], + config['password'] + ) + + # Step 2: Fetch asset metadata + print("Step 2: Fetching asset metadata...") + asset_data = client.get_asset_details(config['asset_id']) + table_name = asset_data['summary']['core.name'] + print(f" Found table: {table_name}") + + # Step 3: Fetch column metadata + print("Step 3: Fetching column metadata...") + column_ids = [col['core.identity'] for col in asset_data.get('hierarchy', [])] + print(f" Found {len(column_ids)} columns") + + column_details = [] + with ThreadPoolExecutor(max_workers=10) as executor: + futures = [executor.submit(client.get_column_details, col_id) for col_id in column_ids] + for future in as_completed(futures): + try: + column_details.append(future.result()) + except Exception as e: + print(f" Warning: {e}") + + column_details.sort(key=extract_column_position) + + # Step 4: Generate ODCS + print("Step 4: Generating ODCS...") + odcs_data = generate_odcs_yaml(asset_data, column_details, client.base_url) + + # Step 5: Process or save ODCS + print("Step 5: Saving ODCS...") + output_file = f"{table_name.lower()}-programmatic-odcs.yaml" + write_yaml_file(output_file, odcs_data) + + print(f"\n✓ ODCS generated successfully: {output_file}") + + # You can now use odcs_data in your application + print(f"\nODCS Summary:") + print(f" - Contract ID: {odcs_data['id']}") + print(f" - Table: {odcs_data['schema'][0]['name']}") + print(f" - Columns: {len(odcs_data['schema'][0]['properties'])}") + print(f" - Server Type: {odcs_data['servers'][0]['type']}") + + return odcs_data + + except Exception as e: + print(f"\n✗ Error: {e}") + return None + + +def main(): + """Run all examples""" + + print("\n" + "=" * 70) + print("INFORMATICA ODCS GENERATOR - EXAMPLES") + print("=" * 70) + + examples = [ + ("Basic Usage", example_basic_usage), + ("Custom Processing", example_custom_processing), + ("Batch Processing", example_batch_processing), + ("Programmatic Usage", example_programmatic_usage) + ] + + print("\nAvailable examples:") + for i, (name, _) in enumerate(examples, 1): + print(f" {i}. {name}") + + print("\nTo run a specific example, uncomment the corresponding line below:") + print("Or run all examples by uncommenting the loop.\n") + + # Uncomment to run specific examples: + # example_basic_usage() + # example_custom_processing() + # example_batch_processing() + # example_programmatic_usage() + + # Uncomment to run all examples: + # for name, example_func in examples: + # print(f"\n{'=' * 70}") + # print(f"Running: {name}") + # print('=' * 70) + # try: + # example_func() + # except Exception as e: + # print(f"Error in {name}: {e}") + # print() + + print("\nNote: Make sure to set the following environment variables:") + print(" - INFORMATICA_CDGC_URL") + print(" - INFORMATICA_USERNAME") + print(" - INFORMATICA_PASSWORD") + print("\nAnd update the asset IDs in the examples with your actual asset IDs.") + + +if __name__ == '__main__': + main() + diff --git a/examples/pandas_dataframe_usage.py b/examples/pandas_dataframe_usage.py index 84f90a0..fcd994f 100644 --- a/examples/pandas_dataframe_usage.py +++ b/examples/pandas_dataframe_usage.py @@ -33,7 +33,7 @@ def main(): print("=" * 70) print("Pandas DataFrame Validation Example") print("=" * 70) - + # Step 1: Define metadata metadata = AssetMetadata( table_name='employees', @@ -46,24 +46,24 @@ def main(): ColumnMetadata('min_salary', DataType.DECIMAL, precision=10, scale=2), ] ) - + print(f"\nAsset: {metadata.table_name}") print(f"Columns: {len(metadata.columns)}") - + # Step 2: Create validator with rules validator = Validator(metadata) - + # Add validation rules validator.add_rule( ValidationRule('emp_id') .add_check(LengthCheck(min_length=4, max_length=6)) ) - + validator.add_rule( ValidationRule('name') .add_check(LengthCheck(min_length=2, max_length=100)) ) - + validator.add_rule( ValidationRule('age') .add_check(ComparisonCheck( @@ -75,7 +75,7 @@ def main(): target_value=65 )) ) - + validator.add_rule( ValidationRule('department') .add_check(ValidValuesCheck( @@ -83,7 +83,7 @@ def main(): case_sensitive=False )) ) - + validator.add_rule( ValidationRule('salary') .add_check(ComparisonCheck( @@ -91,9 +91,9 @@ def main(): target_column='min_salary' )) ) - + print(f"\nValidator configured with {len(validator.rules)} rules") - + # Step 3: Create sample DataFrame df = pd.DataFrame({ 'emp_id': [1001, 12, 1003, 1004, 1005], @@ -103,20 +103,20 @@ def main(): 'salary': [75000.00, 65000.00, 50000.00, 80000.00, 55000.00], 'min_salary': [60000.00, 55000.00, 45000.00, 70000.00, 60000.00] }) - + print(f"\nOriginal DataFrame ({len(df)} rows):") print(df.to_string(index=False)) - + # Step 4: Create Pandas validator pandas_validator = PandasValidator(validator, chunk_size=1000) - + print(f"\nPandas Validator: {pandas_validator}") - + # Step 5: Get summary statistics (memory efficient) print("\n" + "=" * 70) print("Validation Summary Statistics") print("=" * 70) - + summary = pandas_validator.get_summary_statistics(df) print(f"Total Rows: {summary['total_rows']}") print(f"Valid Rows: {summary['valid_rows']}") @@ -125,14 +125,14 @@ def main(): print(f"Total Checks: {summary['total_checks']}") print(f"Passed Checks: {summary['passed_checks']}") print(f"Failed Checks: {summary['failed_checks']}") - + # Step 6: Add validation column print("\n" + "=" * 70) print("DataFrame with Validation Column") print("=" * 70) - + df_validated = pandas_validator.add_validation_column(df) - + print(f"\nColumns: {list(df_validated.columns)}") print(f"\nValidation results (struct column):") for idx, result in enumerate(df_validated['dq_validation_result']): @@ -140,15 +140,15 @@ def main(): print(f"Row {idx}: {status} - Score: {result['score']}, Pass Rate: {result['pass_rate']:.1f}%") if not result['is_valid']: print(f" Errors: {result['error_count']}") - + # Step 7: Get invalid rows print("\n" + "=" * 70) print("Invalid Rows") print("=" * 70) - + invalid_df = pandas_validator.get_invalid_rows(df) print(f"\nFound {len(invalid_df)} invalid rows:") - + for idx, row in invalid_df.iterrows(): print(f"\nRow {idx}:") print(f" emp_id: {row['emp_id']}") @@ -158,31 +158,31 @@ def main(): validation = row['dq_validation_result'] print(f" Validation: {validation['score']} ({validation['pass_rate']:.1f}%)") print(f" Errors: {validation['error_count']}") - + # Step 8: Expand validation column print("\n" + "=" * 70) print("Expanded Validation Columns") print("=" * 70) - + df_expanded = pandas_validator.expand_validation_column(df_validated) - + print(f"\nExpanded columns: {[c for c in df_expanded.columns if c.startswith('dq_')]}") print(f"\nSample of expanded data:") print(df_expanded[['name', 'dq_is_valid', 'dq_score', 'dq_pass_rate', 'dq_error_count']].to_string(index=False)) - + # Step 9: Save results print("\n" + "=" * 70) print("Saving Results") print("=" * 70) - + # Save invalid rows invalid_df.to_csv('invalid_employees.csv', index=False) print("✓ Saved invalid rows to: invalid_employees.csv") - + # Save expanded results df_expanded.to_csv('validation_results.csv', index=False) print("✓ Saved validation results to: validation_results.csv") - + print("\n" + "=" * 70) print("Example Complete!") print("=" * 70) diff --git a/examples/spark_dataframe_usage.py b/examples/spark_dataframe_usage.py index 443d26c..37b3851 100644 --- a/examples/spark_dataframe_usage.py +++ b/examples/spark_dataframe_usage.py @@ -35,16 +35,16 @@ def main(): print("=" * 70) print("PySpark DataFrame Validation Example") print("=" * 70) - + # Step 1: Initialize Spark Session spark = SparkSession.builder \ .appName("DataQualityValidation") \ .master("local[*]") \ .getOrCreate() - + print(f"\nSpark Version: {spark.version}") print(f"Spark Master: {spark.sparkContext.master}") - + # Step 2: Define metadata metadata = AssetMetadata( table_name='employees', @@ -57,24 +57,24 @@ def main(): ColumnMetadata('min_salary', DataType.DECIMAL, precision=10, scale=2), ] ) - + print(f"\nAsset: {metadata.table_name}") print(f"Columns: {len(metadata.columns)}") - + # Step 3: Create validator with rules validator = Validator(metadata) - + # Add validation rules validator.add_rule( ValidationRule('emp_id') .add_check(LengthCheck(min_length=4, max_length=6)) ) - + validator.add_rule( ValidationRule('name') .add_check(LengthCheck(min_length=2, max_length=100)) ) - + validator.add_rule( ValidationRule('age') .add_check(ComparisonCheck( @@ -86,7 +86,7 @@ def main(): target_value=65 )) ) - + validator.add_rule( ValidationRule('department') .add_check(ValidValuesCheck( @@ -94,7 +94,7 @@ def main(): case_sensitive=False )) ) - + validator.add_rule( ValidationRule('salary') .add_check(ComparisonCheck( @@ -102,9 +102,9 @@ def main(): target_column='min_salary' )) ) - + print(f"\nValidator configured with {len(validator.rules)} rules") - + # Step 4: Create sample DataFrame schema = StructType([ StructField('emp_id', IntegerType(), True), @@ -114,7 +114,7 @@ def main(): StructField('salary', DecimalType(10, 2), True), StructField('min_salary', DecimalType(10, 2), True), ]) - + data = [ (1001, 'John Doe', 30, 'Engineering', 75000.00, 60000.00), (12, 'Jane Smith', 25, 'SALES', 65000.00, 55000.00), @@ -122,22 +122,22 @@ def main(): (1004, 'Alice Johnson', 35, 'Marketing', 80000.00, 70000.00), (1005, 'Charlie Brown', 40, 'Finance', 55000.00, 60000.00), ] - + df = spark.createDataFrame(data, schema) - + print(f"\nOriginal DataFrame ({df.count()} rows):") df.show(truncate=False) - + # Step 5: Create Spark validator spark_validator = SparkValidator(validator) - + print(f"\nSpark Validator: {spark_validator}") - + # Step 6: Get summary statistics (distributed aggregation) print("\n" + "=" * 70) print("Validation Summary Statistics") print("=" * 70) - + summary = spark_validator.get_summary_statistics(df) print(f"Total Rows: {summary['total_rows']}") print(f"Valid Rows: {summary['valid_rows']}") @@ -146,47 +146,47 @@ def main(): print(f"Total Checks: {summary['total_checks']}") print(f"Passed Checks: {summary['passed_checks']}") print(f"Failed Checks: {summary['failed_checks']}") - + # Step 7: Add validation column print("\n" + "=" * 70) print("DataFrame with Validation Column") print("=" * 70) - + df_validated = spark_validator.add_validation_column(df) - + print(f"\nColumns: {df_validated.columns}") print(f"\nSchema of validation column:") df_validated.select('dq_validation_result').printSchema() - + print(f"\nValidation results (struct column):") df_validated.select('name', 'dq_validation_result').show(truncate=False) - + # Step 8: Get invalid rows print("\n" + "=" * 70) print("Invalid Rows") print("=" * 70) - + invalid_df = spark_validator.get_invalid_rows(df) print(f"\nFound {invalid_df.count()} invalid rows:") invalid_df.show(truncate=False) - + # Step 9: Expand validation column print("\n" + "=" * 70) print("Expanded Validation Columns") print("=" * 70) - + df_expanded = spark_validator.expand_validation_column(df_validated) - + expanded_cols = [c for c in df_expanded.columns if c.startswith('dq_')] print(f"\nExpanded columns: {expanded_cols}") print(f"\nSample of expanded data:") df_expanded.select('name', 'dq_is_valid', 'dq_score', 'dq_pass_rate', 'dq_error_count').show(truncate=False) - + # Step 10: Get error samples print("\n" + "=" * 70) print("Error Samples") print("=" * 70) - + error_samples = spark_validator.get_error_sample(df, limit=10) print(f"\nFound {len(error_samples)} error samples:") for i, errors in enumerate(error_samples, 1): @@ -194,12 +194,12 @@ def main(): print(f" Errors:") for error in errors: print(f" - {error}") - + # Step 11: Write validation report print("\n" + "=" * 70) print("Writing Validation Report") print("=" * 70) - + output_path = "validation_report" spark_validator.write_validation_report( df, @@ -207,51 +207,51 @@ def main(): format='parquet' ) print(f"✓ Validation report written to: {output_path}") - + # Step 12: Save results print("\n" + "=" * 70) print("Saving Results") print("=" * 70) - + # Save invalid rows invalid_df.write.mode('overwrite').parquet('invalid_employees_spark') print("✓ Saved invalid rows to: invalid_employees_spark/") - + # Save expanded results df_expanded.write.mode('overwrite').parquet('validation_results_spark') print("✓ Saved validation results to: validation_results_spark/") - + # Step 13: Performance demonstration with larger dataset print("\n" + "=" * 70) print("Performance Test with Larger Dataset") print("=" * 70) - + # Create a larger dataset by repeating the data large_df = df for _ in range(10): # 5 * 2^10 = 5,120 rows large_df = large_df.union(df) - + print(f"\nLarge DataFrame: {large_df.count()} rows") - + # Validate large dataset (distributed processing) import time start_time = time.time() - + large_summary = spark_validator.get_summary_statistics(large_df) - + elapsed_time = time.time() - start_time - + print(f"\nValidation completed in {elapsed_time:.2f} seconds") print(f"Total Rows: {large_summary['total_rows']}") print(f"Valid Rows: {large_summary['valid_rows']}") print(f"Invalid Rows: {large_summary['invalid_rows']}") print(f"Pass Rate: {large_summary['pass_rate']:.2f}%") print(f"Throughput: {large_summary['total_rows'] / elapsed_time:.0f} rows/second") - + print("\n" + "=" * 70) print("Example Complete!") print("=" * 70) - + # Stop Spark session spark.stop() diff --git a/examples/test_dph_v1_examples.py b/examples/test_dph_v1_examples.py new file mode 100644 index 0000000..f89041b --- /dev/null +++ b/examples/test_dph_v1_examples.py @@ -0,0 +1,1600 @@ +# -*- coding: utf-8 -*- +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Examples for DphV1 +""" + +from ibm_cloud_sdk_core import ApiException, read_external_sources +from ibm_cloud_sdk_core.utils import datetime_to_string, string_to_datetime +import io +import os +import pytest +from wxdi.dph_services.dph_v1 import * + +# +# This file provides an example of how to use the DPH service. +# +# The following configuration properties are assumed to be defined: +# DPH_URL= +# DPH_AUTH_TYPE=iam +# DPH_APIKEY= +# DPH_AUTH_URL= +# +# These configuration properties can be exported as environment variables, or stored +# in a configuration file and then: +# export IBM_CREDENTIALS_FILE= +# +config_file = 'dph_v1.env' + +dph_service = None + +config = None + +# Variables to hold link values +complete_a_draft_by_contract_terms_id_link = None +complete_a_draft_by_draft_id_link = None +complete_contract_terms_document_by_document_id_link = None +complete_draft_contract_terms_by_data_product_id_link = None +create_a_contract_terms_doc_by_contract_terms_id_link = None +create_a_contract_terms_doc_by_draft_id_link = None +create_data_product_by_catalog_id_link = None +create_draft_by_container_id_link = None +create_new_draft_by_data_product_id_link = None +delete_a_contract_document_by_draft_id_link = None +delete_a_draft_by_contract_terms_id_link = None +delete_a_draft_by_draft_id_link = None +delete_contract_document_by_data_product_id_link = None +delete_contract_terms_document_by_document_id_link = None +delete_draft_of_data_product_by_data_product_id_link = None +get_a_draft_by_contract_terms_id_link = None +get_a_draft_contract_document_by_draft_id_link = None +get_a_draft_of_data_product_by_data_product_id_link = None +get_a_release_by_release_id_link = None +get_a_release_contract_terms_by_contract_terms_id_link = None +get_a_release_contract_terms_by_release_id_link = None +get_a_release_of_data_product_by_data_product_id_link = None +get_contract_document_by_data_product_id_link = None +get_contract_terms_document_by_id_document_id_link = None +get_data_product_by_data_product_id_link = None +get_draft_by_draft_id_link = None +get_list_of_data_product_drafts_by_data_product_id_link = None +get_list_of_releases_of_data_product_by_data_product_id_link = None +get_release_contract_document_by_data_product_id_link = None +get_release_contract_document_by_document_id_link = None +get_status_by_catalog_id_link = None +publish_a_draft_by_draft_id_link = None +publish_a_draft_of_data_product_by_data_product_id_link = None +retire_a_release_contract_terms_by_release_id_link = None +retire_a_releases_of_data_product_by_data_product_id_link = None +update_a_draft_by_contract_terms_id_link = None +update_a_draft_by_draft_id_link = None +update_a_release_by_release_id_link = None +update_contract_document_by_data_product_id_link = None +update_contract_document_by_draft_id_link = None +update_contract_terms_document_by_document_id_link = None +update_draft_of_data_product_by_data_product_id_link = None +update_release_of_data_product_by_data_product_id_link = None +upload_contract_terms_doc_by_data_product_id_link = None + + +############################################################################## +# Start of Examples for Service: DphV1 +############################################################################## +# region +class TestDphV1Examples: + """ + Example Test Class for DphV1 + """ + + @classmethod + def setup_class(cls): + global dph_service + if os.path.exists(config_file): + os.environ['IBM_CREDENTIALS_FILE'] = config_file + + # begin-common + + dph_service = DphV1.new_instance( + ) + + # end-common + assert dph_service is not None + + # Load the configuration + global config + config = read_external_sources(DphV1.DEFAULT_SERVICE_NAME) + + print('Setup complete.') + + needscredentials = pytest.mark.skipif( + not os.path.exists(config_file), reason="External configuration not available, skipping..." + ) + + @needscredentials + def test_initialize_example(self): + """ + initialize request example + """ + try: + global create_draft_by_container_id_link + global create_data_product_by_catalog_id_link + global get_status_by_catalog_id_link + + print('\ninitialize() result:') + + # begin-initialize + + response = dph_service.initialize( + include=['delivery_methods', 'domains_multi_industry', 'data_product_samples', 'workflows', 'project', 'catalog_configurations'], + ) + initialize_resource = response.get_result() + + print(json.dumps(initialize_resource, indent=2)) + + # end-initialize + + create_draft_by_container_id_link = initialize_resource['container']['id'] + create_data_product_by_catalog_id_link = initialize_resource['container']['id'] + get_status_by_catalog_id_link = initialize_resource['container']['id'] + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_create_data_product_example(self): + """ + create_data_product request example + """ + try: + global create_new_draft_by_data_product_id_link + global get_contract_document_by_data_product_id_link + global retire_a_releases_of_data_product_by_data_product_id_link + global get_data_product_by_data_product_id_link + global update_draft_of_data_product_by_data_product_id_link + global update_contract_document_by_data_product_id_link + global delete_draft_of_data_product_by_data_product_id_link + global get_a_release_of_data_product_by_data_product_id_link + global complete_draft_contract_terms_by_data_product_id_link + global delete_contract_document_by_data_product_id_link + global get_list_of_data_product_drafts_by_data_product_id_link + global get_a_draft_of_data_product_by_data_product_id_link + global get_release_contract_document_by_data_product_id_link + global publish_a_draft_of_data_product_by_data_product_id_link + global get_list_of_releases_of_data_product_by_data_product_id_link + global update_release_of_data_product_by_data_product_id_link + global upload_contract_terms_doc_by_data_product_id_link + + print('\ncreate_data_product() result:') + + # begin-create_data_product + + container_identity_model = { + 'id': 'd29c42eb-7100-4b7a-8257-c196dbcca1cd', + } + + asset_prototype_model = { + 'container': container_identity_model, + } + + data_product_draft_prototype_model = { + 'name': 'My New Data Product', + 'asset': asset_prototype_model, + } + + response = dph_service.create_data_product( + drafts=[data_product_draft_prototype_model], + ) + data_product = response.get_result() + + print(json.dumps(data_product, indent=2)) + + # end-create_data_product + + create_new_draft_by_data_product_id_link = data_product['id'] + get_contract_document_by_data_product_id_link = data_product['id'] + retire_a_releases_of_data_product_by_data_product_id_link = data_product['id'] + get_data_product_by_data_product_id_link = data_product['id'] + update_draft_of_data_product_by_data_product_id_link = data_product['id'] + update_contract_document_by_data_product_id_link = data_product['id'] + delete_draft_of_data_product_by_data_product_id_link = data_product['id'] + get_a_release_of_data_product_by_data_product_id_link = data_product['id'] + complete_draft_contract_terms_by_data_product_id_link = data_product['id'] + delete_contract_document_by_data_product_id_link = data_product['id'] + get_list_of_data_product_drafts_by_data_product_id_link = data_product['id'] + get_a_draft_of_data_product_by_data_product_id_link = data_product['id'] + get_release_contract_document_by_data_product_id_link = data_product['id'] + publish_a_draft_of_data_product_by_data_product_id_link = data_product['id'] + get_list_of_releases_of_data_product_by_data_product_id_link = data_product['id'] + update_release_of_data_product_by_data_product_id_link = data_product['id'] + upload_contract_terms_doc_by_data_product_id_link = data_product['id'] + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_create_data_product_draft_example(self): + """ + create_data_product_draft request example + """ + try: + global get_a_draft_contract_document_by_draft_id_link + global update_a_draft_by_contract_terms_id_link + global create_a_contract_terms_doc_by_contract_terms_id_link + global update_contract_document_by_draft_id_link + global get_a_release_contract_terms_by_contract_terms_id_link + global complete_a_draft_by_contract_terms_id_link + global get_draft_by_draft_id_link + global publish_a_draft_by_draft_id_link + global update_a_draft_by_draft_id_link + global create_a_contract_terms_doc_by_draft_id_link + global delete_a_contract_document_by_draft_id_link + global delete_a_draft_by_contract_terms_id_link + global delete_a_draft_by_draft_id_link + global complete_a_draft_by_draft_id_link + global get_a_draft_by_contract_terms_id_link + + print('\ncreate_data_product_draft() result:') + + # begin-create_data_product_draft + + container_identity_model = { + 'id': 'd29c42eb-7100-4b7a-8257-c196dbcca1cd', + } + + asset_prototype_model = { + 'container': container_identity_model, + } + + data_product_draft_version_release_model = { + 'id': '8bf83660-11fe-4427-a72a-8d8359af24e3', + } + + data_product_identity_model = { + 'id': 'b38df608-d34b-4d58-8136-ed25e6c6684e', + 'release': data_product_draft_version_release_model, + } + + response = dph_service.create_data_product_draft( + data_product_id=create_new_draft_by_data_product_id_link, + asset=asset_prototype_model, + version='1.2.0', + data_product=data_product_identity_model, + ) + data_product_draft = response.get_result() + + print(json.dumps(data_product_draft, indent=2)) + + # end-create_data_product_draft + + get_a_draft_contract_document_by_draft_id_link = data_product_draft['id'] + update_a_draft_by_contract_terms_id_link = data_product_draft['contract_terms'][0]['id'] + create_a_contract_terms_doc_by_contract_terms_id_link = data_product_draft['contract_terms'][0]['id'] + update_contract_document_by_draft_id_link = data_product_draft['id'] + get_a_release_contract_terms_by_contract_terms_id_link = data_product_draft['contract_terms'][0]['id'] + complete_a_draft_by_contract_terms_id_link = data_product_draft['contract_terms'][0]['id'] + get_draft_by_draft_id_link = data_product_draft['id'] + publish_a_draft_by_draft_id_link = data_product_draft['id'] + update_a_draft_by_draft_id_link = data_product_draft['id'] + create_a_contract_terms_doc_by_draft_id_link = data_product_draft['id'] + delete_a_contract_document_by_draft_id_link = data_product_draft['id'] + delete_a_draft_by_contract_terms_id_link = data_product_draft['contract_terms'][0]['id'] + delete_a_draft_by_draft_id_link = data_product_draft['id'] + complete_a_draft_by_draft_id_link = data_product_draft['id'] + get_a_draft_by_contract_terms_id_link = data_product_draft['contract_terms'][0]['id'] + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_create_draft_contract_terms_document_example(self): + """ + create_draft_contract_terms_document request example + """ + try: + global get_release_contract_document_by_document_id_link + global delete_contract_terms_document_by_document_id_link + global get_contract_terms_document_by_id_document_id_link + global update_contract_terms_document_by_document_id_link + global complete_contract_terms_document_by_document_id_link + + print('\ncreate_draft_contract_terms_document() result:') + + # begin-create_draft_contract_terms_document + + response = dph_service.create_draft_contract_terms_document( + data_product_id=upload_contract_terms_doc_by_data_product_id_link, + draft_id=create_a_contract_terms_doc_by_draft_id_link, + contract_terms_id=create_a_contract_terms_doc_by_contract_terms_id_link, + type='terms_and_conditions', + name='Terms and conditions document', + ) + contract_terms_document = response.get_result() + + print(json.dumps(contract_terms_document, indent=2)) + + # end-create_draft_contract_terms_document + + get_release_contract_document_by_document_id_link = contract_terms_document['id'] + delete_contract_terms_document_by_document_id_link = contract_terms_document['id'] + get_contract_terms_document_by_id_document_id_link = contract_terms_document['id'] + update_contract_terms_document_by_document_id_link = contract_terms_document['id'] + complete_contract_terms_document_by_document_id_link = contract_terms_document['id'] + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_publish_data_product_draft_example(self): + """ + publish_data_product_draft request example + """ + try: + global update_a_release_by_release_id_link + global get_a_release_contract_terms_by_release_id_link + global retire_a_release_contract_terms_by_release_id_link + global get_a_release_by_release_id_link + + print('\npublish_data_product_draft() result:') + + # begin-publish_data_product_draft + + response = dph_service.publish_data_product_draft( + data_product_id=publish_a_draft_of_data_product_by_data_product_id_link, + draft_id=publish_a_draft_by_draft_id_link, + ) + data_product_release = response.get_result() + + print(json.dumps(data_product_release, indent=2)) + + # end-publish_data_product_draft + + update_a_release_by_release_id_link = data_product_release['id'] + get_a_release_contract_terms_by_release_id_link = data_product_release['id'] + retire_a_release_contract_terms_by_release_id_link = data_product_release['id'] + get_a_release_by_release_id_link = data_product_release['id'] + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_initialize_status_example(self): + """ + get_initialize_status request example + """ + try: + print('\nget_initialize_status() result:') + + # begin-get_initialize_status + + response = dph_service.get_initialize_status() + initialize_resource = response.get_result() + + print(json.dumps(initialize_resource, indent=2)) + + # end-get_initialize_status + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_service_id_credentials_example(self): + """ + get_service_id_credentials request example + """ + try: + print('\nget_service_id_credentials() result:') + + # begin-get_service_id_credentials + + response = dph_service.get_service_id_credentials() + service_id_credentials = response.get_result() + + print(json.dumps(service_id_credentials, indent=2)) + + # end-get_service_id_credentials + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_manage_api_keys_example(self): + """ + manage_api_keys request example + """ + try: + # begin-manage_api_keys + + response = dph_service.manage_api_keys() + + # end-manage_api_keys + print('\nmanage_api_keys() response status code: ', response.get_status_code()) + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_create_data_asset_visualization_example(self): + """ + create_data_asset_visualization request example + """ + try: + print('\ncreate_data_asset_visualization() result:') + + # begin-create_data_asset_visualization + + container_reference_model = { + 'id': '2be8f727-c5d2-4cb0-9216-f9888f428048', + 'type': 'catalog', + } + + asset_reference_model = { + 'id': 'caeee3f3-756e-47d5-846d-da4600809e22', + 'container': container_reference_model, + } + + data_asset_relationship_model = { + 'asset': asset_reference_model, + 'related_asset': asset_reference_model, + } + + response = dph_service.create_data_asset_visualization( + assets=[data_asset_relationship_model], + ) + data_asset_visualization_res = response.get_result() + + print(json.dumps(data_asset_visualization_res, indent=2)) + + # end-create_data_asset_visualization + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_reinitiate_data_asset_visualization_example(self): + """ + reinitiate_data_asset_visualization request example + """ + try: + print('\nreinitiate_data_asset_visualization() result:') + + # begin-reinitiate_data_asset_visualization + + container_reference_model = { + 'id': '2be8f727-c5d2-4cb0-9216-f9888f428048', + 'type': 'catalog', + } + + asset_reference_model = { + 'id': 'caeee3f3-756e-47d5-846d-da4600809e22', + 'container': container_reference_model, + } + + data_asset_relationship_model = { + 'asset': asset_reference_model, + 'related_asset': asset_reference_model, + } + + response = dph_service.reinitiate_data_asset_visualization( + assets=[data_asset_relationship_model], + ) + data_asset_visualization_res = response.get_result() + + print(json.dumps(data_asset_visualization_res, indent=2)) + + # end-reinitiate_data_asset_visualization + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_list_data_products_example(self): + """ + list_data_products request example + """ + try: + print('\nlist_data_products() result:') + + # begin-list_data_products + + all_results = [] + pager = DataProductsPager( + client=dph_service, + limit=10, + ) + while pager.has_next(): + next_page = pager.get_next() + assert next_page is not None + all_results.extend(next_page) + + print(json.dumps(all_results, indent=2)) + + # end-list_data_products + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_data_product_example(self): + """ + get_data_product request example + """ + try: + print('\nget_data_product() result:') + + # begin-get_data_product + + response = dph_service.get_data_product( + data_product_id=get_data_product_by_data_product_id_link, + ) + data_product = response.get_result() + + print(json.dumps(data_product, indent=2)) + + # end-get_data_product + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_complete_draft_contract_terms_document_example(self): + """ + complete_draft_contract_terms_document request example + """ + try: + print('\ncomplete_draft_contract_terms_document() result:') + + # begin-complete_draft_contract_terms_document + + response = dph_service.complete_draft_contract_terms_document( + data_product_id=complete_draft_contract_terms_by_data_product_id_link, + draft_id=complete_a_draft_by_draft_id_link, + contract_terms_id=complete_a_draft_by_contract_terms_id_link, + document_id=complete_contract_terms_document_by_document_id_link, + ) + contract_terms_document = response.get_result() + + print(json.dumps(contract_terms_document, indent=2)) + + # end-complete_draft_contract_terms_document + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_list_data_product_drafts_example(self): + """ + list_data_product_drafts request example + """ + try: + print('\nlist_data_product_drafts() result:') + + # begin-list_data_product_drafts + + all_results = [] + pager = DataProductDraftsPager( + client=dph_service, + data_product_id=get_list_of_data_product_drafts_by_data_product_id_link, + asset_container_id='testString', + version='testString', + limit=10, + ) + while pager.has_next(): + next_page = pager.get_next() + assert next_page is not None + all_results.extend(next_page) + + print(json.dumps(all_results, indent=2)) + + # end-list_data_product_drafts + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_data_product_draft_example(self): + """ + get_data_product_draft request example + """ + try: + print('\nget_data_product_draft() result:') + + # begin-get_data_product_draft + + response = dph_service.get_data_product_draft( + data_product_id=get_a_draft_of_data_product_by_data_product_id_link, + draft_id=get_draft_by_draft_id_link, + ) + data_product_draft = response.get_result() + + print(json.dumps(data_product_draft, indent=2)) + + # end-get_data_product_draft + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_update_data_product_draft_example(self): + """ + update_data_product_draft request example + """ + try: + print('\nupdate_data_product_draft() result:') + + # begin-update_data_product_draft + + json_patch_operation_model = { + 'op': 'add', + 'path': 'testString', + } + + response = dph_service.update_data_product_draft( + data_product_id=update_draft_of_data_product_by_data_product_id_link, + draft_id=update_a_draft_by_draft_id_link, + json_patch_instructions=[json_patch_operation_model], + ) + data_product_draft = response.get_result() + + print(json.dumps(data_product_draft, indent=2)) + + # end-update_data_product_draft + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_draft_contract_terms_document_example(self): + """ + get_draft_contract_terms_document request example + """ + try: + print('\nget_draft_contract_terms_document() result:') + + # begin-get_draft_contract_terms_document + + response = dph_service.get_draft_contract_terms_document( + data_product_id=get_contract_document_by_data_product_id_link, + draft_id=get_a_draft_contract_document_by_draft_id_link, + contract_terms_id=get_a_draft_by_contract_terms_id_link, + document_id=get_contract_terms_document_by_id_document_id_link, + ) + contract_terms_document = response.get_result() + + print(json.dumps(contract_terms_document, indent=2)) + + # end-get_draft_contract_terms_document + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_update_draft_contract_terms_document_example(self): + """ + update_draft_contract_terms_document request example + """ + try: + print('\nupdate_draft_contract_terms_document() result:') + + # begin-update_draft_contract_terms_document + + json_patch_operation_model = { + 'op': 'add', + 'path': 'testString', + } + + response = dph_service.update_draft_contract_terms_document( + data_product_id=update_contract_document_by_data_product_id_link, + draft_id=update_contract_document_by_draft_id_link, + contract_terms_id=update_a_draft_by_contract_terms_id_link, + document_id=update_contract_terms_document_by_document_id_link, + json_patch_instructions=[json_patch_operation_model], + ) + contract_terms_document = response.get_result() + + print(json.dumps(contract_terms_document, indent=2)) + + # end-update_draft_contract_terms_document + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_data_product_draft_contract_terms_example(self): + """ + get_data_product_draft_contract_terms request example + """ + try: + print('\nget_data_product_draft_contract_terms() result:') + + # begin-get_data_product_draft_contract_terms + + response = dph_service.get_data_product_draft_contract_terms( + data_product_id='testString', + draft_id='testString', + contract_terms_id='testString', + ) + contract_terms = response.get_result() + + print(json.dumps(contract_terms, indent=2)) + + # end-get_data_product_draft_contract_terms + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_replace_data_product_draft_contract_terms_example(self): + """ + replace_data_product_draft_contract_terms request example + """ + try: + print('\nreplace_data_product_draft_contract_terms() result:') + + # begin-replace_data_product_draft_contract_terms + + contract_terms_document_model = { + 'url': 'https://ibm.com/document', + 'type': 'terms_and_conditions', + 'name': 'Terms and Conditions', + 'id': 'b38df608-d34b-4d58-8136-ed25e6c6684e', + } + + domain_model = { + 'id': 'b38df608-d34b-4d58-8136-ed25e6c6684e', + 'name': 'domain_name', + } + + overview_model = { + 'api_version': 'v3.0.1', + 'kind': 'DataContract', + 'name': 'Sample Data Contract', + 'version': 'v0.0', + 'domain': domain_model, + 'more_info': 'List of links to sources that provide more details on the data contract.', + } + + contract_terms_more_info_model = { + 'type': 'privacy-statement', + 'url': 'https://www.moreinfo.example.coms', + } + + description_model = { + 'purpose': 'Intended purpose for the provided data.', + 'limitations': 'Technical, compliance, and legal limitations for data use.', + 'usage': 'Recommended usage of the data.', + 'more_info': [contract_terms_more_info_model], + 'custom_properties': 'Custom properties that are not part of the standard.', + } + + contract_template_organization_model = { + 'user_id': 'IBMid-691000IN4G', + 'role': 'owner', + } + + roles_model = { + 'role': 'IAM Role', + } + + pricing_model = { + 'amount': 'Amount', + 'currency': 'Currency', + 'unit': 'Unit', + } + + contract_template_sla_property_model = { + 'property': 'slaproperty', + 'value': 'slavalue', + } + + contract_template_sla_model = { + 'default_element': 'sladefaultelement', + 'properties': [contract_template_sla_property_model], + } + + contract_template_support_and_communication_model = { + 'channel': 'channel', + 'url': 'https://www.example.coms', + } + + contract_template_custom_property_model = { + 'key': 'The name of the key.', + 'value': 'The value of the key.', + } + + contract_asset_model = { + 'id': '684d6aa0-9f93-4564-8a20-e354bc469857', + 'name': 'PAYMENT_TRANSACTIONS1', + } + + contract_server_model = { + 'server': 'snowflake-server-01', + 'asset': contract_asset_model, + 'connection_id': '8d7701be-709a-49c0-ae4e-a7daeaae6def', + 'type': 'snowflake', + 'description': 'Snowflake analytics server', + 'environment': 'dev', + 'account': 'acc-456', + 'catalog': 'analytics_cat', + 'database': 'analytics_db', + 'dataset': 'customer_data', + 'delimiter': ',', + 'endpoint_url': 'https://xy12345.snowflakecomputing.com', + 'format': 'parquet', + 'host': 'xy12345.snowflakecomputing.com', + 'location': 'Mumbai', + 'path': '/analytics/data', + 'port': '443', + 'project': 'projectY', + 'region': 'ap-south-1', + 'region_name': 'Asia South 1', + 'schema': 'PAYMENT_TRANSACTIONS1', + 'service_name': 'snowflake', + 'staging_dir': '/snowflake/staging', + 'stream': 'stream_analytics', + 'warehouse': 'wh_xlarge', + 'custom_properties': [contract_template_custom_property_model], + } + + contract_schema_property_type_model = { + 'type': 'varchar', + 'length': '1024', + 'scale': '0', + 'nullable': 'true', + 'signed': 'false', + } + + contract_schema_property_model = { + 'name': 'product_brand_code', + 'type': contract_schema_property_type_model, + } + + contract_schema_model = { + 'asset_id': '09ca6b40-7c89-412a-8951-ad820da709d1', + 'connection_id': '6cc57d4d-2229-438f-91a0-2c455556422b', + 'name': '000000_0-2025-06-20-20-28-52.csv', + 'connection_path': '/dpx-test-bucket/000000_0-2025-06-20-20-28-52.csv', + 'physical_type': 'text/csv', + 'properties': [contract_schema_property_model], + } + + response = dph_service.replace_data_product_draft_contract_terms( + data_product_id='testString', + draft_id='testString', + contract_terms_id='testString', + documents=[contract_terms_document_model], + overview=overview_model, + description=description_model, + organization=[contract_template_organization_model], + roles=[roles_model], + price=pricing_model, + sla=[contract_template_sla_model], + support_and_communication=[contract_template_support_and_communication_model], + custom_properties=[contract_template_custom_property_model], + servers=[contract_server_model], + schema=[contract_schema_model], + ) + contract_terms = response.get_result() + + print(json.dumps(contract_terms, indent=2)) + + # end-replace_data_product_draft_contract_terms + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_update_data_product_draft_contract_terms_example(self): + """ + update_data_product_draft_contract_terms request example + """ + try: + print('\nupdate_data_product_draft_contract_terms() result:') + + # begin-update_data_product_draft_contract_terms + + json_patch_operation_model = { + 'op': 'add', + 'path': 'testString', + } + + response = dph_service.update_data_product_draft_contract_terms( + data_product_id='testString', + draft_id='testString', + contract_terms_id='testString', + json_patch_instructions=[json_patch_operation_model], + ) + contract_terms = response.get_result() + + print(json.dumps(contract_terms, indent=2)) + + # end-update_data_product_draft_contract_terms + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_contract_terms_in_specified_format_example(self): + """ + get_contract_terms_in_specified_format request example + """ + try: + print('\nget_contract_terms_in_specified_format() result:') + + # begin-get_contract_terms_in_specified_format + + response = dph_service.get_contract_terms_in_specified_format( + data_product_id='testString', + draft_id='testString', + contract_terms_id='testString', + format='testString', + format_version='testString', + ) + result = response.get_result() + + with open('/tmp/result.out', 'wb') as fp: + fp.write(result) + + # end-get_contract_terms_in_specified_format + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_data_product_release_example(self): + """ + get_data_product_release request example + """ + try: + print('\nget_data_product_release() result:') + + # begin-get_data_product_release + + response = dph_service.get_data_product_release( + data_product_id=get_a_release_of_data_product_by_data_product_id_link, + release_id=get_a_release_by_release_id_link, + ) + data_product_release = response.get_result() + + print(json.dumps(data_product_release, indent=2)) + + # end-get_data_product_release + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_update_data_product_release_example(self): + """ + update_data_product_release request example + """ + try: + print('\nupdate_data_product_release() result:') + + # begin-update_data_product_release + + json_patch_operation_model = { + 'op': 'add', + 'path': 'testString', + } + + response = dph_service.update_data_product_release( + data_product_id=update_release_of_data_product_by_data_product_id_link, + release_id='testString', + json_patch_instructions=[json_patch_operation_model], + ) + data_product_release = response.get_result() + + print(json.dumps(data_product_release, indent=2)) + + # end-update_data_product_release + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_release_contract_terms_document_example(self): + """ + get_release_contract_terms_document request example + """ + try: + print('\nget_release_contract_terms_document() result:') + + # begin-get_release_contract_terms_document + + response = dph_service.get_release_contract_terms_document( + data_product_id=get_release_contract_document_by_data_product_id_link, + release_id=get_a_release_contract_terms_by_release_id_link, + contract_terms_id=get_a_release_contract_terms_by_contract_terms_id_link, + document_id=get_release_contract_document_by_document_id_link, + ) + contract_terms_document = response.get_result() + + print(json.dumps(contract_terms_document, indent=2)) + + # end-get_release_contract_terms_document + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_published_data_product_draft_contract_terms_example(self): + """ + get_published_data_product_draft_contract_terms request example + """ + try: + print('\nget_published_data_product_draft_contract_terms() result:') + + # begin-get_published_data_product_draft_contract_terms + + response = dph_service.get_published_data_product_draft_contract_terms( + data_product_id='testString', + release_id='testString', + contract_terms_id='testString', + ) + result = response.get_result() + + with open('/tmp/result.out', 'wb') as fp: + fp.write(result) + + # end-get_published_data_product_draft_contract_terms + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_list_data_product_releases_example(self): + """ + list_data_product_releases request example + """ + try: + print('\nlist_data_product_releases() result:') + + # begin-list_data_product_releases + + all_results = [] + pager = DataProductReleasesPager( + client=dph_service, + data_product_id=get_list_of_releases_of_data_product_by_data_product_id_link, + asset_container_id='testString', + state=['available'], + version='testString', + limit=10, + ) + while pager.has_next(): + next_page = pager.get_next() + assert next_page is not None + all_results.extend(next_page) + + print(json.dumps(all_results, indent=2)) + + # end-list_data_product_releases + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_retire_data_product_release_example(self): + """ + retire_data_product_release request example + """ + try: + print('\nretire_data_product_release() result:') + + # begin-retire_data_product_release + + response = dph_service.retire_data_product_release( + data_product_id=retire_a_releases_of_data_product_by_data_product_id_link, + release_id=retire_a_release_contract_terms_by_release_id_link, + ) + data_product_release = response.get_result() + + print(json.dumps(data_product_release, indent=2)) + + # end-retire_data_product_release + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_create_revoke_access_process_example(self): + """ + create_revoke_access_process request example + """ + try: + print('\ncreate_revoke_access_process() result:') + + # begin-create_revoke_access_process + + response = dph_service.create_revoke_access_process( + data_product_id='testString', + release_id='testString', + body=io.BytesIO(b'This is a mock file.').getvalue(), + ) + revoke_access_response = response.get_result() + + print(json.dumps(revoke_access_response, indent=2)) + + # end-create_revoke_access_process + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_list_data_product_contract_template_example(self): + """ + list_data_product_contract_template request example + """ + try: + print('\nlist_data_product_contract_template() result:') + + # begin-list_data_product_contract_template + + response = dph_service.list_data_product_contract_template() + data_product_contract_template_collection = response.get_result() + + print(json.dumps(data_product_contract_template_collection, indent=2)) + + # end-list_data_product_contract_template + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_create_contract_template_example(self): + """ + create_contract_template request example + """ + try: + print('\ncreate_contract_template() result:') + + # begin-create_contract_template + + container_reference_model = { + 'id': '531f74a-01c8-4e91-8e29-b018db683c86', + 'type': 'catalog', + } + + domain_model = { + 'id': '0094ebe9-abc3-473b-80ea-c777ede095ea', + 'name': 'Test Domain New', + } + + overview_model = { + 'name': 'Sample Data Contract', + 'version': '0.0.0', + 'domain': domain_model, + 'more_info': 'List of links to sources that provide more details on the data contract.', + } + + contract_terms_more_info_model = { + 'type': 'privacy-statement', + 'url': 'https://www.moreinfo.example.coms', + } + + description_model = { + 'purpose': 'Intended purpose for the provided data.', + 'limitations': 'Technical, compliance, and legal limitations for data use.', + 'usage': 'Recommended usage of the data.', + 'more_info': [contract_terms_more_info_model], + 'custom_properties': 'Custom properties that are not part of the standard.', + } + + contract_template_organization_model = { + 'user_id': 'IBMid-691000IN4G', + 'role': 'owner', + } + + roles_model = { + 'role': 'IAM Role', + } + + pricing_model = { + 'amount': '100.00', + 'currency': 'USD', + 'unit': 'megabyte', + } + + contract_template_sla_property_model = { + 'property': 'slaproperty', + 'value': 'slavalue', + } + + contract_template_sla_model = { + 'default_element': 'sladefaultelement', + 'properties': [contract_template_sla_property_model], + } + + contract_template_support_and_communication_model = { + 'channel': 'channel', + 'url': 'https://www.example.coms', + } + + contract_template_custom_property_model = { + 'key': 'propertykey', + 'value': 'propertyvalue', + } + + contract_terms_model = { + 'overview': overview_model, + 'description': description_model, + 'organization': [contract_template_organization_model], + 'roles': [roles_model], + 'price': pricing_model, + 'sla': [contract_template_sla_model], + 'support_and_communication': [contract_template_support_and_communication_model], + 'custom_properties': [contract_template_custom_property_model], + } + + response = dph_service.create_contract_template( + container=container_reference_model, + name='Sample Data Contract Template', + contract_terms=contract_terms_model, + ) + data_product_contract_template = response.get_result() + + print(json.dumps(data_product_contract_template, indent=2)) + + # end-create_contract_template + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_contract_template_example(self): + """ + get_contract_template request example + """ + try: + print('\nget_contract_template() result:') + + # begin-get_contract_template + + response = dph_service.get_contract_template( + contract_template_id='testString', + container_id='testString', + ) + data_product_contract_template = response.get_result() + + print(json.dumps(data_product_contract_template, indent=2)) + + # end-get_contract_template + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_update_data_product_contract_template_example(self): + """ + update_data_product_contract_template request example + """ + try: + print('\nupdate_data_product_contract_template() result:') + + # begin-update_data_product_contract_template + + json_patch_operation_model = { + 'op': 'add', + 'path': 'testString', + } + + response = dph_service.update_data_product_contract_template( + contract_template_id='testString', + container_id='testString', + json_patch_instructions=[json_patch_operation_model], + ) + data_product_contract_template = response.get_result() + + print(json.dumps(data_product_contract_template, indent=2)) + + # end-update_data_product_contract_template + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_list_data_product_domains_example(self): + """ + list_data_product_domains request example + """ + try: + print('\nlist_data_product_domains() result:') + + # begin-list_data_product_domains + + response = dph_service.list_data_product_domains() + data_product_domain_collection = response.get_result() + + print(json.dumps(data_product_domain_collection, indent=2)) + + # end-list_data_product_domains + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_create_data_product_domain_example(self): + """ + create_data_product_domain request example + """ + try: + print('\ncreate_data_product_domain() result:') + + # begin-create_data_product_domain + + container_reference_model = { + 'id': 'ed580171-a6e4-4b93-973f-ae2f2f62991b', + 'type': 'catalog', + } + + initialize_sub_domain_model = { + 'name': 'Sub domain 1', + 'description': 'New sub domain 1', + } + + response = dph_service.create_data_product_domain( + container=container_reference_model, + name='Test domain', + description='The sample description for new domain', + sub_domains=[initialize_sub_domain_model], + ) + data_product_domain = response.get_result() + + print(json.dumps(data_product_domain, indent=2)) + + # end-create_data_product_domain + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_create_data_product_subdomain_example(self): + """ + create_data_product_subdomain request example + """ + try: + print('\ncreate_data_product_subdomain() result:') + + # begin-create_data_product_subdomain + + response = dph_service.create_data_product_subdomain( + domain_id='testString', + container_id='testString', + name='Sub domain 1', + description='New sub domain 1', + ) + initialize_sub_domain = response.get_result() + + print(json.dumps(initialize_sub_domain, indent=2)) + + # end-create_data_product_subdomain + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_domain_example(self): + """ + get_domain request example + """ + try: + print('\nget_domain() result:') + + # begin-get_domain + + response = dph_service.get_domain( + domain_id='testString', + ) + data_product_domain = response.get_result() + + print(json.dumps(data_product_domain, indent=2)) + + # end-get_domain + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_update_data_product_domain_example(self): + """ + update_data_product_domain request example + """ + try: + print('\nupdate_data_product_domain() result:') + + # begin-update_data_product_domain + + json_patch_operation_model = { + 'op': 'add', + 'path': 'testString', + } + + response = dph_service.update_data_product_domain( + domain_id='testString', + container_id='testString', + json_patch_instructions=[json_patch_operation_model], + ) + data_product_domain = response.get_result() + + print(json.dumps(data_product_domain, indent=2)) + + # end-update_data_product_domain + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_data_product_by_domain_example(self): + """ + get_data_product_by_domain request example + """ + try: + print('\nget_data_product_by_domain() result:') + + # begin-get_data_product_by_domain + + response = dph_service.get_data_product_by_domain( + domain_id='testString', + container_id='testString', + ) + data_product_version_collection = response.get_result() + + print(json.dumps(data_product_version_collection, indent=2)) + + # end-get_data_product_by_domain + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_create_s3_bucket_example(self): + """ + create_s3_bucket request example + """ + try: + print('\ncreate_s3_bucket() result:') + + # begin-create_s3_bucket + + response = dph_service.create_s3_bucket( + is_shared=True, + ) + bucket_response = response.get_result() + + print(json.dumps(bucket_response, indent=2)) + + # end-create_s3_bucket + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_s3_bucket_validation_example(self): + """ + get_s3_bucket_validation request example + """ + try: + print('\nget_s3_bucket_validation() result:') + + # begin-get_s3_bucket_validation + + response = dph_service.get_s3_bucket_validation( + bucket_name='testString', + ) + bucket_validation_response = response.get_result() + + print(json.dumps(bucket_validation_response, indent=2)) + + # end-get_s3_bucket_validation + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_get_revoke_access_process_state_example(self): + """ + get_revoke_access_process_state request example + """ + try: + print('\nget_revoke_access_process_state() result:') + + # begin-get_revoke_access_process_state + + response = dph_service.get_revoke_access_process_state( + release_id='testString', + ) + revoke_access_state_response = response.get_result() + + print(json.dumps(revoke_access_state_response, indent=2)) + + # end-get_revoke_access_process_state + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_delete_draft_contract_terms_document_example(self): + """ + delete_draft_contract_terms_document request example + """ + try: + # begin-delete_draft_contract_terms_document + + response = dph_service.delete_draft_contract_terms_document( + data_product_id=delete_contract_document_by_data_product_id_link, + draft_id=delete_a_contract_document_by_draft_id_link, + contract_terms_id=delete_a_draft_by_contract_terms_id_link, + document_id=delete_contract_terms_document_by_document_id_link, + ) + + # end-delete_draft_contract_terms_document + print('\ndelete_draft_contract_terms_document() response status code: ', response.get_status_code()) + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_delete_data_product_draft_example(self): + """ + delete_data_product_draft request example + """ + try: + # begin-delete_data_product_draft + + response = dph_service.delete_data_product_draft( + data_product_id=delete_draft_of_data_product_by_data_product_id_link, + draft_id=delete_a_draft_by_draft_id_link, + ) + + # end-delete_data_product_draft + print('\ndelete_data_product_draft() response status code: ', response.get_status_code()) + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_delete_data_product_contract_template_example(self): + """ + delete_data_product_contract_template request example + """ + try: + # begin-delete_data_product_contract_template + + response = dph_service.delete_data_product_contract_template( + contract_template_id='testString', + container_id='testString', + ) + + # end-delete_data_product_contract_template + print('\ndelete_data_product_contract_template() response status code: ', response.get_status_code()) + + except ApiException as e: + pytest.fail(str(e)) + + @needscredentials + def test_delete_domain_example(self): + """ + delete_domain request example + """ + try: + # begin-delete_domain + + response = dph_service.delete_domain( + domain_id='testString', + ) + + # end-delete_domain + print('\ndelete_domain() response status code: ', response.get_status_code()) + + except ApiException as e: + pytest.fail(str(e)) + + +# endregion +############################################################################## +# End of Examples for Service: DphV1 +############################################################################## diff --git a/requirements-dev.txt b/requirements-dev.txt new file mode 100644 index 0000000..0da4263 --- /dev/null +++ b/requirements-dev.txt @@ -0,0 +1,17 @@ +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +ibm-cloud-sdk-core==3.24.4 +black>=26.3.1 +pylint>=3.0.0 diff --git a/requirements.txt b/requirements.txt index 1b9de16..0951bc4 100644 --- a/requirements.txt +++ b/requirements.txt @@ -14,18 +14,24 @@ # Core dependencies (defined in setup.py, included here for development convenience) pydantic>=2.12.0 -requests>=2.28.0 +requests>=2.32.4 regex>=2023.0.0 +urllib3>=2.6.3 +python-dateutil>=2.5.3,<3.0.0 +PyJWT>=2.12.1 +pyyaml>=5.4.0,<7.0.0 +numpy>=1.24.0 # Note: ibm-cloud-sdk-core is defined in setup.py with exact version pin # Development dependencies pytest>=7.0.0 pytest-cov>=4.0.0 pytest-mock>=3.7.0 +responses>=0.20.0 # black is defined in setup.py extras_require['dev'] to avoid BOM conflicts mypy>=1.0.0 flake8>=6.0.0 # Optional dependencies for development/testing -pandas>=1.3.0 -pyspark>=3.0.0 \ No newline at end of file +pandas>=2.0.0 +pyspark>=3.0.0 diff --git a/setup.py b/setup.py index 03e0d0c..0f2301a 100644 --- a/setup.py +++ b/setup.py @@ -27,10 +27,11 @@ setup( name="data-intelligence-sdk", - version='0.5.3', + version='1.0.0', author="IBM", author_email="Data_Intelligence_SDK@wwpdl.vnet.ibm.com", - description="A Python SDK for performing data quality validations on streaming data records and DataFrames", + description="A Python SDK for IBM watsonx.data intelligence that provides data quality validation for streaming records and DataFrames, " \ + "wrapper methods to access Data Product Hub REST API services, ODCS file generation from data catalogs, and data product recommendations from query log analysis.", long_description=long_description, long_description_content_type="text/markdown", url="https://github.com/IBM/data-intelligence-sdk", @@ -49,11 +50,15 @@ "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", ], - python_requires=">=3.8", + python_requires=">=3.10", install_requires=[ "pydantic>=2.12.0", - "requests>=2.28.0", + "requests>=2.33.1", "regex>=2023.0.0", + "urllib3>=2.6.3", + "python-dateutil>=2.5.3,<3.0.0", + "pyyaml>=5.4.0,<7.0.0", + "numpy>=1.24.0", # Pinned to exact version to avoid CRA bom-generate pip resolver conflict. # CRA sees ibm-cloud-sdk-core from both setup.py and requirements.txt and # fails with ResolutionImpossible when constraints differ (bare vs >=). @@ -64,29 +69,33 @@ "pytest>=7.0.0", "pytest-cov>=4.0.0", "pytest-mock>=3.7.0", + "responses>=0.20.0", "black>=26.3.1", "mypy>=1.0.0", "flake8>=6.0.0", + "pylint>=3.0.0", ], "pandas": [ - "pandas>=1.3.0", + "pandas>=2.0.0", ], "spark": [ "pyspark>=3.0.0", ], "dataframes": [ - "pandas>=1.3.0", + "pandas>=2.0.0", "pyspark>=3.0.0", ], "all": [ - "pandas>=1.3.0", + "pandas>=2.0.0", "pyspark>=3.0.0", "pytest>=7.0.0", "pytest-cov>=4.0.0", "pytest-mock>=3.7.0", + "responses>=0.20.0", "black>=26.3.1", "mypy>=1.0.0", "flake8>=6.0.0", + "pylint>=3.0.0", ], }, entry_points={ diff --git a/src/wxdi/__init__.py b/src/wxdi/__init__.py index 497cd9a..63261f8 100644 --- a/src/wxdi/__init__.py +++ b/src/wxdi/__init__.py @@ -16,7 +16,8 @@ """ WXDI - IBM watsonx.data intelligence SDK -A comprehensive Python SDK for data quality validation and data intelligence operations. +A comprehensive Python SDK for data quality validation, data intelligence operations, +data product hub services, ODCS generation, and data product recommendations. """ # Re-export commonly used modules for convenience @@ -98,4 +99,10 @@ "DataQualityDimension", ] +# Note: dph_services, odcs_generator, and data_product_recommender are available as submodules +# Import them explicitly: +# from wxdi.dph_services import DphV1 +# from wxdi.odcs_generator import CollibraClient, ODCSGenerator, InformaticaClient +# from wxdi.data_product_recommender import DataProductRecommender + # Made with Bob diff --git a/src/wxdi/data_product_recommender/README.md b/src/wxdi/data_product_recommender/README.md new file mode 100644 index 0000000..5c71fbc --- /dev/null +++ b/src/wxdi/data_product_recommender/README.md @@ -0,0 +1,218 @@ + + +# Data Product Recommender + +A tool that analyzes database query logs from files to identify high-value tables and logical groupings that should be prioritized as data products in a data marketplace. + +## Purpose + +**Accelerate data product onboarding** by leveraging existing usage patterns rather than starting from scratch. + +Organizations often have valuable data assets already in active use across various teams and use cases. Instead of building new data pipelines or guessing which tables to promote, this tool analyzes actual query patterns to identify: + +- **High-value tables**: Frequently queried by diverse user groups +- **Logical groupings**: Tables that are commonly used together +- **Proven assets**: Tables with demonstrated business value through real usage + +## Features + +- **Multi-platform support**: Snowflake, Databricks, BigQuery, watsonx.data query log formats +- **File-based input**: Supports CSV and JSON query log files (no direct database connection required) +- **Intelligent scoring**: Combines query frequency, user diversity, recency, and consistency +- **Table grouping**: Identifies tables frequently used together +- **Query pattern analysis**: Shows common query patterns for each recommendation +- **Multiple output formats**: Markdown (human-readable) and JSON (agent-consumable) +- **CLI and Python API**: Use from command line or integrate into your applications + +## Installation + +From the data-intelligence-sdk repository: + +```bash +# Install the package with dependencies +pip install -e . +``` + +## Quick Start + +### Command Line Interface + +```bash +# Analyze Snowflake query logs from CSV +python -m wxdi.data_product_recommender.cli \ + --platform snowflake \ + --input-file samples/query_logs.csv \ + --output output \ + --num-recommendations 20 + +# Analyze with minimum score threshold +python -m wxdi.data_product_recommender.cli \ + --platform databricks \ + --input-file samples/query_logs.json \ + --min-score 60.0 \ + --output-format json +``` + +### Python API + +```python +from data_product_recommender.platforms import SnowflakeQueryParser +from data_product_recommender.recommender import DataProductRecommender + +# Initialize with platform-specific parser +parser = SnowflakeQueryParser() +recommender = DataProductRecommender(parser) + +# Load and analyze query logs from file +recommender.load_query_logs_from_csv_file('query_logs.csv') +recommender.calculate_metrics() +recommendations = recommender.recommend_data_products(num_recommendations=20) + +# Export results +recommender.export_recommendations_markdown(recommendations, 'output/recommendations.md') +recommender.export_recommendations_json(recommendations, 'output/recommendations.json') +``` + +## Methodology + +### Individual Table Scoring (0-100) + +- **37.5%** Query Count - Volume of usage +- **37.5%** User Diversity - Breadth of usage across teams +- **15%** Recency - Recent activity (prioritizes currently active tables) +- **10%** Consistency - Regular usage patterns (identifies stable, ongoing value) + +### Table Group Scoring (0-100) + +- **30%** Cohesion - How tightly tables are connected +- **20%** Usage - Relative usage compared to other groups +- **15%** User Reach - Percentage of users querying the group +- **20%** Recency - Recent activity across tables +- **10%** Consistency - Regular usage patterns +- **5%** Size - Number of tables in the group + +### Star Rating Scale + +- ⭐⭐⭐⭐⭐ **Excellent (80-100)**: Implement immediately +- ⭐⭐⭐⭐ **Good (60-79)**: Medium priority +- ⭐⭐⭐ **Fair (40-59)**: Consider splitting or implement later +- ⭐⭐ **Weak (20-39)**: Reconsider grouping +- ⭐ **Poor (0-19)**: Do not implement + +## CLI Options + +``` +--platform Database platform (snowflake, databricks, bigquery, watsonxdata) +--input-file Path to CSV or JSON query log file +--output Output directory (default: output) +--output-format Output format: markdown or json (default: markdown) +--num-recommendations Number of recommendations (default: 20) +--min-score Minimum score threshold 0-100 (filters low-scoring tables) +``` + +## Output Formats + +### Markdown (Human-Readable) +- Rich formatting with tables and collapsible sections +- Star ratings and detailed explanations +- Query pattern examples with syntax highlighting +- Comprehensive coverage statistics + +### JSON (Agent-Consumable) +- Clean, parseable structure +- All metrics as numeric values +- Rating labels (excellent, good, fair, weak, poor) +- Suitable for AI agents and automated workflows + +## Platform Support + +This tool supports query log files from the following platforms: + +- ✅ **Snowflake** - Export from `SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY` +- ✅ **Databricks** - Export from `system.query.history` (Unity Catalog) +- ✅ **BigQuery** - Export from `INFORMATION_SCHEMA.JOBS_BY_PROJECT` +- ✅ **watsonx.data** - Export from `system.runtime.queries` (Presto) + +**Note**: This tool requires pre-exported query log files in CSV or JSON format. It does not connect directly to databases. + +## Examples + +See the `examples/` directory for detailed usage examples: + +- `data_product_recommender_example.py` - Basic usage with different platforms +- Custom scoring weights +- JSON and CSV input formats + +## Testing + +```bash +# Run unit tests +pytest tests/src/data_product_recommender/ -v + +# Run with coverage +pytest tests/src/data_product_recommender/ --cov=wxdi.data_product_recommender --cov-report=html +``` + +## Architecture + +The tool uses an extensible design with abstract base classes: + +``` +QueryLogParser (Abstract) +├── SnowflakeQueryParser +├── DatabricksQueryParser +├── BigQueryQueryParser +└── WatsonxDataQueryParser + +DataProductRecommender +└── Uses parser for platform-specific query log formats +``` + +## Important Notes + +### Query Log Export + +To use this tool, you need to export query logs from your data platform: + +**Snowflake Example:** +```sql +SELECT query_text, user_name, start_time, end_time, execution_status +FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY +WHERE start_time >= DATEADD(day, -30, CURRENT_TIMESTAMP()) +``` + +**Databricks Example:** +```sql +SELECT statement_text, user_name, start_time, end_time, status +FROM system.query.history +WHERE start_time >= current_timestamp() - INTERVAL 30 DAYS +``` + +### Required Columns + +Query log files must contain these columns (column names will be normalized by the parser): +- `query_text` - The SQL query text +- `user` - User who executed the query +- `start_time` - Query execution timestamp + +### Data Privacy + +Query logs may contain sensitive information including user identities, table names, and query patterns. Ensure proper data handling and security measures are in place. + +## License + +Apache License 2.0 - See LICENSE file in the root directory. \ No newline at end of file diff --git a/src/wxdi/data_product_recommender/__init__.py b/src/wxdi/data_product_recommender/__init__.py new file mode 100644 index 0000000..d226660 --- /dev/null +++ b/src/wxdi/data_product_recommender/__init__.py @@ -0,0 +1,24 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Data Product Recommender - Query Log Analysis Tool + +Analyzes query logs from various data platforms to recommend data products +based on usage patterns, user diversity, and table relationships. +""" + +__version__ = "0.1.0" \ No newline at end of file diff --git a/src/wxdi/data_product_recommender/base.py b/src/wxdi/data_product_recommender/base.py new file mode 100644 index 0000000..89c8916 --- /dev/null +++ b/src/wxdi/data_product_recommender/base.py @@ -0,0 +1,37 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Abstract base class for query log parsers. +""" + +from abc import ABC, abstractmethod +from typing import List +import pandas as pd + + +class QueryLogParser(ABC): + """Abstract base class for parsing query logs""" + + @abstractmethod + def normalize_columns(self, df: pd.DataFrame) -> pd.DataFrame: + """Normalize column names to standard format""" + pass + + @abstractmethod + def extract_tables(self, query_text: str) -> List[str]: + """Extract table names from SQL query""" + pass \ No newline at end of file diff --git a/src/wxdi/data_product_recommender/cli.py b/src/wxdi/data_product_recommender/cli.py new file mode 100644 index 0000000..d884a90 --- /dev/null +++ b/src/wxdi/data_product_recommender/cli.py @@ -0,0 +1,152 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Command-line interface for the Data Product Recommender +""" + +import argparse +import json +from datetime import datetime, timedelta +from pathlib import Path + +from .platforms import ( + SnowflakeQueryParser, + DatabricksQueryParser, + BigQueryQueryParser, + WatsonxDataQueryParser +) +from .recommender import DataProductRecommender + + +def main(): + parser = argparse.ArgumentParser( + description='Analyze query logs to recommend data products' + ) + + # Platform selection + parser.add_argument( + '--platform', + choices=['snowflake', 'databricks', 'bigquery', 'watsonxdata'], + required=True, + help='Data platform to analyze' + ) + + # Input source (file-based only) + parser.add_argument( + '--input-file', + type=str, + required=True, + help='Path to CSV or JSON file containing query logs' + ) + + # Output options + parser.add_argument( + '--output', + type=str, + default='output', + help='Output directory for recommendations (default: output)' + ) + + parser.add_argument( + '--output-format', + choices=['markdown', 'json'], + default='markdown', + help='Output format: markdown (human-readable) or json (agent-consumable) (default: markdown)' + ) + + parser.add_argument( + '--num-recommendations', + type=int, + default=20, + help='Number of recommendations to generate (default: 20)' + ) + + parser.add_argument( + '--min-score', + type=float, + default=None, + help='Minimum recommendation score threshold (0-100). Tables below this score will be excluded.' + ) + + args = parser.parse_args() + + # Initialize platform-specific parser + print(f"Initializing {args.platform} query parser...") + + if args.platform == 'snowflake': + parser = SnowflakeQueryParser() + elif args.platform == 'databricks': + parser = DatabricksQueryParser() + elif args.platform == 'bigquery': + parser = BigQueryQueryParser() + elif args.platform == 'watsonxdata': + parser = WatsonxDataQueryParser() + + # Initialize recommender + recommender = DataProductRecommender(parser) + + # Load query logs from file + print(f"Loading query logs from file: {args.input_file}") + + if args.input_file.lower().endswith('.json'): + recommender.load_query_logs_from_json_file(args.input_file) + elif args.input_file.lower().endswith('.csv'): + recommender.load_query_logs_from_csv_file(args.input_file) + else: + raise ValueError("Invalid --input-file type. Supported file types are JSON and CSV.") + + # Calculate metrics + print("\nCalculating metrics...") + recommender.calculate_metrics() + + # Generate recommendations + print("\nGenerating recommendations...") + recommendations = recommender.recommend_data_products( + num_recommendations=args.num_recommendations, + min_score=args.min_score + ) + + # Create output directory + output_dir = Path(args.output) + output_dir.mkdir(parents=True, exist_ok=True) + + # Generate output filename based on format + timestamp = datetime.now().strftime('%Y%m%d_%H%M%S') + input_name = Path(args.input_file).stem if args.input_file else 'database' + file_extension = 'json' if args.output_format == 'json' else 'md' + output_file = output_dir / f"recommendations_{input_name}_{timestamp}.{file_extension}" + + # Export recommendations in selected format + print(f"\nExporting recommendations to {output_file}...") + if args.output_format == 'json': + recommender.export_recommendations_json(recommendations, str(output_file)) + else: + recommender.export_recommendations_markdown(recommendations, str(output_file)) + + print("\n✓ Analysis complete!") + print(f" - Queries analyzed: {len(recommender.query_logs):,}") + print(f" - Tables identified: {len(recommender.table_metrics)}") + print(f" - Recommendations: {len(recommendations['individual_tables'])}") + if 'table_groups' in recommendations: + print(f" - Table groups: {len(recommendations['table_groups'])}") + print(f" - Output file: {output_file}") + + +if __name__ == '__main__': + main() + +# Made with Bob diff --git a/src/wxdi/data_product_recommender/platforms.py b/src/wxdi/data_product_recommender/platforms.py new file mode 100644 index 0000000..b0c6d7b --- /dev/null +++ b/src/wxdi/data_product_recommender/platforms.py @@ -0,0 +1,167 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Platform-specific query log parsers for Snowflake, Databricks, BigQuery, and watsonx.data + +Note: This module only provides query parsing functionality. +Database connections are not supported - use file-based input instead. +""" + +import re +from typing import List +import pandas as pd + +from .base import QueryLogParser + +# Regex pattern for extracting schema.table from FROM/JOIN clauses +TABLE_PATTERN = r'(?:FROM|JOIN)\s+([\w]+\.[\w]+)(?:\s+[a-zA-Z])?' + + +# ============================================================================ +# SNOWFLAKE QUERY PARSER +# ============================================================================ + +class SnowflakeQueryParser(QueryLogParser): + """Snowflake-specific query parser""" + + def normalize_columns(self, df: pd.DataFrame) -> pd.DataFrame: + """Normalize Snowflake column names""" + return df.rename(columns={ + 'query_text': 'query_text', + 'user_name': 'user', + 'start_time': 'start_time' + }) + + def extract_tables(self, query_text: str) -> List[str]: + """Extract table names from SQL query using regex patterns""" + if not query_text or not isinstance(query_text, str): + return [] + + query_upper = query_text.upper() + tables = set() + + # Pattern: Match FROM and JOIN clauses with schema.table format + matches = re.findall(TABLE_PATTERN, query_upper) + tables.update(matches) + + return list(tables) + + +# ============================================================================ +# DATABRICKS QUERY PARSER +# ============================================================================ + +class DatabricksQueryParser(QueryLogParser): + """Databricks-specific query parser""" + + def normalize_columns(self, df: pd.DataFrame) -> pd.DataFrame: + """Normalize Databricks column names""" + return df.rename(columns={ + 'statement_text': 'query_text', + 'executed_by': 'user', + 'start_time': 'start_time' + }) + + def extract_tables(self, query_text: str) -> List[str]: + """Extract table names from SQL query using regex patterns""" + if not query_text or not isinstance(query_text, str): + return [] + + query_upper = query_text.upper() + tables = set() + + # Pattern: Match FROM and JOIN clauses with schema.table format + matches = re.findall(TABLE_PATTERN, query_upper) + tables.update(matches) + + return list(tables) + + +# ============================================================================ +# BIGQUERY QUERY PARSER +# ============================================================================ + +class BigQueryQueryParser(QueryLogParser): + """BigQuery-specific query parser""" + + def normalize_columns(self, df: pd.DataFrame) -> pd.DataFrame: + """Normalize BigQuery column names""" + return df.rename(columns={ + 'query': 'query_text', + 'user_email': 'user', + 'start_time': 'start_time' + }) + + def extract_tables(self, query_text: str) -> List[str]: + """Extract table names from BigQuery SQL query using regex patterns""" + if not query_text or not isinstance(query_text, str): + return [] + + query_upper = query_text.upper() + tables = set() + + # BigQuery pattern: Match backtick-quoted tables with project.dataset.table format + # Example: `retailco-project.PRODUCT.CATALOG` + pattern1 = r'`([\w-]+\.[\w]+\.[\w]+)`' + matches1 = re.findall(pattern1, query_text) # Use original case for project names + + # Simplify to DATASET.TABLE format (remove project prefix) + for match in matches1: + parts = match.split('.') + if len(parts) == 3: + # Keep only dataset.table + simplified = f"{parts[1]}.{parts[2]}" + tables.add(simplified.upper()) + + # Also try standard pattern for dataset.table without backticks + matches2 = re.findall(TABLE_PATTERN, query_upper) + tables.update(matches2) + + return list(tables) + + +# ============================================================================ +# WATSONX.DATA (PRESTO) QUERY PARSER +# ============================================================================ + +class WatsonxDataQueryParser(QueryLogParser): + """watsonx.data-specific query parser""" + + def normalize_columns(self, df: pd.DataFrame) -> pd.DataFrame: + """Normalize watsonx.data column names""" + return df.rename(columns={ + 'query': 'query_text', + 'user': 'user', + 'created': 'start_time' + }) + + def extract_tables(self, query_text: str) -> List[str]: + """Extract table names from SQL query using regex patterns""" + if not query_text or not isinstance(query_text, str): + return [] + + query_upper = query_text.upper() + tables = set() + + # Pattern: Match FROM and JOIN clauses with schema.table format + matches = re.findall(TABLE_PATTERN, query_upper) + tables.update(matches) + + return list(tables) + + +# Made with Bob \ No newline at end of file diff --git a/src/wxdi/data_product_recommender/recommender.py b/src/wxdi/data_product_recommender/recommender.py new file mode 100644 index 0000000..65f9732 --- /dev/null +++ b/src/wxdi/data_product_recommender/recommender.py @@ -0,0 +1,1292 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Data Product Recommender - Core recommendation engine +""" + +import json +import re +import pandas as pd +from datetime import datetime +from collections import Counter, defaultdict +from typing import List, Dict, Tuple, Optional + +from .base import QueryLogParser + +# Star rating constants +FIVE_STARS = "⭐⭐⭐⭐⭐" +FOUR_STARS = "⭐⭐⭐⭐" +THREE_STARS = "⭐⭐⭐" +TWO_STARS = "⭐⭐" +ONE_STAR = "⭐" + +EXCELLENT_CANDIDATE = "Excellent Candidate" +GOOD_CANDIDATE = "Good Candidate" +FAIR_CANDIDATE = "Fair Candidate" +WEAK_CANDIDATE = "Weak Candidate" +POOR_CANDIDATE = "Poor Candidate" + +# Report section headers +QUERY_LOG_METRICS_HEADER = "**Query Log Metrics:**\n" +TABLES_IN_GROUP_HEADER = "**Tables in Group:**\n\n" +TABLES_IN_GROUP_SIMPLE = "**Tables in Group:**\n" +TABLE_HEADER_ROW = "| Table | Individual Score | Queries | Users | Recency | Consistency |\n" +TABLE_SEPARATOR_ROW = "|-------|-----------------|---------|-------|---------|-------------|\n" + +# SQL code block markers +SQL_CODE_BLOCK_START = " ```sql\n" +SQL_CODE_BLOCK_END = " ```\n" +SQL_EXAMPLE_LABEL = " - Example:\n" +SQL_EXAMPLE_CODE_START = " ```sql\n" +SQL_EXAMPLE_CODE_END = " ```\n" + +# HTML collapsible section markers +DETAILS_START = "
\n" +DETAILS_SUMMARY = "Frequent Query Patterns (click to expand)\n\n" +DETAILS_END = "
\n\n" + + +def normalize_query_pattern(query_text: str) -> str: + """ + Normalize a SQL query into a pattern by replacing literals with placeholders. + + This helps group similar queries together by removing variable parts like: + - Numbers (123 -> ?) + - Quoted strings ('value' -> ?) + - Date literals ('2024-01-01' -> ?) + + Args: + query_text: The SQL query text to normalize + + Returns: + Normalized query pattern with literals replaced by ? + """ + if not query_text or not isinstance(query_text, str): + return "" + + # Convert to uppercase for consistency + pattern = query_text.upper() + + # Replace quoted strings (both single and double quotes) + # Handles escaped quotes within strings + pattern = re.sub(r"'(?:[^'\\]|\\.)*'", "?", pattern) + pattern = re.sub(r'"(?:[^"\\]|\\.)*"', "?", pattern) + + # Replace numbers (integers and decimals) + pattern = re.sub(r'\b\d+\.?\d*\b', '?', pattern) + + # Replace date/timestamp patterns that might remain + # Format: YYYY-MM-DD or YYYY-MM-DD HH:MM:SS + pattern = re.sub(r'\b\?-\?-\?\b', '?', pattern) + pattern = re.sub(r'\b\? \?:\?:\?\b', '?', pattern) + + # Normalize whitespace (multiple spaces/tabs/newlines to single space) + pattern = re.sub(r'\s+', ' ', pattern) + + # Trim leading/trailing whitespace + pattern = pattern.strip() + + return pattern + + +class DataProductRecommender: + """Analyzes query logs and recommends data products""" + + # Error message constants + ERROR_QUERY_LOGS_NOT_LOADED = "Query logs not loaded" + + def __init__(self, parser: QueryLogParser): + self.parser = parser + self.query_logs = None + self.table_metrics = None + self.query_patterns = None # Store query pattern information + + def load_query_logs_from_json_file(self, file_path: str): + """Load and normalize query logs from JSON file""" + print(f"Reading query logs from {file_path}...") + with open(file_path, 'r') as f: + data = json.load(f) + + df = pd.DataFrame(data) + df = self.parser.normalize_columns(df) + + print("Extracting table names from queries...") + df['tables'] = df['query_text'].apply(self.parser.extract_tables) + df = df[df['tables'].apply(len) > 0] + + print("Normalizing query patterns...") + df['query_pattern'] = df['query_text'].apply(normalize_query_pattern) + + self.query_logs = df + print(f"Processed {len(df):,} queries with table references") + return df + + def load_query_logs_from_csv_file(self, file_path: str): + """Load and normalize query logs from CSV or JSON file""" + print(f"Reading query logs from {file_path}...") + + # Read file based on extension + if file_path.lower().endswith('.json'): + df = pd.read_json(file_path) + else: + df = pd.read_csv(file_path) + + # Normalize column names based on platform + df = self.parser.normalize_columns(df) + + # Verify required columns exist + required_cols = ['query_text', 'user', 'start_time'] + missing_cols = [col for col in required_cols if col not in df.columns] + if missing_cols: + raise ValueError(f"CSV file missing required columns after normalization: {missing_cols}") + + print("Extracting table names from queries...") + df['tables'] = df['query_text'].apply(self.parser.extract_tables) + + # Filter out queries with no table references + df = df[df['tables'].apply(len) > 0] + + print("Normalizing query patterns...") + df['query_pattern'] = df['query_text'].apply(normalize_query_pattern) + + self.query_logs = df + print(f"Processed {len(df):,} queries with table references") + return df + + def _calculate_table_query_counts(self) -> Counter: + """Calculate query count per table""" + table_query_count = Counter() + for tables in self.query_logs['tables']: + table_query_count.update(tables) + return table_query_count + + def _calculate_table_users(self) -> dict: + """Calculate user diversity per table""" + table_users = defaultdict(set) + for _, row in self.query_logs.iterrows(): + for table in row['tables']: + table_users[table].add(row['user']) + return table_users + + def _calculate_table_cooccurrence(self) -> dict: + """Calculate table co-occurrence""" + table_cooccurrence = defaultdict(Counter) + for tables in self.query_logs['tables']: + for table1 in tables: + for table2 in tables: + if table1 != table2: + table_cooccurrence[table1][table2] += 1 + return table_cooccurrence + + def _calculate_table_timestamps(self) -> dict: + """Calculate temporal metrics per table""" + table_timestamps = defaultdict(list) + for _, row in self.query_logs.iterrows(): + for table in row['tables']: + table_timestamps[table].append(row['start_time']) + return table_timestamps + + def _calculate_recency_score(self, timestamps: list, reference_time) -> tuple: + """Calculate recency score and days since last query""" + days_since_last_query = max(0, (reference_time - max(timestamps)).days) + recency_score = 1.0 / (1.0 + days_since_last_query) + return recency_score, days_since_last_query + + def _calculate_consistency_score(self, timestamps: list) -> float: + """Calculate consistency score based on query intervals""" + if len(timestamps) <= 1: + return 0.0 + + sorted_times = sorted(timestamps) + intervals = [(sorted_times[i+1] - sorted_times[i]).days + for i in range(len(sorted_times)-1)] + total_interval = sum(intervals) + + if len(intervals) == 0 or total_interval == 0: + return 0.5 + + mean_interval = total_interval / len(intervals) + std_interval = (sum((x - mean_interval)**2 for x in intervals) / len(intervals))**0.5 + cv = std_interval / mean_interval if mean_interval > 0 else 0 + return 1.0 / (1.0 + cv) + + def calculate_metrics(self) -> pd.DataFrame: + """Calculate metrics for each table""" + if self.query_logs is None: + raise ValueError(self.ERROR_QUERY_LOGS_NOT_LOADED) + + print("Calculating table metrics...") + + # Convert start_time to datetime if it's not already + self.query_logs['start_time'] = pd.to_datetime(self.query_logs['start_time']) + + # Calculate all base metrics + table_query_count = self._calculate_table_query_counts() + table_users = self._calculate_table_users() + table_cooccurrence = self._calculate_table_cooccurrence() + table_timestamps = self._calculate_table_timestamps() + + # Use the most recent query in the dataset as reference for recency calculations + reference_time = self.query_logs['start_time'].max() + + # Build metrics dataframe + metrics = [] + for table, query_count in table_query_count.items(): + user_count = len(table_users[table]) + related_tables = [t for t, c in table_cooccurrence[table].most_common(10)] + timestamps = table_timestamps[table] + + recency_score, days_since_last_query = self._calculate_recency_score(timestamps, reference_time) + consistency_score = self._calculate_consistency_score(timestamps) + + metrics.append({ + 'table': table, + 'query_count': query_count, + 'unique_users': user_count, + 'related_tables': related_tables, + 'related_table_count': len(related_tables), + 'recency_score': recency_score, + 'consistency_score': consistency_score, + 'days_since_last_query': days_since_last_query, + 'first_query_date': min(timestamps), + 'last_query_date': max(timestamps) + }) + + self.table_metrics = pd.DataFrame(metrics) + print(f"Calculated metrics for {len(self.table_metrics):,} tables") + return self.table_metrics + + def score_tables(self, + query_weight=0.375, + user_weight=0.375, + recency_weight=0.15, + consistency_weight=0.10): + """ + Score tables based on weighted metrics + + Args: + query_weight: Weight for query frequency (default 0.375) + user_weight: Weight for user diversity (default 0.375) + recency_weight: Weight for recent activity (default 0.15) + consistency_weight: Weight for consistent usage over time (default 0.10) + + Returns: + DataFrame with scored tables sorted by recommendation_score + + Note: + Relationship metrics are not included as standalone tables are packaged + in isolation without their related tables. + """ + if self.table_metrics is None: + raise ValueError("Metrics not calculated") + + # Validate weights sum to 1.0 + total_weight = query_weight + user_weight + recency_weight + consistency_weight + if abs(total_weight - 1.0) > 0.001: + raise ValueError(f"Weights must sum to 1.0, got {total_weight}") + + df = self.table_metrics.copy() + + # Normalize metrics to 0-1 scale, handling edge cases + max_queries = df['query_count'].max() + max_users = df['unique_users'].max() + + df['query_score'] = df['query_count'] / max_queries if max_queries > 0 else 0 + df['user_score'] = df['unique_users'] / max_users if max_users > 0 else 0 + + # Temporal scores are already 0-1, but ensure no NaN values + df['recency_score'] = df['recency_score'].fillna(0) + df['consistency_score'] = df['consistency_score'].fillna(0) + + # Calculate weighted score + df['recommendation_score'] = ( + df['query_score'] * query_weight + + df['user_score'] * user_weight + + df['recency_score'] * recency_weight + + df['consistency_score'] * consistency_weight + ) * 100 + + return df.sort_values('recommendation_score', ascending=False) + def get_top_query_patterns(self, tables: List[str], top_n: int = 5) -> List[Dict]: + """ + Get the most frequent query patterns for a set of tables. + + Args: + tables: List of table names to analyze + top_n: Number of top patterns to return (default 5) + + Returns: + List of dictionaries with pattern info: + - pattern: The normalized query pattern + - count: Number of times this pattern appears + - example: An actual query text example + - tables_used: Tables from the group that appear in this pattern + """ + if self.query_logs is None: + raise ValueError(self.ERROR_QUERY_LOGS_NOT_LOADED) + + if 'query_pattern' not in self.query_logs.columns: + raise ValueError("Query patterns not computed. Reload query logs.") + + # Filter queries that reference any of the specified tables + relevant_queries = self.query_logs[ + self.query_logs['tables'].apply( + lambda query_tables: any(t in tables for t in query_tables) + ) + ].copy() + + if len(relevant_queries) == 0: + return [] + + # Count pattern frequencies + pattern_counts = Counter(relevant_queries['query_pattern']) + + # Get top N patterns + top_patterns = [] + for pattern, count in pattern_counts.most_common(top_n): + # Find an example query for this pattern + example_row = relevant_queries[relevant_queries['query_pattern'] == pattern].iloc[0] + example_query = example_row['query_text'] + + # Determine which tables from the group are used in this pattern + tables_in_pattern = [t for t in tables if t in example_row['tables']] + + top_patterns.append({ + 'pattern': pattern, + 'count': count, + 'example': example_query, + 'tables_used': tables_in_pattern + }) + + return top_patterns + + + def _count_group_queries(self, table_group: tuple) -> int: + """Count queries involving any table in the group""" + if self.query_logs is None: + return 1 # Avoid division by zero + + count = sum(1 for tables_in_query in self.query_logs['tables'] + if any(t in tables_in_query for t in table_group)) + return max(count, 1) # Ensure at least 1 to avoid division by zero + + def _get_pairwise_frequencies(self, table_group: tuple, table_pair_counts: dict, + group_query_count: int) -> tuple: + """Get pairwise join frequencies and percentages for a table group""" + frequencies = [] + percentages = [] + + for i, table1 in enumerate(table_group): + for table2 in table_group[i+1:]: + pair = tuple(sorted([table1, table2])) + freq = table_pair_counts.get(pair, 0) + frequencies.append(freq) + percentages.append(freq / group_query_count) + + return frequencies, percentages + + def _calculate_group_cohesion(self, table_group: tuple, table_pair_counts: dict) -> dict: + """ + Calculate cohesion metrics for a table group + + Args: + table_group: Tuple of table names in the group + table_pair_counts: Dictionary mapping (table1, table2) -> count + + Returns: + Dictionary with cohesion metrics including: + - avg_join_frequency: Average absolute co-occurrence count + - avg_join_percentage: Average co-occurrence as percentage of group queries (0-1) + - min_join_frequency: Minimum co-occurrence count + - max_join_frequency: Maximum co-occurrence count + - total_pairs: Number of table pairs in the group + """ + if len(table_group) < 2: + return { + 'avg_join_frequency': 0, + 'avg_join_percentage': 0, + 'min_join_frequency': 0, + 'max_join_frequency': 0, + 'total_pairs': 0 + } + + group_query_count = self._count_group_queries(table_group) + frequencies, percentages = self._get_pairwise_frequencies( + table_group, table_pair_counts, group_query_count + ) + + return { + 'avg_join_frequency': sum(frequencies) / len(frequencies) if frequencies else 0, + 'avg_join_percentage': sum(percentages) / len(percentages) if percentages else 0, + 'min_join_frequency': min(frequencies) if frequencies else 0, + 'max_join_frequency': max(frequencies) if frequencies else 0, + 'total_pairs': len(frequencies) + } + + def _count_table_occurrences(self) -> Counter: + """Count how many times each table appears in queries""" + table_query_counts = Counter() + for tables in self.query_logs['tables']: + table_query_counts.update(tables) + return table_query_counts + + def _count_table_pairs(self) -> Counter: + """Count pairwise co-occurrences of tables""" + table_pair_counts = Counter() + for tables in self.query_logs['tables']: + if len(tables) >= 2: + for i, table1 in enumerate(tables): + for table2 in tables[i+1:]: + pair = tuple(sorted([table1, table2])) + table_pair_counts[pair] += 1 + return table_pair_counts + + def _build_strong_connections(self, table_pair_counts: Counter, table_query_counts: Counter, + min_frequency_threshold: float) -> dict: + """Build adjacency list of strong table connections""" + strong_connections = defaultdict(set) + + for (table1, table2), count in table_pair_counts.items(): + freq1 = count / table_query_counts[table1] if table_query_counts[table1] > 0 else 0 + freq2 = count / table_query_counts[table2] if table_query_counts[table2] > 0 else 0 + min_freq = min(freq1, freq2) + + if min_freq >= min_frequency_threshold: + strong_connections[table1].add(table2) + strong_connections[table2].add(table1) + + return strong_connections + + def _find_best_candidate(self, candidates: set, cluster: set, strong_connections: dict) -> tuple: + """Find the best candidate to add to a cluster""" + best_candidate = None + best_connection_count = 0 + + for candidate in candidates: + connection_count = len(cluster & strong_connections[candidate]) + if connection_count > best_connection_count: + best_candidate = candidate + best_connection_count = connection_count + + return best_candidate, best_connection_count + + def _grow_cluster(self, core_table: str, strong_connections: dict, visited: set, + max_group_size: int) -> set: + """Grow a cluster starting from a core table""" + cluster = {core_table} + candidates = strong_connections[core_table] - visited + + while candidates and len(cluster) < max_group_size: + best_candidate, best_connection_count = self._find_best_candidate( + candidates, cluster, strong_connections + ) + + if best_candidate and best_connection_count > 0: + cluster.add(best_candidate) + candidates.remove(best_candidate) + candidates.update(strong_connections[best_candidate] - visited - cluster) + else: + break + + return cluster + + def _create_cluster_dict(self, cluster_tuple: tuple, table_pair_counts: Counter) -> dict: + """Create a cluster dictionary with cohesion metrics""" + cohesion = self._calculate_group_cohesion(cluster_tuple, table_pair_counts) + + return { + 'tables': list(cluster_tuple), + 'size': len(cluster_tuple), + 'avg_join_frequency': cohesion['avg_join_frequency'], + 'avg_join_percentage': cohesion['avg_join_percentage'], + 'min_join_frequency': cohesion['min_join_frequency'], + 'max_join_frequency': cohesion['max_join_frequency'] + } + + def _build_frequency_clusters(self, min_frequency_threshold=0.10, min_group_size=2, max_group_size=10): + """ + Build table clusters using frequency-based threshold approach + + Args: + min_frequency_threshold: Minimum join frequency as percentage (0.0-1.0). + Tables must join together in at least this % of queries + where either table appears. + min_group_size: Minimum number of tables in a group + max_group_size: Maximum number of tables in a group + + Returns: + List of dictionaries with cluster information + """ + if self.query_logs is None: + raise ValueError(self.ERROR_QUERY_LOGS_NOT_LOADED) + + print(f"Building frequency-based clusters (threshold: {min_frequency_threshold*100:.1f}%)...") + + table_query_counts = self._count_table_occurrences() + table_pair_counts = self._count_table_pairs() + strong_connections = self._build_strong_connections( + table_pair_counts, table_query_counts, min_frequency_threshold + ) + + visited = set() + clusters = [] + + sorted_tables = sorted(strong_connections.keys(), + key=lambda t: len(strong_connections[t]), + reverse=True) + + for core_table in sorted_tables: + if core_table in visited: + continue + + cluster = self._grow_cluster(core_table, strong_connections, visited, max_group_size) + + if len(cluster) >= min_group_size: + visited.update(cluster) + cluster_tuple = tuple(sorted(cluster)) + clusters.append(self._create_cluster_dict(cluster_tuple, table_pair_counts)) + + clusters.sort(key=lambda x: x['avg_join_frequency'], reverse=True) + + print(f"Built {len(clusters)} frequency-based clusters") + return clusters + + def _calculate_cluster_query_counts(self, clusters: list) -> list: + """Calculate query counts for all clusters""" + cluster_query_counts = [] + for cluster in clusters: + tables = cluster['tables'] + queries_with_group = sum( + 1 for tables_in_query in self.query_logs['tables'] + if any(t in tables_in_query for t in tables) + ) + cluster_query_counts.append(queries_with_group) + return cluster_query_counts + + def _get_cluster_users(self, tables: list) -> set: + """Get unique users querying a cluster""" + users_querying_group = set() + for _, row in self.query_logs.iterrows(): + if any(t in row['tables'] for t in tables): + users_querying_group.add(row['user']) + return users_querying_group + + def _get_temporal_scores(self, tables: list) -> tuple: + """Extract temporal metrics for tables in a cluster""" + recency_scores = [] + consistency_scores = [] + + for table in tables: + table_data = self.table_metrics[self.table_metrics['table'] == table] + if not table_data.empty: + recency_scores.append(table_data['recency_score'].values[0]) + consistency_scores.append(table_data['consistency_score'].values[0]) + + avg_recency = sum(recency_scores) / len(recency_scores) if recency_scores else 0.0 + avg_consistency = sum(consistency_scores) / len(consistency_scores) if consistency_scores else 0.0 + + return avg_recency, avg_consistency + + def _calculate_normalized_scores(self, cluster: dict, queries_with_group: int, + users_count: int, max_queries: int, max_group_size: int) -> dict: + """Calculate normalized scores for a cluster""" + cohesion_score = cluster['avg_join_percentage'] + usage_score = queries_with_group / max_queries if max_queries > 0 else 0 + total_users = self.query_logs['user'].nunique() + user_score = users_count / total_users if total_users > 0 else 0 + size_score = cluster['size'] / max_group_size if max_group_size > 0 else 0 + + return { + 'cohesion_score': cohesion_score, + 'usage_score': usage_score, + 'user_score': user_score, + 'size_score': size_score + } + + def _calculate_group_score(self, scores: dict, avg_recency: float, avg_consistency: float) -> float: + """Calculate weighted group score (0-100 scale)""" + return ( + scores['cohesion_score'] * 0.30 + + scores['usage_score'] * 0.20 + + scores['user_score'] * 0.15 + + avg_recency * 0.20 + + avg_consistency * 0.10 + + scores['size_score'] * 0.05 + ) * 100 + + def _score_table_groups(self, clusters: list) -> list: + """ + Calculate group-specific scores for table clusters + + Args: + clusters: List of cluster dictionaries from _build_frequency_clusters + + Returns: + List of clusters with added 'group_score' field + """ + if self.query_logs is None or self.table_metrics is None: + raise ValueError("Query logs and table metrics must be loaded") + + max_group_size = max((c['size'] for c in clusters), default=1) + cluster_query_counts = self._calculate_cluster_query_counts(clusters) + max_queries = max(cluster_query_counts) if cluster_query_counts else 1 + + scored_clusters = [] + for i, cluster in enumerate(clusters): + tables = cluster['tables'] + queries_with_group = cluster_query_counts[i] + + users_querying_group = self._get_cluster_users(tables) + avg_recency, avg_consistency = self._get_temporal_scores(tables) + + scores = self._calculate_normalized_scores( + cluster, queries_with_group, len(users_querying_group), + max_queries, max_group_size + ) + + group_score = self._calculate_group_score(scores, avg_recency, avg_consistency) + + scored_clusters.append({ + **cluster, + 'group_score': group_score, + 'total_queries': queries_with_group, + 'unique_users': len(users_querying_group), + 'avg_recency_score': avg_recency, + 'avg_consistency_score': avg_consistency, + 'cohesion_score': scores['cohesion_score'] * 100, + 'usage_score': scores['usage_score'] * 100, + 'user_score': scores['user_score'] * 100, + 'size_score': scores['size_score'] * 100 + }) + + scored_clusters.sort(key=lambda x: x['group_score'], reverse=True) + return scored_clusters + + def identify_table_groups(self, min_cooccurrence=None): + """ + Identify groups of frequently co-occurring tables + + Args: + min_cooccurrence: Minimum number of times tables must appear together. + If None, automatically calculated as 0.01% of total queries + (minimum 2, maximum 100) + + Returns: + List of tuples: (table_group, count) sorted by count descending + """ + if self.query_logs is None: + raise ValueError(self.ERROR_QUERY_LOGS_NOT_LOADED) + + print("Identifying table groups...") + + # Calculate dynamic threshold if not provided + if min_cooccurrence is None: + total_queries = len(self.query_logs) + # Use 0.01% of queries as threshold (1 in 10,000) + min_cooccurrence = max(2, min(100, int(total_queries * 0.0001))) + print(f"Auto-calculated co-occurrence threshold: {min_cooccurrence} (based on {total_queries:,} queries)") + + table_groups = Counter() + for tables in self.query_logs['tables']: + if len(tables) >= 2: + # Sort tables to ensure consistent grouping + table_groups[tuple(sorted(tables))] += 1 + + # Filter by minimum co-occurrence + filtered_groups = [(group, count) for group, count in table_groups.items() + if count >= min_cooccurrence] + + # Sort by count descending + filtered_groups.sort(key=lambda x: x[1], reverse=True) + + print(f"Found {len(filtered_groups)} table groups with {min_cooccurrence}+ co-occurrences") + return filtered_groups + + def recommend_data_products(self, + num_recommendations=10, + min_score=None, + min_frequency_threshold=0.10, + min_group_size=2, + max_group_size=10): + """ + Generate final data product recommendations using frequency-based clustering + + Args: + num_recommendations: Maximum number of top recommendations to return + min_score: Minimum recommendation score threshold (0-100). + Tables below this score will be excluded from all recommendations. + Also used to filter standalone (unclustered) tables. + If None, no score filtering is applied. + min_frequency_threshold: Minimum join frequency for clustering (0.0-1.0) + min_group_size: Minimum tables in a cluster + max_group_size: Maximum tables in a cluster + + Returns: + Dictionary with 'individual_tables', 'table_groups', and optionally 'standalone_tables' + """ + scored_tables = self.score_tables() + + # Store metadata before filtering + total_tables = len(scored_tables) + highest_score = scored_tables['recommendation_score'].max() if len(scored_tables) > 0 else 0 + + # Apply score threshold if specified + if min_score is not None: + scored_tables = scored_tables[scored_tables['recommendation_score'] >= min_score] + print(f"Applied score threshold: {min_score} (kept {len(scored_tables)} tables)") + + # Limit to top N + top_tables = scored_tables.head(num_recommendations) + + recommendations = { + 'individual_tables': top_tables.to_dict('records'), + 'metadata': { + 'total_tables': total_tables, + 'recommended_tables': len(top_tables), + 'highest_score': highest_score, + 'min_score_threshold': min_score, + 'clustering_enabled': True # Always enabled now + } + } + + # Build frequency-based clusters + clusters = self._build_frequency_clusters( + min_frequency_threshold=min_frequency_threshold, + min_group_size=min_group_size, + max_group_size=max_group_size + ) + scored_clusters = self._score_table_groups(clusters) + + # Track which tables are in clusters + tables_in_clusters = set() + for cluster in scored_clusters: + tables_in_clusters.update(cluster['tables']) + + # Add individual table details to each group + for cluster in scored_clusters[:num_recommendations]: + cluster['table_details'] = [] + for table_name in cluster['tables']: + table_info = scored_tables[scored_tables['table'] == table_name] + if not table_info.empty: + cluster['table_details'].append(table_info.iloc[0].to_dict()) + + recommendations['table_groups'] = scored_clusters[:num_recommendations] + + # Identify high-value standalone tables (not in any cluster) + # Use the same min_score threshold if provided + standalone_threshold = min_score if min_score is not None else 0 + standalone_tables = scored_tables[ + (~scored_tables['table'].isin(tables_in_clusters)) & + (scored_tables['recommendation_score'] >= standalone_threshold) + ] + + if len(standalone_tables) > 0: + recommendations['standalone_tables'] = standalone_tables.head(num_recommendations).to_dict('records') + recommendations['metadata']['standalone_tables_count'] = len(standalone_tables) + + return recommendations + + def _write_markdown_header(self, f): + """Write markdown file header""" + f.write("# Data Product Recommendations\n\n") + f.write(f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n\n") + + def _write_summary_stats(self, f): + """Write summary statistics section""" + f.write("## Summary Statistics\n\n") + if self.query_logs is not None: + f.write(f"- Total queries analyzed: {len(self.query_logs):,}\n") + f.write(f"- Unique users: {self.query_logs['user'].nunique()}\n") + if self.table_metrics is not None: + f.write(f"- Unique tables: {len(self.table_metrics)}\n") + + def _calculate_clustering_stats(self, recommendations: dict, metadata: dict) -> dict: + """Calculate statistics for clustering mode""" + num_groups = len(recommendations['table_groups']) + total_tables = metadata['total_tables'] + + tables_in_groups = set() + for group in recommendations['table_groups']: + tables_in_groups.update(group['tables']) + + tables_standalone = set() + num_standalone = 0 + if 'standalone_tables' in recommendations: + num_standalone = len(recommendations['standalone_tables']) + tables_standalone = {t['table'] for t in recommendations['standalone_tables']} + + total_recommended = len(tables_in_groups) + len(tables_standalone) + + return { + 'num_groups': num_groups, + 'num_standalone': num_standalone, + 'tables_in_groups_count': len(tables_in_groups), + 'tables_standalone_count': len(tables_standalone), + 'total_recommended': total_recommended, + 'total_tables': total_tables + } + + def _write_clustering_stats(self, f, stats: dict, metadata: dict): + """Write clustering statistics""" + total_coverage_pct = (stats['total_recommended'] / stats['total_tables'] * 100) if stats['total_tables'] > 0 else 0 + groups_coverage_pct = (stats['tables_in_groups_count'] / stats['total_tables'] * 100) if stats['total_tables'] > 0 else 0 + standalone_coverage_pct = (stats['tables_standalone_count'] / stats['total_tables'] * 100) if stats['total_tables'] > 0 else 0 + + f.write(f"- Total data products recommended: {stats['num_groups'] + stats['num_standalone']}\n") + f.write(f" - Multi-table products: {stats['num_groups']}\n") + f.write(f" - Single-table products: {stats['num_standalone']}\n") + f.write(f"- Recommendation coverage: {stats['total_recommended']} of {stats['total_tables']} ({total_coverage_pct:.1f}%)\n") + f.write(f" - In groups: {stats['tables_in_groups_count']} ({groups_coverage_pct:.1f}%)\n") + f.write(f" - Standalone: {stats['tables_standalone_count']} ({standalone_coverage_pct:.1f}%)\n") + + if metadata.get('min_score_threshold') is not None: + f.write(f"- Minimum score threshold: {metadata['min_score_threshold']:.1f}\n") + + def _write_non_clustering_stats(self, f, metadata: dict): + """Write non-clustering statistics""" + total = metadata['total_tables'] + recommended = metadata['recommended_tables'] + percentage = (recommended / total * 100) if total > 0 else 0 + f.write(f"- Total recommended tables: {recommended} ({percentage:.1f}% of all tables)\n") + f.write(f"- Highest table score: {metadata['highest_score']:.1f}\n") + if metadata.get('min_score_threshold') is not None: + f.write(f"- Minimum score threshold: {metadata['min_score_threshold']:.1f}\n") + + def _write_recommendation_stats(self, f, recommendations: dict): + """Write recommendation statistics section""" + if 'metadata' not in recommendations: + return + + metadata = recommendations['metadata'] + + if metadata.get('clustering_enabled', False) and 'table_groups' in recommendations: + stats = self._calculate_clustering_stats(recommendations, metadata) + self._write_clustering_stats(f, stats, metadata) + else: + self._write_non_clustering_stats(f, metadata) + + f.write("\n") + + def _write_individual_tables_header(self, f): + """Write individual tables section header""" + f.write("## Top Recommended Tables\n\n") + f.write("| Rank | Table | Score | Queries | Users | Recency | Consistency | Last Query | Related Tables |\n") + f.write("|------|-------|-------|---------|-------|---------|-------------|------------|----------------|\n") + + def _format_table_row(self, i: int, table: dict) -> str: + """Format a single table row for markdown""" + related = ', '.join(table['related_tables'][:3]) if table['related_tables'] else 'None' + recency_pct = f"{table.get('recency_score', 0)*100:.0f}%" + consistency_pct = f"{table.get('consistency_score', 0)*100:.0f}%" + days_ago = table.get('days_since_last_query', 'N/A') + days_ago_str = f"{days_ago}d ago" if isinstance(days_ago, (int, float)) else days_ago + + return (f"| {i} | {table['table']} | {table['recommendation_score']:.1f} | " + f"{table['query_count']} | {table['unique_users']} | " + f"{recency_pct} | {consistency_pct} | {days_ago_str} | {related} |\n") + + def _write_table_metric_definitions(self, f): + """Write table metric definitions""" + f.write("\n### Metric Definitions\n\n") + f.write("- **Score**: Overall recommendation score (0-100)\n") + f.write("- **Queries**: Total number of queries referencing this table\n") + f.write("- **Users**: Number of unique users querying this table\n") + f.write("- **Recency**: How recently the table was queried (100% = queried today)\n") + f.write("- **Consistency**: How consistently the table is queried over time (100% = perfectly regular)\n") + f.write("- **Last Query**: Days since the most recent query\n") + f.write("- **Related Tables**: Tables frequently queried together with this one\n") + f.write("\n") + + def _write_individual_tables_section(self, f, recommendations: dict): + """Write individual tables section""" + self._write_individual_tables_header(f) + + for i, table in enumerate(recommendations['individual_tables'], 1): + f.write(self._format_table_row(i, table)) + + self._write_table_metric_definitions(f) + + def _write_data_product_metric_definitions(self, f): + """Write data product metric definitions""" + f.write("\n### Data Product Metric Definitions\n\n") + f.write("- **Score**: Overall recommendation score for the data product (0-100)\n") + f.write(" - Star Rating Scale:\n") + f.write(f" - {FIVE_STARS} {EXCELLENT_CANDIDATE} (80-100): Strong data product candidate, implement immediately\n") + f.write(f" - {FOUR_STARS} {GOOD_CANDIDATE} (60-79): Solid candidate, implement as medium priority\n") + f.write(f" - {THREE_STARS} {FAIR_CANDIDATE} (40-59): Consider splitting or implement later\n") + f.write(f" - {TWO_STARS} {WEAK_CANDIDATE} (20-39): Reconsider grouping\n") + f.write(f" - {ONE_STAR} {POOR_CANDIDATE} (0-19): Do not implement as data product\n") + f.write("- **General Metrics:**\n") + f.write(" - **Total Queries**: Total number of queries that touched the table(s)\n") + f.write(" - **Unique Users**: Number of distinct users who queried the table(s)\n") + f.write("- **Table Metrics:**\n") + f.write(" - **Recency Score**: How recently the table was queried (relative to dataset's most recent query)\n") + f.write(" - **Consistency Score**: What % of days (in the log timespan) the table was queried\n") + f.write("- **Table Group Metrics:**\n") + f.write(" - **Group Cohesion Score**: What % of queries join multiple tables from the group together\n") + f.write(" - **Avg Join Frequency**: Average number of times per day tables in the group are joined together\n") + + def _get_star_rating(self, score: float) -> tuple: + """Get star rating and label for a score""" + if score >= 80: + return FIVE_STARS, EXCELLENT_CANDIDATE + elif score >= 60: + return FOUR_STARS, GOOD_CANDIDATE + elif score >= 40: + return THREE_STARS, FAIR_CANDIDATE + elif score >= 20: + return TWO_STARS, WEAK_CANDIDATE + else: + return ONE_STAR, POOR_CANDIDATE + + def _merge_and_sort_products(self, recommendations: dict) -> list: + """Merge groups and standalone tables, sorted by score""" + all_products = [] + + if 'table_groups' in recommendations: + for group in recommendations['table_groups']: + all_products.append({ + 'type': 'group', + 'score': group['group_score'], + 'data': group + }) + + if 'standalone_tables' in recommendations: + for table in recommendations['standalone_tables']: + all_products.append({ + 'type': 'standalone', + 'score': table['recommendation_score'], + 'data': table + }) + + all_products.sort(key=lambda x: x['score'], reverse=True) + return all_products + + def _write_group_metrics(self, f, group: dict): + """Write group-level metrics""" + f.write(QUERY_LOG_METRICS_HEADER) + f.write(f"- Total Queries: {group['total_queries']:,}\n") + f.write(f"- Unique Users: {group['unique_users']}\n") + f.write(f"- Tables in Group: {group['size']}\n") + f.write("- Group Metrics:\n") + f.write(f" - Cohesion Score: {group['cohesion_score']:.1f}%\n") + f.write(f" - Average Join Frequency: {group['avg_join_frequency']:.1f}\n") + f.write("\n") + + def _write_group_tables(self, f, group: dict): + """Write tables in group section""" + if 'table_details' in group and group['table_details']: + f.write(TABLES_IN_GROUP_HEADER) + f.write(TABLE_HEADER_ROW) + f.write(TABLE_SEPARATOR_ROW) + + for table in group['table_details']: + recency_pct = f"{table.get('recency_score', 0)*100:.0f}%" + consistency_pct = f"{table.get('consistency_score', 0)*100:.0f}%" + f.write(f"| {table['table']} | {table['recommendation_score']:.1f} | " + f"{table['query_count']} | {table['unique_users']} | " + f"{recency_pct} | {consistency_pct} |\n") + else: + f.write(TABLES_IN_GROUP_SIMPLE) + for table in group['tables']: + f.write(f"- {table}\n") + f.write("\n") + + def _truncate_text(self, text: str, max_length: int = 200) -> str: + """Truncate text if too long""" + if len(text) > max_length: + return text[:max_length] + "..." + return text + + def _write_query_pattern(self, f, idx: int, pattern_info: dict): + """Write a single query pattern""" + f.write(f"{idx}. **Pattern** (used {pattern_info['count']} times):\n") + f.write(SQL_CODE_BLOCK_START) + f.write(f" {self._truncate_text(pattern_info['pattern'])}\n") + f.write(SQL_CODE_BLOCK_END) + + if 'tables_used' in pattern_info: + f.write(f" - Tables used: {', '.join(pattern_info['tables_used'])}\n") + + f.write(SQL_EXAMPLE_LABEL) + f.write(SQL_EXAMPLE_CODE_START) + f.write(f" {self._truncate_text(pattern_info['example'])}\n") + f.write(SQL_EXAMPLE_CODE_END) + f.write("\n") + + def _write_query_patterns(self, f, tables: list, product_id: int): + """Write query patterns section""" + try: + top_patterns = self.get_top_query_patterns(tables, top_n=5) + if top_patterns: + f.write(DETAILS_START) + f.write(DETAILS_SUMMARY) + for idx, pattern_info in enumerate(top_patterns, 1): + self._write_query_pattern(f, idx, pattern_info) + f.write(DETAILS_END) + except Exception as e: + print(f"Warning: Could not generate query patterns for product {product_id}: {e}") + f.write("\n") + + def _write_group_product(self, f, i: int, group: dict): + """Write a group data product""" + score = group['group_score'] + stars, rating = self._get_star_rating(score) + + f.write(f"### Data Product {i} - Score: {score:.1f} {stars} ({rating})\n\n") + self._write_group_metrics(f, group) + self._write_group_tables(f, group) + self._write_query_patterns(f, group['tables'], i) + + def _write_standalone_product(self, f, i: int, table: dict): + """Write a standalone table data product""" + score = table['recommendation_score'] + stars, rating = self._get_star_rating(score) + + f.write(f"### Data Product {i} - Score: {score:.1f} {stars} ({rating})\n\n") + + f.write(QUERY_LOG_METRICS_HEADER) + f.write(f"- Total Queries: {table['query_count']:,}\n") + f.write(f"- Unique Users: {table['unique_users']}\n") + f.write("- Tables in Group: 1\n") + f.write("\n") + + recency_pct = f"{table.get('recency_score', 0)*100:.0f}%" + consistency_pct = f"{table.get('consistency_score', 0)*100:.0f}%" + + f.write(TABLES_IN_GROUP_HEADER) + f.write(TABLE_HEADER_ROW) + f.write(TABLE_SEPARATOR_ROW) + f.write(f"| {table['table']} | {table['recommendation_score']:.1f} | " + f"{table['query_count']} | {table['unique_users']} | " + f"{recency_pct} | {consistency_pct} |\n") + f.write("\n") + + self._write_query_patterns(f, [table['table']], i) + + def _write_clustered_products(self, f, recommendations: dict): + """Write clustered data products section""" + f.write("\n## Recommended Data Products\n\n") + f.write("*Sorted by recommendation score (descending). Groups identified using frequency-based clustering.*\n\n") + f.write("*Note: Group scores and individual table scores use different weighting formulas and are not directly comparable, but both indicate relative value within their category.*\n\n") + + all_products = self._merge_and_sort_products(recommendations) + + for i, product in enumerate(all_products, 1): + if product['type'] == 'group': + self._write_group_product(f, i, product['data']) + else: + self._write_standalone_product(f, i, product['data']) + + def _write_original_groups(self, f, recommendations: dict): + """Write original co-occurrence format groups""" + f.write("\n## Recommended Table Groups\n\n") + f.write("Tables that are frequently queried together:\n\n") + + for i, group in enumerate(recommendations['table_groups'], 1): + f.write(f"### Group {i} (Co-occurrence: {group['co_occurrence_count']})\n\n") + for table in group['tables']: + f.write(f"- {table}\n") + f.write("\n") + + def export_recommendations_markdown(self, recommendations: dict, output_file: str): + """Export recommendations to markdown file""" + with open(output_file, 'w', encoding='utf-8') as f: + self._write_markdown_header(f) + self._write_summary_stats(f) + self._write_recommendation_stats(f, recommendations) + + clustering_enabled = recommendations.get('metadata', {}).get('clustering_enabled', False) + + if not clustering_enabled: + self._write_individual_tables_section(f, recommendations) + + if 'table_groups' in recommendations: + self._write_data_product_metric_definitions(f) + + if clustering_enabled: + self._write_clustered_products(f, recommendations) + else: + self._write_original_groups(f, recommendations) + + + def _build_json_metadata(self, recommendations: dict) -> dict: + """Build metadata section for JSON export""" + from datetime import datetime + return { + "generated_at": datetime.now().isoformat(), + "total_queries_analyzed": len(self.query_logs) if self.query_logs is not None else 0, + "unique_users": int(self.query_logs['user'].nunique()) if self.query_logs is not None else 0, + "unique_tables": len(self.table_metrics) if self.table_metrics is not None else 0, + "platform": self.parser.__class__.__name__.replace('QueryParser', '').lower(), + "clustering_enabled": recommendations.get('metadata', {}).get('clustering_enabled', False), + "min_score_threshold": recommendations.get('metadata', {}).get('min_score_threshold') + } + + def _add_table_details_to_product(self, product: dict, group: dict): + """Add table details to a group product""" + if 'table_details' in group and group['table_details']: + for table_info in group['table_details']: + product["table_details"].append({ + "table": table_info['table'], + "score": round(table_info['recommendation_score'], 1), + "queries": table_info['query_count'], + "users": table_info['unique_users'], + "recency_score": round(table_info.get('recency_score', 0), 2), + "consistency_score": round(table_info.get('consistency_score', 0), 2) + }) + + def _add_query_patterns_to_product(self, product: dict, tables: list, include_tables_used: bool = True): + """Add query patterns to a product""" + try: + patterns = self.get_top_query_patterns(tables, top_n=5) + for pattern_info in patterns: + pattern_dict = { + "pattern": pattern_info['pattern'], + "count": pattern_info['count'], + "example": pattern_info['example'] + } + if include_tables_used and 'tables_used' in pattern_info: + pattern_dict["tables_used"] = pattern_info['tables_used'] + product["query_patterns"].append(pattern_dict) + except Exception: + pass # Skip patterns if error + + def _create_group_product_json(self, idx: int, group: dict) -> dict: + """Create JSON product for a group""" + product = { + "id": f"dp_{idx:03d}", + "type": "group", + "score": round(group['group_score'], 1), + "rating": self._get_rating_label(group['group_score']), + "tables": group['tables'], + "metrics": { + "total_queries": group['total_queries'], + "unique_users": group['unique_users'], + "table_count": group['size'], + "cohesion_score": round(group['cohesion_score'], 1), + "avg_join_frequency": round(group['avg_join_frequency'], 1), + "recency_score": round(group.get('avg_recency_score', 0), 2), + "consistency_score": round(group.get('avg_consistency_score', 0), 2) + }, + "table_details": [], + "query_patterns": [] + } + + self._add_table_details_to_product(product, group) + self._add_query_patterns_to_product(product, group['tables'], include_tables_used=True) + + return product + + def _create_standalone_product_json(self, idx: int, table: dict) -> dict: + """Create JSON product for a standalone table""" + product = { + "id": f"dp_{idx:03d}", + "type": "standalone", + "score": round(table['recommendation_score'], 1), + "rating": self._get_rating_label(table['recommendation_score']), + "tables": [table['table']], + "metrics": { + "total_queries": table['query_count'], + "unique_users": table['unique_users'], + "table_count": 1, + "recency_score": round(table.get('recency_score', 0), 2), + "consistency_score": round(table.get('consistency_score', 0), 2) + }, + "query_patterns": [] + } + + self._add_query_patterns_to_product(product, [table['table']], include_tables_used=False) + + return product + + def _process_clustered_recommendations(self, recommendations: dict) -> list: + """Process recommendations in clustering mode""" + all_products = [] + + if 'table_groups' in recommendations: + for idx, group in enumerate(recommendations['table_groups'], 1): + product = self._create_group_product_json(idx, group) + all_products.append(product) + + if 'standalone_tables' in recommendations: + start_idx = len(all_products) + 1 + for idx, table in enumerate(recommendations['standalone_tables'], start_idx): + product = self._create_standalone_product_json(idx, table) + all_products.append(product) + + all_products.sort(key=lambda x: x['score'], reverse=True) + return all_products + + def _process_non_clustered_recommendations(self, recommendations: dict) -> list: + """Process recommendations in non-clustering mode""" + products = [] + + if 'individual_tables' in recommendations: + for idx, table in enumerate(recommendations['individual_tables'], 1): + product = self._create_standalone_product_json(idx, table) + # Add tables_used for non-clustered mode + try: + patterns = self.get_top_query_patterns([table['table']], top_n=5) + product["query_patterns"] = [] + for pattern_info in patterns: + product["query_patterns"].append({ + "pattern": pattern_info['pattern'], + "count": pattern_info['count'], + "tables_used": pattern_info.get('tables_used', []), + "example": pattern_info['example'] + }) + except Exception: + pass + + products.append(product) + + return products + + def export_recommendations_json(self, recommendations: dict, output_file: str): + """Export recommendations to JSON file for agent consumption""" + import json + + output = { + "recommendations": [], + "metadata": self._build_json_metadata(recommendations) + } + + clustering_enabled = recommendations.get('metadata', {}).get('clustering_enabled', False) + + if clustering_enabled: + output["recommendations"] = self._process_clustered_recommendations(recommendations) + else: + output["recommendations"] = self._process_non_clustered_recommendations(recommendations) + + with open(output_file, 'w', encoding='utf-8') as f: + json.dump(output, f, indent=2, ensure_ascii=False) + + def _get_rating_label(self, score: float) -> str: + """Convert numeric score to rating label""" + if score >= 80: + return "excellent" + elif score >= 60: + return "good" + elif score >= 40: + return "fair" + elif score >= 20: + return "weak" + else: + return "poor" + +# Made with Bob diff --git a/src/wxdi/dph_services/README.md b/src/wxdi/dph_services/README.md new file mode 100644 index 0000000..3610050 --- /dev/null +++ b/src/wxdi/dph_services/README.md @@ -0,0 +1,458 @@ + + +# Data Product Hub Services (DPH Services) + +Python client library for IBM Data Product Hub API, providing programmatic access to data product management, container operations, contract terms, and asset visualization. + +## Overview + +The `dph_services` module provides a complete Python SDK for interacting with IBM Data Product Hub services. It enables developers to: + +- Initialize and manage data product containers +- Create, update, and publish data products +- Manage data product drafts and releases +- Handle contract terms and documents +- Create and manage data asset visualizations +- Manage domains and subdomains +- Work with contract templates + +## Installation + +The module is included in the data-intelligence-sdk package: + +```bash +pip install -e . +``` + +## Quick Start + +### Basic Setup + +```python +from wxdi.dph_services import DphV1 +from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + +# Initialize authenticator +authenticator = IAMAuthenticator('your-api-key') + +# Create service instance +dph_service = DphV1(authenticator=authenticator) +dph_service.set_service_url('https://your-dph-instance.com') +``` + +### Initialize a Container + +```python +# Initialize container with default settings +response = dph_service.initialize( + include=['delivery_methods', 'data_product_samples', 'domains_multi_industry'] +) + +print(f"Container initialized: {response.result}") +``` + +### Create a Data Product + +```python +# Create a new data product with a draft +data_product = dph_service.create_data_product( + drafts=[{ + 'version': '1.0.0', + 'name': 'Customer Analytics Data Product', + 'description': 'Comprehensive customer analytics dataset', + 'asset': { + 'id': 'asset-123', + 'container': {'id': 'container-456'} + }, + 'domain': { + 'id': 'domain-789', + 'name': 'Customer Analytics' + } + }] +) + +print(f"Data product created: {data_product.result['id']}") +``` + +### List Data Products + +```python +# List all data products +response = dph_service.list_data_products(limit=50) + +for product in response.result['data_products']: + print(f"- {product['name']} (v{product['version']})") +``` + +### Get a Specific Data Product + +```python +# Get data product by ID +data_product_id = 'your-data-product-id' +response = dph_service.get_data_product(data_product_id=data_product_id) + +print(f"Data Product: {response.result['name']}") +print(f"Description: {response.result['description']}") +``` + +## Core Features + +### 1. Container Management + +Initialize and manage data product containers: + +```python +# Initialize container +response = dph_service.initialize( + include=['delivery_methods', 'data_product_samples'] +) + +# Get initialization status +status = dph_service.get_initialize_status() +print(f"Status: {status.result['status']}") +``` + +### 2. Data Product Operations + +Complete lifecycle management: + +```python +# Create data product +product = dph_service.create_data_product(drafts=[...]) + +# Update data product +updated = dph_service.update_data_product( + data_product_id=product_id, + json_patch_instructions=[ + {'op': 'replace', 'path': '/description', 'value': 'Updated description'} + ] +) + +# Delete data product (if needed) +dph_service.delete_data_product(data_product_id=product_id) +``` + +### 3. Draft Management + +Work with data product drafts: + +```python +# Create a draft +draft = dph_service.create_data_product_draft( + data_product_id=product_id, + asset={'id': 'asset-123', 'container': {'id': 'container-456'}}, + version='1.1.0', + name='Updated Version' +) + +# List drafts +drafts = dph_service.list_data_product_drafts(data_product_id=product_id) + +# Get specific draft +draft_detail = dph_service.get_data_product_draft( + data_product_id=product_id, + draft_id=draft_id +) + +# Update draft +updated_draft = dph_service.update_data_product_draft( + data_product_id=product_id, + draft_id=draft_id, + json_patch_instructions=[...] +) + +# Publish draft +published = dph_service.publish_data_product_draft( + data_product_id=product_id, + draft_id=draft_id +) +``` + +### 4. Contract Terms Management + +Manage contract terms and documents: + +```python +# Create contract terms document +doc = dph_service.create_draft_contract_terms_document( + data_product_id=product_id, + draft_id=draft_id, + contract_terms_id=terms_id, + type='terms_and_conditions', + name='Terms and Conditions', + url='https://example.com/terms.pdf' +) + +# Get contract terms +terms = dph_service.get_data_product_draft_contract_terms( + data_product_id=product_id, + draft_id=draft_id +) + +# Update contract terms document +updated_doc = dph_service.update_draft_contract_terms_document( + data_product_id=product_id, + draft_id=draft_id, + contract_terms_id=terms_id, + document_id=doc_id, + json_patch_instructions=[...] +) + +# Delete contract terms document +dph_service.delete_draft_contract_terms_document( + data_product_id=product_id, + draft_id=draft_id, + contract_terms_id=terms_id, + document_id=doc_id +) +``` + +### 5. Release Management + +Manage data product releases: + +```python +# List releases +releases = dph_service.list_data_product_releases( + data_product_id=product_id +) + +# Get specific release +release = dph_service.get_data_product_release( + data_product_id=product_id, + release_id=release_id +) + +# Update release +updated_release = dph_service.update_data_product_release( + data_product_id=product_id, + release_id=release_id, + json_patch_instructions=[...] +) + +# Retire release +retired = dph_service.retire_data_product_release( + data_product_id=product_id, + release_id=release_id +) +``` + +### 6. Asset Visualization + +Create and manage data asset visualizations: + +```python +# Create visualization +visualization = dph_service.create_data_asset_visualization( + container={'id': 'container-123'}, + assets=[ + {'id': 'asset-1', 'container': {'id': 'container-123'}}, + {'id': 'asset-2', 'container': {'id': 'container-123'}} + ] +) + +# Reinitiate visualization +reinitiated = dph_service.reinitiate_data_asset_visualization( + container={'id': 'container-123'}, + assets=[...] +) +``` + +### 7. Domain Management + +Organize data products by domains: + +```python +# List domains +domains = dph_service.list_data_product_domains(limit=50) + +# Create domain +domain = dph_service.create_data_product_domain( + name='Customer Analytics', + description='Customer-related data products', + container={'id': 'container-123'} +) + +# Create subdomain +subdomain = dph_service.create_data_product_subdomain( + domain_id=domain_id, + name='Customer Segmentation', + description='Customer segmentation datasets' +) + +# Get domain +domain_detail = dph_service.get_domain(domain_id=domain_id) + +# Update domain +updated_domain = dph_service.update_data_product_domain( + domain_id=domain_id, + json_patch_instructions=[...] +) + +# Delete domain +dph_service.delete_domain(domain_id=domain_id) +``` + +### 8. Contract Templates + +Manage reusable contract templates: + +```python +# Create contract template +template = dph_service.create_contract_template( + name='Standard Terms Template', + description='Standard contract terms for data products', + contract_terms_documents=[...] +) + +# List templates +templates = dph_service.list_data_product_contract_template(limit=50) + +# Get template +template_detail = dph_service.get_contract_template( + contract_template_id=template_id +) + +# Update template +updated_template = dph_service.update_data_product_contract_template( + contract_template_id=template_id, + json_patch_instructions=[...] +) + +# Delete template +dph_service.delete_data_product_contract_template( + contract_template_id=template_id +) +``` + +## Advanced Usage + +### Pagination + +Handle large result sets with pagination: + +```python +# Using pager for data products +all_products = [] +pager = dph_service.list_data_products_with_pager(limit=50) + +for page in pager: + all_products.extend(page['data_products']) + +print(f"Total products: {len(all_products)}") +``` + +### Error Handling + +```python +from ibm_cloud_sdk_core import ApiException + +try: + response = dph_service.get_data_product(data_product_id='invalid-id') +except ApiException as e: + print(f"Error: {e.code} - {e.message}") +``` + +### Custom Headers + +```python +# Add custom headers to requests +response = dph_service.get_data_product( + data_product_id=product_id, + headers={'Custom-Header': 'value'} +) +``` + +## API Reference + +### Main Classes + +- **`DphV1`**: Main service class for Data Product Hub operations +- **`DataProduct`**: Data product model +- **`DataProductDraft`**: Draft model +- **`ContractTerms`**: Contract terms model +- **`Domain`**: Domain model + +### Key Methods + +#### Container Operations +- `initialize()` - Initialize container +- `get_initialize_status()` - Get initialization status +- `get_service_id_credentials()` - Get service credentials +- `manage_api_keys()` - Manage API keys + +#### Data Product Operations +- `create_data_product()` - Create new data product +- `list_data_products()` - List all data products +- `get_data_product()` - Get specific data product +- `update_data_product()` - Update data product +- `delete_data_product()` - Delete data product + +#### Draft Operations +- `create_data_product_draft()` - Create draft +- `list_data_product_drafts()` - List drafts +- `get_data_product_draft()` - Get draft details +- `update_data_product_draft()` - Update draft +- `delete_data_product_draft()` - Delete draft +- `publish_data_product_draft()` - Publish draft + +#### Release Operations +- `list_data_product_releases()` - List releases +- `get_data_product_release()` - Get release details +- `update_data_product_release()` - Update release +- `retire_data_product_release()` - Retire release + +## Examples + +See the `examples/test_dph_v1_examples.py` file for comprehensive usage examples. + +## Testing + +Run the unit tests: + +```bash +pytest tests/src/dph_services/ -v +``` + +Run integration tests (requires service credentials): + +```bash +pytest tests/src/integration/test_dph_v1.py -v +``` + +## Requirements + +- Python 3.8+ +- ibm-cloud-sdk-core >= 3.16.7 +- requests >= 2.32.4 +- python-dateutil >= 2.5.3 + +## License + +Apache License 2.0 + +## Support + +For issues and questions: +- GitHub Issues: https://github.com/IBM/data-intelligence-sdk/issues +- Documentation: See main README.md + +## Related Modules + +- **dq_validator**: Data quality validation +- **odcs_generator**: ODCS file generation +- **data_product_recommender**: Query log analysis \ No newline at end of file diff --git a/src/wxdi/dph_services/__init__.py b/src/wxdi/dph_services/__init__.py new file mode 100644 index 0000000..ad33907 --- /dev/null +++ b/src/wxdi/dph_services/__init__.py @@ -0,0 +1,22 @@ +# coding: utf-8 +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Python client library for the DPH Services""" + +from ibm_cloud_sdk_core import IAMTokenManager, DetailedResponse, BaseService, ApiException + +from .common import get_sdk_headers +from .version import __version__ +from .dph_v1 import DphV1 diff --git a/src/wxdi/dph_services/common.py b/src/wxdi/dph_services/common.py new file mode 100644 index 0000000..cb538ca --- /dev/null +++ b/src/wxdi/dph_services/common.py @@ -0,0 +1,75 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +This module provides common methods for use across all service modules. +""" + +import platform +from wxdi.dph_services.version import __version__ + +HEADER_NAME_USER_AGENT = 'User-Agent' +SDK_NAME = 'data-product-python-sdk' + + +def get_system_info(): + """ + Get information about the system to be inserted into the User-Agent header. + """ + return 'lang={0}; arch={1}; os={2}; python.version={3}'.format( + 'python', platform.machine(), platform.system(), platform.python_version() # Architecture # OS + ) # Python version + + +def get_user_agent(): + """ + Get the value to be sent in the User-Agent header. + """ + return USER_AGENT + + +USER_AGENT = '{0}/{1} ({2})'.format(SDK_NAME, __version__, get_system_info()) + + +def get_sdk_headers(service_name=None, service_version=None, operation_id=None): + """ + Get the request headers to be sent in requests by the SDK. + + If you plan to gather metrics for your SDK, the User-Agent header value must + be a string similar to the following: + my-python-sdk/0.0.1 (lang=python; arch=x86_64; os=Linux; python.version=3.7.4) + + In the example above, the analytics tool will parse the user-agent header and + use the following properties: + "my-python-sdk" - the name of your sdk + "0.0.1"- the version of your sdk + "lang=python" - the language of the current sdk + "arch=x86_64; os=Linux; python.version=3.7.4" - system information + + Note: It is very important that the sdk name ends with the string `-sdk`, + as the analytics data collector uses this to gather usage data. + + :param service_name: The name of the service (optional, for future use) + :param service_version: The version of the service (optional, for future use) + :param operation_id: The operation ID (optional, for future use) + """ + # Parameters are accepted for API compatibility + # They may be utilized in future versions for enhanced telemetry + _ = (service_name, service_version, operation_id) + + headers = {} + headers[HEADER_NAME_USER_AGENT] = get_user_agent() + return headers diff --git a/src/wxdi/dph_services/common_constants.py b/src/wxdi/dph_services/common_constants.py new file mode 100644 index 0000000..585c79c --- /dev/null +++ b/src/wxdi/dph_services/common_constants.py @@ -0,0 +1,48 @@ +# coding: utf-8 + +# Copyright 2019, 2020 IBM All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +This module provides common constants for use across all service modules. +""" + +# Dph Api Paths +URL_GET_INITIALIZE_STATUS = '/data_product_exchange/v1/configuration/initialize/status' +URL_GET_SERVICEID_CREDENTIALS = '/data_product_exchange/v1/configuration/credentials' +URL_INITIALIZE = '/data_product_exchange/v1/configuration/initialize' +URL_MANAGE_APIKEYS = '/data_product_exchange/v1/configuration/rotate_credentials' +URL_LIST_DATA_PRODUCTS = '/data_product_exchange/v1/data_products' +URL_CREATE_DATA_PRODUCT = '/data_product_exchange/v1/data_products' +URL_GET_DATA_PRODUCT = '/data_product_exchange/v1/data_products/{data_product_id}' +URL_COMPLETE_DRAFT_CONTRACT_TERMS_DOCUMENT = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}/documents/{document_id}/complete' +URL_LIST_DATA_PRODUCT_DRAFTS = '/data_product_exchange/v1/data_products/{data_product_id}/drafts' +URL_CREATE_DATA_PRODUCT_DRAFT = '/data_product_exchange/v1/data_products/{data_product_id}/drafts' +URL_CREATE_DRAFT_CONTRACT_TERMS_DOCUMENT = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}/documents' +URL_GET_DATA_PRODUCT_DRAFT = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}' +URL_GET_DRAFT_CONTRACT_TERMS_DOCUMENT = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}/documents/{document_id}' +URL_PUBLISH_DATA_PRODUCT_DRAFT = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/publish' +URL_GET_DATA_PRODUCT_RELEASE = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}' +URL_UPDATE_DATA_PRODUCT_RELEASE = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}' +URL_GET_RELEASE_CONTRACT_TERMS_DOCUMENT = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}/contract_terms/{contract_terms_id}/documents/{document_id}' +URL_LIST_DATA_PRODUCT_RELEASES = '/data_product_exchange/v1/data_products/{data_product_id}/releases' +URL_RETIRE_DATA_PRODUCT_RELEASE = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}/retire' + +# Dph Api Headers +CONTENT_TYPE_JSON = 'application/json' +CONTENT_TYPE_PATCH_JSON = 'application/json-patch+json' + +SERVICE_NAME = 'data_product_hub_api_service' +SERVICE_VERSION = 'V1' + diff --git a/src/wxdi/dph_services/dph_v1.py b/src/wxdi/dph_services/dph_v1.py new file mode 100644 index 0000000..a05cec9 --- /dev/null +++ b/src/wxdi/dph_services/dph_v1.py @@ -0,0 +1,12411 @@ +# coding: utf-8 + +# (C) Copyright IBM Corp. 2025. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# IBM OpenAPI SDK Code Generator Version: 3.96.0-d6dec9d7-20241008-212902 + +""" +Data Product Hub API Service + +API Version: 1 +""" + +from datetime import datetime +from enum import Enum +from typing import BinaryIO, Dict, List, Optional +import json + +from ibm_cloud_sdk_core import BaseService, DetailedResponse +from ibm_cloud_sdk_core.authenticators.authenticator import Authenticator +from ibm_cloud_sdk_core.get_authenticator import get_authenticator_from_environment +from ibm_cloud_sdk_core.utils import convert_list, convert_model, datetime_to_string, string_to_datetime + +from .common import get_sdk_headers +from .common_constants import * + +############################################################################## +# Service +############################################################################## + + +class DphV1(BaseService): + """The DPH V1 service.""" + + DEFAULT_SERVICE_URL = 'https://api.dataplatform.dev.cloud.ibm.com/' + DEFAULT_SERVICE_NAME = SERVICE_NAME + + @classmethod + def new_instance( + cls, + service_name: str = DEFAULT_SERVICE_NAME, + ) -> 'DphV1': + """ + Return a new client for the DPH service using the specified parameters and + external configuration. + """ + authenticator = get_authenticator_from_environment(service_name) + service = cls( + authenticator + ) + service.configure_service(service_name) + return service + + def __init__( + self, + authenticator: Authenticator = None, + ) -> None: + """ + Construct a new client for the DPH service. + + :param Authenticator authenticator: The authenticator specifies the authentication mechanism. + Get up to date information from https://github.com/IBM/python-sdk-core/blob/main/README.md + about initializing the authenticator of your choice. + """ + BaseService.__init__(self, service_url=self.DEFAULT_SERVICE_URL, authenticator=authenticator) + + ######################### + # Helper Methods + ######################### + + def _prepare_headers(self, operation_id: str, **kwargs) -> Dict: + """ + Prepare request headers with SDK headers and custom headers from kwargs. + + :param operation_id: The operation ID for SDK headers + :param kwargs: Additional keyword arguments that may contain 'headers' + :return: Dictionary of prepared headers + """ + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id=operation_id, + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + + return headers + + def _set_accept_header(self, headers: Dict, accept_type: Optional[str] = CONTENT_TYPE_JSON) -> None: + """ + Set the Accept header if not already present. + + :param headers: Headers dictionary to update + :param accept_type: The accept type to set (default: CONTENT_TYPE_JSON) + """ + if accept_type and 'Accept' not in headers: + headers['Accept'] = accept_type + + def _prepare_json_data(self, data: Dict) -> str: + """ + Prepare JSON data by removing None values and converting to JSON string. + + :param data: Dictionary of data to prepare + :return: JSON string + """ + data = {k: v for (k, v) in data.items() if v is not None} + return json.dumps(data) + + ######################### + # Configuration + ######################### + + def get_initialize_status( + self, + *, + container_id: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Get resource initialization status. + + Use this API to get the status of resource initialization in Data Product + Hub.

If the data product catalog exists but has never been initialized, + the status will be "not_started".

If the data product catalog exists and + has been or is being initialized, the response will contain the status of the last + or current initialization. If the initialization failed, the "errors" and "trace" + fields will contain the error(s) encountered during the initialization, including + the ID to trace the error(s).

If the data product catalog doesn't exist, + an HTTP 404 response is returned. + + :param str container_id: (optional) Container ID of the data product + catalog. If not supplied, the data product catalog is looked up by using + the uid of the default data product catalog. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `InitializeResource` object + """ + + headers = self._prepare_headers('get_initialize_status', **kwargs) + self._set_accept_header(headers) + + params = { + 'container.id': container_id, + } + + url = '/data_product_exchange/v1/configuration/initialize/status' + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def get_service_id_credentials( + self, + **kwargs, + ) -> DetailedResponse: + """ + Get service id credentials. + + Use this API to get the information of service id credentials in Data Product Hub. + + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `ServiceIdCredentials` object + """ + + headers = self._prepare_headers('get_service_id_credentials', **kwargs) + self._set_accept_header(headers) + + url = '/data_product_exchange/v1/configuration/credentials' + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + def initialize( + self, + *, + container: Optional['ContainerReference'] = None, + include: Optional[List[str]] = None, + **kwargs, + ) -> DetailedResponse: + """ + Initialize resources. + + Use this API to initialize default assets for data product hub.

You can + initialize:
  • `delivery_methods` - Methods through which data product + parts can be delivered to consumers of the data product + hub
  • `domains_multi_industry` - Taxonomy of domains and use cases + applicable to multiple industries
  • `data_product_samples` - Sample data + products used to illustrate capabilities of the data product + hub
  • `workflows` - Workflows to enable restricted data + products
  • `project` - A default project for exporting data assets to + files
  • `catalog_configurations` - Catalog configurations for the default + data product catalog


If a resource depends on resources that + are not specified in the request, these dependent resources will be automatically + initialized. E.g., initializing `data_product_samples` will also initialize + `domains_multi_industry` and `delivery_methods` even if they are not specified in + the request because it depends on them.

If initializing the data product + hub for the first time, do not specify a container. The default data product + catalog will be created.
For first time initialization, it is recommended that + at least `delivery_methods` and `domains_multi_industry` is included in the + initialize operation.

If the data product hub has already been + initialized, you may call this API again to initialize new resources, such as new + delivery methods. In this case, specify the default data product catalog container + information. + + :param ContainerReference container: (optional) Container reference. + :param List[str] include: (optional) List of configuration options to + (re-)initialize. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `InitializeResource` object + """ + + if container is not None: + container = convert_model(container) + + headers = self._prepare_headers('initialize', **kwargs) + self._set_accept_header(headers) + + data = self._prepare_json_data({ + 'container': container, + 'include': include, + }) + headers['content-type'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/configuration/initialize' + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def manage_api_keys( + self, + **kwargs, + ) -> DetailedResponse: + """ + Rotate credentials for a Data Product Hub instance. + + Use this API to rotate credentials for a Data Product Hub instance. + + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse + """ + + headers = self._prepare_headers('manage_api_keys', **kwargs) + + url = '/data_product_exchange/v1/configuration/rotate_credentials' + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + ######################### + # Data Asset Visualization + ######################### + + def create_data_asset_visualization( + self, + *, + assets: Optional[List['DataAssetRelationship']] = None, + **kwargs, + ) -> DetailedResponse: + """ + Create visualization asset and initialize profiling for the provided data assets. + + Use this API to create visualization asset and initialize profiling for the + provided data assets

Provide the below required fields

Required + fields:

- catalog_id
- Collection of assetId with it's related asset + id

. + + :param List[DataAssetRelationship] assets: (optional) Data product hub + asset and it's related part asset. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataAssetVisualizationRes` object + """ + + if assets is not None: + assets = [convert_model(x) for x in assets] + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='create_data_asset_visualization', + ) + headers.update(sdk_headers) + + data = { + 'assets': assets, + } + data = {k: v for (k, v) in data.items() if v is not None} + data = json.dumps(data) + headers['content-type'] = CONTENT_TYPE_JSON + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/data_asset/visualization' + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def reinitiate_data_asset_visualization( + self, + *, + assets: Optional[List['DataAssetRelationship']] = None, + **kwargs, + ) -> DetailedResponse: + """ + Reinitiate visualization for an asset. + + Use this API to Reinitiate visualization for an asset which is in below + scenarios

- Previous bucket got deleted and new bucket is created.
- + Data visualization attachment is missing in asset details.
- Visualization + asset reference is missing in related asset details.

. + + :param List[DataAssetRelationship] assets: (optional) Data product hub + asset and it's related part asset. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataAssetVisualizationRes` object + """ + + if assets is not None: + assets = [convert_model(x) for x in assets] + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='reinitiate_data_asset_visualization', + ) + headers.update(sdk_headers) + + data = { + 'assets': assets, + } + data = {k: v for (k, v) in data.items() if v is not None} + data = json.dumps(data) + headers['content-type'] = CONTENT_TYPE_JSON + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/data_asset/visualization/reinitiate' + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + ######################### + # Data Products + ######################### + + def list_data_products( + self, + *, + limit: Optional[int] = None, + start: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a list of data products. + + Retrieve a list of data products. + + :param int limit: (optional) Limit the number of data products in the + results. The maximum limit is 200. + :param str start: (optional) Start token for pagination. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductCollection` object + """ + + headers = self._prepare_headers('list_data_products', **kwargs) + self._set_accept_header(headers) + + params = { + 'limit': limit, + 'start': start, + } + + url = '/data_product_exchange/v1/data_products' + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def create_data_product( + self, + drafts: List['DataProductDraftPrototype'], + *, + limit: Optional[int] = None, + start: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Create a new data product. + + Use this API to create a new data product.

Provide the initial draft of + the data product.

Required fields:

- name
- + container

If `version` is not specified, the default version **1.0.0** + will be used.

The `domain` is optional. + + :param List[DataProductDraftPrototype] drafts: Collection of data products + drafts to add to data product. + :param int limit: (optional) Limit the number of data products in the + results. The maximum limit is 200. + :param str start: (optional) Start token for pagination. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProduct` object + """ + + if drafts is None: + raise ValueError('drafts must be provided') + drafts = [convert_model(x) for x in drafts] + + headers = self._prepare_headers('create_data_product', **kwargs) + self._set_accept_header(headers) + + params = { + 'limit': limit, + 'start': start, + } + + data = self._prepare_json_data({'drafts': drafts}) + headers['content-type'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/data_products' + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + params=params, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def get_data_product( + self, + data_product_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a data product identified by id. + + Retrieve a data product identified by id. + + :param str data_product_id: Data product ID. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProduct` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + + headers = self._prepare_headers('get_data_product', **kwargs) + self._set_accept_header(headers) + + path_param_keys = ['data_product_id'] + path_param_values = self.encode_path_vars(data_product_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + ######################### + # Data Product Drafts + ######################### + + def complete_draft_contract_terms_document( + self, + data_product_id: str, + draft_id: str, + contract_terms_id: str, + document_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Complete a contract document upload operation. + + After uploading a file to the provided signed URL, call this endpoint to mark the + upload as complete. After the upload operation is marked as complete, the file is + available to download. Use '-' for the `data_product_id` to skip specifying the + data product ID explicitly. + - After the upload is marked as complete, the returned URL is displayed in the + "url" field. The signed URL is used to download the document. + - Calling complete on referential documents results in an error. + - Calling complete on attachment documents for which the file has not been + uploaded will result in an error. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param str contract_terms_id: Contract terms id. + :param str document_id: Document id. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `ContractTermsDocument` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + if not document_id: + raise ValueError('document_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='complete_draft_contract_terms_document', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'draft_id', 'contract_terms_id', 'document_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id, contract_terms_id, document_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}/documents/{document_id}/complete'.format(**path_param_dict) + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + def list_data_product_drafts( + self, + data_product_id: str, + *, + asset_container_id: Optional[str] = None, + version: Optional[str] = None, + limit: Optional[int] = None, + start: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a list of data product drafts. + + Retrieve a list of data product drafts. Use '-' for the `data_product_id` to skip + specifying the data product ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str asset_container_id: (optional) Filter the list of data product + drafts by container id. + :param str version: (optional) Filter the list of data product drafts by + version number. + :param int limit: (optional) Limit the number of data product drafts in the + results. The maximum limit is 200. + :param str start: (optional) Start token for pagination. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductDraftCollection` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='list_data_product_drafts', + ) + headers.update(sdk_headers) + + params = { + 'asset.container.id': asset_container_id, + 'version': version, + 'limit': limit, + 'start': start, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id'] + path_param_values = self.encode_path_vars(data_product_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def create_data_product_draft( + self, + data_product_id: str, + asset: 'AssetPrototype', + *, + version: Optional[str] = None, + state: Optional[str] = None, + data_product: Optional['DataProductIdentity'] = None, + name: Optional[str] = None, + description: Optional[str] = None, + tags: Optional[List[str]] = None, + use_cases: Optional[List['UseCase']] = None, + types: Optional[List[str]] = None, + contract_terms: Optional[List['ContractTerms']] = None, + domain: Optional['Domain'] = None, + parts_out: Optional[List['DataProductPart']] = None, + workflows: Optional['DataProductWorkflows'] = None, + dataview_enabled: Optional[bool] = None, + comments: Optional[str] = None, + access_control: Optional['AssetListAccessControl'] = None, + last_updated_at: Optional[datetime] = None, + sub_container: Optional['ContainerIdentity'] = None, + is_restricted: Optional[bool] = None, + **kwargs, + ) -> DetailedResponse: + """ + Create a new draft of an existing data product. + + Create a new draft of an existing data product. + + :param str data_product_id: Data product ID. + :param AssetPrototype asset: New asset input properties. + :param str version: (optional) The data product version number. + :param str state: (optional) The state of the data product version. If not + specified, the data product version will be created in `draft` state. + :param DataProductIdentity data_product: (optional) Data product + identifier. + :param str name: (optional) The name that refers to the new data product + version. If this is a new data product, this value must be specified. If + this is a new version of an existing data product, the name will default to + the name of the previous data product version. A name can contain letters, + numbers, understores, dashes, spaces or periods. A name must contain at + least one non-space character. + :param str description: (optional) Description of the data product version. + If this is a new version of an existing data product, the description will + default to the description of the previous version of the data product. + :param List[str] tags: (optional) Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated + with the data product version. + :param List[str] types: (optional) Types of parts on the data product. + :param List[ContractTerms] contract_terms: (optional) Contract terms + binding various aspects of the data product. + :param Domain domain: (optional) Domain that the data product version + belongs to. If this is the first version of a data product, this field is + required. If this is a new version of an existing data product, the domain + will default to the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: (optional) The outgoing parts of + this data product version to be delivered to consumers. If this is the + first version of a data product, this field defaults to an empty list. If + this is a new version of an existing data product, the data product parts + will default to the parts list from the previous version of the data + product. + :param DataProductWorkflows workflows: (optional) The workflows associated + with the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided + either at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control + object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for + a IBM knowledge catalog container (catalog/project/space). + :param bool is_restricted: (optional) Indicates whether the data product is + restricted or not. A restricted data product indicates that orders of the + data product requires explicit approval before data is delivered. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductDraft` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if asset is None: + raise ValueError('asset must be provided') + asset = convert_model(asset) + if data_product is not None: + data_product = convert_model(data_product) + if use_cases is not None: + use_cases = [convert_model(x) for x in use_cases] + if contract_terms is not None: + contract_terms = [convert_model(x) for x in contract_terms] + if domain is not None: + domain = convert_model(domain) + if parts_out is not None: + parts_out = [convert_model(x) for x in parts_out] + if workflows is not None: + workflows = convert_model(workflows) + if access_control is not None: + access_control = convert_model(access_control) + if last_updated_at is not None: + last_updated_at = datetime_to_string(last_updated_at) + if sub_container is not None: + sub_container = convert_model(sub_container) + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='create_data_product_draft', + ) + headers.update(sdk_headers) + + data = { + 'asset': asset, + 'version': version, + 'state': state, + 'data_product': data_product, + 'name': name, + 'description': description, + 'tags': tags, + 'use_cases': use_cases, + 'types': types, + 'contract_terms': contract_terms, + 'domain': domain, + 'parts_out': parts_out, + 'workflows': workflows, + 'dataview_enabled': dataview_enabled, + 'comments': comments, + 'access_control': access_control, + 'last_updated_at': last_updated_at, + 'sub_container': sub_container, + 'is_restricted': is_restricted, + } + data = {k: v for (k, v) in data.items() if v is not None} + data = json.dumps(data) + headers['content-type'] = CONTENT_TYPE_JSON + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id'] + path_param_values = self.encode_path_vars(data_product_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts'.format(**path_param_dict) + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def create_draft_contract_terms_document( + self, + data_product_id: str, + draft_id: str, + contract_terms_id: str, + type: str, + name: str, + *, + url: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Upload a contract document to the data product draft contract terms. + + Upload a contract document to the data product draft identified by draft_id. Use + '-' for the `data_product_id` to skip specifying the data product ID explicitly. + - If the request object contains a "url" parameter, a referential document is + created to store the provided url. + - If the request object does not contain a "url" parameter, an attachment document + is created, and a signed url will be returned in an "upload_url" parameter. The + data product producer can upload the document using the provided "upload_url". + After the upload is completed, call "complete_contract_terms_document" for the + given document needs to be called to mark the upload as completed. After + completion of the upload, "get_contract_terms_document" for the given document + returns a signed "url" parameter that can be used to download the attachment + document. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param str contract_terms_id: Contract terms id. + :param str type: Type of the contract document. + :param str name: Name of the contract document. + :param str url: (optional) URL that can be used to retrieve the contract + document. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `ContractTermsDocument` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + if type is None: + raise ValueError('type must be provided') + if name is None: + raise ValueError('name must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='create_draft_contract_terms_document', + ) + headers.update(sdk_headers) + + data = { + 'type': type, + 'name': name, + 'url': url, + } + data = {k: v for (k, v) in data.items() if v is not None} + data = json.dumps(data) + headers['content-type'] = CONTENT_TYPE_JSON + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'draft_id', 'contract_terms_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id, contract_terms_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}/documents'.format(**path_param_dict) + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def get_data_product_draft( + self, + data_product_id: str, + draft_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Get a draft of an existing data product. + + Get a draft of an existing data product. Use '-' for the `data_product_id` to skip + specifying the data product ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductDraft` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_data_product_draft', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'draft_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + def delete_data_product_draft( + self, + data_product_id: str, + draft_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Delete a data product draft identified by ID. + + Delete a data product draft identified by a valid ID. Use '-' for the + `data_product_id` to skip specifying the data product ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='delete_data_product_draft', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + + path_param_keys = ['data_product_id', 'draft_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}'.format(**path_param_dict) + request = self.prepare_request( + method='DELETE', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + def update_data_product_draft( + self, + data_product_id: str, + draft_id: str, + json_patch_instructions: List['JsonPatchOperation'], + **kwargs, + ) -> DetailedResponse: + """ + Update the data product draft identified by ID. + + Use this API to update the properties of a data product draft identified by a + valid ID. Use '-' for the `data_product_id` to skip specifying the data product ID + explicitly.

Specify patch operations using http://jsonpatch.com/ + syntax.

Supported patch operations include:

- Update the + properties of a data product

- Add/Remove parts from a data product (up + to 20 parts)

- Add/Remove use cases from a data product

- Update + the data product state

. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param List[JsonPatchOperation] json_patch_instructions: A set of patch + operations as defined in RFC 6902. See http://jsonpatch.com/ for more + information. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductDraft` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if json_patch_instructions is None: + raise ValueError('json_patch_instructions must be provided') + json_patch_instructions = [convert_model(x) for x in json_patch_instructions] + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='update_data_product_draft', + ) + headers.update(sdk_headers) + + data = json.dumps(json_patch_instructions) + headers['content-type'] = 'application/json-patch+json' + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'draft_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}'.format(**path_param_dict) + request = self.prepare_request( + method='PATCH', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def get_draft_contract_terms_document( + self, + data_product_id: str, + draft_id: str, + contract_terms_id: str, + document_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Get a contract document. + + If a document has a completed attachment, the response contains the `url` which + can be used to download the attachment. If a document does not have a completed + attachment, the response contains the `url` which was submitted at document + creation. If a document has an attachment that is incomplete, an error is returned + to prompt the user to upload the document file and complete it. Use '-' for the + `data_product_id` to skip specifying the data product ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param str contract_terms_id: Contract terms id. + :param str document_id: Document id. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `ContractTermsDocument` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + if not document_id: + raise ValueError('document_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_draft_contract_terms_document', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'draft_id', 'contract_terms_id', 'document_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id, contract_terms_id, document_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}/documents/{document_id}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + def delete_draft_contract_terms_document( + self, + data_product_id: str, + draft_id: str, + contract_terms_id: str, + document_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Delete a contract document. + + Delete an existing contract document. + Contract documents can only be deleted for data product versions that are in DRAFT + state. Use '-' for the `data_product_id` to skip specifying the data product ID + explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param str contract_terms_id: Contract terms id. + :param str document_id: Document id. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + if not document_id: + raise ValueError('document_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='delete_draft_contract_terms_document', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + + path_param_keys = ['data_product_id', 'draft_id', 'contract_terms_id', 'document_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id, contract_terms_id, document_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}/documents/{document_id}'.format(**path_param_dict) + request = self.prepare_request( + method='DELETE', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + def update_draft_contract_terms_document( + self, + data_product_id: str, + draft_id: str, + contract_terms_id: str, + document_id: str, + json_patch_instructions: List['JsonPatchOperation'], + **kwargs, + ) -> DetailedResponse: + """ + Update a contract document. + + Use this API to update the properties of a contract document that is identified by + a valid ID. + Specify patch operations using http://jsonpatch.com/ syntax. + Supported patch operations include: + - Update the url of document if it does not have an attachment. + - Update the type of the document. +

Contract terms documents can only be updated if the associated data + product version is in DRAFT state. Use '-' for the `data_product_id` to skip + specifying the data product ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param str contract_terms_id: Contract terms id. + :param str document_id: Document id. + :param List[JsonPatchOperation] json_patch_instructions: A set of patch + operations as defined in RFC 6902. See http://jsonpatch.com/ for more + information. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `ContractTermsDocument` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + if not document_id: + raise ValueError('document_id must be provided') + if json_patch_instructions is None: + raise ValueError('json_patch_instructions must be provided') + json_patch_instructions = [convert_model(x) for x in json_patch_instructions] + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='update_draft_contract_terms_document', + ) + headers.update(sdk_headers) + + data = json.dumps(json_patch_instructions) + headers['content-type'] = 'application/json-patch+json' + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'draft_id', 'contract_terms_id', 'document_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id, contract_terms_id, document_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}/documents/{document_id}'.format(**path_param_dict) + request = self.prepare_request( + method='PATCH', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def get_data_product_draft_contract_terms( + self, + data_product_id: str, + draft_id: str, + contract_terms_id: str, + *, + accept: Optional[str] = None, + include_contract_documents: Optional[bool] = None, + autopopulate_server_information: Optional[bool] = None, + server_asset_id: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a data product contract terms identified by id. + + Retrieve a data product contract terms identified by id. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param str contract_terms_id: Contract terms id. + :param str accept: (optional) The type of the response: application/json or + application/odcs+yaml. + :param bool include_contract_documents: (optional) Set to false to exclude + external contract documents (e.g., Terms and Conditions URLs) from the + response. By default, these are included. + :param bool autopopulate_server_information: (optional) Set to true to + autopopulate server information from connection details. Default is false. + :param str server_asset_id: (optional) Asset ID of the server used for + autopopulating connection details. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `ContractTerms` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + headers = { + 'Accept': accept, + } + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_data_product_draft_contract_terms', + ) + headers.update(sdk_headers) + + params = { + 'include_contract_documents': include_contract_documents, + 'autopopulate_server_information': autopopulate_server_information, + 'server_asset_id': server_asset_id, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + + path_param_keys = ['data_product_id', 'draft_id', 'contract_terms_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id, contract_terms_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def replace_data_product_draft_contract_terms( + self, + data_product_id: str, + draft_id: str, + contract_terms_id: str, + *, + asset: Optional['AssetReference'] = None, + id: Optional[str] = None, + documents: Optional[List['ContractTermsDocument']] = None, + error_msg: Optional[str] = None, + overview: Optional['Overview'] = None, + description: Optional['Description'] = None, + organization: Optional[List['ContractTemplateOrganization']] = None, + roles: Optional[List['Roles']] = None, + price: Optional['Pricing'] = None, + sla: Optional[List['ContractTemplateSLA']] = None, + support_and_communication: Optional[List['ContractTemplateSupportAndCommunication']] = None, + custom_properties: Optional[List['ContractTemplateCustomProperty']] = None, + contract_test: Optional['ContractTest'] = None, + servers: Optional[List['ContractServer']] = None, + schema: Optional[List['ContractSchema']] = None, + **kwargs, + ) -> DetailedResponse: + """ + Update a data product contract terms identified by id. + + Update a data product contract terms identified by id. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param str contract_terms_id: Contract terms id. + :param AssetReference asset: (optional) The reference schema for a asset in + a container. + :param str id: (optional) ID of the contract terms. + :param List[ContractTermsDocument] documents: (optional) Collection of + contract terms documents. + :param str error_msg: (optional) An error message, if existing, relating to + the contract terms. + :param Overview overview: (optional) Overview details of a data contract. + :param Description description: (optional) Description details of a data + contract. + :param List[ContractTemplateOrganization] organization: (optional) List of + sub domains to be added within a domain. + :param List[Roles] roles: (optional) List of roles associated with the + contract. + :param Pricing price: (optional) Represents the pricing details of the + contract. + :param List[ContractTemplateSLA] sla: (optional) Service Level Agreement + details. + :param List[ContractTemplateSupportAndCommunication] + support_and_communication: (optional) Support and communication details for + the contract. + :param List[ContractTemplateCustomProperty] custom_properties: (optional) + Custom properties that are not part of the standard contract. + :param ContractTest contract_test: (optional) Contains the contract test + status and related metadata. + :param List[ContractServer] servers: (optional) List of server definitions. + :param List[ContractSchema] schema: (optional) Schema details of the data + asset. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `ContractTerms` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + if asset is not None: + asset = convert_model(asset) + if documents is not None: + documents = [convert_model(x) for x in documents] + if overview is not None: + overview = convert_model(overview) + if description is not None: + description = convert_model(description) + if organization is not None: + organization = [convert_model(x) for x in organization] + if roles is not None: + roles = [convert_model(x) for x in roles] + if price is not None: + price = convert_model(price) + if sla is not None: + sla = [convert_model(x) for x in sla] + if support_and_communication is not None: + support_and_communication = [convert_model(x) for x in support_and_communication] + if custom_properties is not None: + custom_properties = [convert_model(x) for x in custom_properties] + if contract_test is not None: + contract_test = convert_model(contract_test) + if servers is not None: + servers = [convert_model(x) for x in servers] + if schema is not None: + schema = [convert_model(x) for x in schema] + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='replace_data_product_draft_contract_terms', + ) + headers.update(sdk_headers) + + data = { + 'asset': asset, + 'id': id, + 'documents': documents, + 'error_msg': error_msg, + 'overview': overview, + 'description': description, + 'organization': organization, + 'roles': roles, + 'price': price, + 'sla': sla, + 'support_and_communication': support_and_communication, + 'custom_properties': custom_properties, + 'contract_test': contract_test, + 'servers': servers, + 'schema': schema, + } + data = {k: v for (k, v) in data.items() if v is not None} + data = json.dumps(data) + headers['content-type'] = CONTENT_TYPE_JSON + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'draft_id', 'contract_terms_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id, contract_terms_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}'.format(**path_param_dict) + request = self.prepare_request( + method='PUT', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def update_data_product_draft_contract_terms( + self, + data_product_id: str, + draft_id: str, + contract_terms_id: str, + json_patch_instructions: List['JsonPatchOperation'], + **kwargs, + ) -> DetailedResponse: + """ + Update a contract terms property. + + Use this API to update the properties of a contract terms that is identified by a + valid ID. + Specify patch operations using http://jsonpatch.com/ syntax. + Supported patch operations include: + - Update the contract terms properties. +

Contract terms can only be updated if the associated data product + version is in DRAFT state. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param str contract_terms_id: Contract terms id. + :param List[JsonPatchOperation] json_patch_instructions: A set of patch + operations as defined in RFC 6902. See http://jsonpatch.com/ for more + information. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `ContractTerms` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + if json_patch_instructions is None: + raise ValueError('json_patch_instructions must be provided') + json_patch_instructions = [convert_model(x) for x in json_patch_instructions] + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='update_data_product_draft_contract_terms', + ) + headers.update(sdk_headers) + + data = json.dumps(json_patch_instructions) + headers['content-type'] = 'application/json-patch+json' + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'draft_id', 'contract_terms_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id, contract_terms_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}'.format(**path_param_dict) + request = self.prepare_request( + method='PATCH', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def get_contract_terms_in_specified_format( + self, + data_product_id: str, + draft_id: str, + contract_terms_id: str, + format: str, + format_version: str, + *, + accept: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a data product contract terms identified by id in specified format. + + Retrieve a data product contract terms identified by id in specified format. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param str contract_terms_id: Contract terms id. + :param str format: The format for returning contract terms. For example: + odcs. + :param str format_version: The version of the format for returning contract + terms. For example: 3. + :param str accept: (optional) The type of the response: + application/odcs+yaml or application/json. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `BinaryIO` result + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + if not format: + raise ValueError('format must be provided') + if not format_version: + raise ValueError('format_version must be provided') + headers = { + 'Accept': accept, + } + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_contract_terms_in_specified_format', + ) + headers.update(sdk_headers) + + params = { + 'format': format, + 'format_version': format_version, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + + path_param_keys = ['data_product_id', 'draft_id', 'contract_terms_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id, contract_terms_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/contract_terms/{contract_terms_id}/format'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def publish_data_product_draft( + self, + data_product_id: str, + draft_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Publish a draft of an existing data product. + + Publish a draft of an existing data product. Use '-' for the `data_product_id` to + skip specifying the data product ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str draft_id: Data product draft id. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductRelease` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not draft_id: + raise ValueError('draft_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='publish_data_product_draft', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'draft_id'] + path_param_values = self.encode_path_vars(data_product_id, draft_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/drafts/{draft_id}/publish'.format(**path_param_dict) + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + ######################### + # Data Product Releases + ######################### + + def get_data_product_release( + self, + data_product_id: str, + release_id: str, + *, + check_caller_approval: Optional[bool] = None, + **kwargs, + ) -> DetailedResponse: + """ + Get a release of an existing data product. + + Get a release of an existing data product. Use '-' for the `data_product_id` to + skip specifying the data product ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str release_id: Data product release id. + :param bool check_caller_approval: (optional) If the value is true, then it + will be verfied whether the caller is present in the data access request + pre-approved user list. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductRelease` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not release_id: + raise ValueError('release_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_data_product_release', + ) + headers.update(sdk_headers) + + params = { + 'check_caller_approval': check_caller_approval, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'release_id'] + path_param_values = self.encode_path_vars(data_product_id, release_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def update_data_product_release( + self, + data_product_id: str, + release_id: str, + json_patch_instructions: List['JsonPatchOperation'], + **kwargs, + ) -> DetailedResponse: + """ + Update the data product release identified by ID. + + Use this API to update the properties of a data product release identified by a + valid ID. Use '-' for the `data_product_id` to skip specifying the data product ID + explicitly.

Specify patch operations using http://jsonpatch.com/ + syntax.

Supported patch operations include:

- Update the + properties of a data product

- Add/remove parts from a data product (up + to 20 parts)

- Add/remove use cases from a data product

. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str release_id: Data product release id. + :param List[JsonPatchOperation] json_patch_instructions: A set of patch + operations as defined in RFC 6902. See http://jsonpatch.com/ for more + information. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductRelease` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not release_id: + raise ValueError('release_id must be provided') + if json_patch_instructions is None: + raise ValueError('json_patch_instructions must be provided') + json_patch_instructions = [convert_model(x) for x in json_patch_instructions] + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='update_data_product_release', + ) + headers.update(sdk_headers) + + data = json.dumps(json_patch_instructions) + headers['content-type'] = 'application/json-patch+json' + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'release_id'] + path_param_values = self.encode_path_vars(data_product_id, release_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}'.format(**path_param_dict) + request = self.prepare_request( + method='PATCH', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def get_release_contract_terms_document( + self, + data_product_id: str, + release_id: str, + contract_terms_id: str, + document_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Get a contract document. + + If the document has a completed attachment, the response contains the `url` to + download the attachment.

If the document does not have an attachment, + the response contains the `url` which was submitted at document + creation.

If the document has an incomplete attachment, an error is + returned to prompt the user to upload the document file to complete the + attachment. Use '-' for the `data_product_id` to skip specifying the data product + ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str release_id: Data product release id. + :param str contract_terms_id: Contract terms id. + :param str document_id: Document id. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `ContractTermsDocument` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not release_id: + raise ValueError('release_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + if not document_id: + raise ValueError('document_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_release_contract_terms_document', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'release_id', 'contract_terms_id', 'document_id'] + path_param_values = self.encode_path_vars(data_product_id, release_id, contract_terms_id, document_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}/contract_terms/{contract_terms_id}/documents/{document_id}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + def get_published_data_product_draft_contract_terms( + self, + data_product_id: str, + release_id: str, + contract_terms_id: str, + *, + accept: Optional[str] = None, + include_contract_documents: Optional[bool] = None, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a published data product contract terms identified by id. + + Retrieve a published data product contract terms identified by id. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str release_id: Data product release id. + :param str contract_terms_id: Contract terms id. + :param str accept: (optional) The type of the response: + application/odcs+yaml or application/json. + :param bool include_contract_documents: (optional) Set to false to exclude + external contract documents (e.g., Terms and Conditions URLs) from the + response. By default, these are included. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `BinaryIO` result + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not release_id: + raise ValueError('release_id must be provided') + if not contract_terms_id: + raise ValueError('contract_terms_id must be provided') + headers = { + 'Accept': accept, + } + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_published_data_product_draft_contract_terms', + ) + headers.update(sdk_headers) + + params = { + 'include_contract_documents': include_contract_documents, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + + path_param_keys = ['data_product_id', 'release_id', 'contract_terms_id'] + path_param_values = self.encode_path_vars(data_product_id, release_id, contract_terms_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}/contract_terms/{contract_terms_id}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def list_data_product_releases( + self, + data_product_id: str, + *, + asset_container_id: Optional[str] = None, + state: Optional[List[str]] = None, + version: Optional[str] = None, + limit: Optional[int] = None, + start: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a list of data product releases. + + Retrieve a list of data product releases. Use '-' for the `data_product_id` to + skip specifying the data product ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str asset_container_id: (optional) Filter the list of data product + releases by container id. + :param List[str] state: (optional) Filter the list of data product versions + by state. States are: available and retired. Default is + "available","retired". + :param str version: (optional) Filter the list of data product releases by + version number. + :param int limit: (optional) Limit the number of data product releases in + the results. The maximum is 200. + :param str start: (optional) Start token for pagination. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductReleaseCollection` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='list_data_product_releases', + ) + headers.update(sdk_headers) + + params = { + 'asset.container.id': asset_container_id, + 'state': convert_list(state), + 'version': version, + 'limit': limit, + 'start': start, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id'] + path_param_values = self.encode_path_vars(data_product_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/releases'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def retire_data_product_release( + self, + data_product_id: str, + release_id: str, + *, + revoke_access: Optional[bool] = None, + start_at: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Retire a release of an existing data product. + + Retire a release of an existing data product. Use '-' for the `data_product_id` to + skip specifying the data product ID explicitly. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str release_id: Data product release id. + :param bool revoke_access: (optional) Revoke's Access from all the + Subscriptions of the Data Product. No user's can able to see the subscribed + assets anymore. + :param str start_at: (optional) The date and time when the revoke access + operation should start (ISO 8601 format, e.g., 2025-09-24T06:55:29Z). If + not provided, the operation starts immediately. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductRelease` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not release_id: + raise ValueError('release_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='retire_data_product_release', + ) + headers.update(sdk_headers) + + params = { + 'revoke_access': revoke_access, + 'start_at': start_at, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'release_id'] + path_param_values = self.encode_path_vars(data_product_id, release_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}/retire'.format(**path_param_dict) + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def create_revoke_access_process( + self, + data_product_id: str, + release_id: str, + *, + body: Optional[BinaryIO] = None, + content_type: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Revoke access from Data Product subscriptions. + + Revoke's access from Subscriptions of the data product id passed in the path + parameter. Optionally specify a future date-time when the revoke access operation + should start using the start_at field in ISO 8601 format (e.g., + 2025-09-24T06:55:29Z). If start_at is not provided, the revoke access operation + starts immediately. + + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str release_id: The unique identifier of the data product release. + :param BinaryIO body: (optional) Request parameters to handle revoke access + from subscriptions. The start_at field can be used to schedule the revoke + access operation for a future date-time. + :param str content_type: (optional) The type of the input. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `RevokeAccessResponse` object + """ + + if not data_product_id: + raise ValueError('data_product_id must be provided') + if not release_id: + raise ValueError('release_id must be provided') + headers = { + 'Content-Type': content_type, + } + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='create_revoke_access_process', + ) + headers.update(sdk_headers) + + data = body + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['data_product_id', 'release_id'] + path_param_values = self.encode_path_vars(data_product_id, release_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/data_products/{data_product_id}/releases/{release_id}/revoke_access'.format(**path_param_dict) + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + ######################### + # Data Product Contract Templates + ######################### + + def list_data_product_contract_template( + self, + *, + container_id: Optional[str] = None, + contract_template_name: Optional[str] = None, + domain_ids: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a list of data product contract templates. + + Retrieve a list of data product contract templates. + + :param str container_id: (optional) Container ID of the data product + catalog. If not supplied, the data product catalog is looked up by using + the uid of the default data product catalog. + :param str contract_template_name: (optional) Name of the data product + contract template. If not supplied, the data product templates within the + catalog will returned. + :param str domain_ids: (optional) Comma-separated domain IDs to filter data + product contract templates. If not supplied, the data product templates + within the catalog will returned. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductContractTemplateCollection` object + """ + + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='list_data_product_contract_template', + ) + headers.update(sdk_headers) + + params = { + 'container.id': container_id, + 'contract_template.name': contract_template_name, + 'domain.ids': domain_ids, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/contract_templates' + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def create_contract_template( + self, + container: 'ContainerReference', + *, + id: Optional[str] = None, + creator_id: Optional[str] = None, + created_at: Optional[str] = None, + name: Optional[str] = None, + error: Optional['ErrorMessage'] = None, + contract_terms: Optional['ContractTerms'] = None, + container_id: Optional[str] = None, + contract_template_name: Optional[str] = None, + domain_ids: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Create new data product contract template. + + Create new data product contract template. + + :param ContainerReference container: Container reference. + :param str id: (optional) The identifier of the data product contract + template. + :param str creator_id: (optional) The identifier of the user who created + the data product contract template. + :param str created_at: (optional) The timestamp when the data product + contract template was created. + :param str name: (optional) The name of the contract template. + :param ErrorMessage error: (optional) Contains the code and details. + :param ContractTerms contract_terms: (optional) Defines the complete + structure of a contract terms. + :param str container_id: (optional) Container ID of the data product + catalog. If not supplied, the data product catalog is looked up by using + the uid of the default data product catalog. + :param str contract_template_name: (optional) Name of the data product + contract template. If not supplied, the data product templates within the + catalog will returned. + :param str domain_ids: (optional) Comma-separated domain IDs to filter data + product contract templates. If not supplied, the data product templates + within the catalog will returned. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductContractTemplate` object + """ + + if container is None: + raise ValueError('container must be provided') + container = convert_model(container) + if error is not None: + error = convert_model(error) + if contract_terms is not None: + contract_terms = convert_model(contract_terms) + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='create_contract_template', + ) + headers.update(sdk_headers) + + params = { + 'container.id': container_id, + 'contract_template.name': contract_template_name, + 'domain.ids': domain_ids, + } + + data = { + 'container': container, + 'id': id, + 'creator_id': creator_id, + 'created_at': created_at, + 'name': name, + 'error': error, + 'contract_terms': contract_terms, + } + data = {k: v for (k, v) in data.items() if v is not None} + data = json.dumps(data) + headers['content-type'] = CONTENT_TYPE_JSON + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/contract_templates' + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + params=params, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def get_contract_template( + self, + contract_template_id: str, + container_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a data product contract template identified by id. + + Retrieve a data product contract template identified by id. + + :param str contract_template_id: Data Product Contract Template id. + :param str container_id: Container ID of the data product catalog. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductContractTemplate` object + """ + + if not contract_template_id: + raise ValueError('contract_template_id must be provided') + if not container_id: + raise ValueError('container_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_contract_template', + ) + headers.update(sdk_headers) + + params = { + 'container.id': container_id, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['contract_template_id'] + path_param_values = self.encode_path_vars(contract_template_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/contract_templates/{contract_template_id}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def delete_data_product_contract_template( + self, + contract_template_id: str, + container_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Delete a data product contract template identified by id. + + Delete a data product contract template identified by id. + + :param str contract_template_id: Data Product Contract Template id. + :param str container_id: Container ID of the data product catalog. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse + """ + + if not contract_template_id: + raise ValueError('contract_template_id must be provided') + if not container_id: + raise ValueError('container_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='delete_data_product_contract_template', + ) + headers.update(sdk_headers) + + params = { + 'container.id': container_id, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + + path_param_keys = ['contract_template_id'] + path_param_values = self.encode_path_vars(contract_template_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/contract_templates/{contract_template_id}'.format(**path_param_dict) + request = self.prepare_request( + method='DELETE', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def update_data_product_contract_template( + self, + contract_template_id: str, + container_id: str, + json_patch_instructions: List['JsonPatchOperation'], + **kwargs, + ) -> DetailedResponse: + """ + Update the data product contract template identified by ID. + + Use this API to update the properties of a data product contract template + identified by a valid ID.

Specify patch operations using + http://jsonpatch.com/ syntax.

Supported patch operations + include:

- Update the name of a data product contract template

- + Update the contract terms of data product contract template

. + + :param str contract_template_id: Data Product Contract Template id. + :param str container_id: Container ID of the data product catalog. + :param List[JsonPatchOperation] json_patch_instructions: A set of patch + operations as defined in RFC 6902. See http://jsonpatch.com/ for more + information. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductContractTemplate` object + """ + + if not contract_template_id: + raise ValueError('contract_template_id must be provided') + if not container_id: + raise ValueError('container_id must be provided') + if json_patch_instructions is None: + raise ValueError('json_patch_instructions must be provided') + json_patch_instructions = [convert_model(x) for x in json_patch_instructions] + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='update_data_product_contract_template', + ) + headers.update(sdk_headers) + + params = { + 'container.id': container_id, + } + + data = json.dumps(json_patch_instructions) + headers['content-type'] = 'application/json-patch+json' + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['contract_template_id'] + path_param_values = self.encode_path_vars(contract_template_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/contract_templates/{contract_template_id}'.format(**path_param_dict) + request = self.prepare_request( + method='PATCH', + url=url, + headers=headers, + params=params, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + ######################### + # Data Product Domains + ######################### + + def list_data_product_domains( + self, + *, + container_id: Optional[str] = None, + include_subdomains: Optional[bool] = None, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a list of data product domains. + + Retrieve a list of data product domains. + + :param str container_id: (optional) Container ID of the data product + catalog. If not supplied, the data product catalog is looked up by using + the uid of the default data product catalog. + :param bool include_subdomains: (optional) Include subdomains in the + response. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductDomainCollection` object + """ + + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='list_data_product_domains', + ) + headers.update(sdk_headers) + + params = { + 'container.id': container_id, + 'include_subdomains': include_subdomains, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/domains' + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def create_data_product_domain( + self, + container: 'ContainerReference', + *, + trace: Optional[str] = None, + errors: Optional[List['ErrorModelResource']] = None, + name: Optional[str] = None, + description: Optional[str] = None, + id: Optional[str] = None, + created_by: Optional[str] = None, + member_roles: Optional['MemberRolesSchema'] = None, + properties: Optional['PropertiesSchema'] = None, + sub_domains: Optional[List['InitializeSubDomain']] = None, + sub_container: Optional['ContainerIdentity'] = None, + link_to_subcontainers: Optional[bool] = None, + **kwargs, + ) -> DetailedResponse: + """ + Create new data product domain. + + Create new data product domain. + + :param ContainerReference container: Container reference. + :param str trace: (optional) The id to trace the failed domain creations. + :param List[ErrorModelResource] errors: (optional) Set of errors on the sub + domain creation. + :param str name: (optional) The name of the data product domain. + :param str description: (optional) The description of the data product + domain. + :param str id: (optional) The identifier of the data product domain. + :param str created_by: (optional) The identifier of the creator of the data + product domain. + :param MemberRolesSchema member_roles: (optional) Member roles of a + corresponding asset. + :param PropertiesSchema properties: (optional) Properties of the + corresponding asset. + :param List[InitializeSubDomain] sub_domains: (optional) List of sub + domains to be added within a domain. + :param ContainerIdentity sub_container: (optional) The identity schema for + a IBM knowledge catalog container (catalog/project/space). + :param bool link_to_subcontainers: (optional) Link domains to + subcontainers. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductDomain` object + """ + + if container is None: + raise ValueError('container must be provided') + container = convert_model(container) + if errors is not None: + errors = [convert_model(x) for x in errors] + if member_roles is not None: + member_roles = convert_model(member_roles) + if properties is not None: + properties = convert_model(properties) + if sub_domains is not None: + sub_domains = [convert_model(x) for x in sub_domains] + if sub_container is not None: + sub_container = convert_model(sub_container) + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='create_data_product_domain', + ) + headers.update(sdk_headers) + + params = { + 'link_to_subcontainers': link_to_subcontainers, + } + + data = { + 'container': container, + 'trace': trace, + 'errors': errors, + 'name': name, + 'description': description, + 'id': id, + 'created_by': created_by, + 'member_roles': member_roles, + 'properties': properties, + 'sub_domains': sub_domains, + 'sub_container': sub_container, + } + data = {k: v for (k, v) in data.items() if v is not None} + data = json.dumps(data) + headers['content-type'] = CONTENT_TYPE_JSON + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/domains' + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + params=params, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def create_data_product_subdomain( + self, + domain_id: str, + container_id: str, + *, + name: Optional[str] = None, + id: Optional[str] = None, + description: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Create data product subdomains for a specific domain identified by id. + + Create data product subdomains for a specific domain identified by id. + + :param str domain_id: Domain id. + :param str container_id: Container ID of the data product catalog. + :param str name: (optional) The name of the data product subdomain. + :param str id: (optional) The identifier of the data product subdomain. + :param str description: (optional) The description of the data product + subdomain. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `InitializeSubDomain` object + """ + + if not domain_id: + raise ValueError('domain_id must be provided') + if not container_id: + raise ValueError('container_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='create_data_product_subdomain', + ) + headers.update(sdk_headers) + + params = { + 'container.id': container_id, + } + + data = { + 'name': name, + 'id': id, + 'description': description, + } + data = {k: v for (k, v) in data.items() if v is not None} + data = json.dumps(data) + headers['content-type'] = CONTENT_TYPE_JSON + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['domain_id'] + path_param_values = self.encode_path_vars(domain_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/domains/{domain_id}/subdomains'.format(**path_param_dict) + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + params=params, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def get_domain( + self, + domain_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve a data product domain or subdomain identified by id. + + Retrieve a data product domain or subdomain identified by id. + + :param str domain_id: Domain id. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductDomain` object + """ + + if not domain_id: + raise ValueError('domain_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_domain', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['domain_id'] + path_param_values = self.encode_path_vars(domain_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/domains/{domain_id}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + def delete_domain( + self, + domain_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Delete a data product domain identified by id. + + Delete a data product domain identified by id. + + :param str domain_id: Domain id. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse + """ + + if not domain_id: + raise ValueError('domain_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='delete_domain', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + + path_param_keys = ['domain_id'] + path_param_values = self.encode_path_vars(domain_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/domains/{domain_id}'.format(**path_param_dict) + request = self.prepare_request( + method='DELETE', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + def update_data_product_domain( + self, + domain_id: str, + container_id: str, + json_patch_instructions: List['JsonPatchOperation'], + **kwargs, + ) -> DetailedResponse: + """ + Update the data product domain identified by ID. + + Use this API to update the properties of a data product domain identified by a + valid ID.

Specify patch operations using http://jsonpatch.com/ + syntax.

Supported patch operations include:

- Update the name of + a data product domain

- Update the description of a data product + domain

- Update the rov of a data product domain

. + + :param str domain_id: Domain id. + :param str container_id: Container ID of the data product catalog. + :param List[JsonPatchOperation] json_patch_instructions: A set of patch + operations as defined in RFC 6902. See http://jsonpatch.com/ for more + information. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductDomain` object + """ + + if not domain_id: + raise ValueError('domain_id must be provided') + if not container_id: + raise ValueError('container_id must be provided') + if json_patch_instructions is None: + raise ValueError('json_patch_instructions must be provided') + json_patch_instructions = [convert_model(x) for x in json_patch_instructions] + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='update_data_product_domain', + ) + headers.update(sdk_headers) + + params = { + 'container.id': container_id, + } + + data = json.dumps(json_patch_instructions) + headers['content-type'] = 'application/json-patch+json' + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['domain_id'] + path_param_values = self.encode_path_vars(domain_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/domains/{domain_id}'.format(**path_param_dict) + request = self.prepare_request( + method='PATCH', + url=url, + headers=headers, + params=params, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def get_data_product_by_domain( + self, + domain_id: str, + container_id: str, + **kwargs, + ) -> DetailedResponse: + """ + Retrieve all data products in a domain specified by id or any of it's subdomains. + + Retrieve all the data products tagged to the domain identified by id or any of + it's subdomains. + + :param str domain_id: Domain id. + :param str container_id: Container ID of the data product catalog. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `DataProductVersionCollection` object + """ + + if not domain_id: + raise ValueError('domain_id must be provided') + if not container_id: + raise ValueError('container_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_data_product_by_domain', + ) + headers.update(sdk_headers) + + params = { + 'container.id': container_id, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['domain_id'] + path_param_values = self.encode_path_vars(domain_id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/domains/{domain_id}/data_products'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + ######################### + # Bucket Services + ######################### + + def create_s3_bucket( + self, + is_shared: bool, + **kwargs, + ) -> DetailedResponse: + """ + Create a new Bucket. + + Use this API to create a new S3 Bucket on an AWS Hosting. + + :param bool is_shared: Flag to specify whether the bucket is dedicated or + shared. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `BucketResponse` object + """ + + if is_shared is None: + raise ValueError('is_shared must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='create_s3_bucket', + ) + headers.update(sdk_headers) + + params = { + 'is_shared': is_shared, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/bucket' + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + def get_s3_bucket_validation( + self, + bucket_name: str, + **kwargs, + ) -> DetailedResponse: + """ + Validate the Bucket Existence. + + Use this API to validate the bucket existence on an AWS hosting. + + :param str bucket_name: Name of the bucket to validate. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `BucketValidationResponse` object + """ + + if not bucket_name: + raise ValueError('bucket_name must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_s3_bucket_validation', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + path_param_keys = ['bucket_name'] + path_param_values = self.encode_path_vars(bucket_name) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/data_product_exchange/v1/bucket/validate/{bucket_name}'.format(**path_param_dict) + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response + + ######################### + # Data Product Revoke Access Job Runs + ######################### + + def get_revoke_access_process_state( + self, + release_id: str, + *, + limit: Optional[int] = None, + start: Optional[str] = None, + **kwargs, + ) -> DetailedResponse: + """ + Access revoke status of the subscriptions against the data product release id. + + Retrieves the status of revoke access requests. + + :param str release_id: Pass the data product release version id to retrieve + job runs state for that specific DPV ID. + :param int limit: (optional) Limit the number of tracking assets in the + results. The maximum is 200. + :param str start: (optional) Start token for pagination. + :param dict headers: A `dict` containing the request headers + :return: A `DetailedResponse` containing the result, headers and HTTP status code. + :rtype: DetailedResponse with `dict` result representing a `RevokeAccessStateResponse` object + """ + + if not release_id: + raise ValueError('release_id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='get_revoke_access_process_state', + ) + headers.update(sdk_headers) + + params = { + 'release_id': release_id, + 'limit': limit, + 'start': start, + } + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = CONTENT_TYPE_JSON + + url = '/data_product_exchange/v1/data_product_revoke_access/job_runs' + request = self.prepare_request( + method='GET', + url=url, + headers=headers, + params=params, + ) + + response = self.send(request, **kwargs) + return response + + +class GetDataProductDraftContractTermsEnums: + """ + Enums for get_data_product_draft_contract_terms parameters. + """ + + class Accept(str, Enum): + """ + The type of the response: application/json or application/odcs+yaml. + """ + + APPLICATION_JSON = CONTENT_TYPE_JSON + APPLICATION_ODCS_YAML = 'application/odcs+yaml' + + +class GetContractTermsInSpecifiedFormatEnums: + """ + Enums for get_contract_terms_in_specified_format parameters. + """ + + class Accept(str, Enum): + """ + The type of the response: application/odcs+yaml or application/json. + """ + + APPLICATION_ODCS_YAML = 'application/odcs+yaml' + APPLICATION_JSON = CONTENT_TYPE_JSON + + +class GetPublishedDataProductDraftContractTermsEnums: + """ + Enums for get_published_data_product_draft_contract_terms parameters. + """ + + class Accept(str, Enum): + """ + The type of the response: application/odcs+yaml or application/json. + """ + + APPLICATION_ODCS_YAML = 'application/odcs+yaml' + APPLICATION_JSON = CONTENT_TYPE_JSON + + +class ListDataProductReleasesEnums: + """ + Enums for list_data_product_releases parameters. + """ + + class State(str, Enum): + """ + Filter the list of data product versions by state. States are: available and + retired. Default is "available","retired". + """ + + AVAILABLE = 'available' + RETIRED = 'retired' + + +############################################################################## +# Models +############################################################################## + + +class Asset: + """ + Asset. + + :param dict metadata: (optional) + :param dict entity: (optional) + """ + + def __init__( + self, + *, + metadata: Optional[dict] = None, + entity: Optional[dict] = None, + ) -> None: + """ + Initialize a Asset object. + + :param dict metadata: (optional) + :param dict entity: (optional) + """ + self.metadata = metadata + self.entity = entity + + @classmethod + def from_dict(cls, _dict: Dict) -> 'Asset': + """Initialize a Asset object from a json dictionary.""" + args = {} + if (metadata := _dict.get('metadata')) is not None: + args['metadata'] = metadata + if (entity := _dict.get('entity')) is not None: + args['entity'] = entity + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a Asset object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'metadata') and self.metadata is not None: + _dict['metadata'] = self.metadata + if hasattr(self, 'entity') and self.entity is not None: + _dict['entity'] = self.entity + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this Asset object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'Asset') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'Asset') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class AssetListAccessControl: + """ + Access control object. + + :param str owner: (optional) The owner of the asset. + """ + + def __init__( + self, + *, + owner: Optional[str] = None, + ) -> None: + """ + Initialize a AssetListAccessControl object. + + :param str owner: (optional) The owner of the asset. + """ + self.owner = owner + + @classmethod + def from_dict(cls, _dict: Dict) -> 'AssetListAccessControl': + """Initialize a AssetListAccessControl object from a json dictionary.""" + args = {} + if (owner := _dict.get('owner')) is not None: + args['owner'] = owner + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a AssetListAccessControl object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'owner') and self.owner is not None: + _dict['owner'] = self.owner + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this AssetListAccessControl object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'AssetListAccessControl') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'AssetListAccessControl') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class AssetPartReference: + """ + The asset represented in this part. + + :param str id: (optional) The unique identifier of the asset. + :param str name: (optional) Asset name. + :param ContainerReference container: Container reference. + :param str type: (optional) The type of the asset. + """ + + def __init__( + self, + container: 'ContainerReference', + *, + id: Optional[str] = None, + name: Optional[str] = None, + type: Optional[str] = None, + ) -> None: + """ + Initialize a AssetPartReference object. + + :param ContainerReference container: Container reference. + :param str id: (optional) The unique identifier of the asset. + :param str name: (optional) Asset name. + :param str type: (optional) The type of the asset. + """ + self.id = id + self.name = name + self.container = container + self.type = type + + @classmethod + def from_dict(cls, _dict: Dict) -> 'AssetPartReference': + """Initialize a AssetPartReference object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + if (name := _dict.get('name')) is not None: + args['name'] = name + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in AssetPartReference JSON') + if (type := _dict.get('type')) is not None: + args['type'] = type + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a AssetPartReference object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + if hasattr(self, 'type') and self.type is not None: + _dict['type'] = self.type + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this AssetPartReference object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'AssetPartReference') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'AssetPartReference') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class AssetPrototype: + """ + New asset input properties. + + :param str id: (optional) The unique identifier of the asset. + :param ContainerIdentity container: The identity schema for a IBM knowledge + catalog container (catalog/project/space). + """ + + def __init__( + self, + container: 'ContainerIdentity', + *, + id: Optional[str] = None, + ) -> None: + """ + Initialize a AssetPrototype object. + + :param ContainerIdentity container: The identity schema for a IBM knowledge + catalog container (catalog/project/space). + :param str id: (optional) The unique identifier of the asset. + """ + self.id = id + self.container = container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'AssetPrototype': + """Initialize a AssetPrototype object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + if (container := _dict.get('container')) is not None: + args['container'] = ContainerIdentity.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in AssetPrototype JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a AssetPrototype object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this AssetPrototype object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'AssetPrototype') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'AssetPrototype') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class AssetReference: + """ + The reference schema for a asset in a container. + + :param str id: (optional) The unique identifier of the asset. + :param str name: (optional) Asset name. + :param ContainerReference container: Container reference. + """ + + def __init__( + self, + container: 'ContainerReference', + *, + id: Optional[str] = None, + name: Optional[str] = None, + ) -> None: + """ + Initialize a AssetReference object. + + :param ContainerReference container: Container reference. + :param str id: (optional) The unique identifier of the asset. + :param str name: (optional) Asset name. + """ + self.id = id + self.name = name + self.container = container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'AssetReference': + """Initialize a AssetReference object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + if (name := _dict.get('name')) is not None: + args['name'] = name + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in AssetReference JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a AssetReference object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this AssetReference object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'AssetReference') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'AssetReference') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class BucketResponse: + """ + BucketResponse to hold the Bucket response data. + + :param str bucket_name: (optional) Name of the Bucket. + :param str bucket_location: (optional) Location of the Bucket stored. + :param str role_arn: (optional) Role ARN. + :param str bucket_type: (optional) Bucket Type. + :param bool shared: (optional) Is Shared Bucket. + """ + + def __init__( + self, + *, + bucket_name: Optional[str] = None, + bucket_location: Optional[str] = None, + role_arn: Optional[str] = None, + bucket_type: Optional[str] = None, + shared: Optional[bool] = None, + ) -> None: + """ + Initialize a BucketResponse object. + + :param str bucket_name: (optional) Name of the Bucket. + :param str bucket_location: (optional) Location of the Bucket stored. + :param str role_arn: (optional) Role ARN. + :param str bucket_type: (optional) Bucket Type. + :param bool shared: (optional) Is Shared Bucket. + """ + self.bucket_name = bucket_name + self.bucket_location = bucket_location + self.role_arn = role_arn + self.bucket_type = bucket_type + self.shared = shared + + @classmethod + def from_dict(cls, _dict: Dict) -> 'BucketResponse': + """Initialize a BucketResponse object from a json dictionary.""" + args = {} + if (bucket_name := _dict.get('bucket_name')) is not None: + args['bucket_name'] = bucket_name + if (bucket_location := _dict.get('bucket_location')) is not None: + args['bucket_location'] = bucket_location + if (role_arn := _dict.get('role_arn')) is not None: + args['role_arn'] = role_arn + if (bucket_type := _dict.get('bucket_type')) is not None: + args['bucket_type'] = bucket_type + if (shared := _dict.get('shared')) is not None: + args['shared'] = shared + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a BucketResponse object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'bucket_name') and self.bucket_name is not None: + _dict['bucket_name'] = self.bucket_name + if hasattr(self, 'bucket_location') and self.bucket_location is not None: + _dict['bucket_location'] = self.bucket_location + if hasattr(self, 'role_arn') and self.role_arn is not None: + _dict['role_arn'] = self.role_arn + if hasattr(self, 'bucket_type') and self.bucket_type is not None: + _dict['bucket_type'] = self.bucket_type + if hasattr(self, 'shared') and self.shared is not None: + _dict['shared'] = self.shared + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this BucketResponse object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'BucketResponse') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'BucketResponse') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class BucketValidationResponse: + """ + BucketValidationResponse to hold the bucket validation data. + + :param bool bucket_exists: (optional) Flag of bucket existence. + """ + + def __init__( + self, + *, + bucket_exists: Optional[bool] = None, + ) -> None: + """ + Initialize a BucketValidationResponse object. + + :param bool bucket_exists: (optional) Flag of bucket existence. + """ + self.bucket_exists = bucket_exists + + @classmethod + def from_dict(cls, _dict: Dict) -> 'BucketValidationResponse': + """Initialize a BucketValidationResponse object from a json dictionary.""" + args = {} + if (bucket_exists := _dict.get('bucket_exists')) is not None: + args['bucket_exists'] = bucket_exists + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a BucketValidationResponse object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'bucket_exists') and self.bucket_exists is not None: + _dict['bucket_exists'] = self.bucket_exists + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this BucketValidationResponse object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'BucketValidationResponse') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'BucketValidationResponse') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContainerIdentity: + """ + The identity schema for a IBM knowledge catalog container (catalog/project/space). + + :param str id: Container identifier. + """ + + def __init__( + self, + id: str, + ) -> None: + """ + Initialize a ContainerIdentity object. + + :param str id: Container identifier. + """ + self.id = id + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContainerIdentity': + """Initialize a ContainerIdentity object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in ContainerIdentity JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContainerIdentity object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContainerIdentity object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContainerIdentity') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContainerIdentity') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContainerReference: + """ + Container reference. + + :param str id: Container identifier. + :param str type: Container type. + """ + + def __init__( + self, + id: str, + type: str, + ) -> None: + """ + Initialize a ContainerReference object. + + :param str id: Container identifier. + :param str type: Container type. + """ + self.id = id + self.type = type + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContainerReference': + """Initialize a ContainerReference object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in ContainerReference JSON') + if (type := _dict.get('type')) is not None: + args['type'] = type + else: + raise ValueError('Required property \'type\' not present in ContainerReference JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContainerReference object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'type') and self.type is not None: + _dict['type'] = self.type + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContainerReference object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContainerReference') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContainerReference') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class TypeEnum(str, Enum): + """ + Container type. + """ + + CATALOG = 'catalog' + PROJECT = 'project' + + + +class ContractAsset: + """ + Defines a data asset name and id. + + :param str id: (optional) ID of the data asset. + :param str name: (optional) Name of the data asset. + """ + + def __init__( + self, + *, + id: Optional[str] = None, + name: Optional[str] = None, + ) -> None: + """ + Initialize a ContractAsset object. + + :param str id: (optional) ID of the data asset. + :param str name: (optional) Name of the data asset. + """ + self.id = id + self.name = name + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractAsset': + """Initialize a ContractAsset object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + if (name := _dict.get('name')) is not None: + args['name'] = name + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractAsset object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractAsset object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractAsset') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractAsset') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractQualityRule: + """ + Defines a quality rule for validating data assets. + + :param str type: The type of the quality rule: 'text', 'library', or 'sql'. + :param str description: (optional) A descriptive explanation of the quality + rule. + :param str rule: (optional) The name or identifier of the library-based quality + rule to be applied. + :param str implementation: (optional) A text (non-parsed) block of code required + for the third-party DQ engine to run. + :param str engine: (optional) Required for custom DQ rule: name of the + third-party engine being used. Common values include soda, greatExpectations, + montecarlo, etc. + :param str must_be_less_than: (optional) The threshold value that the quality + check result must be less than. + :param str must_be_less_or_equal_to: (optional) The threshold value that the + quality check result must be less than or equal to. + :param str must_be_greater_than: (optional) The threshold value that the quality + check result must be greater than. + :param str must_be_greater_or_equal_to: (optional) The threshold value that the + quality check result must be greater than or equal to. + :param List[str] must_be_between: (optional) Inclusive range (min and max) for + the quality check result. + :param List[str] must_not_be_between: (optional) Inclusive range (min and max) + the quality check must not fall within. + :param str must_be: (optional) The exact value(s) the quality check result must + match. + :param str must_not_be: (optional) The exact value(s) the quality check result + must not match. + :param str name: (optional) User-friendly name for the quality rule. + :param str unit: (optional) Unit used for evaluating the quality rule (e.g., + rows, records). + :param str query: (optional) The SQL query to execute for validating quality in + case of a 'sql' rule type. + """ + + def __init__( + self, + type: str, + *, + description: Optional[str] = None, + rule: Optional[str] = None, + implementation: Optional[str] = None, + engine: Optional[str] = None, + must_be_less_than: Optional[str] = None, + must_be_less_or_equal_to: Optional[str] = None, + must_be_greater_than: Optional[str] = None, + must_be_greater_or_equal_to: Optional[str] = None, + must_be_between: Optional[List[str]] = None, + must_not_be_between: Optional[List[str]] = None, + must_be: Optional[str] = None, + must_not_be: Optional[str] = None, + name: Optional[str] = None, + unit: Optional[str] = None, + query: Optional[str] = None, + ) -> None: + """ + Initialize a ContractQualityRule object. + + :param str type: The type of the quality rule: 'text', 'library', or 'sql'. + :param str description: (optional) A descriptive explanation of the quality + rule. + :param str rule: (optional) The name or identifier of the library-based + quality rule to be applied. + :param str implementation: (optional) A text (non-parsed) block of code + required for the third-party DQ engine to run. + :param str engine: (optional) Required for custom DQ rule: name of the + third-party engine being used. Common values include soda, + greatExpectations, montecarlo, etc. + :param str must_be_less_than: (optional) The threshold value that the + quality check result must be less than. + :param str must_be_less_or_equal_to: (optional) The threshold value that + the quality check result must be less than or equal to. + :param str must_be_greater_than: (optional) The threshold value that the + quality check result must be greater than. + :param str must_be_greater_or_equal_to: (optional) The threshold value that + the quality check result must be greater than or equal to. + :param List[str] must_be_between: (optional) Inclusive range (min and max) + for the quality check result. + :param List[str] must_not_be_between: (optional) Inclusive range (min and + max) the quality check must not fall within. + :param str must_be: (optional) The exact value(s) the quality check result + must match. + :param str must_not_be: (optional) The exact value(s) the quality check + result must not match. + :param str name: (optional) User-friendly name for the quality rule. + :param str unit: (optional) Unit used for evaluating the quality rule + (e.g., rows, records). + :param str query: (optional) The SQL query to execute for validating + quality in case of a 'sql' rule type. + """ + self.type = type + self.description = description + self.rule = rule + self.implementation = implementation + self.engine = engine + self.must_be_less_than = must_be_less_than + self.must_be_less_or_equal_to = must_be_less_or_equal_to + self.must_be_greater_than = must_be_greater_than + self.must_be_greater_or_equal_to = must_be_greater_or_equal_to + self.must_be_between = must_be_between + self.must_not_be_between = must_not_be_between + self.must_be = must_be + self.must_not_be = must_not_be + self.name = name + self.unit = unit + self.query = query + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractQualityRule': + """Initialize a ContractQualityRule object from a json dictionary.""" + args = {} + if (type := _dict.get('type')) is not None: + args['type'] = type + else: + raise ValueError('Required property \'type\' not present in ContractQualityRule JSON') + if (description := _dict.get('description')) is not None: + args['description'] = description + if (rule := _dict.get('rule')) is not None: + args['rule'] = rule + if (implementation := _dict.get('implementation')) is not None: + args['implementation'] = implementation + if (engine := _dict.get('engine')) is not None: + args['engine'] = engine + if (must_be_less_than := _dict.get('must_be_less_than')) is not None: + args['must_be_less_than'] = must_be_less_than + if (must_be_less_or_equal_to := _dict.get('must_be_less_or_equal_to')) is not None: + args['must_be_less_or_equal_to'] = must_be_less_or_equal_to + if (must_be_greater_than := _dict.get('must_be_greater_than')) is not None: + args['must_be_greater_than'] = must_be_greater_than + if (must_be_greater_or_equal_to := _dict.get('must_be_greater_or_equal_to')) is not None: + args['must_be_greater_or_equal_to'] = must_be_greater_or_equal_to + if (must_be_between := _dict.get('must_be_between')) is not None: + args['must_be_between'] = must_be_between + if (must_not_be_between := _dict.get('must_not_be_between')) is not None: + args['must_not_be_between'] = must_not_be_between + if (must_be := _dict.get('must_be')) is not None: + args['must_be'] = must_be + if (must_not_be := _dict.get('must_not_be')) is not None: + args['must_not_be'] = must_not_be + if (name := _dict.get('name')) is not None: + args['name'] = name + if (unit := _dict.get('unit')) is not None: + args['unit'] = unit + if (query := _dict.get('query')) is not None: + args['query'] = query + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractQualityRule object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'type') and self.type is not None: + _dict['type'] = self.type + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'rule') and self.rule is not None: + _dict['rule'] = self.rule + if hasattr(self, 'implementation') and self.implementation is not None: + _dict['implementation'] = self.implementation + if hasattr(self, 'engine') and self.engine is not None: + _dict['engine'] = self.engine + if hasattr(self, 'must_be_less_than') and self.must_be_less_than is not None: + _dict['must_be_less_than'] = self.must_be_less_than + if hasattr(self, 'must_be_less_or_equal_to') and self.must_be_less_or_equal_to is not None: + _dict['must_be_less_or_equal_to'] = self.must_be_less_or_equal_to + if hasattr(self, 'must_be_greater_than') and self.must_be_greater_than is not None: + _dict['must_be_greater_than'] = self.must_be_greater_than + if hasattr(self, 'must_be_greater_or_equal_to') and self.must_be_greater_or_equal_to is not None: + _dict['must_be_greater_or_equal_to'] = self.must_be_greater_or_equal_to + if hasattr(self, 'must_be_between') and self.must_be_between is not None: + _dict['must_be_between'] = self.must_be_between + if hasattr(self, 'must_not_be_between') and self.must_not_be_between is not None: + _dict['must_not_be_between'] = self.must_not_be_between + if hasattr(self, 'must_be') and self.must_be is not None: + _dict['must_be'] = self.must_be + if hasattr(self, 'must_not_be') and self.must_not_be is not None: + _dict['must_not_be'] = self.must_not_be + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'unit') and self.unit is not None: + _dict['unit'] = self.unit + if hasattr(self, 'query') and self.query is not None: + _dict['query'] = self.query + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractQualityRule object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractQualityRule') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractQualityRule') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractSchema: + """ + Schema definition of the data asset. + + :param str asset_id: Id of the data asset whose schema information is stored. + :param str connection_id: Connection Id of the data asset whose schema + information is stored. + :param str name: (optional) Name of the schema or data asset part. + :param str description: (optional) Description of the schema. + :param str connection_path: (optional) Connection path of the asset. + :param str physical_type: (optional) MIME type or physical type. + :param List[ContractSchemaProperty] properties: (optional) List of properties. + :param List[ContractQualityRule] quality: (optional) List of quality rules + defined for the asset. + """ + + def __init__( + self, + asset_id: str, + connection_id: str, + *, + name: Optional[str] = None, + description: Optional[str] = None, + connection_path: Optional[str] = None, + physical_type: Optional[str] = None, + properties: Optional[List['ContractSchemaProperty']] = None, + quality: Optional[List['ContractQualityRule']] = None, + ) -> None: + """ + Initialize a ContractSchema object. + + :param str asset_id: Id of the data asset whose schema information is + stored. + :param str connection_id: Connection Id of the data asset whose schema + information is stored. + :param str name: (optional) Name of the schema or data asset part. + :param str description: (optional) Description of the schema. + :param str connection_path: (optional) Connection path of the asset. + :param str physical_type: (optional) MIME type or physical type. + :param List[ContractSchemaProperty] properties: (optional) List of + properties. + :param List[ContractQualityRule] quality: (optional) List of quality rules + defined for the asset. + """ + self.asset_id = asset_id + self.connection_id = connection_id + self.name = name + self.description = description + self.connection_path = connection_path + self.physical_type = physical_type + self.properties = properties + self.quality = quality + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractSchema': + """Initialize a ContractSchema object from a json dictionary.""" + args = {} + if (asset_id := _dict.get('asset_id')) is not None: + args['asset_id'] = asset_id + else: + raise ValueError('Required property \'asset_id\' not present in ContractSchema JSON') + if (connection_id := _dict.get('connection_id')) is not None: + args['connection_id'] = connection_id + else: + raise ValueError('Required property \'connection_id\' not present in ContractSchema JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + if (description := _dict.get('description')) is not None: + args['description'] = description + if (connection_path := _dict.get('connection_path')) is not None: + args['connection_path'] = connection_path + if (physical_type := _dict.get('physical_type')) is not None: + args['physical_type'] = physical_type + if (properties := _dict.get('properties')) is not None: + args['properties'] = [ContractSchemaProperty.from_dict(v) for v in properties] + if (quality := _dict.get('quality')) is not None: + args['quality'] = [ContractQualityRule.from_dict(v) for v in quality] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractSchema object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'asset_id') and self.asset_id is not None: + _dict['asset_id'] = self.asset_id + if hasattr(self, 'connection_id') and self.connection_id is not None: + _dict['connection_id'] = self.connection_id + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'connection_path') and self.connection_path is not None: + _dict['connection_path'] = self.connection_path + if hasattr(self, 'physical_type') and self.physical_type is not None: + _dict['physical_type'] = self.physical_type + if hasattr(self, 'properties') and self.properties is not None: + properties_list = [] + for v in self.properties: + if isinstance(v, dict): + properties_list.append(v) + else: + properties_list.append(v.to_dict()) + _dict['properties'] = properties_list + if hasattr(self, 'quality') and self.quality is not None: + quality_list = [] + for v in self.quality: + if isinstance(v, dict): + quality_list.append(v) + else: + quality_list.append(v.to_dict()) + _dict['quality'] = quality_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractSchema object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractSchema') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractSchema') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractSchemaProperty: + """ + Defines a property inside the schema. + + :param str name: Property name. + :param ContractSchemaPropertyType type: (optional) Detailed type definition of a + schema property. + :param List[ContractQualityRule] quality: (optional) List of quality rules + defined for the column. + """ + + def __init__( + self, + name: str, + *, + type: Optional['ContractSchemaPropertyType'] = None, + quality: Optional[List['ContractQualityRule']] = None, + ) -> None: + """ + Initialize a ContractSchemaProperty object. + + :param str name: Property name. + :param ContractSchemaPropertyType type: (optional) Detailed type definition + of a schema property. + :param List[ContractQualityRule] quality: (optional) List of quality rules + defined for the column. + """ + self.name = name + self.type = type + self.quality = quality + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractSchemaProperty': + """Initialize a ContractSchemaProperty object from a json dictionary.""" + args = {} + if (name := _dict.get('name')) is not None: + args['name'] = name + else: + raise ValueError('Required property \'name\' not present in ContractSchemaProperty JSON') + if (type := _dict.get('type')) is not None: + args['type'] = ContractSchemaPropertyType.from_dict(type) + if (quality := _dict.get('quality')) is not None: + args['quality'] = [ContractQualityRule.from_dict(v) for v in quality] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractSchemaProperty object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'type') and self.type is not None: + if isinstance(self.type, dict): + _dict['type'] = self.type + else: + _dict['type'] = self.type.to_dict() + if hasattr(self, 'quality') and self.quality is not None: + quality_list = [] + for v in self.quality: + if isinstance(v, dict): + quality_list.append(v) + else: + quality_list.append(v.to_dict()) + _dict['quality'] = quality_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractSchemaProperty object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractSchemaProperty') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractSchemaProperty') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractSchemaPropertyType: + """ + Detailed type definition of a schema property. + + :param str type: (optional) Type of the field. + :param str length: (optional) Length of the field as string. + :param str scale: (optional) Scale of the field as string. + :param str nullable: (optional) Is field nullable? true/false as string. + :param str signed: (optional) Is field signed? true/false as string. + :param str native_type: (optional) Native type of the field. + """ + + def __init__( + self, + *, + type: Optional[str] = None, + length: Optional[str] = None, + scale: Optional[str] = None, + nullable: Optional[str] = None, + signed: Optional[str] = None, + native_type: Optional[str] = None, + ) -> None: + """ + Initialize a ContractSchemaPropertyType object. + + :param str type: (optional) Type of the field. + :param str length: (optional) Length of the field as string. + :param str scale: (optional) Scale of the field as string. + :param str nullable: (optional) Is field nullable? true/false as string. + :param str signed: (optional) Is field signed? true/false as string. + :param str native_type: (optional) Native type of the field. + """ + self.type = type + self.length = length + self.scale = scale + self.nullable = nullable + self.signed = signed + self.native_type = native_type + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractSchemaPropertyType': + """Initialize a ContractSchemaPropertyType object from a json dictionary.""" + args = {} + if (type := _dict.get('type')) is not None: + args['type'] = type + if (length := _dict.get('length')) is not None: + args['length'] = length + if (scale := _dict.get('scale')) is not None: + args['scale'] = scale + if (nullable := _dict.get('nullable')) is not None: + args['nullable'] = nullable + if (signed := _dict.get('signed')) is not None: + args['signed'] = signed + if (native_type := _dict.get('native_type')) is not None: + args['native_type'] = native_type + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractSchemaPropertyType object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'type') and self.type is not None: + _dict['type'] = self.type + if hasattr(self, 'length') and self.length is not None: + _dict['length'] = self.length + if hasattr(self, 'scale') and self.scale is not None: + _dict['scale'] = self.scale + if hasattr(self, 'nullable') and self.nullable is not None: + _dict['nullable'] = self.nullable + if hasattr(self, 'signed') and self.signed is not None: + _dict['signed'] = self.signed + if hasattr(self, 'native_type') and self.native_type is not None: + _dict['native_type'] = self.native_type + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractSchemaPropertyType object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractSchemaPropertyType') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractSchemaPropertyType') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractServer: + """ + Schema definition of a server configuration for the asset. + + :param str server: Name of the server. + :param ContractAsset asset: (optional) Defines a data asset name and id. + :param str connection_id: (optional) ID of the data source associated with data + asset. + :param str type: (optional) Type of the server. + :param str description: (optional) Description of the server. + :param str environment: (optional) Environment in which the server operates. + :param str account: (optional) Account used by the server. + :param str catalog: (optional) Catalog name. + :param str database: (optional) Database name. + :param str dataset: (optional) Dataset name. + :param str delimiter: (optional) Delimiter. + :param str endpoint_url: (optional) Server endpoint URL. + :param str format: (optional) File format. + :param str host: (optional) Host name or IP address. + :param str location: (optional) Location URL. + :param str path: (optional) Relative or absolute path to the data. + :param str port: (optional) Port to the server. + :param str project: (optional) Project name. + :param str region: (optional) Cloud region. + :param str region_name: (optional) Region name. + :param str schema: (optional) Schema name. + :param str service_name: (optional) Service name. + :param str staging_dir: (optional) Staging directory. + :param str stream: (optional) Data stream name. + :param str warehouse: (optional) Warehouse or cluster name. + :param List[str] roles: (optional) List of roles for the server. + :param List[ContractTemplateCustomProperty] custom_properties: (optional) List + of custom properties for the server. + """ + + def __init__( + self, + server: str, + *, + asset: Optional['ContractAsset'] = None, + connection_id: Optional[str] = None, + type: Optional[str] = None, + description: Optional[str] = None, + environment: Optional[str] = None, + account: Optional[str] = None, + catalog: Optional[str] = None, + database: Optional[str] = None, + dataset: Optional[str] = None, + delimiter: Optional[str] = None, + endpoint_url: Optional[str] = None, + format: Optional[str] = None, + host: Optional[str] = None, + location: Optional[str] = None, + path: Optional[str] = None, + port: Optional[str] = None, + project: Optional[str] = None, + region: Optional[str] = None, + region_name: Optional[str] = None, + schema: Optional[str] = None, + service_name: Optional[str] = None, + staging_dir: Optional[str] = None, + stream: Optional[str] = None, + warehouse: Optional[str] = None, + roles: Optional[List[str]] = None, + custom_properties: Optional[List['ContractTemplateCustomProperty']] = None, + ) -> None: + """ + Initialize a ContractServer object. + + :param str server: Name of the server. + :param ContractAsset asset: (optional) Defines a data asset name and id. + :param str connection_id: (optional) ID of the data source associated with + data asset. + :param str type: (optional) Type of the server. + :param str description: (optional) Description of the server. + :param str environment: (optional) Environment in which the server + operates. + :param str account: (optional) Account used by the server. + :param str catalog: (optional) Catalog name. + :param str database: (optional) Database name. + :param str dataset: (optional) Dataset name. + :param str delimiter: (optional) Delimiter. + :param str endpoint_url: (optional) Server endpoint URL. + :param str format: (optional) File format. + :param str host: (optional) Host name or IP address. + :param str location: (optional) Location URL. + :param str path: (optional) Relative or absolute path to the data. + :param str port: (optional) Port to the server. + :param str project: (optional) Project name. + :param str region: (optional) Cloud region. + :param str region_name: (optional) Region name. + :param str schema: (optional) Schema name. + :param str service_name: (optional) Service name. + :param str staging_dir: (optional) Staging directory. + :param str stream: (optional) Data stream name. + :param str warehouse: (optional) Warehouse or cluster name. + :param List[str] roles: (optional) List of roles for the server. + :param List[ContractTemplateCustomProperty] custom_properties: (optional) + List of custom properties for the server. + """ + self.server = server + self.asset = asset + self.connection_id = connection_id + self.type = type + self.description = description + self.environment = environment + self.account = account + self.catalog = catalog + self.database = database + self.dataset = dataset + self.delimiter = delimiter + self.endpoint_url = endpoint_url + self.format = format + self.host = host + self.location = location + self.path = path + self.port = port + self.project = project + self.region = region + self.region_name = region_name + self.schema = schema + self.service_name = service_name + self.staging_dir = staging_dir + self.stream = stream + self.warehouse = warehouse + self.roles = roles + self.custom_properties = custom_properties + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractServer': + """Initialize a ContractServer object from a json dictionary.""" + args = {} + if (server := _dict.get('server')) is not None: + args['server'] = server + else: + raise ValueError('Required property \'server\' not present in ContractServer JSON') + if (asset := _dict.get('asset')) is not None: + args['asset'] = ContractAsset.from_dict(asset) + if (connection_id := _dict.get('connection_id')) is not None: + args['connection_id'] = connection_id + if (type := _dict.get('type')) is not None: + args['type'] = type + if (description := _dict.get('description')) is not None: + args['description'] = description + if (environment := _dict.get('environment')) is not None: + args['environment'] = environment + if (account := _dict.get('account')) is not None: + args['account'] = account + if (catalog := _dict.get('catalog')) is not None: + args['catalog'] = catalog + if (database := _dict.get('database')) is not None: + args['database'] = database + if (dataset := _dict.get('dataset')) is not None: + args['dataset'] = dataset + if (delimiter := _dict.get('delimiter')) is not None: + args['delimiter'] = delimiter + if (endpoint_url := _dict.get('endpoint_url')) is not None: + args['endpoint_url'] = endpoint_url + if (format := _dict.get('format')) is not None: + args['format'] = format + if (host := _dict.get('host')) is not None: + args['host'] = host + if (location := _dict.get('location')) is not None: + args['location'] = location + if (path := _dict.get('path')) is not None: + args['path'] = path + if (port := _dict.get('port')) is not None: + args['port'] = port + if (project := _dict.get('project')) is not None: + args['project'] = project + if (region := _dict.get('region')) is not None: + args['region'] = region + if (region_name := _dict.get('region_name')) is not None: + args['region_name'] = region_name + if (schema := _dict.get('schema')) is not None: + args['schema'] = schema + if (service_name := _dict.get('service_name')) is not None: + args['service_name'] = service_name + if (staging_dir := _dict.get('staging_dir')) is not None: + args['staging_dir'] = staging_dir + if (stream := _dict.get('stream')) is not None: + args['stream'] = stream + if (warehouse := _dict.get('warehouse')) is not None: + args['warehouse'] = warehouse + if (roles := _dict.get('roles')) is not None: + args['roles'] = roles + if (custom_properties := _dict.get('custom_properties')) is not None: + args['custom_properties'] = [ContractTemplateCustomProperty.from_dict(v) for v in custom_properties] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractServer object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'server') and self.server is not None: + _dict['server'] = self.server + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + if hasattr(self, 'connection_id') and self.connection_id is not None: + _dict['connection_id'] = self.connection_id + if hasattr(self, 'type') and self.type is not None: + _dict['type'] = self.type + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'environment') and self.environment is not None: + _dict['environment'] = self.environment + if hasattr(self, 'account') and self.account is not None: + _dict['account'] = self.account + if hasattr(self, 'catalog') and self.catalog is not None: + _dict['catalog'] = self.catalog + if hasattr(self, 'database') and self.database is not None: + _dict['database'] = self.database + if hasattr(self, 'dataset') and self.dataset is not None: + _dict['dataset'] = self.dataset + if hasattr(self, 'delimiter') and self.delimiter is not None: + _dict['delimiter'] = self.delimiter + if hasattr(self, 'endpoint_url') and self.endpoint_url is not None: + _dict['endpoint_url'] = self.endpoint_url + if hasattr(self, 'format') and self.format is not None: + _dict['format'] = self.format + if hasattr(self, 'host') and self.host is not None: + _dict['host'] = self.host + if hasattr(self, 'location') and self.location is not None: + _dict['location'] = self.location + if hasattr(self, 'path') and self.path is not None: + _dict['path'] = self.path + if hasattr(self, 'port') and self.port is not None: + _dict['port'] = self.port + if hasattr(self, 'project') and self.project is not None: + _dict['project'] = self.project + if hasattr(self, 'region') and self.region is not None: + _dict['region'] = self.region + if hasattr(self, 'region_name') and self.region_name is not None: + _dict['region_name'] = self.region_name + if hasattr(self, 'schema') and self.schema is not None: + _dict['schema'] = self.schema + if hasattr(self, 'service_name') and self.service_name is not None: + _dict['service_name'] = self.service_name + if hasattr(self, 'staging_dir') and self.staging_dir is not None: + _dict['staging_dir'] = self.staging_dir + if hasattr(self, 'stream') and self.stream is not None: + _dict['stream'] = self.stream + if hasattr(self, 'warehouse') and self.warehouse is not None: + _dict['warehouse'] = self.warehouse + if hasattr(self, 'roles') and self.roles is not None: + _dict['roles'] = self.roles + if hasattr(self, 'custom_properties') and self.custom_properties is not None: + custom_properties_list = [] + for v in self.custom_properties: + if isinstance(v, dict): + custom_properties_list.append(v) + else: + custom_properties_list.append(v.to_dict()) + _dict['custom_properties'] = custom_properties_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractServer object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractServer') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractServer') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractTemplateCustomProperty: + """ + Represents a custom property within the contract. + + :param str key: The name of the key. Names should be in camel case–the same as + if they were permanent properties in the contract. + :param str value: The value of the key. + """ + + def __init__( + self, + key: str, + value: str, + ) -> None: + """ + Initialize a ContractTemplateCustomProperty object. + + :param str key: The name of the key. Names should be in camel case–the same + as if they were permanent properties in the contract. + :param str value: The value of the key. + """ + self.key = key + self.value = value + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTemplateCustomProperty': + """Initialize a ContractTemplateCustomProperty object from a json dictionary.""" + args = {} + if (key := _dict.get('key')) is not None: + args['key'] = key + else: + raise ValueError('Required property \'key\' not present in ContractTemplateCustomProperty JSON') + if (value := _dict.get('value')) is not None: + args['value'] = value + else: + raise ValueError('Required property \'value\' not present in ContractTemplateCustomProperty JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTemplateCustomProperty object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'key') and self.key is not None: + _dict['key'] = self.key + if hasattr(self, 'value') and self.value is not None: + _dict['value'] = self.value + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTemplateCustomProperty object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTemplateCustomProperty') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTemplateCustomProperty') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractTemplateOrganization: + """ + Represents a user and their role in the contract. + + :param str user_id: The user ID associated with the contract. + :param str role: The role of the user in the contract. + """ + + def __init__( + self, + user_id: str, + role: str, + ) -> None: + """ + Initialize a ContractTemplateOrganization object. + + :param str user_id: The user ID associated with the contract. + :param str role: The role of the user in the contract. + """ + self.user_id = user_id + self.role = role + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTemplateOrganization': + """Initialize a ContractTemplateOrganization object from a json dictionary.""" + args = {} + if (user_id := _dict.get('user_id')) is not None: + args['user_id'] = user_id + else: + raise ValueError('Required property \'user_id\' not present in ContractTemplateOrganization JSON') + if (role := _dict.get('role')) is not None: + args['role'] = role + else: + raise ValueError('Required property \'role\' not present in ContractTemplateOrganization JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTemplateOrganization object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'user_id') and self.user_id is not None: + _dict['user_id'] = self.user_id + if hasattr(self, 'role') and self.role is not None: + _dict['role'] = self.role + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTemplateOrganization object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTemplateOrganization') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTemplateOrganization') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractTemplateSLA: + """ + Represents the SLA details of the contract. + + :param str default_element: (optional) The default SLA element. + :param List[ContractTemplateSLAProperty] properties: (optional) List of SLA + properties and their values. + """ + + def __init__( + self, + *, + default_element: Optional[str] = None, + properties: Optional[List['ContractTemplateSLAProperty']] = None, + ) -> None: + """ + Initialize a ContractTemplateSLA object. + + :param str default_element: (optional) The default SLA element. + :param List[ContractTemplateSLAProperty] properties: (optional) List of SLA + properties and their values. + """ + self.default_element = default_element + self.properties = properties + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTemplateSLA': + """Initialize a ContractTemplateSLA object from a json dictionary.""" + args = {} + if (default_element := _dict.get('default_element')) is not None: + args['default_element'] = default_element + if (properties := _dict.get('properties')) is not None: + args['properties'] = [ContractTemplateSLAProperty.from_dict(v) for v in properties] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTemplateSLA object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'default_element') and self.default_element is not None: + _dict['default_element'] = self.default_element + if hasattr(self, 'properties') and self.properties is not None: + properties_list = [] + for v in self.properties: + if isinstance(v, dict): + properties_list.append(v) + else: + properties_list.append(v.to_dict()) + _dict['properties'] = properties_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTemplateSLA object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTemplateSLA') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTemplateSLA') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractTemplateSLAProperty: + """ + Represents an SLA property and its value. + + :param str property: The SLA property name. + :param str value: The value associated with the SLA property. + """ + + def __init__( + self, + property: str, + value: str, + ) -> None: + """ + Initialize a ContractTemplateSLAProperty object. + + :param str property: The SLA property name. + :param str value: The value associated with the SLA property. + """ + self.property = property + self.value = value + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTemplateSLAProperty': + """Initialize a ContractTemplateSLAProperty object from a json dictionary.""" + args = {} + if (property := _dict.get('property')) is not None: + args['property'] = property + else: + raise ValueError('Required property \'property\' not present in ContractTemplateSLAProperty JSON') + if (value := _dict.get('value')) is not None: + args['value'] = value + else: + raise ValueError('Required property \'value\' not present in ContractTemplateSLAProperty JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTemplateSLAProperty object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'property') and self.property is not None: + _dict['property'] = self.property + if hasattr(self, 'value') and self.value is not None: + _dict['value'] = self.value + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTemplateSLAProperty object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTemplateSLAProperty') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTemplateSLAProperty') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractTemplateSupportAndCommunication: + """ + Represents a support and communication channel for the contract. + + :param str channel: The communication channel. + :param str url: The URL associated with the communication channel. + """ + + def __init__( + self, + channel: str, + url: str, + ) -> None: + """ + Initialize a ContractTemplateSupportAndCommunication object. + + :param str channel: The communication channel. + :param str url: The URL associated with the communication channel. + """ + self.channel = channel + self.url = url + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTemplateSupportAndCommunication': + """Initialize a ContractTemplateSupportAndCommunication object from a json dictionary.""" + args = {} + if (channel := _dict.get('channel')) is not None: + args['channel'] = channel + else: + raise ValueError('Required property \'channel\' not present in ContractTemplateSupportAndCommunication JSON') + if (url := _dict.get('url')) is not None: + args['url'] = url + else: + raise ValueError('Required property \'url\' not present in ContractTemplateSupportAndCommunication JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTemplateSupportAndCommunication object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'channel') and self.channel is not None: + _dict['channel'] = self.channel + if hasattr(self, 'url') and self.url is not None: + _dict['url'] = self.url + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTemplateSupportAndCommunication object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTemplateSupportAndCommunication') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTemplateSupportAndCommunication') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractTerms: + """ + Defines the complete structure of a contract terms. + + :param AssetReference asset: (optional) The reference schema for a asset in a + container. + :param str id: (optional) ID of the contract terms. + :param List[ContractTermsDocument] documents: (optional) Collection of contract + terms documents. + :param str error_msg: (optional) An error message, if existing, relating to the + contract terms. + :param Overview overview: (optional) Overview details of a data contract. + :param Description description: (optional) Description details of a data + contract. + :param List[ContractTemplateOrganization] organization: (optional) List of sub + domains to be added within a domain. + :param List[Roles] roles: (optional) List of roles associated with the contract. + :param Pricing price: (optional) Represents the pricing details of the contract. + :param List[ContractTemplateSLA] sla: (optional) Service Level Agreement + details. + :param List[ContractTemplateSupportAndCommunication] support_and_communication: + (optional) Support and communication details for the contract. + :param List[ContractTemplateCustomProperty] custom_properties: (optional) Custom + properties that are not part of the standard contract. + :param ContractTest contract_test: (optional) Contains the contract test status + and related metadata. + :param List[ContractServer] servers: (optional) List of server definitions. + :param List[ContractSchema] schema: (optional) Schema details of the data asset. + """ + + def __init__( + self, + *, + asset: Optional['AssetReference'] = None, + id: Optional[str] = None, + documents: Optional[List['ContractTermsDocument']] = None, + error_msg: Optional[str] = None, + overview: Optional['Overview'] = None, + description: Optional['Description'] = None, + organization: Optional[List['ContractTemplateOrganization']] = None, + roles: Optional[List['Roles']] = None, + price: Optional['Pricing'] = None, + sla: Optional[List['ContractTemplateSLA']] = None, + support_and_communication: Optional[List['ContractTemplateSupportAndCommunication']] = None, + custom_properties: Optional[List['ContractTemplateCustomProperty']] = None, + contract_test: Optional['ContractTest'] = None, + servers: Optional[List['ContractServer']] = None, + schema: Optional[List['ContractSchema']] = None, + ) -> None: + """ + Initialize a ContractTerms object. + + :param AssetReference asset: (optional) The reference schema for a asset in + a container. + :param str id: (optional) ID of the contract terms. + :param List[ContractTermsDocument] documents: (optional) Collection of + contract terms documents. + :param str error_msg: (optional) An error message, if existing, relating to + the contract terms. + :param Overview overview: (optional) Overview details of a data contract. + :param Description description: (optional) Description details of a data + contract. + :param List[ContractTemplateOrganization] organization: (optional) List of + sub domains to be added within a domain. + :param List[Roles] roles: (optional) List of roles associated with the + contract. + :param Pricing price: (optional) Represents the pricing details of the + contract. + :param List[ContractTemplateSLA] sla: (optional) Service Level Agreement + details. + :param List[ContractTemplateSupportAndCommunication] + support_and_communication: (optional) Support and communication details for + the contract. + :param List[ContractTemplateCustomProperty] custom_properties: (optional) + Custom properties that are not part of the standard contract. + :param ContractTest contract_test: (optional) Contains the contract test + status and related metadata. + :param List[ContractServer] servers: (optional) List of server definitions. + :param List[ContractSchema] schema: (optional) Schema details of the data + asset. + """ + self.asset = asset + self.id = id + self.documents = documents + self.error_msg = error_msg + self.overview = overview + self.description = description + self.organization = organization + self.roles = roles + self.price = price + self.sla = sla + self.support_and_communication = support_and_communication + self.custom_properties = custom_properties + self.contract_test = contract_test + self.servers = servers + self.schema = schema + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTerms': + """Initialize a ContractTerms object from a json dictionary.""" + args = {} + if (asset := _dict.get('asset')) is not None: + args['asset'] = AssetReference.from_dict(asset) + if (id := _dict.get('id')) is not None: + args['id'] = id + if (documents := _dict.get('documents')) is not None: + args['documents'] = [ContractTermsDocument.from_dict(v) for v in documents] + if (error_msg := _dict.get('error_msg')) is not None: + args['error_msg'] = error_msg + if (overview := _dict.get('overview')) is not None: + args['overview'] = Overview.from_dict(overview) + if (description := _dict.get('description')) is not None: + args['description'] = Description.from_dict(description) + if (organization := _dict.get('organization')) is not None: + args['organization'] = [ContractTemplateOrganization.from_dict(v) for v in organization] + if (roles := _dict.get('roles')) is not None: + args['roles'] = [Roles.from_dict(v) for v in roles] + if (price := _dict.get('price')) is not None: + args['price'] = Pricing.from_dict(price) + if (sla := _dict.get('sla')) is not None: + args['sla'] = [ContractTemplateSLA.from_dict(v) for v in sla] + if (support_and_communication := _dict.get('support_and_communication')) is not None: + args['support_and_communication'] = [ContractTemplateSupportAndCommunication.from_dict(v) for v in support_and_communication] + if (custom_properties := _dict.get('custom_properties')) is not None: + args['custom_properties'] = [ContractTemplateCustomProperty.from_dict(v) for v in custom_properties] + if (contract_test := _dict.get('contract_test')) is not None: + args['contract_test'] = ContractTest.from_dict(contract_test) + if (servers := _dict.get('servers')) is not None: + args['servers'] = [ContractServer.from_dict(v) for v in servers] + if (schema := _dict.get('schema')) is not None: + args['schema'] = [ContractSchema.from_dict(v) for v in schema] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTerms object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'documents') and self.documents is not None: + documents_list = [] + for v in self.documents: + if isinstance(v, dict): + documents_list.append(v) + else: + documents_list.append(v.to_dict()) + _dict['documents'] = documents_list + if hasattr(self, 'error_msg') and self.error_msg is not None: + _dict['error_msg'] = self.error_msg + if hasattr(self, 'overview') and self.overview is not None: + if isinstance(self.overview, dict): + _dict['overview'] = self.overview + else: + _dict['overview'] = self.overview.to_dict() + if hasattr(self, 'description') and self.description is not None: + if isinstance(self.description, dict): + _dict['description'] = self.description + else: + _dict['description'] = self.description.to_dict() + if hasattr(self, 'organization') and self.organization is not None: + organization_list = [] + for v in self.organization: + if isinstance(v, dict): + organization_list.append(v) + else: + organization_list.append(v.to_dict()) + _dict['organization'] = organization_list + if hasattr(self, 'roles') and self.roles is not None: + roles_list = [] + for v in self.roles: + if isinstance(v, dict): + roles_list.append(v) + else: + roles_list.append(v.to_dict()) + _dict['roles'] = roles_list + if hasattr(self, 'price') and self.price is not None: + if isinstance(self.price, dict): + _dict['price'] = self.price + else: + _dict['price'] = self.price.to_dict() + if hasattr(self, 'sla') and self.sla is not None: + sla_list = [] + for v in self.sla: + if isinstance(v, dict): + sla_list.append(v) + else: + sla_list.append(v.to_dict()) + _dict['sla'] = sla_list + if hasattr(self, 'support_and_communication') and self.support_and_communication is not None: + support_and_communication_list = [] + for v in self.support_and_communication: + if isinstance(v, dict): + support_and_communication_list.append(v) + else: + support_and_communication_list.append(v.to_dict()) + _dict['support_and_communication'] = support_and_communication_list + if hasattr(self, 'custom_properties') and self.custom_properties is not None: + custom_properties_list = [] + for v in self.custom_properties: + if isinstance(v, dict): + custom_properties_list.append(v) + else: + custom_properties_list.append(v.to_dict()) + _dict['custom_properties'] = custom_properties_list + if hasattr(self, 'contract_test') and self.contract_test is not None: + if isinstance(self.contract_test, dict): + _dict['contract_test'] = self.contract_test + else: + _dict['contract_test'] = self.contract_test.to_dict() + if hasattr(self, 'servers') and self.servers is not None: + servers_list = [] + for v in self.servers: + if isinstance(v, dict): + servers_list.append(v) + else: + servers_list.append(v.to_dict()) + _dict['servers'] = servers_list + if hasattr(self, 'schema') and self.schema is not None: + schema_list = [] + for v in self.schema: + if isinstance(v, dict): + schema_list.append(v) + else: + schema_list.append(v.to_dict()) + _dict['schema'] = schema_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTerms object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTerms') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTerms') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractTermsDocument: + """ + Standard contract terms document, which is used for get and list contract terms + responses. + + :param str url: (optional) URL that can be used to retrieve the contract + document. + :param str type: Type of the contract document. + :param str name: Name of the contract document. + :param str id: Id uniquely identifying this document within the contract terms + instance. + :param ContractTermsDocumentAttachment attachment: (optional) Attachment + associated witht the document. + :param str upload_url: (optional) URL which can be used to upload document file. + """ + + def __init__( + self, + type: str, + name: str, + id: str, + *, + url: Optional[str] = None, + attachment: Optional['ContractTermsDocumentAttachment'] = None, + upload_url: Optional[str] = None, + ) -> None: + """ + Initialize a ContractTermsDocument object. + + :param str type: Type of the contract document. + :param str name: Name of the contract document. + :param str id: Id uniquely identifying this document within the contract + terms instance. + :param str url: (optional) URL that can be used to retrieve the contract + document. + :param ContractTermsDocumentAttachment attachment: (optional) Attachment + associated witht the document. + :param str upload_url: (optional) URL which can be used to upload document + file. + """ + self.url = url + self.type = type + self.name = name + self.id = id + self.attachment = attachment + self.upload_url = upload_url + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTermsDocument': + """Initialize a ContractTermsDocument object from a json dictionary.""" + args = {} + if (url := _dict.get('url')) is not None: + args['url'] = url + if (type := _dict.get('type')) is not None: + args['type'] = type + else: + raise ValueError('Required property \'type\' not present in ContractTermsDocument JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + else: + raise ValueError('Required property \'name\' not present in ContractTermsDocument JSON') + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in ContractTermsDocument JSON') + if (attachment := _dict.get('attachment')) is not None: + args['attachment'] = ContractTermsDocumentAttachment.from_dict(attachment) + if (upload_url := _dict.get('upload_url')) is not None: + args['upload_url'] = upload_url + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTermsDocument object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'url') and self.url is not None: + _dict['url'] = self.url + if hasattr(self, 'type') and self.type is not None: + _dict['type'] = self.type + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'attachment') and self.attachment is not None: + if isinstance(self.attachment, dict): + _dict['attachment'] = self.attachment + else: + _dict['attachment'] = self.attachment.to_dict() + if hasattr(self, 'upload_url') and self.upload_url is not None: + _dict['upload_url'] = self.upload_url + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTermsDocument object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTermsDocument') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTermsDocument') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class TypeEnum(str, Enum): + """ + Type of the contract document. + """ + + TERMS_AND_CONDITIONS = 'terms_and_conditions' + SLA = 'sla' + + + +class ContractTermsDocumentAttachment: + """ + Attachment associated witht the document. + + :param str id: (optional) Id representing the corresponding attachment. + """ + + def __init__( + self, + *, + id: Optional[str] = None, + ) -> None: + """ + Initialize a ContractTermsDocumentAttachment object. + + :param str id: (optional) Id representing the corresponding attachment. + """ + self.id = id + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTermsDocumentAttachment': + """Initialize a ContractTermsDocumentAttachment object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTermsDocumentAttachment object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTermsDocumentAttachment object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTermsDocumentAttachment') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTermsDocumentAttachment') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractTermsMoreInfo: + """ + List of links to sources that provide more details on the dataset. + + :param str type: Type of Source Link. + :param str url: Link to source that provide more details on the dataset. + """ + + def __init__( + self, + type: str, + url: str, + ) -> None: + """ + Initialize a ContractTermsMoreInfo object. + + :param str type: Type of Source Link. + :param str url: Link to source that provide more details on the dataset. + """ + self.type = type + self.url = url + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTermsMoreInfo': + """Initialize a ContractTermsMoreInfo object from a json dictionary.""" + args = {} + if (type := _dict.get('type')) is not None: + args['type'] = type + else: + raise ValueError('Required property \'type\' not present in ContractTermsMoreInfo JSON') + if (url := _dict.get('url')) is not None: + args['url'] = url + else: + raise ValueError('Required property \'url\' not present in ContractTermsMoreInfo JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTermsMoreInfo object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'type') and self.type is not None: + _dict['type'] = self.type + if hasattr(self, 'url') and self.url is not None: + _dict['url'] = self.url + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTermsMoreInfo object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTermsMoreInfo') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTermsMoreInfo') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ContractTest: + """ + Contains the contract test status and related metadata. + + :param str status: Status of the contract test (pass or fail). + :param str last_tested_time: Timestamp of when the contract was last tested. + :param str message: (optional) Optional message or details about the contract + test. + """ + + def __init__( + self, + status: str, + last_tested_time: str, + *, + message: Optional[str] = None, + ) -> None: + """ + Initialize a ContractTest object. + + :param str status: Status of the contract test (pass or fail). + :param str last_tested_time: Timestamp of when the contract was last + tested. + :param str message: (optional) Optional message or details about the + contract test. + """ + self.status = status + self.last_tested_time = last_tested_time + self.message = message + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ContractTest': + """Initialize a ContractTest object from a json dictionary.""" + args = {} + if (status := _dict.get('status')) is not None: + args['status'] = status + else: + raise ValueError('Required property \'status\' not present in ContractTest JSON') + if (last_tested_time := _dict.get('last_tested_time')) is not None: + args['last_tested_time'] = last_tested_time + else: + raise ValueError('Required property \'last_tested_time\' not present in ContractTest JSON') + if (message := _dict.get('message')) is not None: + args['message'] = message + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ContractTest object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'status') and self.status is not None: + _dict['status'] = self.status + if hasattr(self, 'last_tested_time') and self.last_tested_time is not None: + _dict['last_tested_time'] = self.last_tested_time + if hasattr(self, 'message') and self.message is not None: + _dict['message'] = self.message + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ContractTest object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ContractTest') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ContractTest') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class StatusEnum(str, Enum): + """ + Status of the contract test (pass or fail). + """ + + PASS = 'pass' + FAIL = 'fail' + + + +class DataAssetRelationship: + """ + Data members for visualization process. + + :param Visualization visualization: (optional) Data members for visualization. + :param AssetReference asset: The reference schema for a asset in a container. + :param AssetReference related_asset: The reference schema for a asset in a + container. + :param ErrorMessage error: (optional) Contains the code and details. + """ + + def __init__( + self, + asset: 'AssetReference', + related_asset: 'AssetReference', + *, + visualization: Optional['Visualization'] = None, + error: Optional['ErrorMessage'] = None, + ) -> None: + """ + Initialize a DataAssetRelationship object. + + :param AssetReference asset: The reference schema for a asset in a + container. + :param AssetReference related_asset: The reference schema for a asset in a + container. + :param Visualization visualization: (optional) Data members for + visualization. + :param ErrorMessage error: (optional) Contains the code and details. + """ + self.visualization = visualization + self.asset = asset + self.related_asset = related_asset + self.error = error + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataAssetRelationship': + """Initialize a DataAssetRelationship object from a json dictionary.""" + args = {} + if (visualization := _dict.get('visualization')) is not None: + args['visualization'] = Visualization.from_dict(visualization) + if (asset := _dict.get('asset')) is not None: + args['asset'] = AssetReference.from_dict(asset) + else: + raise ValueError('Required property \'asset\' not present in DataAssetRelationship JSON') + if (related_asset := _dict.get('related_asset')) is not None: + args['related_asset'] = AssetReference.from_dict(related_asset) + else: + raise ValueError('Required property \'related_asset\' not present in DataAssetRelationship JSON') + if (error := _dict.get('error')) is not None: + args['error'] = ErrorMessage.from_dict(error) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataAssetRelationship object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'visualization') and self.visualization is not None: + if isinstance(self.visualization, dict): + _dict['visualization'] = self.visualization + else: + _dict['visualization'] = self.visualization.to_dict() + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + if hasattr(self, 'related_asset') and self.related_asset is not None: + if isinstance(self.related_asset, dict): + _dict['related_asset'] = self.related_asset + else: + _dict['related_asset'] = self.related_asset.to_dict() + if hasattr(self, 'error') and self.error is not None: + if isinstance(self.error, dict): + _dict['error'] = self.error + else: + _dict['error'] = self.error.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataAssetRelationship object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataAssetRelationship') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataAssetRelationship') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataAssetVisualizationRes: + """ + Data relationships for the visualization process response. + + :param List[DataAssetRelationship] results: (optional) Data asset Ids and their + related asset Ids. + """ + + def __init__( + self, + *, + results: Optional[List['DataAssetRelationship']] = None, + ) -> None: + """ + Initialize a DataAssetVisualizationRes object. + + :param List[DataAssetRelationship] results: (optional) Data asset Ids and + their related asset Ids. + """ + self.results = results + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataAssetVisualizationRes': + """Initialize a DataAssetVisualizationRes object from a json dictionary.""" + args = {} + if (results := _dict.get('results')) is not None: + args['results'] = [DataAssetRelationship.from_dict(v) for v in results] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataAssetVisualizationRes object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'results') and self.results is not None: + results_list = [] + for v in self.results: + if isinstance(v, dict): + results_list.append(v) + else: + results_list.append(v.to_dict()) + _dict['results'] = results_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataAssetVisualizationRes object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataAssetVisualizationRes') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataAssetVisualizationRes') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProduct: + """ + Data Product. + + :param str id: Data product identifier. + :param DataProductDraftVersionRelease release: (optional) A data product draft + version object. + :param ContainerReference container: Container reference. + :param str name: (optional) Data product name. + :param DataProductVersionSummary latest_release: (optional) Summary of Data + Product Version object. + :param List[DataProductVersionSummary] drafts: (optional) List of draft + summaries of this data product. + """ + + def __init__( + self, + id: str, + container: 'ContainerReference', + *, + release: Optional['DataProductDraftVersionRelease'] = None, + name: Optional[str] = None, + latest_release: Optional['DataProductVersionSummary'] = None, + drafts: Optional[List['DataProductVersionSummary']] = None, + ) -> None: + """ + Initialize a DataProduct object. + + :param str id: Data product identifier. + :param ContainerReference container: Container reference. + :param DataProductDraftVersionRelease release: (optional) A data product + draft version object. + :param str name: (optional) Data product name. + :param DataProductVersionSummary latest_release: (optional) Summary of Data + Product Version object. + :param List[DataProductVersionSummary] drafts: (optional) List of draft + summaries of this data product. + """ + self.id = id + self.release = release + self.container = container + self.name = name + self.latest_release = latest_release + self.drafts = drafts + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProduct': + """Initialize a DataProduct object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProduct JSON') + if (release := _dict.get('release')) is not None: + args['release'] = DataProductDraftVersionRelease.from_dict(release) + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DataProduct JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + if (latest_release := _dict.get('latest_release')) is not None: + args['latest_release'] = DataProductVersionSummary.from_dict(latest_release) + if (drafts := _dict.get('drafts')) is not None: + args['drafts'] = [DataProductVersionSummary.from_dict(v) for v in drafts] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProduct object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'release') and self.release is not None: + if isinstance(self.release, dict): + _dict['release'] = self.release + else: + _dict['release'] = self.release.to_dict() + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'latest_release') and self.latest_release is not None: + if isinstance(self.latest_release, dict): + _dict['latest_release'] = self.latest_release + else: + _dict['latest_release'] = self.latest_release.to_dict() + if hasattr(self, 'drafts') and self.drafts is not None: + drafts_list = [] + for v in self.drafts: + if isinstance(v, dict): + drafts_list.append(v) + else: + drafts_list.append(v.to_dict()) + _dict['drafts'] = drafts_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProduct object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProduct') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProduct') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductCollection: + """ + A collection of data product summaries. + + :param int limit: Set a limit on the number of results returned. + :param FirstPage first: First page in the collection. + :param NextPage next: (optional) Next page in the collection. + :param int total_results: (optional) Indicates the total number of results + returned. + :param List[DataProductSummary] data_products: Collection of data product + summaries. + """ + + def __init__( + self, + limit: int, + first: 'FirstPage', + data_products: List['DataProductSummary'], + *, + next: Optional['NextPage'] = None, + total_results: Optional[int] = None, + ) -> None: + """ + Initialize a DataProductCollection object. + + :param int limit: Set a limit on the number of results returned. + :param FirstPage first: First page in the collection. + :param List[DataProductSummary] data_products: Collection of data product + summaries. + :param NextPage next: (optional) Next page in the collection. + :param int total_results: (optional) Indicates the total number of results + returned. + """ + self.limit = limit + self.first = first + self.next = next + self.total_results = total_results + self.data_products = data_products + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductCollection': + """Initialize a DataProductCollection object from a json dictionary.""" + args = {} + if (limit := _dict.get('limit')) is not None: + args['limit'] = limit + else: + raise ValueError('Required property \'limit\' not present in DataProductCollection JSON') + if (first := _dict.get('first')) is not None: + args['first'] = FirstPage.from_dict(first) + else: + raise ValueError('Required property \'first\' not present in DataProductCollection JSON') + if (next := _dict.get('next')) is not None: + args['next'] = NextPage.from_dict(next) + if (total_results := _dict.get('total_results')) is not None: + args['total_results'] = total_results + if (data_products := _dict.get('data_products')) is not None: + args['data_products'] = [DataProductSummary.from_dict(v) for v in data_products] + else: + raise ValueError('Required property \'data_products\' not present in DataProductCollection JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductCollection object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'limit') and self.limit is not None: + _dict['limit'] = self.limit + if hasattr(self, 'first') and self.first is not None: + if isinstance(self.first, dict): + _dict['first'] = self.first + else: + _dict['first'] = self.first.to_dict() + if hasattr(self, 'next') and self.next is not None: + if isinstance(self.next, dict): + _dict['next'] = self.next + else: + _dict['next'] = self.next.to_dict() + if hasattr(self, 'total_results') and self.total_results is not None: + _dict['total_results'] = self.total_results + if hasattr(self, 'data_products') and self.data_products is not None: + data_products_list = [] + for v in self.data_products: + if isinstance(v, dict): + data_products_list.append(v) + else: + data_products_list.append(v.to_dict()) + _dict['data_products'] = data_products_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductCollection object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductCollection') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductCollection') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductContractTemplate: + """ + Defines the complete structure of a contract template. + + :param ContainerReference container: Container reference. + :param str id: (optional) The identifier of the data product contract template. + :param str creator_id: (optional) The identifier of the user who created the + data product contract template. + :param str created_at: (optional) The timestamp when the data product contract + template was created. + :param str name: (optional) The name of the contract template. + :param ErrorMessage error: (optional) Contains the code and details. + :param ContractTerms contract_terms: (optional) Defines the complete structure + of a contract terms. + """ + + def __init__( + self, + container: 'ContainerReference', + *, + id: Optional[str] = None, + creator_id: Optional[str] = None, + created_at: Optional[str] = None, + name: Optional[str] = None, + error: Optional['ErrorMessage'] = None, + contract_terms: Optional['ContractTerms'] = None, + ) -> None: + """ + Initialize a DataProductContractTemplate object. + + :param ContainerReference container: Container reference. + :param str id: (optional) The identifier of the data product contract + template. + :param str creator_id: (optional) The identifier of the user who created + the data product contract template. + :param str created_at: (optional) The timestamp when the data product + contract template was created. + :param str name: (optional) The name of the contract template. + :param ErrorMessage error: (optional) Contains the code and details. + :param ContractTerms contract_terms: (optional) Defines the complete + structure of a contract terms. + """ + self.container = container + self.id = id + self.creator_id = creator_id + self.created_at = created_at + self.name = name + self.error = error + self.contract_terms = contract_terms + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductContractTemplate': + """Initialize a DataProductContractTemplate object from a json dictionary.""" + args = {} + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DataProductContractTemplate JSON') + if (id := _dict.get('id')) is not None: + args['id'] = id + if (creator_id := _dict.get('creator_id')) is not None: + args['creator_id'] = creator_id + if (created_at := _dict.get('created_at')) is not None: + args['created_at'] = created_at + if (name := _dict.get('name')) is not None: + args['name'] = name + if (error := _dict.get('error')) is not None: + args['error'] = ErrorMessage.from_dict(error) + if (contract_terms := _dict.get('contract_terms')) is not None: + args['contract_terms'] = ContractTerms.from_dict(contract_terms) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductContractTemplate object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'creator_id') and self.creator_id is not None: + _dict['creator_id'] = self.creator_id + if hasattr(self, 'created_at') and self.created_at is not None: + _dict['created_at'] = self.created_at + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'error') and self.error is not None: + if isinstance(self.error, dict): + _dict['error'] = self.error + else: + _dict['error'] = self.error.to_dict() + if hasattr(self, 'contract_terms') and self.contract_terms is not None: + if isinstance(self.contract_terms, dict): + _dict['contract_terms'] = self.contract_terms + else: + _dict['contract_terms'] = self.contract_terms.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductContractTemplate object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductContractTemplate') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductContractTemplate') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductContractTemplateCollection: + """ + A collection of data product contract templates. + + :param List[DataProductContractTemplate] contract_templates: Collection of data + product contract templates. + """ + + def __init__( + self, + contract_templates: List['DataProductContractTemplate'], + ) -> None: + """ + Initialize a DataProductContractTemplateCollection object. + + :param List[DataProductContractTemplate] contract_templates: Collection of + data product contract templates. + """ + self.contract_templates = contract_templates + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductContractTemplateCollection': + """Initialize a DataProductContractTemplateCollection object from a json dictionary.""" + args = {} + if (contract_templates := _dict.get('contract_templates')) is not None: + args['contract_templates'] = [DataProductContractTemplate.from_dict(v) for v in contract_templates] + else: + raise ValueError('Required property \'contract_templates\' not present in DataProductContractTemplateCollection JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductContractTemplateCollection object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'contract_templates') and self.contract_templates is not None: + contract_templates_list = [] + for v in self.contract_templates: + if isinstance(v, dict): + contract_templates_list.append(v) + else: + contract_templates_list.append(v.to_dict()) + _dict['contract_templates'] = contract_templates_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductContractTemplateCollection object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductContractTemplateCollection') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductContractTemplateCollection') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductCustomWorkflowDefinition: + """ + A custom workflow definition to be used to create a workflow to approve a data product + subscription. + + :param str id: (optional) ID of a workflow definition. + """ + + def __init__( + self, + *, + id: Optional[str] = None, + ) -> None: + """ + Initialize a DataProductCustomWorkflowDefinition object. + + :param str id: (optional) ID of a workflow definition. + """ + self.id = id + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductCustomWorkflowDefinition': + """Initialize a DataProductCustomWorkflowDefinition object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductCustomWorkflowDefinition object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductCustomWorkflowDefinition object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductCustomWorkflowDefinition') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductCustomWorkflowDefinition') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductDomain: + """ + The data product domain. + + :param ContainerReference container: Container reference. + :param str trace: (optional) The id to trace the failed domain creations. + :param List[ErrorModelResource] errors: (optional) Set of errors on the sub + domain creation. + :param str name: (optional) The name of the data product domain. + :param str description: (optional) The description of the data product domain. + :param str id: (optional) The identifier of the data product domain. + :param str created_by: (optional) The identifier of the creator of the data + product domain. + :param MemberRolesSchema member_roles: (optional) Member roles of a + corresponding asset. + :param PropertiesSchema properties: (optional) Properties of the corresponding + asset. + :param List[InitializeSubDomain] sub_domains: (optional) List of sub domains to + be added within a domain. + :param ContainerIdentity sub_container: (optional) The identity schema for a IBM + knowledge catalog container (catalog/project/space). + """ + + def __init__( + self, + container: 'ContainerReference', + *, + trace: Optional[str] = None, + errors: Optional[List['ErrorModelResource']] = None, + name: Optional[str] = None, + description: Optional[str] = None, + id: Optional[str] = None, + created_by: Optional[str] = None, + member_roles: Optional['MemberRolesSchema'] = None, + properties: Optional['PropertiesSchema'] = None, + sub_domains: Optional[List['InitializeSubDomain']] = None, + sub_container: Optional['ContainerIdentity'] = None, + ) -> None: + """ + Initialize a DataProductDomain object. + + :param ContainerReference container: Container reference. + :param str trace: (optional) The id to trace the failed domain creations. + :param List[ErrorModelResource] errors: (optional) Set of errors on the sub + domain creation. + :param str name: (optional) The name of the data product domain. + :param str description: (optional) The description of the data product + domain. + :param str id: (optional) The identifier of the data product domain. + :param str created_by: (optional) The identifier of the creator of the data + product domain. + :param MemberRolesSchema member_roles: (optional) Member roles of a + corresponding asset. + :param PropertiesSchema properties: (optional) Properties of the + corresponding asset. + :param List[InitializeSubDomain] sub_domains: (optional) List of sub + domains to be added within a domain. + :param ContainerIdentity sub_container: (optional) The identity schema for + a IBM knowledge catalog container (catalog/project/space). + """ + self.container = container + self.trace = trace + self.errors = errors + self.name = name + self.description = description + self.id = id + self.created_by = created_by + self.member_roles = member_roles + self.properties = properties + self.sub_domains = sub_domains + self.sub_container = sub_container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductDomain': + """Initialize a DataProductDomain object from a json dictionary.""" + args = {} + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DataProductDomain JSON') + if (trace := _dict.get('trace')) is not None: + args['trace'] = trace + if (errors := _dict.get('errors')) is not None: + args['errors'] = [ErrorModelResource.from_dict(v) for v in errors] + if (name := _dict.get('name')) is not None: + args['name'] = name + if (description := _dict.get('description')) is not None: + args['description'] = description + if (id := _dict.get('id')) is not None: + args['id'] = id + if (created_by := _dict.get('created_by')) is not None: + args['created_by'] = created_by + if (member_roles := _dict.get('member_roles')) is not None: + args['member_roles'] = MemberRolesSchema.from_dict(member_roles) + if (properties := _dict.get('properties')) is not None: + args['properties'] = PropertiesSchema.from_dict(properties) + if (sub_domains := _dict.get('sub_domains')) is not None: + args['sub_domains'] = [InitializeSubDomain.from_dict(v) for v in sub_domains] + if (sub_container := _dict.get('sub_container')) is not None: + args['sub_container'] = ContainerIdentity.from_dict(sub_container) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductDomain object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + if hasattr(self, 'trace') and self.trace is not None: + _dict['trace'] = self.trace + if hasattr(self, 'errors') and self.errors is not None: + errors_list = [] + for v in self.errors: + if isinstance(v, dict): + errors_list.append(v) + else: + errors_list.append(v.to_dict()) + _dict['errors'] = errors_list + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'created_by') and self.created_by is not None: + _dict['created_by'] = self.created_by + if hasattr(self, 'member_roles') and self.member_roles is not None: + if isinstance(self.member_roles, dict): + _dict['member_roles'] = self.member_roles + else: + _dict['member_roles'] = self.member_roles.to_dict() + if hasattr(self, 'properties') and self.properties is not None: + if isinstance(self.properties, dict): + _dict['properties'] = self.properties + else: + _dict['properties'] = self.properties.to_dict() + if hasattr(self, 'sub_domains') and self.sub_domains is not None: + sub_domains_list = [] + for v in self.sub_domains: + if isinstance(v, dict): + sub_domains_list.append(v) + else: + sub_domains_list.append(v.to_dict()) + _dict['sub_domains'] = sub_domains_list + if hasattr(self, 'sub_container') and self.sub_container is not None: + if isinstance(self.sub_container, dict): + _dict['sub_container'] = self.sub_container + else: + _dict['sub_container'] = self.sub_container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductDomain object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductDomain') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductDomain') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductDomainCollection: + """ + A collection of data product domains. + + :param List[DataProductDomain] domains: Collection of data product domains. + """ + + def __init__( + self, + domains: List['DataProductDomain'], + ) -> None: + """ + Initialize a DataProductDomainCollection object. + + :param List[DataProductDomain] domains: Collection of data product domains. + """ + self.domains = domains + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductDomainCollection': + """Initialize a DataProductDomainCollection object from a json dictionary.""" + args = {} + if (domains := _dict.get('domains')) is not None: + args['domains'] = [DataProductDomain.from_dict(v) for v in domains] + else: + raise ValueError('Required property \'domains\' not present in DataProductDomainCollection JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductDomainCollection object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'domains') and self.domains is not None: + domains_list = [] + for v in self.domains: + if isinstance(v, dict): + domains_list.append(v) + else: + domains_list.append(v.to_dict()) + _dict['domains'] = domains_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductDomainCollection object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductDomainCollection') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductDomainCollection') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductDraft: + """ + Data Product version draft. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductDraftDataProduct data_product: Data product reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable and + reusable. + :param str description: The description of the data product version. + :param List[str] tags: Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated with + the data product version. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param Domain domain: Domain that the data product version belongs to. If this + is the first version of a data product, this field is required. If this is a new + version of an existing data product, the domain will default to the domain of + the previous version of the data product. + :param List[DataProductPart] parts_out: The outgoing parts of this data product + version to be delivered to consumers. If this is the first version of a data + product, this field defaults to an empty list. If this is a new version of an + existing data product, the data product parts will default to the parts list + from the previous version of the data product. + :param DataProductWorkflows workflows: (optional) The workflows associated with + the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided either + at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for a IBM + knowledge catalog container (catalog/project/space). + :param bool is_restricted: Indicates whether the data product is restricted or + not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a container. + :param str published_by: (optional) The user who published this data product + version. + :param datetime published_at: (optional) The time when this data product version + was published. + :param str created_by: The creator of this data product version. + :param datetime created_at: The time when this data product version was created. + :param dict properties: (optional) Metadata properties on data products. + :param List[DataAssetRelationship] visualization_errors: (optional) Errors + encountered during the visualization creation process. + """ + + def __init__( + self, + version: str, + state: str, + data_product: 'DataProductDraftDataProduct', + name: str, + description: str, + tags: List[str], + types: List[str], + contract_terms: List['ContractTerms'], + domain: 'Domain', + parts_out: List['DataProductPart'], + is_restricted: bool, + id: str, + asset: 'AssetReference', + created_by: str, + created_at: datetime, + *, + use_cases: Optional[List['UseCase']] = None, + workflows: Optional['DataProductWorkflows'] = None, + dataview_enabled: Optional[bool] = None, + comments: Optional[str] = None, + access_control: Optional['AssetListAccessControl'] = None, + last_updated_at: Optional[datetime] = None, + sub_container: Optional['ContainerIdentity'] = None, + published_by: Optional[str] = None, + published_at: Optional[datetime] = None, + properties: Optional[dict] = None, + visualization_errors: Optional[List['DataAssetRelationship']] = None, + ) -> None: + """ + Initialize a DataProductDraft object. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductDraftDataProduct data_product: Data product reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable + and reusable. + :param str description: The description of the data product version. + :param List[str] tags: Tags on the data product. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param Domain domain: Domain that the data product version belongs to. If + this is the first version of a data product, this field is required. If + this is a new version of an existing data product, the domain will default + to the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: The outgoing parts of this data + product version to be delivered to consumers. If this is the first version + of a data product, this field defaults to an empty list. If this is a new + version of an existing data product, the data product parts will default to + the parts list from the previous version of the data product. + :param bool is_restricted: Indicates whether the data product is restricted + or not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a + container. + :param str created_by: The creator of this data product version. + :param datetime created_at: The time when this data product version was + created. + :param List[UseCase] use_cases: (optional) A list of use cases associated + with the data product version. + :param DataProductWorkflows workflows: (optional) The workflows associated + with the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided + either at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control + object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for + a IBM knowledge catalog container (catalog/project/space). + :param str published_by: (optional) The user who published this data + product version. + :param datetime published_at: (optional) The time when this data product + version was published. + :param dict properties: (optional) Metadata properties on data products. + :param List[DataAssetRelationship] visualization_errors: (optional) Errors + encountered during the visualization creation process. + """ + self.version = version + self.state = state + self.data_product = data_product + self.name = name + self.description = description + self.tags = tags + self.use_cases = use_cases + self.types = types + self.contract_terms = contract_terms + self.domain = domain + self.parts_out = parts_out + self.workflows = workflows + self.dataview_enabled = dataview_enabled + self.comments = comments + self.access_control = access_control + self.last_updated_at = last_updated_at + self.sub_container = sub_container + self.is_restricted = is_restricted + self.id = id + self.asset = asset + self.published_by = published_by + self.published_at = published_at + self.created_by = created_by + self.created_at = created_at + self.properties = properties + self.visualization_errors = visualization_errors + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductDraft': + """Initialize a DataProductDraft object from a json dictionary.""" + args = {} + if (version := _dict.get('version')) is not None: + args['version'] = version + else: + raise ValueError('Required property \'version\' not present in DataProductDraft JSON') + if (state := _dict.get('state')) is not None: + args['state'] = state + else: + raise ValueError('Required property \'state\' not present in DataProductDraft JSON') + if (data_product := _dict.get('data_product')) is not None: + args['data_product'] = DataProductDraftDataProduct.from_dict(data_product) + else: + raise ValueError('Required property \'data_product\' not present in DataProductDraft JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + else: + raise ValueError('Required property \'name\' not present in DataProductDraft JSON') + if (description := _dict.get('description')) is not None: + args['description'] = description + else: + raise ValueError('Required property \'description\' not present in DataProductDraft JSON') + if (tags := _dict.get('tags')) is not None: + args['tags'] = tags + else: + raise ValueError('Required property \'tags\' not present in DataProductDraft JSON') + if (use_cases := _dict.get('use_cases')) is not None: + args['use_cases'] = [UseCase.from_dict(v) for v in use_cases] + if (types := _dict.get('types')) is not None: + args['types'] = types + else: + raise ValueError('Required property \'types\' not present in DataProductDraft JSON') + if (contract_terms := _dict.get('contract_terms')) is not None: + args['contract_terms'] = [ContractTerms.from_dict(v) for v in contract_terms] + else: + raise ValueError('Required property \'contract_terms\' not present in DataProductDraft JSON') + if (domain := _dict.get('domain')) is not None: + args['domain'] = Domain.from_dict(domain) + else: + raise ValueError('Required property \'domain\' not present in DataProductDraft JSON') + if (parts_out := _dict.get('parts_out')) is not None: + args['parts_out'] = [DataProductPart.from_dict(v) for v in parts_out] + else: + raise ValueError('Required property \'parts_out\' not present in DataProductDraft JSON') + if (workflows := _dict.get('workflows')) is not None: + args['workflows'] = DataProductWorkflows.from_dict(workflows) + if (dataview_enabled := _dict.get('dataview_enabled')) is not None: + args['dataview_enabled'] = dataview_enabled + if (comments := _dict.get('comments')) is not None: + args['comments'] = comments + if (access_control := _dict.get('access_control')) is not None: + args['access_control'] = AssetListAccessControl.from_dict(access_control) + if (last_updated_at := _dict.get('last_updated_at')) is not None: + args['last_updated_at'] = string_to_datetime(last_updated_at) + if (sub_container := _dict.get('sub_container')) is not None: + args['sub_container'] = ContainerIdentity.from_dict(sub_container) + if (is_restricted := _dict.get('is_restricted')) is not None: + args['is_restricted'] = is_restricted + else: + raise ValueError('Required property \'is_restricted\' not present in DataProductDraft JSON') + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductDraft JSON') + if (asset := _dict.get('asset')) is not None: + args['asset'] = AssetReference.from_dict(asset) + else: + raise ValueError('Required property \'asset\' not present in DataProductDraft JSON') + if (published_by := _dict.get('published_by')) is not None: + args['published_by'] = published_by + if (published_at := _dict.get('published_at')) is not None: + args['published_at'] = string_to_datetime(published_at) + if (created_by := _dict.get('created_by')) is not None: + args['created_by'] = created_by + else: + raise ValueError('Required property \'created_by\' not present in DataProductDraft JSON') + if (created_at := _dict.get('created_at')) is not None: + args['created_at'] = string_to_datetime(created_at) + else: + raise ValueError('Required property \'created_at\' not present in DataProductDraft JSON') + if (properties := _dict.get('properties')) is not None: + args['properties'] = properties + if (visualization_errors := _dict.get('visualization_errors')) is not None: + args['visualization_errors'] = [DataAssetRelationship.from_dict(v) for v in visualization_errors] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductDraft object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'version') and self.version is not None: + _dict['version'] = self.version + if hasattr(self, 'state') and self.state is not None: + _dict['state'] = self.state + if hasattr(self, 'data_product') and self.data_product is not None: + if isinstance(self.data_product, dict): + _dict['data_product'] = self.data_product + else: + _dict['data_product'] = self.data_product.to_dict() + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'tags') and self.tags is not None: + _dict['tags'] = self.tags + if hasattr(self, 'use_cases') and self.use_cases is not None: + use_cases_list = [] + for v in self.use_cases: + if isinstance(v, dict): + use_cases_list.append(v) + else: + use_cases_list.append(v.to_dict()) + _dict['use_cases'] = use_cases_list + if hasattr(self, 'types') and self.types is not None: + _dict['types'] = self.types + if hasattr(self, 'contract_terms') and self.contract_terms is not None: + contract_terms_list = [] + for v in self.contract_terms: + if isinstance(v, dict): + contract_terms_list.append(v) + else: + contract_terms_list.append(v.to_dict()) + _dict['contract_terms'] = contract_terms_list + if hasattr(self, 'domain') and self.domain is not None: + if isinstance(self.domain, dict): + _dict['domain'] = self.domain + else: + _dict['domain'] = self.domain.to_dict() + if hasattr(self, 'parts_out') and self.parts_out is not None: + parts_out_list = [] + for v in self.parts_out: + if isinstance(v, dict): + parts_out_list.append(v) + else: + parts_out_list.append(v.to_dict()) + _dict['parts_out'] = parts_out_list + if hasattr(self, 'workflows') and self.workflows is not None: + if isinstance(self.workflows, dict): + _dict['workflows'] = self.workflows + else: + _dict['workflows'] = self.workflows.to_dict() + if hasattr(self, 'dataview_enabled') and self.dataview_enabled is not None: + _dict['dataview_enabled'] = self.dataview_enabled + if hasattr(self, 'comments') and self.comments is not None: + _dict['comments'] = self.comments + if hasattr(self, 'access_control') and self.access_control is not None: + if isinstance(self.access_control, dict): + _dict['access_control'] = self.access_control + else: + _dict['access_control'] = self.access_control.to_dict() + if hasattr(self, 'last_updated_at') and self.last_updated_at is not None: + _dict['last_updated_at'] = datetime_to_string(self.last_updated_at) + if hasattr(self, 'sub_container') and self.sub_container is not None: + if isinstance(self.sub_container, dict): + _dict['sub_container'] = self.sub_container + else: + _dict['sub_container'] = self.sub_container.to_dict() + if hasattr(self, 'is_restricted') and self.is_restricted is not None: + _dict['is_restricted'] = self.is_restricted + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + if hasattr(self, 'published_by') and self.published_by is not None: + _dict['published_by'] = self.published_by + if hasattr(self, 'published_at') and self.published_at is not None: + _dict['published_at'] = datetime_to_string(self.published_at) + if hasattr(self, 'created_by') and self.created_by is not None: + _dict['created_by'] = self.created_by + if hasattr(self, 'created_at') and self.created_at is not None: + _dict['created_at'] = datetime_to_string(self.created_at) + if hasattr(self, 'properties') and self.properties is not None: + _dict['properties'] = self.properties + if hasattr(self, 'visualization_errors') and self.visualization_errors is not None: + visualization_errors_list = [] + for v in self.visualization_errors: + if isinstance(v, dict): + visualization_errors_list.append(v) + else: + visualization_errors_list.append(v.to_dict()) + _dict['visualization_errors'] = visualization_errors_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductDraft object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductDraft') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductDraft') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class StateEnum(str, Enum): + """ + The state of the data product version. + """ + + DRAFT = 'draft' + AVAILABLE = 'available' + RETIRED = 'retired' + + + class TypesEnum(str, Enum): + """ + types. + """ + + DATA = 'data' + CODE = 'code' + + + +class DataProductDraftCollection: + """ + A collection of data product draft summaries. + + :param int limit: Set a limit on the number of results returned. + :param FirstPage first: First page in the collection. + :param NextPage next: (optional) Next page in the collection. + :param int total_results: (optional) Indicates the total number of results + returned. + :param List[DataProductDraftSummary] drafts: Collection of data product drafts. + """ + + def __init__( + self, + limit: int, + first: 'FirstPage', + drafts: List['DataProductDraftSummary'], + *, + next: Optional['NextPage'] = None, + total_results: Optional[int] = None, + ) -> None: + """ + Initialize a DataProductDraftCollection object. + + :param int limit: Set a limit on the number of results returned. + :param FirstPage first: First page in the collection. + :param List[DataProductDraftSummary] drafts: Collection of data product + drafts. + :param NextPage next: (optional) Next page in the collection. + :param int total_results: (optional) Indicates the total number of results + returned. + """ + self.limit = limit + self.first = first + self.next = next + self.total_results = total_results + self.drafts = drafts + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductDraftCollection': + """Initialize a DataProductDraftCollection object from a json dictionary.""" + args = {} + if (limit := _dict.get('limit')) is not None: + args['limit'] = limit + else: + raise ValueError('Required property \'limit\' not present in DataProductDraftCollection JSON') + if (first := _dict.get('first')) is not None: + args['first'] = FirstPage.from_dict(first) + else: + raise ValueError('Required property \'first\' not present in DataProductDraftCollection JSON') + if (next := _dict.get('next')) is not None: + args['next'] = NextPage.from_dict(next) + if (total_results := _dict.get('total_results')) is not None: + args['total_results'] = total_results + if (drafts := _dict.get('drafts')) is not None: + args['drafts'] = [DataProductDraftSummary.from_dict(v) for v in drafts] + else: + raise ValueError('Required property \'drafts\' not present in DataProductDraftCollection JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductDraftCollection object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'limit') and self.limit is not None: + _dict['limit'] = self.limit + if hasattr(self, 'first') and self.first is not None: + if isinstance(self.first, dict): + _dict['first'] = self.first + else: + _dict['first'] = self.first.to_dict() + if hasattr(self, 'next') and self.next is not None: + if isinstance(self.next, dict): + _dict['next'] = self.next + else: + _dict['next'] = self.next.to_dict() + if hasattr(self, 'total_results') and self.total_results is not None: + _dict['total_results'] = self.total_results + if hasattr(self, 'drafts') and self.drafts is not None: + drafts_list = [] + for v in self.drafts: + if isinstance(v, dict): + drafts_list.append(v) + else: + drafts_list.append(v.to_dict()) + _dict['drafts'] = drafts_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductDraftCollection object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductDraftCollection') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductDraftCollection') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductDraftDataProduct: + """ + Data product reference. + + :param str id: Data product identifier. + :param DataProductDraftVersionRelease release: (optional) A data product draft + version object. + :param ContainerReference container: Container reference. + """ + + def __init__( + self, + id: str, + container: 'ContainerReference', + *, + release: Optional['DataProductDraftVersionRelease'] = None, + ) -> None: + """ + Initialize a DataProductDraftDataProduct object. + + :param str id: Data product identifier. + :param ContainerReference container: Container reference. + :param DataProductDraftVersionRelease release: (optional) A data product + draft version object. + """ + self.id = id + self.release = release + self.container = container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductDraftDataProduct': + """Initialize a DataProductDraftDataProduct object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductDraftDataProduct JSON') + if (release := _dict.get('release')) is not None: + args['release'] = DataProductDraftVersionRelease.from_dict(release) + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DataProductDraftDataProduct JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductDraftDataProduct object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'release') and self.release is not None: + if isinstance(self.release, dict): + _dict['release'] = self.release + else: + _dict['release'] = self.release.to_dict() + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductDraftDataProduct object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductDraftDataProduct') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductDraftDataProduct') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductDraftPrototype: + """ + New data product version input properties. + + :param str version: (optional) The data product version number. + :param str state: (optional) The state of the data product version. If not + specified, the data product version will be created in `draft` state. + :param DataProductIdentity data_product: (optional) Data product identifier. + :param str name: (optional) The name that refers to the new data product + version. If this is a new data product, this value must be specified. If this is + a new version of an existing data product, the name will default to the name of + the previous data product version. A name can contain letters, numbers, + understores, dashes, spaces or periods. A name must contain at least one + non-space character. + :param str description: (optional) Description of the data product version. If + this is a new version of an existing data product, the description will default + to the description of the previous version of the data product. + :param List[str] tags: (optional) Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated with + the data product version. + :param List[str] types: (optional) Types of parts on the data product. + :param List[ContractTerms] contract_terms: (optional) Contract terms binding + various aspects of the data product. + :param Domain domain: (optional) Domain that the data product version belongs + to. If this is the first version of a data product, this field is required. If + this is a new version of an existing data product, the domain will default to + the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: (optional) The outgoing parts of this + data product version to be delivered to consumers. If this is the first version + of a data product, this field defaults to an empty list. If this is a new + version of an existing data product, the data product parts will default to the + parts list from the previous version of the data product. + :param DataProductWorkflows workflows: (optional) The workflows associated with + the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided either + at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for a IBM + knowledge catalog container (catalog/project/space). + :param bool is_restricted: (optional) Indicates whether the data product is + restricted or not. A restricted data product indicates that orders of the data + product requires explicit approval before data is delivered. + :param AssetPrototype asset: New asset input properties. + """ + + def __init__( + self, + asset: 'AssetPrototype', + *, + version: Optional[str] = None, + state: Optional[str] = None, + data_product: Optional['DataProductIdentity'] = None, + name: Optional[str] = None, + description: Optional[str] = None, + tags: Optional[List[str]] = None, + use_cases: Optional[List['UseCase']] = None, + types: Optional[List[str]] = None, + contract_terms: Optional[List['ContractTerms']] = None, + domain: Optional['Domain'] = None, + parts_out: Optional[List['DataProductPart']] = None, + workflows: Optional['DataProductWorkflows'] = None, + dataview_enabled: Optional[bool] = None, + comments: Optional[str] = None, + access_control: Optional['AssetListAccessControl'] = None, + last_updated_at: Optional[datetime] = None, + sub_container: Optional['ContainerIdentity'] = None, + is_restricted: Optional[bool] = None, + ) -> None: + """ + Initialize a DataProductDraftPrototype object. + + :param AssetPrototype asset: New asset input properties. + :param str version: (optional) The data product version number. + :param str state: (optional) The state of the data product version. If not + specified, the data product version will be created in `draft` state. + :param DataProductIdentity data_product: (optional) Data product + identifier. + :param str name: (optional) The name that refers to the new data product + version. If this is a new data product, this value must be specified. If + this is a new version of an existing data product, the name will default to + the name of the previous data product version. A name can contain letters, + numbers, understores, dashes, spaces or periods. A name must contain at + least one non-space character. + :param str description: (optional) Description of the data product version. + If this is a new version of an existing data product, the description will + default to the description of the previous version of the data product. + :param List[str] tags: (optional) Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated + with the data product version. + :param List[str] types: (optional) Types of parts on the data product. + :param List[ContractTerms] contract_terms: (optional) Contract terms + binding various aspects of the data product. + :param Domain domain: (optional) Domain that the data product version + belongs to. If this is the first version of a data product, this field is + required. If this is a new version of an existing data product, the domain + will default to the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: (optional) The outgoing parts of + this data product version to be delivered to consumers. If this is the + first version of a data product, this field defaults to an empty list. If + this is a new version of an existing data product, the data product parts + will default to the parts list from the previous version of the data + product. + :param DataProductWorkflows workflows: (optional) The workflows associated + with the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided + either at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control + object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for + a IBM knowledge catalog container (catalog/project/space). + :param bool is_restricted: (optional) Indicates whether the data product is + restricted or not. A restricted data product indicates that orders of the + data product requires explicit approval before data is delivered. + """ + self.version = version + self.state = state + self.data_product = data_product + self.name = name + self.description = description + self.tags = tags + self.use_cases = use_cases + self.types = types + self.contract_terms = contract_terms + self.domain = domain + self.parts_out = parts_out + self.workflows = workflows + self.dataview_enabled = dataview_enabled + self.comments = comments + self.access_control = access_control + self.last_updated_at = last_updated_at + self.sub_container = sub_container + self.is_restricted = is_restricted + self.asset = asset + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductDraftPrototype': + """Initialize a DataProductDraftPrototype object from a json dictionary.""" + args = {} + if (version := _dict.get('version')) is not None: + args['version'] = version + if (state := _dict.get('state')) is not None: + args['state'] = state + if (data_product := _dict.get('data_product')) is not None: + args['data_product'] = DataProductIdentity.from_dict(data_product) + if (name := _dict.get('name')) is not None: + args['name'] = name + if (description := _dict.get('description')) is not None: + args['description'] = description + if (tags := _dict.get('tags')) is not None: + args['tags'] = tags + if (use_cases := _dict.get('use_cases')) is not None: + args['use_cases'] = [UseCase.from_dict(v) for v in use_cases] + if (types := _dict.get('types')) is not None: + args['types'] = types + if (contract_terms := _dict.get('contract_terms')) is not None: + args['contract_terms'] = [ContractTerms.from_dict(v) for v in contract_terms] + if (domain := _dict.get('domain')) is not None: + args['domain'] = Domain.from_dict(domain) + if (parts_out := _dict.get('parts_out')) is not None: + args['parts_out'] = [DataProductPart.from_dict(v) for v in parts_out] + if (workflows := _dict.get('workflows')) is not None: + args['workflows'] = DataProductWorkflows.from_dict(workflows) + if (dataview_enabled := _dict.get('dataview_enabled')) is not None: + args['dataview_enabled'] = dataview_enabled + if (comments := _dict.get('comments')) is not None: + args['comments'] = comments + if (access_control := _dict.get('access_control')) is not None: + args['access_control'] = AssetListAccessControl.from_dict(access_control) + if (last_updated_at := _dict.get('last_updated_at')) is not None: + args['last_updated_at'] = string_to_datetime(last_updated_at) + if (sub_container := _dict.get('sub_container')) is not None: + args['sub_container'] = ContainerIdentity.from_dict(sub_container) + if (is_restricted := _dict.get('is_restricted')) is not None: + args['is_restricted'] = is_restricted + if (asset := _dict.get('asset')) is not None: + args['asset'] = AssetPrototype.from_dict(asset) + else: + raise ValueError('Required property \'asset\' not present in DataProductDraftPrototype JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductDraftPrototype object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'version') and self.version is not None: + _dict['version'] = self.version + if hasattr(self, 'state') and self.state is not None: + _dict['state'] = self.state + if hasattr(self, 'data_product') and self.data_product is not None: + if isinstance(self.data_product, dict): + _dict['data_product'] = self.data_product + else: + _dict['data_product'] = self.data_product.to_dict() + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'tags') and self.tags is not None: + _dict['tags'] = self.tags + if hasattr(self, 'use_cases') and self.use_cases is not None: + use_cases_list = [] + for v in self.use_cases: + if isinstance(v, dict): + use_cases_list.append(v) + else: + use_cases_list.append(v.to_dict()) + _dict['use_cases'] = use_cases_list + if hasattr(self, 'types') and self.types is not None: + _dict['types'] = self.types + if hasattr(self, 'contract_terms') and self.contract_terms is not None: + contract_terms_list = [] + for v in self.contract_terms: + if isinstance(v, dict): + contract_terms_list.append(v) + else: + contract_terms_list.append(v.to_dict()) + _dict['contract_terms'] = contract_terms_list + if hasattr(self, 'domain') and self.domain is not None: + if isinstance(self.domain, dict): + _dict['domain'] = self.domain + else: + _dict['domain'] = self.domain.to_dict() + if hasattr(self, 'parts_out') and self.parts_out is not None: + parts_out_list = [] + for v in self.parts_out: + if isinstance(v, dict): + parts_out_list.append(v) + else: + parts_out_list.append(v.to_dict()) + _dict['parts_out'] = parts_out_list + if hasattr(self, 'workflows') and self.workflows is not None: + if isinstance(self.workflows, dict): + _dict['workflows'] = self.workflows + else: + _dict['workflows'] = self.workflows.to_dict() + if hasattr(self, 'dataview_enabled') and self.dataview_enabled is not None: + _dict['dataview_enabled'] = self.dataview_enabled + if hasattr(self, 'comments') and self.comments is not None: + _dict['comments'] = self.comments + if hasattr(self, 'access_control') and self.access_control is not None: + if isinstance(self.access_control, dict): + _dict['access_control'] = self.access_control + else: + _dict['access_control'] = self.access_control.to_dict() + if hasattr(self, 'last_updated_at') and self.last_updated_at is not None: + _dict['last_updated_at'] = datetime_to_string(self.last_updated_at) + if hasattr(self, 'sub_container') and self.sub_container is not None: + if isinstance(self.sub_container, dict): + _dict['sub_container'] = self.sub_container + else: + _dict['sub_container'] = self.sub_container.to_dict() + if hasattr(self, 'is_restricted') and self.is_restricted is not None: + _dict['is_restricted'] = self.is_restricted + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductDraftPrototype object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductDraftPrototype') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductDraftPrototype') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class StateEnum(str, Enum): + """ + The state of the data product version. If not specified, the data product version + will be created in `draft` state. + """ + + DRAFT = 'draft' + AVAILABLE = 'available' + RETIRED = 'retired' + + + class TypesEnum(str, Enum): + """ + types. + """ + + DATA = 'data' + CODE = 'code' + + + +class DataProductDraftSummary: + """ + Summary of Data Product Version object. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductDraftSummaryDataProduct data_product: Data product reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable and + reusable. + :param str description: The description of the data product version. + :param List[str] tags: Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated with + the data product version. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param Domain domain: Domain that the data product version belongs to. If this + is the first version of a data product, this field is required. If this is a new + version of an existing data product, the domain will default to the domain of + the previous version of the data product. + :param List[DataProductPart] parts_out: The outgoing parts of this data product + version to be delivered to consumers. If this is the first version of a data + product, this field defaults to an empty list. If this is a new version of an + existing data product, the data product parts will default to the parts list + from the previous version of the data product. + :param DataProductWorkflows workflows: (optional) The workflows associated with + the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided either + at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for a IBM + knowledge catalog container (catalog/project/space). + :param bool is_restricted: Indicates whether the data product is restricted or + not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a container. + """ + + def __init__( + self, + version: str, + state: str, + data_product: 'DataProductDraftSummaryDataProduct', + name: str, + description: str, + tags: List[str], + types: List[str], + contract_terms: List['ContractTerms'], + domain: 'Domain', + parts_out: List['DataProductPart'], + is_restricted: bool, + id: str, + asset: 'AssetReference', + *, + use_cases: Optional[List['UseCase']] = None, + workflows: Optional['DataProductWorkflows'] = None, + dataview_enabled: Optional[bool] = None, + comments: Optional[str] = None, + access_control: Optional['AssetListAccessControl'] = None, + last_updated_at: Optional[datetime] = None, + sub_container: Optional['ContainerIdentity'] = None, + ) -> None: + """ + Initialize a DataProductDraftSummary object. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductDraftSummaryDataProduct data_product: Data product + reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable + and reusable. + :param str description: The description of the data product version. + :param List[str] tags: Tags on the data product. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param Domain domain: Domain that the data product version belongs to. If + this is the first version of a data product, this field is required. If + this is a new version of an existing data product, the domain will default + to the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: The outgoing parts of this data + product version to be delivered to consumers. If this is the first version + of a data product, this field defaults to an empty list. If this is a new + version of an existing data product, the data product parts will default to + the parts list from the previous version of the data product. + :param bool is_restricted: Indicates whether the data product is restricted + or not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a + container. + :param List[UseCase] use_cases: (optional) A list of use cases associated + with the data product version. + :param DataProductWorkflows workflows: (optional) The workflows associated + with the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided + either at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control + object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for + a IBM knowledge catalog container (catalog/project/space). + """ + self.version = version + self.state = state + self.data_product = data_product + self.name = name + self.description = description + self.tags = tags + self.use_cases = use_cases + self.types = types + self.contract_terms = contract_terms + self.domain = domain + self.parts_out = parts_out + self.workflows = workflows + self.dataview_enabled = dataview_enabled + self.comments = comments + self.access_control = access_control + self.last_updated_at = last_updated_at + self.sub_container = sub_container + self.is_restricted = is_restricted + self.id = id + self.asset = asset + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductDraftSummary': + """Initialize a DataProductDraftSummary object from a json dictionary.""" + args = {} + if (version := _dict.get('version')) is not None: + args['version'] = version + else: + raise ValueError('Required property \'version\' not present in DataProductDraftSummary JSON') + if (state := _dict.get('state')) is not None: + args['state'] = state + else: + raise ValueError('Required property \'state\' not present in DataProductDraftSummary JSON') + if (data_product := _dict.get('data_product')) is not None: + args['data_product'] = DataProductDraftSummaryDataProduct.from_dict(data_product) + else: + raise ValueError('Required property \'data_product\' not present in DataProductDraftSummary JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + else: + raise ValueError('Required property \'name\' not present in DataProductDraftSummary JSON') + if (description := _dict.get('description')) is not None: + args['description'] = description + else: + raise ValueError('Required property \'description\' not present in DataProductDraftSummary JSON') + if (tags := _dict.get('tags')) is not None: + args['tags'] = tags + else: + raise ValueError('Required property \'tags\' not present in DataProductDraftSummary JSON') + if (use_cases := _dict.get('use_cases')) is not None: + args['use_cases'] = [UseCase.from_dict(v) for v in use_cases] + if (types := _dict.get('types')) is not None: + args['types'] = types + else: + raise ValueError('Required property \'types\' not present in DataProductDraftSummary JSON') + if (contract_terms := _dict.get('contract_terms')) is not None: + args['contract_terms'] = [ContractTerms.from_dict(v) for v in contract_terms] + else: + raise ValueError('Required property \'contract_terms\' not present in DataProductDraftSummary JSON') + if (domain := _dict.get('domain')) is not None: + args['domain'] = Domain.from_dict(domain) + else: + raise ValueError('Required property \'domain\' not present in DataProductDraftSummary JSON') + if (parts_out := _dict.get('parts_out')) is not None: + args['parts_out'] = [DataProductPart.from_dict(v) for v in parts_out] + else: + raise ValueError('Required property \'parts_out\' not present in DataProductDraftSummary JSON') + if (workflows := _dict.get('workflows')) is not None: + args['workflows'] = DataProductWorkflows.from_dict(workflows) + if (dataview_enabled := _dict.get('dataview_enabled')) is not None: + args['dataview_enabled'] = dataview_enabled + if (comments := _dict.get('comments')) is not None: + args['comments'] = comments + if (access_control := _dict.get('access_control')) is not None: + args['access_control'] = AssetListAccessControl.from_dict(access_control) + if (last_updated_at := _dict.get('last_updated_at')) is not None: + args['last_updated_at'] = string_to_datetime(last_updated_at) + if (sub_container := _dict.get('sub_container')) is not None: + args['sub_container'] = ContainerIdentity.from_dict(sub_container) + if (is_restricted := _dict.get('is_restricted')) is not None: + args['is_restricted'] = is_restricted + else: + raise ValueError('Required property \'is_restricted\' not present in DataProductDraftSummary JSON') + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductDraftSummary JSON') + if (asset := _dict.get('asset')) is not None: + args['asset'] = AssetReference.from_dict(asset) + else: + raise ValueError('Required property \'asset\' not present in DataProductDraftSummary JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductDraftSummary object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'version') and self.version is not None: + _dict['version'] = self.version + if hasattr(self, 'state') and self.state is not None: + _dict['state'] = self.state + if hasattr(self, 'data_product') and self.data_product is not None: + if isinstance(self.data_product, dict): + _dict['data_product'] = self.data_product + else: + _dict['data_product'] = self.data_product.to_dict() + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'tags') and self.tags is not None: + _dict['tags'] = self.tags + if hasattr(self, 'use_cases') and self.use_cases is not None: + use_cases_list = [] + for v in self.use_cases: + if isinstance(v, dict): + use_cases_list.append(v) + else: + use_cases_list.append(v.to_dict()) + _dict['use_cases'] = use_cases_list + if hasattr(self, 'types') and self.types is not None: + _dict['types'] = self.types + if hasattr(self, 'contract_terms') and self.contract_terms is not None: + contract_terms_list = [] + for v in self.contract_terms: + if isinstance(v, dict): + contract_terms_list.append(v) + else: + contract_terms_list.append(v.to_dict()) + _dict['contract_terms'] = contract_terms_list + if hasattr(self, 'domain') and self.domain is not None: + if isinstance(self.domain, dict): + _dict['domain'] = self.domain + else: + _dict['domain'] = self.domain.to_dict() + if hasattr(self, 'parts_out') and self.parts_out is not None: + parts_out_list = [] + for v in self.parts_out: + if isinstance(v, dict): + parts_out_list.append(v) + else: + parts_out_list.append(v.to_dict()) + _dict['parts_out'] = parts_out_list + if hasattr(self, 'workflows') and self.workflows is not None: + if isinstance(self.workflows, dict): + _dict['workflows'] = self.workflows + else: + _dict['workflows'] = self.workflows.to_dict() + if hasattr(self, 'dataview_enabled') and self.dataview_enabled is not None: + _dict['dataview_enabled'] = self.dataview_enabled + if hasattr(self, 'comments') and self.comments is not None: + _dict['comments'] = self.comments + if hasattr(self, 'access_control') and self.access_control is not None: + if isinstance(self.access_control, dict): + _dict['access_control'] = self.access_control + else: + _dict['access_control'] = self.access_control.to_dict() + if hasattr(self, 'last_updated_at') and self.last_updated_at is not None: + _dict['last_updated_at'] = datetime_to_string(self.last_updated_at) + if hasattr(self, 'sub_container') and self.sub_container is not None: + if isinstance(self.sub_container, dict): + _dict['sub_container'] = self.sub_container + else: + _dict['sub_container'] = self.sub_container.to_dict() + if hasattr(self, 'is_restricted') and self.is_restricted is not None: + _dict['is_restricted'] = self.is_restricted + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductDraftSummary object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductDraftSummary') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductDraftSummary') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class StateEnum(str, Enum): + """ + The state of the data product version. + """ + + DRAFT = 'draft' + AVAILABLE = 'available' + RETIRED = 'retired' + + + class TypesEnum(str, Enum): + """ + types. + """ + + DATA = 'data' + CODE = 'code' + + + +class DataProductDraftSummaryDataProduct: + """ + Data product reference. + + :param str id: Data product identifier. + :param DataProductDraftVersionRelease release: (optional) A data product draft + version object. + :param ContainerReference container: Container reference. + """ + + def __init__( + self, + id: str, + container: 'ContainerReference', + *, + release: Optional['DataProductDraftVersionRelease'] = None, + ) -> None: + """ + Initialize a DataProductDraftSummaryDataProduct object. + + :param str id: Data product identifier. + :param ContainerReference container: Container reference. + :param DataProductDraftVersionRelease release: (optional) A data product + draft version object. + """ + self.id = id + self.release = release + self.container = container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductDraftSummaryDataProduct': + """Initialize a DataProductDraftSummaryDataProduct object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductDraftSummaryDataProduct JSON') + if (release := _dict.get('release')) is not None: + args['release'] = DataProductDraftVersionRelease.from_dict(release) + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DataProductDraftSummaryDataProduct JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductDraftSummaryDataProduct object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'release') and self.release is not None: + if isinstance(self.release, dict): + _dict['release'] = self.release + else: + _dict['release'] = self.release.to_dict() + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductDraftSummaryDataProduct object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductDraftSummaryDataProduct') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductDraftSummaryDataProduct') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductDraftVersionRelease: + """ + A data product draft version object. + + :param str id: (optional) ID of a draft version of data product. + """ + + def __init__( + self, + *, + id: Optional[str] = None, + ) -> None: + """ + Initialize a DataProductDraftVersionRelease object. + + :param str id: (optional) ID of a draft version of data product. + """ + self.id = id + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductDraftVersionRelease': + """Initialize a DataProductDraftVersionRelease object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductDraftVersionRelease object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductDraftVersionRelease object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductDraftVersionRelease') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductDraftVersionRelease') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductIdentity: + """ + Data product identifier. + + :param str id: Data product identifier. + :param DataProductDraftVersionRelease release: (optional) A data product draft + version object. + """ + + def __init__( + self, + id: str, + *, + release: Optional['DataProductDraftVersionRelease'] = None, + ) -> None: + """ + Initialize a DataProductIdentity object. + + :param str id: Data product identifier. + :param DataProductDraftVersionRelease release: (optional) A data product + draft version object. + """ + self.id = id + self.release = release + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductIdentity': + """Initialize a DataProductIdentity object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductIdentity JSON') + if (release := _dict.get('release')) is not None: + args['release'] = DataProductDraftVersionRelease.from_dict(release) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductIdentity object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'release') and self.release is not None: + if isinstance(self.release, dict): + _dict['release'] = self.release + else: + _dict['release'] = self.release.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductIdentity object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductIdentity') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductIdentity') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductOrderAccessRequest: + """ + The approval workflows associated with the data product version. + + :param List[str] task_assignee_users: (optional) The workflow approvers + associated with the data product version. + :param List[str] pre_approved_users: (optional) The list of users or groups + whose request will get pre-approved associated with the data product version. + :param DataProductCustomWorkflowDefinition custom_workflow_definition: + (optional) A custom workflow definition to be used to create a workflow to + approve a data product subscription. + """ + + def __init__( + self, + *, + task_assignee_users: Optional[List[str]] = None, + pre_approved_users: Optional[List[str]] = None, + custom_workflow_definition: Optional['DataProductCustomWorkflowDefinition'] = None, + ) -> None: + """ + Initialize a DataProductOrderAccessRequest object. + + :param List[str] task_assignee_users: (optional) The workflow approvers + associated with the data product version. + :param List[str] pre_approved_users: (optional) The list of users or groups + whose request will get pre-approved associated with the data product + version. + :param DataProductCustomWorkflowDefinition custom_workflow_definition: + (optional) A custom workflow definition to be used to create a workflow to + approve a data product subscription. + """ + self.task_assignee_users = task_assignee_users + self.pre_approved_users = pre_approved_users + self.custom_workflow_definition = custom_workflow_definition + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductOrderAccessRequest': + """Initialize a DataProductOrderAccessRequest object from a json dictionary.""" + args = {} + if (task_assignee_users := _dict.get('task_assignee_users')) is not None: + args['task_assignee_users'] = task_assignee_users + if (pre_approved_users := _dict.get('pre_approved_users')) is not None: + args['pre_approved_users'] = pre_approved_users + if (custom_workflow_definition := _dict.get('custom_workflow_definition')) is not None: + args['custom_workflow_definition'] = DataProductCustomWorkflowDefinition.from_dict(custom_workflow_definition) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductOrderAccessRequest object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'task_assignee_users') and self.task_assignee_users is not None: + _dict['task_assignee_users'] = self.task_assignee_users + if hasattr(self, 'pre_approved_users') and self.pre_approved_users is not None: + _dict['pre_approved_users'] = self.pre_approved_users + if hasattr(self, 'custom_workflow_definition') and self.custom_workflow_definition is not None: + if isinstance(self.custom_workflow_definition, dict): + _dict['custom_workflow_definition'] = self.custom_workflow_definition + else: + _dict['custom_workflow_definition'] = self.custom_workflow_definition.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductOrderAccessRequest object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductOrderAccessRequest') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductOrderAccessRequest') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductPart: + """ + Data Product Part. + + :param AssetPartReference asset: The asset represented in this part. + :param List[DeliveryMethod] delivery_methods: (optional) Delivery methods + describing the delivery options available for this part. + """ + + def __init__( + self, + asset: 'AssetPartReference', + *, + delivery_methods: Optional[List['DeliveryMethod']] = None, + ) -> None: + """ + Initialize a DataProductPart object. + + :param AssetPartReference asset: The asset represented in this part. + :param List[DeliveryMethod] delivery_methods: (optional) Delivery methods + describing the delivery options available for this part. + """ + self.asset = asset + self.delivery_methods = delivery_methods + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductPart': + """Initialize a DataProductPart object from a json dictionary.""" + args = {} + if (asset := _dict.get('asset')) is not None: + args['asset'] = AssetPartReference.from_dict(asset) + else: + raise ValueError('Required property \'asset\' not present in DataProductPart JSON') + if (delivery_methods := _dict.get('delivery_methods')) is not None: + args['delivery_methods'] = [DeliveryMethod.from_dict(v) for v in delivery_methods] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductPart object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + if hasattr(self, 'delivery_methods') and self.delivery_methods is not None: + delivery_methods_list = [] + for v in self.delivery_methods: + if isinstance(v, dict): + delivery_methods_list.append(v) + else: + delivery_methods_list.append(v.to_dict()) + _dict['delivery_methods'] = delivery_methods_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductPart object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductPart') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductPart') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductRelease: + """ + Data Product version release. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductReleaseDataProduct data_product: Data product reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable and + reusable. + :param str description: The description of the data product version. + :param List[str] tags: Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated with + the data product version. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param Domain domain: Domain that the data product version belongs to. If this + is the first version of a data product, this field is required. If this is a new + version of an existing data product, the domain will default to the domain of + the previous version of the data product. + :param List[DataProductPart] parts_out: The outgoing parts of this data product + version to be delivered to consumers. If this is the first version of a data + product, this field defaults to an empty list. If this is a new version of an + existing data product, the data product parts will default to the parts list + from the previous version of the data product. + :param DataProductWorkflows workflows: (optional) The workflows associated with + the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided either + at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for a IBM + knowledge catalog container (catalog/project/space). + :param bool is_restricted: Indicates whether the data product is restricted or + not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a container. + :param str published_by: (optional) The user who published this data product + version. + :param datetime published_at: (optional) The time when this data product version + was published. + :param str created_by: The creator of this data product version. + :param datetime created_at: The time when this data product version was created. + :param dict properties: (optional) Metadata properties on data products. + :param List[DataAssetRelationship] visualization_errors: (optional) Errors + encountered during the visualization creation process. + """ + + def __init__( + self, + version: str, + state: str, + data_product: 'DataProductReleaseDataProduct', + name: str, + description: str, + tags: List[str], + types: List[str], + contract_terms: List['ContractTerms'], + domain: 'Domain', + parts_out: List['DataProductPart'], + is_restricted: bool, + id: str, + asset: 'AssetReference', + created_by: str, + created_at: datetime, + *, + use_cases: Optional[List['UseCase']] = None, + workflows: Optional['DataProductWorkflows'] = None, + dataview_enabled: Optional[bool] = None, + comments: Optional[str] = None, + access_control: Optional['AssetListAccessControl'] = None, + last_updated_at: Optional[datetime] = None, + sub_container: Optional['ContainerIdentity'] = None, + published_by: Optional[str] = None, + published_at: Optional[datetime] = None, + properties: Optional[dict] = None, + visualization_errors: Optional[List['DataAssetRelationship']] = None, + ) -> None: + """ + Initialize a DataProductRelease object. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductReleaseDataProduct data_product: Data product reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable + and reusable. + :param str description: The description of the data product version. + :param List[str] tags: Tags on the data product. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param Domain domain: Domain that the data product version belongs to. If + this is the first version of a data product, this field is required. If + this is a new version of an existing data product, the domain will default + to the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: The outgoing parts of this data + product version to be delivered to consumers. If this is the first version + of a data product, this field defaults to an empty list. If this is a new + version of an existing data product, the data product parts will default to + the parts list from the previous version of the data product. + :param bool is_restricted: Indicates whether the data product is restricted + or not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a + container. + :param str created_by: The creator of this data product version. + :param datetime created_at: The time when this data product version was + created. + :param List[UseCase] use_cases: (optional) A list of use cases associated + with the data product version. + :param DataProductWorkflows workflows: (optional) The workflows associated + with the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided + either at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control + object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for + a IBM knowledge catalog container (catalog/project/space). + :param str published_by: (optional) The user who published this data + product version. + :param datetime published_at: (optional) The time when this data product + version was published. + :param dict properties: (optional) Metadata properties on data products. + :param List[DataAssetRelationship] visualization_errors: (optional) Errors + encountered during the visualization creation process. + """ + self.version = version + self.state = state + self.data_product = data_product + self.name = name + self.description = description + self.tags = tags + self.use_cases = use_cases + self.types = types + self.contract_terms = contract_terms + self.domain = domain + self.parts_out = parts_out + self.workflows = workflows + self.dataview_enabled = dataview_enabled + self.comments = comments + self.access_control = access_control + self.last_updated_at = last_updated_at + self.sub_container = sub_container + self.is_restricted = is_restricted + self.id = id + self.asset = asset + self.published_by = published_by + self.published_at = published_at + self.created_by = created_by + self.created_at = created_at + self.properties = properties + self.visualization_errors = visualization_errors + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductRelease': + """Initialize a DataProductRelease object from a json dictionary.""" + args = {} + if (version := _dict.get('version')) is not None: + args['version'] = version + else: + raise ValueError('Required property \'version\' not present in DataProductRelease JSON') + if (state := _dict.get('state')) is not None: + args['state'] = state + else: + raise ValueError('Required property \'state\' not present in DataProductRelease JSON') + if (data_product := _dict.get('data_product')) is not None: + args['data_product'] = DataProductReleaseDataProduct.from_dict(data_product) + else: + raise ValueError('Required property \'data_product\' not present in DataProductRelease JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + else: + raise ValueError('Required property \'name\' not present in DataProductRelease JSON') + if (description := _dict.get('description')) is not None: + args['description'] = description + else: + raise ValueError('Required property \'description\' not present in DataProductRelease JSON') + if (tags := _dict.get('tags')) is not None: + args['tags'] = tags + else: + raise ValueError('Required property \'tags\' not present in DataProductRelease JSON') + if (use_cases := _dict.get('use_cases')) is not None: + args['use_cases'] = [UseCase.from_dict(v) for v in use_cases] + if (types := _dict.get('types')) is not None: + args['types'] = types + else: + raise ValueError('Required property \'types\' not present in DataProductRelease JSON') + if (contract_terms := _dict.get('contract_terms')) is not None: + args['contract_terms'] = [ContractTerms.from_dict(v) for v in contract_terms] + else: + raise ValueError('Required property \'contract_terms\' not present in DataProductRelease JSON') + if (domain := _dict.get('domain')) is not None: + args['domain'] = Domain.from_dict(domain) + else: + raise ValueError('Required property \'domain\' not present in DataProductRelease JSON') + if (parts_out := _dict.get('parts_out')) is not None: + args['parts_out'] = [DataProductPart.from_dict(v) for v in parts_out] + else: + raise ValueError('Required property \'parts_out\' not present in DataProductRelease JSON') + if (workflows := _dict.get('workflows')) is not None: + args['workflows'] = DataProductWorkflows.from_dict(workflows) + if (dataview_enabled := _dict.get('dataview_enabled')) is not None: + args['dataview_enabled'] = dataview_enabled + if (comments := _dict.get('comments')) is not None: + args['comments'] = comments + if (access_control := _dict.get('access_control')) is not None: + args['access_control'] = AssetListAccessControl.from_dict(access_control) + if (last_updated_at := _dict.get('last_updated_at')) is not None: + args['last_updated_at'] = string_to_datetime(last_updated_at) + if (sub_container := _dict.get('sub_container')) is not None: + args['sub_container'] = ContainerIdentity.from_dict(sub_container) + if (is_restricted := _dict.get('is_restricted')) is not None: + args['is_restricted'] = is_restricted + else: + raise ValueError('Required property \'is_restricted\' not present in DataProductRelease JSON') + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductRelease JSON') + if (asset := _dict.get('asset')) is not None: + args['asset'] = AssetReference.from_dict(asset) + else: + raise ValueError('Required property \'asset\' not present in DataProductRelease JSON') + if (published_by := _dict.get('published_by')) is not None: + args['published_by'] = published_by + if (published_at := _dict.get('published_at')) is not None: + args['published_at'] = string_to_datetime(published_at) + if (created_by := _dict.get('created_by')) is not None: + args['created_by'] = created_by + else: + raise ValueError('Required property \'created_by\' not present in DataProductRelease JSON') + if (created_at := _dict.get('created_at')) is not None: + args['created_at'] = string_to_datetime(created_at) + else: + raise ValueError('Required property \'created_at\' not present in DataProductRelease JSON') + if (properties := _dict.get('properties')) is not None: + args['properties'] = properties + if (visualization_errors := _dict.get('visualization_errors')) is not None: + args['visualization_errors'] = [DataAssetRelationship.from_dict(v) for v in visualization_errors] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductRelease object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'version') and self.version is not None: + _dict['version'] = self.version + if hasattr(self, 'state') and self.state is not None: + _dict['state'] = self.state + if hasattr(self, 'data_product') and self.data_product is not None: + if isinstance(self.data_product, dict): + _dict['data_product'] = self.data_product + else: + _dict['data_product'] = self.data_product.to_dict() + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'tags') and self.tags is not None: + _dict['tags'] = self.tags + if hasattr(self, 'use_cases') and self.use_cases is not None: + use_cases_list = [] + for v in self.use_cases: + if isinstance(v, dict): + use_cases_list.append(v) + else: + use_cases_list.append(v.to_dict()) + _dict['use_cases'] = use_cases_list + if hasattr(self, 'types') and self.types is not None: + _dict['types'] = self.types + if hasattr(self, 'contract_terms') and self.contract_terms is not None: + contract_terms_list = [] + for v in self.contract_terms: + if isinstance(v, dict): + contract_terms_list.append(v) + else: + contract_terms_list.append(v.to_dict()) + _dict['contract_terms'] = contract_terms_list + if hasattr(self, 'domain') and self.domain is not None: + if isinstance(self.domain, dict): + _dict['domain'] = self.domain + else: + _dict['domain'] = self.domain.to_dict() + if hasattr(self, 'parts_out') and self.parts_out is not None: + parts_out_list = [] + for v in self.parts_out: + if isinstance(v, dict): + parts_out_list.append(v) + else: + parts_out_list.append(v.to_dict()) + _dict['parts_out'] = parts_out_list + if hasattr(self, 'workflows') and self.workflows is not None: + if isinstance(self.workflows, dict): + _dict['workflows'] = self.workflows + else: + _dict['workflows'] = self.workflows.to_dict() + if hasattr(self, 'dataview_enabled') and self.dataview_enabled is not None: + _dict['dataview_enabled'] = self.dataview_enabled + if hasattr(self, 'comments') and self.comments is not None: + _dict['comments'] = self.comments + if hasattr(self, 'access_control') and self.access_control is not None: + if isinstance(self.access_control, dict): + _dict['access_control'] = self.access_control + else: + _dict['access_control'] = self.access_control.to_dict() + if hasattr(self, 'last_updated_at') and self.last_updated_at is not None: + _dict['last_updated_at'] = datetime_to_string(self.last_updated_at) + if hasattr(self, 'sub_container') and self.sub_container is not None: + if isinstance(self.sub_container, dict): + _dict['sub_container'] = self.sub_container + else: + _dict['sub_container'] = self.sub_container.to_dict() + if hasattr(self, 'is_restricted') and self.is_restricted is not None: + _dict['is_restricted'] = self.is_restricted + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + if hasattr(self, 'published_by') and self.published_by is not None: + _dict['published_by'] = self.published_by + if hasattr(self, 'published_at') and self.published_at is not None: + _dict['published_at'] = datetime_to_string(self.published_at) + if hasattr(self, 'created_by') and self.created_by is not None: + _dict['created_by'] = self.created_by + if hasattr(self, 'created_at') and self.created_at is not None: + _dict['created_at'] = datetime_to_string(self.created_at) + if hasattr(self, 'properties') and self.properties is not None: + _dict['properties'] = self.properties + if hasattr(self, 'visualization_errors') and self.visualization_errors is not None: + visualization_errors_list = [] + for v in self.visualization_errors: + if isinstance(v, dict): + visualization_errors_list.append(v) + else: + visualization_errors_list.append(v.to_dict()) + _dict['visualization_errors'] = visualization_errors_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductRelease object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductRelease') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductRelease') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class StateEnum(str, Enum): + """ + The state of the data product version. + """ + + DRAFT = 'draft' + AVAILABLE = 'available' + RETIRED = 'retired' + + + class TypesEnum(str, Enum): + """ + types. + """ + + DATA = 'data' + CODE = 'code' + + + +class DataProductReleaseCollection: + """ + A collection of data product release summaries. + + :param int limit: Set a limit on the number of results returned. + :param FirstPage first: First page in the collection. + :param NextPage next: (optional) Next page in the collection. + :param int total_results: (optional) Indicates the total number of results + returned. + :param List[DataProductReleaseSummary] releases: Collection of data product + releases. + """ + + def __init__( + self, + limit: int, + first: 'FirstPage', + releases: List['DataProductReleaseSummary'], + *, + next: Optional['NextPage'] = None, + total_results: Optional[int] = None, + ) -> None: + """ + Initialize a DataProductReleaseCollection object. + + :param int limit: Set a limit on the number of results returned. + :param FirstPage first: First page in the collection. + :param List[DataProductReleaseSummary] releases: Collection of data product + releases. + :param NextPage next: (optional) Next page in the collection. + :param int total_results: (optional) Indicates the total number of results + returned. + """ + self.limit = limit + self.first = first + self.next = next + self.total_results = total_results + self.releases = releases + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductReleaseCollection': + """Initialize a DataProductReleaseCollection object from a json dictionary.""" + args = {} + if (limit := _dict.get('limit')) is not None: + args['limit'] = limit + else: + raise ValueError('Required property \'limit\' not present in DataProductReleaseCollection JSON') + if (first := _dict.get('first')) is not None: + args['first'] = FirstPage.from_dict(first) + else: + raise ValueError('Required property \'first\' not present in DataProductReleaseCollection JSON') + if (next := _dict.get('next')) is not None: + args['next'] = NextPage.from_dict(next) + if (total_results := _dict.get('total_results')) is not None: + args['total_results'] = total_results + if (releases := _dict.get('releases')) is not None: + args['releases'] = [DataProductReleaseSummary.from_dict(v) for v in releases] + else: + raise ValueError('Required property \'releases\' not present in DataProductReleaseCollection JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductReleaseCollection object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'limit') and self.limit is not None: + _dict['limit'] = self.limit + if hasattr(self, 'first') and self.first is not None: + if isinstance(self.first, dict): + _dict['first'] = self.first + else: + _dict['first'] = self.first.to_dict() + if hasattr(self, 'next') and self.next is not None: + if isinstance(self.next, dict): + _dict['next'] = self.next + else: + _dict['next'] = self.next.to_dict() + if hasattr(self, 'total_results') and self.total_results is not None: + _dict['total_results'] = self.total_results + if hasattr(self, 'releases') and self.releases is not None: + releases_list = [] + for v in self.releases: + if isinstance(v, dict): + releases_list.append(v) + else: + releases_list.append(v.to_dict()) + _dict['releases'] = releases_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductReleaseCollection object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductReleaseCollection') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductReleaseCollection') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductReleaseDataProduct: + """ + Data product reference. + + :param str id: Data product identifier. + :param DataProductDraftVersionRelease release: (optional) A data product draft + version object. + :param ContainerReference container: Container reference. + """ + + def __init__( + self, + id: str, + container: 'ContainerReference', + *, + release: Optional['DataProductDraftVersionRelease'] = None, + ) -> None: + """ + Initialize a DataProductReleaseDataProduct object. + + :param str id: Data product identifier. + :param ContainerReference container: Container reference. + :param DataProductDraftVersionRelease release: (optional) A data product + draft version object. + """ + self.id = id + self.release = release + self.container = container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductReleaseDataProduct': + """Initialize a DataProductReleaseDataProduct object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductReleaseDataProduct JSON') + if (release := _dict.get('release')) is not None: + args['release'] = DataProductDraftVersionRelease.from_dict(release) + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DataProductReleaseDataProduct JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductReleaseDataProduct object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'release') and self.release is not None: + if isinstance(self.release, dict): + _dict['release'] = self.release + else: + _dict['release'] = self.release.to_dict() + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductReleaseDataProduct object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductReleaseDataProduct') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductReleaseDataProduct') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductReleaseSummary: + """ + Summary of Data Product Version object. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductReleaseSummaryDataProduct data_product: Data product + reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable and + reusable. + :param str description: The description of the data product version. + :param List[str] tags: (optional) Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated with + the data product version. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param Domain domain: (optional) Domain that the data product version belongs + to. If this is the first version of a data product, this field is required. If + this is a new version of an existing data product, the domain will default to + the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: (optional) The outgoing parts of this + data product version to be delivered to consumers. If this is the first version + of a data product, this field defaults to an empty list. If this is a new + version of an existing data product, the data product parts will default to the + parts list from the previous version of the data product. + :param DataProductWorkflows workflows: (optional) The workflows associated with + the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided either + at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for a IBM + knowledge catalog container (catalog/project/space). + :param bool is_restricted: Indicates whether the data product is restricted or + not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a container. + """ + + def __init__( + self, + version: str, + state: str, + data_product: 'DataProductReleaseSummaryDataProduct', + name: str, + description: str, + types: List[str], + contract_terms: List['ContractTerms'], + is_restricted: bool, + id: str, + asset: 'AssetReference', + *, + tags: Optional[List[str]] = None, + use_cases: Optional[List['UseCase']] = None, + domain: Optional['Domain'] = None, + parts_out: Optional[List['DataProductPart']] = None, + workflows: Optional['DataProductWorkflows'] = None, + dataview_enabled: Optional[bool] = None, + comments: Optional[str] = None, + access_control: Optional['AssetListAccessControl'] = None, + last_updated_at: Optional[datetime] = None, + sub_container: Optional['ContainerIdentity'] = None, + ) -> None: + """ + Initialize a DataProductReleaseSummary object. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductReleaseSummaryDataProduct data_product: Data product + reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable + and reusable. + :param str description: The description of the data product version. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param bool is_restricted: Indicates whether the data product is restricted + or not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a + container. + :param List[str] tags: (optional) Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated + with the data product version. + :param Domain domain: (optional) Domain that the data product version + belongs to. If this is the first version of a data product, this field is + required. If this is a new version of an existing data product, the domain + will default to the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: (optional) The outgoing parts of + this data product version to be delivered to consumers. If this is the + first version of a data product, this field defaults to an empty list. If + this is a new version of an existing data product, the data product parts + will default to the parts list from the previous version of the data + product. + :param DataProductWorkflows workflows: (optional) The workflows associated + with the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided + either at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control + object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for + a IBM knowledge catalog container (catalog/project/space). + """ + self.version = version + self.state = state + self.data_product = data_product + self.name = name + self.description = description + self.tags = tags + self.use_cases = use_cases + self.types = types + self.contract_terms = contract_terms + self.domain = domain + self.parts_out = parts_out + self.workflows = workflows + self.dataview_enabled = dataview_enabled + self.comments = comments + self.access_control = access_control + self.last_updated_at = last_updated_at + self.sub_container = sub_container + self.is_restricted = is_restricted + self.id = id + self.asset = asset + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductReleaseSummary': + """Initialize a DataProductReleaseSummary object from a json dictionary.""" + args = {} + if (version := _dict.get('version')) is not None: + args['version'] = version + else: + raise ValueError('Required property \'version\' not present in DataProductReleaseSummary JSON') + if (state := _dict.get('state')) is not None: + args['state'] = state + else: + raise ValueError('Required property \'state\' not present in DataProductReleaseSummary JSON') + if (data_product := _dict.get('data_product')) is not None: + args['data_product'] = DataProductReleaseSummaryDataProduct.from_dict(data_product) + else: + raise ValueError('Required property \'data_product\' not present in DataProductReleaseSummary JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + else: + raise ValueError('Required property \'name\' not present in DataProductReleaseSummary JSON') + if (description := _dict.get('description')) is not None: + args['description'] = description + else: + raise ValueError('Required property \'description\' not present in DataProductReleaseSummary JSON') + if (tags := _dict.get('tags')) is not None: + args['tags'] = tags + if (use_cases := _dict.get('use_cases')) is not None: + args['use_cases'] = [UseCase.from_dict(v) for v in use_cases] + if (types := _dict.get('types')) is not None: + args['types'] = types + else: + raise ValueError('Required property \'types\' not present in DataProductReleaseSummary JSON') + if (contract_terms := _dict.get('contract_terms')) is not None: + args['contract_terms'] = [ContractTerms.from_dict(v) for v in contract_terms] + else: + raise ValueError('Required property \'contract_terms\' not present in DataProductReleaseSummary JSON') + if (domain := _dict.get('domain')) is not None: + args['domain'] = Domain.from_dict(domain) + if (parts_out := _dict.get('parts_out')) is not None: + args['parts_out'] = [DataProductPart.from_dict(v) for v in parts_out] + if (workflows := _dict.get('workflows')) is not None: + args['workflows'] = DataProductWorkflows.from_dict(workflows) + if (dataview_enabled := _dict.get('dataview_enabled')) is not None: + args['dataview_enabled'] = dataview_enabled + if (comments := _dict.get('comments')) is not None: + args['comments'] = comments + if (access_control := _dict.get('access_control')) is not None: + args['access_control'] = AssetListAccessControl.from_dict(access_control) + if (last_updated_at := _dict.get('last_updated_at')) is not None: + args['last_updated_at'] = string_to_datetime(last_updated_at) + if (sub_container := _dict.get('sub_container')) is not None: + args['sub_container'] = ContainerIdentity.from_dict(sub_container) + if (is_restricted := _dict.get('is_restricted')) is not None: + args['is_restricted'] = is_restricted + else: + raise ValueError('Required property \'is_restricted\' not present in DataProductReleaseSummary JSON') + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductReleaseSummary JSON') + if (asset := _dict.get('asset')) is not None: + args['asset'] = AssetReference.from_dict(asset) + else: + raise ValueError('Required property \'asset\' not present in DataProductReleaseSummary JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductReleaseSummary object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'version') and self.version is not None: + _dict['version'] = self.version + if hasattr(self, 'state') and self.state is not None: + _dict['state'] = self.state + if hasattr(self, 'data_product') and self.data_product is not None: + if isinstance(self.data_product, dict): + _dict['data_product'] = self.data_product + else: + _dict['data_product'] = self.data_product.to_dict() + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'tags') and self.tags is not None: + _dict['tags'] = self.tags + if hasattr(self, 'use_cases') and self.use_cases is not None: + use_cases_list = [] + for v in self.use_cases: + if isinstance(v, dict): + use_cases_list.append(v) + else: + use_cases_list.append(v.to_dict()) + _dict['use_cases'] = use_cases_list + if hasattr(self, 'types') and self.types is not None: + _dict['types'] = self.types + if hasattr(self, 'contract_terms') and self.contract_terms is not None: + contract_terms_list = [] + for v in self.contract_terms: + if isinstance(v, dict): + contract_terms_list.append(v) + else: + contract_terms_list.append(v.to_dict()) + _dict['contract_terms'] = contract_terms_list + if hasattr(self, 'domain') and self.domain is not None: + if isinstance(self.domain, dict): + _dict['domain'] = self.domain + else: + _dict['domain'] = self.domain.to_dict() + if hasattr(self, 'parts_out') and self.parts_out is not None: + parts_out_list = [] + for v in self.parts_out: + if isinstance(v, dict): + parts_out_list.append(v) + else: + parts_out_list.append(v.to_dict()) + _dict['parts_out'] = parts_out_list + if hasattr(self, 'workflows') and self.workflows is not None: + if isinstance(self.workflows, dict): + _dict['workflows'] = self.workflows + else: + _dict['workflows'] = self.workflows.to_dict() + if hasattr(self, 'dataview_enabled') and self.dataview_enabled is not None: + _dict['dataview_enabled'] = self.dataview_enabled + if hasattr(self, 'comments') and self.comments is not None: + _dict['comments'] = self.comments + if hasattr(self, 'access_control') and self.access_control is not None: + if isinstance(self.access_control, dict): + _dict['access_control'] = self.access_control + else: + _dict['access_control'] = self.access_control.to_dict() + if hasattr(self, 'last_updated_at') and self.last_updated_at is not None: + _dict['last_updated_at'] = datetime_to_string(self.last_updated_at) + if hasattr(self, 'sub_container') and self.sub_container is not None: + if isinstance(self.sub_container, dict): + _dict['sub_container'] = self.sub_container + else: + _dict['sub_container'] = self.sub_container.to_dict() + if hasattr(self, 'is_restricted') and self.is_restricted is not None: + _dict['is_restricted'] = self.is_restricted + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductReleaseSummary object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductReleaseSummary') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductReleaseSummary') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class StateEnum(str, Enum): + """ + The state of the data product version. + """ + + DRAFT = 'draft' + AVAILABLE = 'available' + RETIRED = 'retired' + + + class TypesEnum(str, Enum): + """ + types. + """ + + DATA = 'data' + CODE = 'code' + + + +class DataProductReleaseSummaryDataProduct: + """ + Data product reference. + + :param str id: Data product identifier. + :param DataProductDraftVersionRelease release: (optional) A data product draft + version object. + :param ContainerReference container: Container reference. + """ + + def __init__( + self, + id: str, + container: 'ContainerReference', + *, + release: Optional['DataProductDraftVersionRelease'] = None, + ) -> None: + """ + Initialize a DataProductReleaseSummaryDataProduct object. + + :param str id: Data product identifier. + :param ContainerReference container: Container reference. + :param DataProductDraftVersionRelease release: (optional) A data product + draft version object. + """ + self.id = id + self.release = release + self.container = container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductReleaseSummaryDataProduct': + """Initialize a DataProductReleaseSummaryDataProduct object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductReleaseSummaryDataProduct JSON') + if (release := _dict.get('release')) is not None: + args['release'] = DataProductDraftVersionRelease.from_dict(release) + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DataProductReleaseSummaryDataProduct JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductReleaseSummaryDataProduct object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'release') and self.release is not None: + if isinstance(self.release, dict): + _dict['release'] = self.release + else: + _dict['release'] = self.release.to_dict() + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductReleaseSummaryDataProduct object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductReleaseSummaryDataProduct') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductReleaseSummaryDataProduct') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductSummary: + """ + Data Product Summary. + + :param str id: Data product identifier. + :param DataProductDraftVersionRelease release: (optional) A data product draft + version object. + :param ContainerReference container: Container reference. + :param str name: (optional) Data product name. + """ + + def __init__( + self, + id: str, + container: 'ContainerReference', + *, + release: Optional['DataProductDraftVersionRelease'] = None, + name: Optional[str] = None, + ) -> None: + """ + Initialize a DataProductSummary object. + + :param str id: Data product identifier. + :param ContainerReference container: Container reference. + :param DataProductDraftVersionRelease release: (optional) A data product + draft version object. + :param str name: (optional) Data product name. + """ + self.id = id + self.release = release + self.container = container + self.name = name + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductSummary': + """Initialize a DataProductSummary object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductSummary JSON') + if (release := _dict.get('release')) is not None: + args['release'] = DataProductDraftVersionRelease.from_dict(release) + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DataProductSummary JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductSummary object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'release') and self.release is not None: + if isinstance(self.release, dict): + _dict['release'] = self.release + else: + _dict['release'] = self.release.to_dict() + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductSummary object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductSummary') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductSummary') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductVersionCollection: + """ + A collection of data product version summaries. + + :param int limit: Set a limit on the number of results returned. + :param FirstPage first: First page in the collection. + :param NextPage next: (optional) Next page in the collection. + :param int total_results: (optional) Indicates the total number of results + returned. + :param List[DataProductVersionSummary] data_product_versions: Collection of data + product versions. + """ + + def __init__( + self, + limit: int, + first: 'FirstPage', + data_product_versions: List['DataProductVersionSummary'], + *, + next: Optional['NextPage'] = None, + total_results: Optional[int] = None, + ) -> None: + """ + Initialize a DataProductVersionCollection object. + + :param int limit: Set a limit on the number of results returned. + :param FirstPage first: First page in the collection. + :param List[DataProductVersionSummary] data_product_versions: Collection of + data product versions. + :param NextPage next: (optional) Next page in the collection. + :param int total_results: (optional) Indicates the total number of results + returned. + """ + self.limit = limit + self.first = first + self.next = next + self.total_results = total_results + self.data_product_versions = data_product_versions + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductVersionCollection': + """Initialize a DataProductVersionCollection object from a json dictionary.""" + args = {} + if (limit := _dict.get('limit')) is not None: + args['limit'] = limit + else: + raise ValueError('Required property \'limit\' not present in DataProductVersionCollection JSON') + if (first := _dict.get('first')) is not None: + args['first'] = FirstPage.from_dict(first) + else: + raise ValueError('Required property \'first\' not present in DataProductVersionCollection JSON') + if (next := _dict.get('next')) is not None: + args['next'] = NextPage.from_dict(next) + if (total_results := _dict.get('total_results')) is not None: + args['total_results'] = total_results + if (data_product_versions := _dict.get('data_product_versions')) is not None: + args['data_product_versions'] = [DataProductVersionSummary.from_dict(v) for v in data_product_versions] + else: + raise ValueError('Required property \'data_product_versions\' not present in DataProductVersionCollection JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductVersionCollection object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'limit') and self.limit is not None: + _dict['limit'] = self.limit + if hasattr(self, 'first') and self.first is not None: + if isinstance(self.first, dict): + _dict['first'] = self.first + else: + _dict['first'] = self.first.to_dict() + if hasattr(self, 'next') and self.next is not None: + if isinstance(self.next, dict): + _dict['next'] = self.next + else: + _dict['next'] = self.next.to_dict() + if hasattr(self, 'total_results') and self.total_results is not None: + _dict['total_results'] = self.total_results + if hasattr(self, 'data_product_versions') and self.data_product_versions is not None: + data_product_versions_list = [] + for v in self.data_product_versions: + if isinstance(v, dict): + data_product_versions_list.append(v) + else: + data_product_versions_list.append(v.to_dict()) + _dict['data_product_versions'] = data_product_versions_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductVersionCollection object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductVersionCollection') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductVersionCollection') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductVersionSummary: + """ + Summary of Data Product Version object. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductVersionSummaryDataProduct data_product: Data product + reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable and + reusable. + :param str description: The description of the data product version. + :param List[str] tags: (optional) Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated with + the data product version. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param Domain domain: (optional) Domain that the data product version belongs + to. If this is the first version of a data product, this field is required. If + this is a new version of an existing data product, the domain will default to + the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: (optional) The outgoing parts of this + data product version to be delivered to consumers. If this is the first version + of a data product, this field defaults to an empty list. If this is a new + version of an existing data product, the data product parts will default to the + parts list from the previous version of the data product. + :param DataProductWorkflows workflows: (optional) The workflows associated with + the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided either + at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for a IBM + knowledge catalog container (catalog/project/space). + :param bool is_restricted: Indicates whether the data product is restricted or + not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a container. + """ + + def __init__( + self, + version: str, + state: str, + data_product: 'DataProductVersionSummaryDataProduct', + name: str, + description: str, + types: List[str], + contract_terms: List['ContractTerms'], + is_restricted: bool, + id: str, + asset: 'AssetReference', + *, + tags: Optional[List[str]] = None, + use_cases: Optional[List['UseCase']] = None, + domain: Optional['Domain'] = None, + parts_out: Optional[List['DataProductPart']] = None, + workflows: Optional['DataProductWorkflows'] = None, + dataview_enabled: Optional[bool] = None, + comments: Optional[str] = None, + access_control: Optional['AssetListAccessControl'] = None, + last_updated_at: Optional[datetime] = None, + sub_container: Optional['ContainerIdentity'] = None, + ) -> None: + """ + Initialize a DataProductVersionSummary object. + + :param str version: The data product version number. + :param str state: The state of the data product version. + :param DataProductVersionSummaryDataProduct data_product: Data product + reference. + :param str name: The name of the data product version. A name can contain + letters, numbers, understores, dashes, spaces or periods. Names are mutable + and reusable. + :param str description: The description of the data product version. + :param List[str] types: Types of parts on the data product. + :param List[ContractTerms] contract_terms: Contract terms binding various + aspects of the data product. + :param bool is_restricted: Indicates whether the data product is restricted + or not. A restricted data product indicates that orders of the data product + requires explicit approval before data is delivered. + :param str id: The identifier of the data product version. + :param AssetReference asset: The reference schema for a asset in a + container. + :param List[str] tags: (optional) Tags on the data product. + :param List[UseCase] use_cases: (optional) A list of use cases associated + with the data product version. + :param Domain domain: (optional) Domain that the data product version + belongs to. If this is the first version of a data product, this field is + required. If this is a new version of an existing data product, the domain + will default to the domain of the previous version of the data product. + :param List[DataProductPart] parts_out: (optional) The outgoing parts of + this data product version to be delivered to consumers. If this is the + first version of a data product, this field defaults to an empty list. If + this is a new version of an existing data product, the data product parts + will default to the parts list from the previous version of the data + product. + :param DataProductWorkflows workflows: (optional) The workflows associated + with the data product version. + :param bool dataview_enabled: (optional) Indicates whether the dataView has + enabled for data product. + :param str comments: (optional) Comments by a producer that are provided + either at the time of data product version creation or retiring. + :param AssetListAccessControl access_control: (optional) Access control + object. + :param datetime last_updated_at: (optional) Timestamp of last asset update. + :param ContainerIdentity sub_container: (optional) The identity schema for + a IBM knowledge catalog container (catalog/project/space). + """ + self.version = version + self.state = state + self.data_product = data_product + self.name = name + self.description = description + self.tags = tags + self.use_cases = use_cases + self.types = types + self.contract_terms = contract_terms + self.domain = domain + self.parts_out = parts_out + self.workflows = workflows + self.dataview_enabled = dataview_enabled + self.comments = comments + self.access_control = access_control + self.last_updated_at = last_updated_at + self.sub_container = sub_container + self.is_restricted = is_restricted + self.id = id + self.asset = asset + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductVersionSummary': + """Initialize a DataProductVersionSummary object from a json dictionary.""" + args = {} + if (version := _dict.get('version')) is not None: + args['version'] = version + else: + raise ValueError('Required property \'version\' not present in DataProductVersionSummary JSON') + if (state := _dict.get('state')) is not None: + args['state'] = state + else: + raise ValueError('Required property \'state\' not present in DataProductVersionSummary JSON') + if (data_product := _dict.get('data_product')) is not None: + args['data_product'] = DataProductVersionSummaryDataProduct.from_dict(data_product) + else: + raise ValueError('Required property \'data_product\' not present in DataProductVersionSummary JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + else: + raise ValueError('Required property \'name\' not present in DataProductVersionSummary JSON') + if (description := _dict.get('description')) is not None: + args['description'] = description + else: + raise ValueError('Required property \'description\' not present in DataProductVersionSummary JSON') + if (tags := _dict.get('tags')) is not None: + args['tags'] = tags + if (use_cases := _dict.get('use_cases')) is not None: + args['use_cases'] = [UseCase.from_dict(v) for v in use_cases] + if (types := _dict.get('types')) is not None: + args['types'] = types + else: + raise ValueError('Required property \'types\' not present in DataProductVersionSummary JSON') + if (contract_terms := _dict.get('contract_terms')) is not None: + args['contract_terms'] = [ContractTerms.from_dict(v) for v in contract_terms] + else: + raise ValueError('Required property \'contract_terms\' not present in DataProductVersionSummary JSON') + if (domain := _dict.get('domain')) is not None: + args['domain'] = Domain.from_dict(domain) + if (parts_out := _dict.get('parts_out')) is not None: + args['parts_out'] = [DataProductPart.from_dict(v) for v in parts_out] + if (workflows := _dict.get('workflows')) is not None: + args['workflows'] = DataProductWorkflows.from_dict(workflows) + if (dataview_enabled := _dict.get('dataview_enabled')) is not None: + args['dataview_enabled'] = dataview_enabled + if (comments := _dict.get('comments')) is not None: + args['comments'] = comments + if (access_control := _dict.get('access_control')) is not None: + args['access_control'] = AssetListAccessControl.from_dict(access_control) + if (last_updated_at := _dict.get('last_updated_at')) is not None: + args['last_updated_at'] = string_to_datetime(last_updated_at) + if (sub_container := _dict.get('sub_container')) is not None: + args['sub_container'] = ContainerIdentity.from_dict(sub_container) + if (is_restricted := _dict.get('is_restricted')) is not None: + args['is_restricted'] = is_restricted + else: + raise ValueError('Required property \'is_restricted\' not present in DataProductVersionSummary JSON') + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductVersionSummary JSON') + if (asset := _dict.get('asset')) is not None: + args['asset'] = AssetReference.from_dict(asset) + else: + raise ValueError('Required property \'asset\' not present in DataProductVersionSummary JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductVersionSummary object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'version') and self.version is not None: + _dict['version'] = self.version + if hasattr(self, 'state') and self.state is not None: + _dict['state'] = self.state + if hasattr(self, 'data_product') and self.data_product is not None: + if isinstance(self.data_product, dict): + _dict['data_product'] = self.data_product + else: + _dict['data_product'] = self.data_product.to_dict() + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + if hasattr(self, 'tags') and self.tags is not None: + _dict['tags'] = self.tags + if hasattr(self, 'use_cases') and self.use_cases is not None: + use_cases_list = [] + for v in self.use_cases: + if isinstance(v, dict): + use_cases_list.append(v) + else: + use_cases_list.append(v.to_dict()) + _dict['use_cases'] = use_cases_list + if hasattr(self, 'types') and self.types is not None: + _dict['types'] = self.types + if hasattr(self, 'contract_terms') and self.contract_terms is not None: + contract_terms_list = [] + for v in self.contract_terms: + if isinstance(v, dict): + contract_terms_list.append(v) + else: + contract_terms_list.append(v.to_dict()) + _dict['contract_terms'] = contract_terms_list + if hasattr(self, 'domain') and self.domain is not None: + if isinstance(self.domain, dict): + _dict['domain'] = self.domain + else: + _dict['domain'] = self.domain.to_dict() + if hasattr(self, 'parts_out') and self.parts_out is not None: + parts_out_list = [] + for v in self.parts_out: + if isinstance(v, dict): + parts_out_list.append(v) + else: + parts_out_list.append(v.to_dict()) + _dict['parts_out'] = parts_out_list + if hasattr(self, 'workflows') and self.workflows is not None: + if isinstance(self.workflows, dict): + _dict['workflows'] = self.workflows + else: + _dict['workflows'] = self.workflows.to_dict() + if hasattr(self, 'dataview_enabled') and self.dataview_enabled is not None: + _dict['dataview_enabled'] = self.dataview_enabled + if hasattr(self, 'comments') and self.comments is not None: + _dict['comments'] = self.comments + if hasattr(self, 'access_control') and self.access_control is not None: + if isinstance(self.access_control, dict): + _dict['access_control'] = self.access_control + else: + _dict['access_control'] = self.access_control.to_dict() + if hasattr(self, 'last_updated_at') and self.last_updated_at is not None: + _dict['last_updated_at'] = datetime_to_string(self.last_updated_at) + if hasattr(self, 'sub_container') and self.sub_container is not None: + if isinstance(self.sub_container, dict): + _dict['sub_container'] = self.sub_container + else: + _dict['sub_container'] = self.sub_container.to_dict() + if hasattr(self, 'is_restricted') and self.is_restricted is not None: + _dict['is_restricted'] = self.is_restricted + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'asset') and self.asset is not None: + if isinstance(self.asset, dict): + _dict['asset'] = self.asset + else: + _dict['asset'] = self.asset.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductVersionSummary object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductVersionSummary') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductVersionSummary') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class StateEnum(str, Enum): + """ + The state of the data product version. + """ + + DRAFT = 'draft' + AVAILABLE = 'available' + RETIRED = 'retired' + + + class TypesEnum(str, Enum): + """ + types. + """ + + DATA = 'data' + CODE = 'code' + + + +class DataProductVersionSummaryDataProduct: + """ + Data product reference. + + :param str id: Data product identifier. + :param DataProductDraftVersionRelease release: (optional) A data product draft + version object. + :param ContainerReference container: Container reference. + """ + + def __init__( + self, + id: str, + container: 'ContainerReference', + *, + release: Optional['DataProductDraftVersionRelease'] = None, + ) -> None: + """ + Initialize a DataProductVersionSummaryDataProduct object. + + :param str id: Data product identifier. + :param ContainerReference container: Container reference. + :param DataProductDraftVersionRelease release: (optional) A data product + draft version object. + """ + self.id = id + self.release = release + self.container = container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductVersionSummaryDataProduct': + """Initialize a DataProductVersionSummaryDataProduct object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DataProductVersionSummaryDataProduct JSON') + if (release := _dict.get('release')) is not None: + args['release'] = DataProductDraftVersionRelease.from_dict(release) + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DataProductVersionSummaryDataProduct JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductVersionSummaryDataProduct object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'release') and self.release is not None: + if isinstance(self.release, dict): + _dict['release'] = self.release + else: + _dict['release'] = self.release.to_dict() + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductVersionSummaryDataProduct object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductVersionSummaryDataProduct') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductVersionSummaryDataProduct') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DataProductWorkflows: + """ + The workflows associated with the data product version. + + :param DataProductOrderAccessRequest order_access_request: (optional) The + approval workflows associated with the data product version. + """ + + def __init__( + self, + *, + order_access_request: Optional['DataProductOrderAccessRequest'] = None, + ) -> None: + """ + Initialize a DataProductWorkflows object. + + :param DataProductOrderAccessRequest order_access_request: (optional) The + approval workflows associated with the data product version. + """ + self.order_access_request = order_access_request + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DataProductWorkflows': + """Initialize a DataProductWorkflows object from a json dictionary.""" + args = {} + if (order_access_request := _dict.get('order_access_request')) is not None: + args['order_access_request'] = DataProductOrderAccessRequest.from_dict(order_access_request) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DataProductWorkflows object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'order_access_request') and self.order_access_request is not None: + if isinstance(self.order_access_request, dict): + _dict['order_access_request'] = self.order_access_request + else: + _dict['order_access_request'] = self.order_access_request.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DataProductWorkflows object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DataProductWorkflows') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DataProductWorkflows') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DeliveryMethod: + """ + DeliveryMethod. + + :param str id: The ID of the delivery method. + :param ContainerReference container: Container reference. + :param DeliveryMethodPropertiesModel getproperties: (optional) The propertiess + of the delivery method. + """ + + def __init__( + self, + id: str, + container: 'ContainerReference', + *, + getproperties: Optional['DeliveryMethodPropertiesModel'] = None, + ) -> None: + """ + Initialize a DeliveryMethod object. + + :param str id: The ID of the delivery method. + :param ContainerReference container: Container reference. + :param DeliveryMethodPropertiesModel getproperties: (optional) The + propertiess of the delivery method. + """ + self.id = id + self.container = container + self.getproperties = getproperties + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DeliveryMethod': + """Initialize a DeliveryMethod object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in DeliveryMethod JSON') + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + else: + raise ValueError('Required property \'container\' not present in DeliveryMethod JSON') + if (getproperties := _dict.get('getproperties')) is not None: + args['getproperties'] = DeliveryMethodPropertiesModel.from_dict(getproperties) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DeliveryMethod object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + if hasattr(self, 'getproperties') and self.getproperties is not None: + if isinstance(self.getproperties, dict): + _dict['getproperties'] = self.getproperties + else: + _dict['getproperties'] = self.getproperties.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DeliveryMethod object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DeliveryMethod') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DeliveryMethod') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class DeliveryMethodPropertiesModel: + """ + The propertiess of the delivery method. + + :param ProducerInputModel producer_input: (optional) Parameters for delivery + that are set by a data product producer. + """ + + def __init__( + self, + *, + producer_input: Optional['ProducerInputModel'] = None, + ) -> None: + """ + Initialize a DeliveryMethodPropertiesModel object. + + :param ProducerInputModel producer_input: (optional) Parameters for + delivery that are set by a data product producer. + """ + self.producer_input = producer_input + + @classmethod + def from_dict(cls, _dict: Dict) -> 'DeliveryMethodPropertiesModel': + """Initialize a DeliveryMethodPropertiesModel object from a json dictionary.""" + args = {} + if (producer_input := _dict.get('producer_input')) is not None: + args['producer_input'] = ProducerInputModel.from_dict(producer_input) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a DeliveryMethodPropertiesModel object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'producer_input') and self.producer_input is not None: + if isinstance(self.producer_input, dict): + _dict['producer_input'] = self.producer_input + else: + _dict['producer_input'] = self.producer_input.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this DeliveryMethodPropertiesModel object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'DeliveryMethodPropertiesModel') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'DeliveryMethodPropertiesModel') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class Description: + """ + Description details of a data contract. + + :param str purpose: (optional) Intended purpose for the provided data. + :param str limitations: (optional) Technical, compliance, and legal limitations + for data use. + :param str usage: (optional) Recommended usage of the data. + :param List[ContractTermsMoreInfo] more_info: (optional) List of links to + sources that provide more details on the dataset. + :param str custom_properties: (optional) Custom properties that are not part of + the standard. + """ + + def __init__( + self, + *, + purpose: Optional[str] = None, + limitations: Optional[str] = None, + usage: Optional[str] = None, + more_info: Optional[List['ContractTermsMoreInfo']] = None, + custom_properties: Optional[str] = None, + ) -> None: + """ + Initialize a Description object. + + :param str purpose: (optional) Intended purpose for the provided data. + :param str limitations: (optional) Technical, compliance, and legal + limitations for data use. + :param str usage: (optional) Recommended usage of the data. + :param List[ContractTermsMoreInfo] more_info: (optional) List of links to + sources that provide more details on the dataset. + :param str custom_properties: (optional) Custom properties that are not + part of the standard. + """ + self.purpose = purpose + self.limitations = limitations + self.usage = usage + self.more_info = more_info + self.custom_properties = custom_properties + + @classmethod + def from_dict(cls, _dict: Dict) -> 'Description': + """Initialize a Description object from a json dictionary.""" + args = {} + if (purpose := _dict.get('purpose')) is not None: + args['purpose'] = purpose + if (limitations := _dict.get('limitations')) is not None: + args['limitations'] = limitations + if (usage := _dict.get('usage')) is not None: + args['usage'] = usage + if (more_info := _dict.get('more_info')) is not None: + args['more_info'] = [ContractTermsMoreInfo.from_dict(v) for v in more_info] + if (custom_properties := _dict.get('custom_properties')) is not None: + args['custom_properties'] = custom_properties + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a Description object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'purpose') and self.purpose is not None: + _dict['purpose'] = self.purpose + if hasattr(self, 'limitations') and self.limitations is not None: + _dict['limitations'] = self.limitations + if hasattr(self, 'usage') and self.usage is not None: + _dict['usage'] = self.usage + if hasattr(self, 'more_info') and self.more_info is not None: + more_info_list = [] + for v in self.more_info: + if isinstance(v, dict): + more_info_list.append(v) + else: + more_info_list.append(v.to_dict()) + _dict['more_info'] = more_info_list + if hasattr(self, 'custom_properties') and self.custom_properties is not None: + _dict['custom_properties'] = self.custom_properties + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this Description object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'Description') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'Description') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class Domain: + """ + Domain that the data product version belongs to. If this is the first version of a + data product, this field is required. If this is a new version of an existing data + product, the domain will default to the domain of the previous version of the data + product. + + :param str id: The ID of the domain. + :param str name: (optional) The display name of the domain. + :param ContainerReference container: (optional) Container reference. + """ + + def __init__( + self, + id: str, + *, + name: Optional[str] = None, + container: Optional['ContainerReference'] = None, + ) -> None: + """ + Initialize a Domain object. + + :param str id: The ID of the domain. + :param str name: (optional) The display name of the domain. + :param ContainerReference container: (optional) Container reference. + """ + self.id = id + self.name = name + self.container = container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'Domain': + """Initialize a Domain object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in Domain JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a Domain object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this Domain object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'Domain') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'Domain') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class EngineDetailsModel: + """ + Engine details as defined by the data product producer. + + :param str display_name: (optional) The name of the engine defined by the data + product producer. + :param str engine_id: (optional) The id of the engine defined by the data + product producer. + :param str engine_port: (optional) The port of the engine defined by the data + product producer. + :param str engine_host: (optional) The host of the engine defined by the data + product producer. + :param str engine_type: The type of the engine (eg: Presto/Spark). + :param List[str] associated_catalogs: (optional) The list of associated + catalogs. + """ + + def __init__( + self, + engine_type: str, + *, + display_name: Optional[str] = None, + engine_id: Optional[str] = None, + engine_port: Optional[str] = None, + engine_host: Optional[str] = None, + associated_catalogs: Optional[List[str]] = None, + ) -> None: + """ + Initialize a EngineDetailsModel object. + + :param str engine_type: The type of the engine (eg: Presto/Spark). + :param str display_name: (optional) The name of the engine defined by the + data product producer. + :param str engine_id: (optional) The id of the engine defined by the data + product producer. + :param str engine_port: (optional) The port of the engine defined by the + data product producer. + :param str engine_host: (optional) The host of the engine defined by the + data product producer. + :param List[str] associated_catalogs: (optional) The list of associated + catalogs. + """ + self.display_name = display_name + self.engine_id = engine_id + self.engine_port = engine_port + self.engine_host = engine_host + self.engine_type = engine_type + self.associated_catalogs = associated_catalogs + + @classmethod + def from_dict(cls, _dict: Dict) -> 'EngineDetailsModel': + """Initialize a EngineDetailsModel object from a json dictionary.""" + args = {} + if (display_name := _dict.get('display_name')) is not None: + args['display_name'] = display_name + if (engine_id := _dict.get('engine_id')) is not None: + args['engine_id'] = engine_id + if (engine_port := _dict.get('engine_port')) is not None: + args['engine_port'] = engine_port + if (engine_host := _dict.get('engine_host')) is not None: + args['engine_host'] = engine_host + if (engine_type := _dict.get('engine_type')) is not None: + args['engine_type'] = engine_type + else: + raise ValueError('Required property \'engine_type\' not present in EngineDetailsModel JSON') + if (associated_catalogs := _dict.get('associated_catalogs')) is not None: + args['associated_catalogs'] = associated_catalogs + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a EngineDetailsModel object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'display_name') and self.display_name is not None: + _dict['display_name'] = self.display_name + if hasattr(self, 'engine_id') and self.engine_id is not None: + _dict['engine_id'] = self.engine_id + if hasattr(self, 'engine_port') and self.engine_port is not None: + _dict['engine_port'] = self.engine_port + if hasattr(self, 'engine_host') and self.engine_host is not None: + _dict['engine_host'] = self.engine_host + if hasattr(self, 'engine_type') and self.engine_type is not None: + _dict['engine_type'] = self.engine_type + if hasattr(self, 'associated_catalogs') and self.associated_catalogs is not None: + _dict['associated_catalogs'] = self.associated_catalogs + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this EngineDetailsModel object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'EngineDetailsModel') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'EngineDetailsModel') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class EngineTypeEnum(str, Enum): + """ + The type of the engine (eg: Presto/Spark). + """ + + SPARK = 'spark' + PRESTO = 'presto' + + + +class ErrorExtraResource: + """ + Detailed error information. + + :param str id: (optional) Error id. + :param datetime timestamp: (optional) Timestamp of the error. + :param str environment_name: (optional) Environment where the error occurred. + :param int http_status: (optional) Http status code. + :param int source_cluster: (optional) Source cluster of the error. + :param int source_component: (optional) Source component of the error. + :param int transaction_id: (optional) Transaction id of the request. + """ + + def __init__( + self, + *, + id: Optional[str] = None, + timestamp: Optional[datetime] = None, + environment_name: Optional[str] = None, + http_status: Optional[int] = None, + source_cluster: Optional[int] = None, + source_component: Optional[int] = None, + transaction_id: Optional[int] = None, + ) -> None: + """ + Initialize a ErrorExtraResource object. + + :param str id: (optional) Error id. + :param datetime timestamp: (optional) Timestamp of the error. + :param str environment_name: (optional) Environment where the error + occurred. + :param int http_status: (optional) Http status code. + :param int source_cluster: (optional) Source cluster of the error. + :param int source_component: (optional) Source component of the error. + :param int transaction_id: (optional) Transaction id of the request. + """ + self.id = id + self.timestamp = timestamp + self.environment_name = environment_name + self.http_status = http_status + self.source_cluster = source_cluster + self.source_component = source_component + self.transaction_id = transaction_id + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ErrorExtraResource': + """Initialize a ErrorExtraResource object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + if (timestamp := _dict.get('timestamp')) is not None: + args['timestamp'] = string_to_datetime(timestamp) + if (environment_name := _dict.get('environment_name')) is not None: + args['environment_name'] = environment_name + if (http_status := _dict.get('http_status')) is not None: + args['http_status'] = http_status + if (source_cluster := _dict.get('source_cluster')) is not None: + args['source_cluster'] = source_cluster + if (source_component := _dict.get('source_component')) is not None: + args['source_component'] = source_component + if (transaction_id := _dict.get('transaction_id')) is not None: + args['transaction_id'] = transaction_id + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ErrorExtraResource object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'timestamp') and self.timestamp is not None: + _dict['timestamp'] = datetime_to_string(self.timestamp) + if hasattr(self, 'environment_name') and self.environment_name is not None: + _dict['environment_name'] = self.environment_name + if hasattr(self, 'http_status') and self.http_status is not None: + _dict['http_status'] = self.http_status + if hasattr(self, 'source_cluster') and self.source_cluster is not None: + _dict['source_cluster'] = self.source_cluster + if hasattr(self, 'source_component') and self.source_component is not None: + _dict['source_component'] = self.source_component + if hasattr(self, 'transaction_id') and self.transaction_id is not None: + _dict['transaction_id'] = self.transaction_id + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ErrorExtraResource object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ErrorExtraResource') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ErrorExtraResource') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ErrorMessage: + """ + Contains the code and details. + + :param str code: The error code. + :param str message: The error details. + """ + + def __init__( + self, + code: str, + message: str, + ) -> None: + """ + Initialize a ErrorMessage object. + + :param str code: The error code. + :param str message: The error details. + """ + self.code = code + self.message = message + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ErrorMessage': + """Initialize a ErrorMessage object from a json dictionary.""" + args = {} + if (code := _dict.get('code')) is not None: + args['code'] = code + else: + raise ValueError('Required property \'code\' not present in ErrorMessage JSON') + if (message := _dict.get('message')) is not None: + args['message'] = message + else: + raise ValueError('Required property \'message\' not present in ErrorMessage JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ErrorMessage object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'code') and self.code is not None: + _dict['code'] = self.code + if hasattr(self, 'message') and self.message is not None: + _dict['message'] = self.message + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ErrorMessage object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ErrorMessage') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ErrorMessage') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ErrorModelResource: + """ + Detailed error information. + + :param str code: Error code. + :param str message: (optional) Error message. + :param ErrorExtraResource extra: (optional) Detailed error information. + :param str more_info: (optional) More info message. + """ + + def __init__( + self, + code: str, + *, + message: Optional[str] = None, + extra: Optional['ErrorExtraResource'] = None, + more_info: Optional[str] = None, + ) -> None: + """ + Initialize a ErrorModelResource object. + + :param str code: Error code. + :param str message: (optional) Error message. + :param ErrorExtraResource extra: (optional) Detailed error information. + :param str more_info: (optional) More info message. + """ + self.code = code + self.message = message + self.extra = extra + self.more_info = more_info + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ErrorModelResource': + """Initialize a ErrorModelResource object from a json dictionary.""" + args = {} + if (code := _dict.get('code')) is not None: + args['code'] = code + else: + raise ValueError('Required property \'code\' not present in ErrorModelResource JSON') + if (message := _dict.get('message')) is not None: + args['message'] = message + if (extra := _dict.get('extra')) is not None: + args['extra'] = ErrorExtraResource.from_dict(extra) + if (more_info := _dict.get('more_info')) is not None: + args['more_info'] = more_info + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ErrorModelResource object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'code') and self.code is not None: + _dict['code'] = self.code + if hasattr(self, 'message') and self.message is not None: + _dict['message'] = self.message + if hasattr(self, 'extra') and self.extra is not None: + if isinstance(self.extra, dict): + _dict['extra'] = self.extra + else: + _dict['extra'] = self.extra.to_dict() + if hasattr(self, 'more_info') and self.more_info is not None: + _dict['more_info'] = self.more_info + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ErrorModelResource object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ErrorModelResource') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ErrorModelResource') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class CodeEnum(str, Enum): + """ + Error code. + """ + + REQUEST_BODY_ERROR = 'request_body_error' + MISSING_REQUIRED_VALUE = 'missing_required_value' + INVALID_PARAMETER = 'invalid_parameter' + DOES_NOT_EXIST = 'does_not_exist' + ALREADY_EXISTS = 'already_exists' + NOT_AUTHENTICATED = 'not_authenticated' + NOT_AUTHORIZED = 'not_authorized' + FORBIDDEN = 'forbidden' + CONFLICT = 'conflict' + CREATE_ERROR = 'create_error' + FETCH_ERROR = 'fetch_error' + UPDATE_ERROR = 'update_error' + DELETE_ERROR = 'delete_error' + PATCH_ERROR = 'patch_error' + DATA_ERROR = 'data_error' + DATABASE_ERROR = 'database_error' + DATABASE_QUERY_ERROR = 'database_query_error' + CONSTRAINT_VIOLATION = 'constraint_violation' + UNABLE_TO_PERFORM = 'unable_to_perform' + TOO_MANY_REQUESTS = 'too_many_requests' + DEPENDENT_SERVICE_ERROR = 'dependent_service_error' + CONFIGURATION_ERROR = 'configuration_error' + UNEXPECTED_EXCEPTION = 'unexpected_exception' + GOVERNANCE_POLICY_DENIAL = 'governance_policy_denial' + DATABASE_USAGE_LIMITS = 'database_usage_limits' + INACTIVE_USER = 'inactive_user' + ENTITLEMENT_ENFORCEMENT = 'entitlement_enforcement' + DELETED = 'deleted' + NOT_IMPLEMENTED = 'not_implemented' + FEATURE_NOT_ENABLED = 'feature_not_enabled' + MISSING_ASSET_DETAILS = 'missing_asset_details' + + + +class FirstPage: + """ + First page in the collection. + + :param str href: Link to the first page in the collection. + """ + + def __init__( + self, + href: str, + ) -> None: + """ + Initialize a FirstPage object. + + :param str href: Link to the first page in the collection. + """ + self.href = href + + @classmethod + def from_dict(cls, _dict: Dict) -> 'FirstPage': + """Initialize a FirstPage object from a json dictionary.""" + args = {} + if (href := _dict.get('href')) is not None: + args['href'] = href + else: + raise ValueError('Required property \'href\' not present in FirstPage JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a FirstPage object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'href') and self.href is not None: + _dict['href'] = self.href + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this FirstPage object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'FirstPage') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'FirstPage') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class InitializeResource: + """ + Resource defining initialization parameters. + + :param ContainerReference container: (optional) Container reference. + :param str href: (optional) Link to monitor the status of the initialize + operation. + :param str status: Status of the initialize operation. + :param str trace: (optional) The id to trace the failed initialization + operation. + :param List[ErrorModelResource] errors: (optional) Set of errors on the latest + initialization request. + :param datetime last_started_at: (optional) Start time of the last + initialization. + :param datetime last_finished_at: (optional) End time of the last + initialization. + :param List[InitializedOption] initialized_options: (optional) Initialized + options. + :param ProvidedCatalogWorkflows workflows: (optional) Resource defining provided + workflow definitions. + """ + + def __init__( + self, + status: str, + *, + container: Optional['ContainerReference'] = None, + href: Optional[str] = None, + trace: Optional[str] = None, + errors: Optional[List['ErrorModelResource']] = None, + last_started_at: Optional[datetime] = None, + last_finished_at: Optional[datetime] = None, + initialized_options: Optional[List['InitializedOption']] = None, + workflows: Optional['ProvidedCatalogWorkflows'] = None, + ) -> None: + """ + Initialize a InitializeResource object. + + :param str status: Status of the initialize operation. + :param ContainerReference container: (optional) Container reference. + :param str href: (optional) Link to monitor the status of the initialize + operation. + :param str trace: (optional) The id to trace the failed initialization + operation. + :param List[ErrorModelResource] errors: (optional) Set of errors on the + latest initialization request. + :param datetime last_started_at: (optional) Start time of the last + initialization. + :param datetime last_finished_at: (optional) End time of the last + initialization. + :param List[InitializedOption] initialized_options: (optional) Initialized + options. + :param ProvidedCatalogWorkflows workflows: (optional) Resource defining + provided workflow definitions. + """ + self.container = container + self.href = href + self.status = status + self.trace = trace + self.errors = errors + self.last_started_at = last_started_at + self.last_finished_at = last_finished_at + self.initialized_options = initialized_options + self.workflows = workflows + + @classmethod + def from_dict(cls, _dict: Dict) -> 'InitializeResource': + """Initialize a InitializeResource object from a json dictionary.""" + args = {} + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + if (href := _dict.get('href')) is not None: + args['href'] = href + if (status := _dict.get('status')) is not None: + args['status'] = status + else: + raise ValueError('Required property \'status\' not present in InitializeResource JSON') + if (trace := _dict.get('trace')) is not None: + args['trace'] = trace + if (errors := _dict.get('errors')) is not None: + args['errors'] = [ErrorModelResource.from_dict(v) for v in errors] + if (last_started_at := _dict.get('last_started_at')) is not None: + args['last_started_at'] = string_to_datetime(last_started_at) + if (last_finished_at := _dict.get('last_finished_at')) is not None: + args['last_finished_at'] = string_to_datetime(last_finished_at) + if (initialized_options := _dict.get('initialized_options')) is not None: + args['initialized_options'] = [InitializedOption.from_dict(v) for v in initialized_options] + if (workflows := _dict.get('workflows')) is not None: + args['workflows'] = ProvidedCatalogWorkflows.from_dict(workflows) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a InitializeResource object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + if hasattr(self, 'href') and self.href is not None: + _dict['href'] = self.href + if hasattr(self, 'status') and self.status is not None: + _dict['status'] = self.status + if hasattr(self, 'trace') and self.trace is not None: + _dict['trace'] = self.trace + if hasattr(self, 'errors') and self.errors is not None: + errors_list = [] + for v in self.errors: + if isinstance(v, dict): + errors_list.append(v) + else: + errors_list.append(v.to_dict()) + _dict['errors'] = errors_list + if hasattr(self, 'last_started_at') and self.last_started_at is not None: + _dict['last_started_at'] = datetime_to_string(self.last_started_at) + if hasattr(self, 'last_finished_at') and self.last_finished_at is not None: + _dict['last_finished_at'] = datetime_to_string(self.last_finished_at) + if hasattr(self, 'initialized_options') and self.initialized_options is not None: + initialized_options_list = [] + for v in self.initialized_options: + if isinstance(v, dict): + initialized_options_list.append(v) + else: + initialized_options_list.append(v.to_dict()) + _dict['initialized_options'] = initialized_options_list + if hasattr(self, 'workflows') and self.workflows is not None: + if isinstance(self.workflows, dict): + _dict['workflows'] = self.workflows + else: + _dict['workflows'] = self.workflows.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this InitializeResource object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'InitializeResource') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'InitializeResource') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class StatusEnum(str, Enum): + """ + Status of the initialize operation. + """ + + NOT_STARTED = 'not_started' + IN_PROGRESS = 'in_progress' + SUCCEEDED = 'succeeded' + FAILED = 'failed' + + + +class InitializeSubDomain: + """ + The subdomain for a data product domain. + + :param str name: (optional) The name of the data product subdomain. + :param str id: (optional) The identifier of the data product subdomain. + :param str description: (optional) The description of the data product + subdomain. + """ + + def __init__( + self, + *, + name: Optional[str] = None, + id: Optional[str] = None, + description: Optional[str] = None, + ) -> None: + """ + Initialize a InitializeSubDomain object. + + :param str name: (optional) The name of the data product subdomain. + :param str id: (optional) The identifier of the data product subdomain. + :param str description: (optional) The description of the data product + subdomain. + """ + self.name = name + self.id = id + self.description = description + + @classmethod + def from_dict(cls, _dict: Dict) -> 'InitializeSubDomain': + """Initialize a InitializeSubDomain object from a json dictionary.""" + args = {} + if (name := _dict.get('name')) is not None: + args['name'] = name + if (id := _dict.get('id')) is not None: + args['id'] = id + if (description := _dict.get('description')) is not None: + args['description'] = description + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a InitializeSubDomain object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'description') and self.description is not None: + _dict['description'] = self.description + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this InitializeSubDomain object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'InitializeSubDomain') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'InitializeSubDomain') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class InitializedOption: + """ + List of options successfully initialized. + + :param str name: (optional) The name of the option. + :param int version: (optional) The version of the option. + """ + + def __init__( + self, + *, + name: Optional[str] = None, + version: Optional[int] = None, + ) -> None: + """ + Initialize a InitializedOption object. + + :param str name: (optional) The name of the option. + :param int version: (optional) The version of the option. + """ + self.name = name + self.version = version + + @classmethod + def from_dict(cls, _dict: Dict) -> 'InitializedOption': + """Initialize a InitializedOption object from a json dictionary.""" + args = {} + if (name := _dict.get('name')) is not None: + args['name'] = name + if (version := _dict.get('version')) is not None: + args['version'] = version + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a InitializedOption object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'version') and self.version is not None: + _dict['version'] = self.version + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this InitializedOption object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'InitializedOption') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'InitializedOption') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class JsonPatchOperation: + """ + This model represents an individual patch operation to be performed on a JSON + document, as defined by RFC 6902. + + :param str op: The operation to be performed. + :param str path: The JSON Pointer that identifies the field that is the target + of the operation. + :param str from_: (optional) The JSON Pointer that identifies the field that is + the source of the operation. + :param object value: (optional) The value to be used within the operation. + """ + + def __init__( + self, + op: str, + path: str, + *, + from_: Optional[str] = None, + value: Optional[object] = None, + ) -> None: + """ + Initialize a JsonPatchOperation object. + + :param str op: The operation to be performed. + :param str path: The JSON Pointer that identifies the field that is the + target of the operation. + :param str from_: (optional) The JSON Pointer that identifies the field + that is the source of the operation. + :param object value: (optional) The value to be used within the operation. + """ + self.op = op + self.path = path + self.from_ = from_ + self.value = value + + @classmethod + def from_dict(cls, _dict: Dict) -> 'JsonPatchOperation': + """Initialize a JsonPatchOperation object from a json dictionary.""" + args = {} + if (op := _dict.get('op')) is not None: + args['op'] = op + else: + raise ValueError('Required property \'op\' not present in JsonPatchOperation JSON') + if (path := _dict.get('path')) is not None: + args['path'] = path + else: + raise ValueError('Required property \'path\' not present in JsonPatchOperation JSON') + if (from_ := _dict.get('from')) is not None: + args['from_'] = from_ + if (value := _dict.get('value')) is not None: + args['value'] = value + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a JsonPatchOperation object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'op') and self.op is not None: + _dict['op'] = self.op + if hasattr(self, 'path') and self.path is not None: + _dict['path'] = self.path + if hasattr(self, 'from_') and self.from_ is not None: + _dict['from'] = self.from_ + if hasattr(self, 'value') and self.value is not None: + _dict['value'] = self.value + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this JsonPatchOperation object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'JsonPatchOperation') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'JsonPatchOperation') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + class OpEnum(str, Enum): + """ + The operation to be performed. + """ + + ADD = 'add' + REMOVE = 'remove' + REPLACE = 'replace' + MOVE = 'move' + COPY = 'copy' + TEST = 'test' + + + +class MemberRolesSchema: + """ + Member roles of a corresponding asset. + + :param str user_iam_id: (optional) User id. + :param List[str] roles: (optional) Roles of the given user. + """ + + def __init__( + self, + *, + user_iam_id: Optional[str] = None, + roles: Optional[List[str]] = None, + ) -> None: + """ + Initialize a MemberRolesSchema object. + + :param str user_iam_id: (optional) User id. + :param List[str] roles: (optional) Roles of the given user. + """ + self.user_iam_id = user_iam_id + self.roles = roles + + @classmethod + def from_dict(cls, _dict: Dict) -> 'MemberRolesSchema': + """Initialize a MemberRolesSchema object from a json dictionary.""" + args = {} + if (user_iam_id := _dict.get('user_iam_id')) is not None: + args['user_iam_id'] = user_iam_id + if (roles := _dict.get('roles')) is not None: + args['roles'] = roles + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a MemberRolesSchema object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'user_iam_id') and self.user_iam_id is not None: + _dict['user_iam_id'] = self.user_iam_id + if hasattr(self, 'roles') and self.roles is not None: + _dict['roles'] = self.roles + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this MemberRolesSchema object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'MemberRolesSchema') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'MemberRolesSchema') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class NextPage: + """ + Next page in the collection. + + :param str href: Link to the next page in the collection. + :param str start: Start token for pagination to the next page in the collection. + """ + + def __init__( + self, + href: str, + start: str, + ) -> None: + """ + Initialize a NextPage object. + + :param str href: Link to the next page in the collection. + :param str start: Start token for pagination to the next page in the + collection. + """ + self.href = href + self.start = start + + @classmethod + def from_dict(cls, _dict: Dict) -> 'NextPage': + """Initialize a NextPage object from a json dictionary.""" + args = {} + if (href := _dict.get('href')) is not None: + args['href'] = href + else: + raise ValueError('Required property \'href\' not present in NextPage JSON') + if (start := _dict.get('start')) is not None: + args['start'] = start + else: + raise ValueError('Required property \'start\' not present in NextPage JSON') + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a NextPage object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'href') and self.href is not None: + _dict['href'] = self.href + if hasattr(self, 'start') and self.start is not None: + _dict['start'] = self.start + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this NextPage object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'NextPage') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'NextPage') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class Overview: + """ + Overview details of a data contract. + + :param str api_version: (optional) The API version of the contract. + :param str kind: (optional) The kind of contract. + :param str name: (optional) The name of the contract. + :param str version: The version of the contract. + :param Domain domain: Domain that the data product version belongs to. If this + is the first version of a data product, this field is required. If this is a new + version of an existing data product, the domain will default to the domain of + the previous version of the data product. + :param str more_info: (optional) Additional information links about the + contract. + """ + + def __init__( + self, + version: str, + domain: 'Domain', + *, + api_version: Optional[str] = None, + kind: Optional[str] = None, + name: Optional[str] = None, + more_info: Optional[str] = None, + ) -> None: + """ + Initialize a Overview object. + + :param str version: The version of the contract. + :param Domain domain: Domain that the data product version belongs to. If + this is the first version of a data product, this field is required. If + this is a new version of an existing data product, the domain will default + to the domain of the previous version of the data product. + :param str api_version: (optional) The API version of the contract. + :param str kind: (optional) The kind of contract. + :param str name: (optional) The name of the contract. + :param str more_info: (optional) Additional information links about the + contract. + """ + self.api_version = api_version + self.kind = kind + self.name = name + self.version = version + self.domain = domain + self.more_info = more_info + + @classmethod + def from_dict(cls, _dict: Dict) -> 'Overview': + """Initialize a Overview object from a json dictionary.""" + args = {} + if (api_version := _dict.get('api_version')) is not None: + args['api_version'] = api_version + if (kind := _dict.get('kind')) is not None: + args['kind'] = kind + if (name := _dict.get('name')) is not None: + args['name'] = name + if (version := _dict.get('version')) is not None: + args['version'] = version + else: + raise ValueError('Required property \'version\' not present in Overview JSON') + if (domain := _dict.get('domain')) is not None: + args['domain'] = Domain.from_dict(domain) + else: + raise ValueError('Required property \'domain\' not present in Overview JSON') + if (more_info := _dict.get('more_info')) is not None: + args['more_info'] = more_info + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a Overview object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'api_version') and self.api_version is not None: + _dict['api_version'] = self.api_version + if hasattr(self, 'kind') and self.kind is not None: + _dict['kind'] = self.kind + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'version') and self.version is not None: + _dict['version'] = self.version + if hasattr(self, 'domain') and self.domain is not None: + if isinstance(self.domain, dict): + _dict['domain'] = self.domain + else: + _dict['domain'] = self.domain.to_dict() + if hasattr(self, 'more_info') and self.more_info is not None: + _dict['more_info'] = self.more_info + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this Overview object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'Overview') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'Overview') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class Pricing: + """ + Represents the pricing details of the contract. + + :param str amount: (optional) The amount for the contract pricing. + :param str currency: (optional) The currency for the pricing amount. + :param str unit: (optional) The unit associated with the pricing. + """ + + def __init__( + self, + *, + amount: Optional[str] = None, + currency: Optional[str] = None, + unit: Optional[str] = None, + ) -> None: + """ + Initialize a Pricing object. + + :param str amount: (optional) The amount for the contract pricing. + :param str currency: (optional) The currency for the pricing amount. + :param str unit: (optional) The unit associated with the pricing. + """ + self.amount = amount + self.currency = currency + self.unit = unit + + @classmethod + def from_dict(cls, _dict: Dict) -> 'Pricing': + """Initialize a Pricing object from a json dictionary.""" + args = {} + if (amount := _dict.get('amount')) is not None: + args['amount'] = amount + if (currency := _dict.get('currency')) is not None: + args['currency'] = currency + if (unit := _dict.get('unit')) is not None: + args['unit'] = unit + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a Pricing object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'amount') and self.amount is not None: + _dict['amount'] = self.amount + if hasattr(self, 'currency') and self.currency is not None: + _dict['currency'] = self.currency + if hasattr(self, 'unit') and self.unit is not None: + _dict['unit'] = self.unit + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this Pricing object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'Pricing') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'Pricing') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ProducerInputModel: + """ + Parameters for delivery that are set by a data product producer. + + :param EngineDetailsModel engine_details: (optional) Engine details as defined + by the data product producer. + :param List[EngineDetailsModel] engines: (optional) List of engines defined by + the data product producer. + """ + + def __init__( + self, + *, + engine_details: Optional['EngineDetailsModel'] = None, + engines: Optional[List['EngineDetailsModel']] = None, + ) -> None: + """ + Initialize a ProducerInputModel object. + + :param EngineDetailsModel engine_details: (optional) Engine details as + defined by the data product producer. + :param List[EngineDetailsModel] engines: (optional) List of engines defined + by the data product producer. + """ + self.engine_details = engine_details + self.engines = engines + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ProducerInputModel': + """Initialize a ProducerInputModel object from a json dictionary.""" + args = {} + if (engine_details := _dict.get('engine_details')) is not None: + args['engine_details'] = EngineDetailsModel.from_dict(engine_details) + if (engines := _dict.get('engines')) is not None: + args['engines'] = [EngineDetailsModel.from_dict(v) for v in engines] + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ProducerInputModel object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'engine_details') and self.engine_details is not None: + if isinstance(self.engine_details, dict): + _dict['engine_details'] = self.engine_details + else: + _dict['engine_details'] = self.engine_details.to_dict() + if hasattr(self, 'engines') and self.engines is not None: + engines_list = [] + for v in self.engines: + if isinstance(v, dict): + engines_list.append(v) + else: + engines_list.append(v.to_dict()) + _dict['engines'] = engines_list + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ProducerInputModel object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ProducerInputModel') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ProducerInputModel') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class PropertiesSchema: + """ + Properties of the corresponding asset. + + :param str value: (optional) Value of the property object. + """ + + def __init__( + self, + *, + value: Optional[str] = None, + ) -> None: + """ + Initialize a PropertiesSchema object. + + :param str value: (optional) Value of the property object. + """ + self.value = value + + @classmethod + def from_dict(cls, _dict: Dict) -> 'PropertiesSchema': + """Initialize a PropertiesSchema object from a json dictionary.""" + args = {} + if (value := _dict.get('value')) is not None: + args['value'] = value + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a PropertiesSchema object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'value') and self.value is not None: + _dict['value'] = self.value + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this PropertiesSchema object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'PropertiesSchema') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'PropertiesSchema') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ProvidedCatalogWorkflows: + """ + Resource defining provided workflow definitions. + + :param ProvidedWorkflowResource data_access: (optional) A reference to a + workflow definition. + :param ProvidedWorkflowResource request_new_product: (optional) A reference to a + workflow definition. + """ + + def __init__( + self, + *, + data_access: Optional['ProvidedWorkflowResource'] = None, + request_new_product: Optional['ProvidedWorkflowResource'] = None, + ) -> None: + """ + Initialize a ProvidedCatalogWorkflows object. + + :param ProvidedWorkflowResource data_access: (optional) A reference to a + workflow definition. + :param ProvidedWorkflowResource request_new_product: (optional) A reference + to a workflow definition. + """ + self.data_access = data_access + self.request_new_product = request_new_product + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ProvidedCatalogWorkflows': + """Initialize a ProvidedCatalogWorkflows object from a json dictionary.""" + args = {} + if (data_access := _dict.get('data_access')) is not None: + args['data_access'] = ProvidedWorkflowResource.from_dict(data_access) + if (request_new_product := _dict.get('request_new_product')) is not None: + args['request_new_product'] = ProvidedWorkflowResource.from_dict(request_new_product) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ProvidedCatalogWorkflows object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'data_access') and self.data_access is not None: + if isinstance(self.data_access, dict): + _dict['data_access'] = self.data_access + else: + _dict['data_access'] = self.data_access.to_dict() + if hasattr(self, 'request_new_product') and self.request_new_product is not None: + if isinstance(self.request_new_product, dict): + _dict['request_new_product'] = self.request_new_product + else: + _dict['request_new_product'] = self.request_new_product.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ProvidedCatalogWorkflows object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ProvidedCatalogWorkflows') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ProvidedCatalogWorkflows') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ProvidedWorkflowResource: + """ + A reference to a workflow definition. + + :param WorkflowDefinitionReference definition: (optional) Reference to a + workflow definition. + """ + + def __init__( + self, + *, + definition: Optional['WorkflowDefinitionReference'] = None, + ) -> None: + """ + Initialize a ProvidedWorkflowResource object. + + :param WorkflowDefinitionReference definition: (optional) Reference to a + workflow definition. + """ + self.definition = definition + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ProvidedWorkflowResource': + """Initialize a ProvidedWorkflowResource object from a json dictionary.""" + args = {} + if (definition := _dict.get('definition')) is not None: + args['definition'] = WorkflowDefinitionReference.from_dict(definition) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ProvidedWorkflowResource object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'definition') and self.definition is not None: + if isinstance(self.definition, dict): + _dict['definition'] = self.definition + else: + _dict['definition'] = self.definition.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ProvidedWorkflowResource object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ProvidedWorkflowResource') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ProvidedWorkflowResource') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class RevokeAccessResponse: + """ + This class holds the response message from the revoke access operation. + + :param str message: (optional) Response message of revoke access. + """ + + def __init__( + self, + *, + message: Optional[str] = None, + ) -> None: + """ + Initialize a RevokeAccessResponse object. + + :param str message: (optional) Response message of revoke access. + """ + self.message = message + + @classmethod + def from_dict(cls, _dict: Dict) -> 'RevokeAccessResponse': + """Initialize a RevokeAccessResponse object from a json dictionary.""" + args = {} + if (message := _dict.get('message')) is not None: + args['message'] = message + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a RevokeAccessResponse object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'message') and self.message is not None: + _dict['message'] = self.message + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this RevokeAccessResponse object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'RevokeAccessResponse') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'RevokeAccessResponse') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class RevokeAccessStateResponse: + """ + Revoke access states with pagination support. + + :param List[Asset] results: (optional) Holds revoke access state. + :param int total_count: (optional) Total number of rows available. + :param SearchAssetPaginationInfo next: (optional) Pagination information for the + next page of results. + """ + + def __init__( + self, + *, + results: Optional[List['Asset']] = None, + total_count: Optional[int] = None, + next: Optional['SearchAssetPaginationInfo'] = None, + ) -> None: + """ + Initialize a RevokeAccessStateResponse object. + + :param List[Asset] results: (optional) Holds revoke access state. + :param int total_count: (optional) Total number of rows available. + :param SearchAssetPaginationInfo next: (optional) Pagination information + for the next page of results. + """ + self.results = results + self.total_count = total_count + self.next = next + + @classmethod + def from_dict(cls, _dict: Dict) -> 'RevokeAccessStateResponse': + """Initialize a RevokeAccessStateResponse object from a json dictionary.""" + args = {} + if (results := _dict.get('results')) is not None: + args['results'] = [Asset.from_dict(v) for v in results] + if (total_count := _dict.get('total_count')) is not None: + args['total_count'] = total_count + if (next := _dict.get('next')) is not None: + args['next'] = SearchAssetPaginationInfo.from_dict(next) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a RevokeAccessStateResponse object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'results') and self.results is not None: + results_list = [] + for v in self.results: + if isinstance(v, dict): + results_list.append(v) + else: + results_list.append(v.to_dict()) + _dict['results'] = results_list + if hasattr(self, 'total_count') and self.total_count is not None: + _dict['total_count'] = self.total_count + if hasattr(self, 'next') and self.next is not None: + if isinstance(self.next, dict): + _dict['next'] = self.next + else: + _dict['next'] = self.next.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this RevokeAccessStateResponse object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'RevokeAccessStateResponse') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'RevokeAccessStateResponse') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class Roles: + """ + Represents a role associated with the contract. + + :param str role: (optional) The role associated with the contract. + """ + + def __init__( + self, + *, + role: Optional[str] = None, + ) -> None: + """ + Initialize a Roles object. + + :param str role: (optional) The role associated with the contract. + """ + self.role = role + + @classmethod + def from_dict(cls, _dict: Dict) -> 'Roles': + """Initialize a Roles object from a json dictionary.""" + args = {} + if (role := _dict.get('role')) is not None: + args['role'] = role + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a Roles object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'role') and self.role is not None: + _dict['role'] = self.role + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this Roles object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'Roles') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'Roles') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class SearchAssetPaginationInfo: + """ + Pagination information for the next page of results. + + :param str query: (optional) Search query for filtering results. + :param int limit: (optional) Number of items per page. + :param str bookmark: (optional) Bookmark for pagination. + :param str include: (optional) What to include in the results. + :param int skip: (optional) Number of items to skip. + """ + + def __init__( + self, + *, + query: Optional[str] = None, + limit: Optional[int] = None, + bookmark: Optional[str] = None, + include: Optional[str] = None, + skip: Optional[int] = None, + ) -> None: + """ + Initialize a SearchAssetPaginationInfo object. + + :param str query: (optional) Search query for filtering results. + :param int limit: (optional) Number of items per page. + :param str bookmark: (optional) Bookmark for pagination. + :param str include: (optional) What to include in the results. + :param int skip: (optional) Number of items to skip. + """ + self.query = query + self.limit = limit + self.bookmark = bookmark + self.include = include + self.skip = skip + + @classmethod + def from_dict(cls, _dict: Dict) -> 'SearchAssetPaginationInfo': + """Initialize a SearchAssetPaginationInfo object from a json dictionary.""" + args = {} + if (query := _dict.get('query')) is not None: + args['query'] = query + if (limit := _dict.get('limit')) is not None: + args['limit'] = limit + if (bookmark := _dict.get('bookmark')) is not None: + args['bookmark'] = bookmark + if (include := _dict.get('include')) is not None: + args['include'] = include + if (skip := _dict.get('skip')) is not None: + args['skip'] = skip + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a SearchAssetPaginationInfo object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'query') and self.query is not None: + _dict['query'] = self.query + if hasattr(self, 'limit') and self.limit is not None: + _dict['limit'] = self.limit + if hasattr(self, 'bookmark') and self.bookmark is not None: + _dict['bookmark'] = self.bookmark + if hasattr(self, 'include') and self.include is not None: + _dict['include'] = self.include + if hasattr(self, 'skip') and self.skip is not None: + _dict['skip'] = self.skip + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this SearchAssetPaginationInfo object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'SearchAssetPaginationInfo') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'SearchAssetPaginationInfo') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class ServiceIdCredentials: + """ + Service id credentials. + + :param str name: (optional) Name of the api key of the service id. + :param datetime created_at: (optional) Created date of the api key of the + service id. + """ + + def __init__( + self, + *, + name: Optional[str] = None, + created_at: Optional[datetime] = None, + ) -> None: + """ + Initialize a ServiceIdCredentials object. + + :param str name: (optional) Name of the api key of the service id. + :param datetime created_at: (optional) Created date of the api key of the + service id. + """ + self.name = name + self.created_at = created_at + + @classmethod + def from_dict(cls, _dict: Dict) -> 'ServiceIdCredentials': + """Initialize a ServiceIdCredentials object from a json dictionary.""" + args = {} + if (name := _dict.get('name')) is not None: + args['name'] = name + if (created_at := _dict.get('created_at')) is not None: + args['created_at'] = string_to_datetime(created_at) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a ServiceIdCredentials object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'created_at') and self.created_at is not None: + _dict['created_at'] = datetime_to_string(self.created_at) + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this ServiceIdCredentials object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'ServiceIdCredentials') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'ServiceIdCredentials') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class UseCase: + """ + UseCase. + + :param str id: The id of the use case associated with the data product. + :param str name: (optional) The display name of the use case associated with the + data product. + :param ContainerReference container: (optional) Container reference. + """ + + def __init__( + self, + id: str, + *, + name: Optional[str] = None, + container: Optional['ContainerReference'] = None, + ) -> None: + """ + Initialize a UseCase object. + + :param str id: The id of the use case associated with the data product. + :param str name: (optional) The display name of the use case associated + with the data product. + :param ContainerReference container: (optional) Container reference. + """ + self.id = id + self.name = name + self.container = container + + @classmethod + def from_dict(cls, _dict: Dict) -> 'UseCase': + """Initialize a UseCase object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + else: + raise ValueError('Required property \'id\' not present in UseCase JSON') + if (name := _dict.get('name')) is not None: + args['name'] = name + if (container := _dict.get('container')) is not None: + args['container'] = ContainerReference.from_dict(container) + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a UseCase object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + if hasattr(self, 'container') and self.container is not None: + if isinstance(self.container, dict): + _dict['container'] = self.container + else: + _dict['container'] = self.container.to_dict() + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this UseCase object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'UseCase') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'UseCase') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class Visualization: + """ + Data members for visualization. + + :param str id: (optional) Visualization identifier. + :param str name: (optional) Visualization name. + """ + + def __init__( + self, + *, + id: Optional[str] = None, + name: Optional[str] = None, + ) -> None: + """ + Initialize a Visualization object. + + :param str id: (optional) Visualization identifier. + :param str name: (optional) Visualization name. + """ + self.id = id + self.name = name + + @classmethod + def from_dict(cls, _dict: Dict) -> 'Visualization': + """Initialize a Visualization object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + if (name := _dict.get('name')) is not None: + args['name'] = name + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a Visualization object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + if hasattr(self, 'name') and self.name is not None: + _dict['name'] = self.name + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this Visualization object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'Visualization') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'Visualization') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + + +class WorkflowDefinitionReference: + """ + Reference to a workflow definition. + + :param str id: (optional) ID of a workflow definition. + """ + + def __init__( + self, + *, + id: Optional[str] = None, + ) -> None: + """ + Initialize a WorkflowDefinitionReference object. + + :param str id: (optional) ID of a workflow definition. + """ + self.id = id + + @classmethod + def from_dict(cls, _dict: Dict) -> 'WorkflowDefinitionReference': + """Initialize a WorkflowDefinitionReference object from a json dictionary.""" + args = {} + if (id := _dict.get('id')) is not None: + args['id'] = id + return cls(**args) + + @classmethod + def _from_dict(cls, _dict): + """Initialize a WorkflowDefinitionReference object from a json dictionary.""" + return cls.from_dict(_dict) + + def to_dict(self) -> Dict: + """Return a json dictionary representing this model.""" + _dict = {} + if hasattr(self, 'id') and self.id is not None: + _dict['id'] = self.id + return _dict + + def _to_dict(self): + """Return a json dictionary representing this model.""" + return self.to_dict() + + def __str__(self) -> str: + """Return a `str` version of this WorkflowDefinitionReference object.""" + return json.dumps(self.to_dict(), indent=2) + + def __eq__(self, other: 'WorkflowDefinitionReference') -> bool: + """Return `true` when self and other are equal, false otherwise.""" + if not isinstance(other, self.__class__): + return False + return self.__dict__ == other.__dict__ + + def __ne__(self, other: 'WorkflowDefinitionReference') -> bool: + """Return `true` when self and other are not equal, false otherwise.""" + return not self.__eq__(other) + +############################################################################## +# Pagers +############################################################################## + + +class DataProductsPager: + """ + DataProductsPager can be used to simplify the use of the "list_data_products" method. + """ + + def __init__( + self, + *, + client: DphV1, + limit: int = None, + ) -> None: + """ + Initialize a DataProductsPager object. + :param int limit: (optional) Limit the number of data products in the + results. The maximum limit is 200. + """ + self._has_next = True + self._client = client + self._page_context = {'next': None} + self._limit = limit + + def has_next(self) -> bool: + """ + Returns true if there are potentially more results to be retrieved. + """ + return self._has_next + + def get_next(self) -> List[dict]: + """ + Returns the next page of results. + :return: A List[dict], where each element is a dict that represents an instance of DataProductSummary. + :rtype: List[dict] + """ + if not self.has_next(): + raise StopIteration(message='No more results available') + + result = self._client.list_data_products( + limit=self._limit, + start=self._page_context.get('next'), + ).get_result() + + next = None + next_page_link = result.get('next') + if next_page_link is not None: + next = next_page_link.get('start') + self._page_context['next'] = next + if next is None: + self._has_next = False + + return result.get('data_products') + + def get_all(self) -> List[dict]: + """ + Returns all results by invoking get_next() repeatedly + until all pages of results have been retrieved. + :return: A List[dict], where each element is a dict that represents an instance of DataProductSummary. + :rtype: List[dict] + """ + results = [] + while self.has_next(): + next_page = self.get_next() + results.extend(next_page) + return results + + +class DataProductDraftsPager: + """ + DataProductDraftsPager can be used to simplify the use of the "list_data_product_drafts" method. + """ + + def __init__( + self, + *, + client: DphV1, + data_product_id: str, + asset_container_id: str = None, + version: str = None, + limit: int = None, + ) -> None: + """ + Initialize a DataProductDraftsPager object. + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str asset_container_id: (optional) Filter the list of data product + drafts by container id. + :param str version: (optional) Filter the list of data product drafts by + version number. + :param int limit: (optional) Limit the number of data product drafts in the + results. The maximum limit is 200. + """ + self._has_next = True + self._client = client + self._page_context = {'next': None} + self._data_product_id = data_product_id + self._asset_container_id = asset_container_id + self._version = version + self._limit = limit + + def has_next(self) -> bool: + """ + Returns true if there are potentially more results to be retrieved. + """ + return self._has_next + + def get_next(self) -> List[dict]: + """ + Returns the next page of results. + :return: A List[dict], where each element is a dict that represents an instance of DataProductDraftSummary. + :rtype: List[dict] + """ + if not self.has_next(): + raise StopIteration(message='No more results available') + + result = self._client.list_data_product_drafts( + data_product_id=self._data_product_id, + asset_container_id=self._asset_container_id, + version=self._version, + limit=self._limit, + start=self._page_context.get('next'), + ).get_result() + + next = None + next_page_link = result.get('next') + if next_page_link is not None: + next = next_page_link.get('start') + self._page_context['next'] = next + if next is None: + self._has_next = False + + return result.get('drafts') + + def get_all(self) -> List[dict]: + """ + Returns all results by invoking get_next() repeatedly + until all pages of results have been retrieved. + :return: A List[dict], where each element is a dict that represents an instance of DataProductDraftSummary. + :rtype: List[dict] + """ + results = [] + while self.has_next(): + next_page = self.get_next() + results.extend(next_page) + return results + + +class DataProductReleasesPager: + """ + DataProductReleasesPager can be used to simplify the use of the "list_data_product_releases" method. + """ + + def __init__( + self, + *, + client: DphV1, + data_product_id: str, + asset_container_id: str = None, + state: List[str] = None, + version: str = None, + limit: int = None, + ) -> None: + """ + Initialize a DataProductReleasesPager object. + :param str data_product_id: Data product ID. Use '-' to skip specifying the + data product ID explicitly. + :param str asset_container_id: (optional) Filter the list of data product + releases by container id. + :param List[str] state: (optional) Filter the list of data product versions + by state. States are: available and retired. Default is + "available","retired". + :param str version: (optional) Filter the list of data product releases by + version number. + :param int limit: (optional) Limit the number of data product releases in + the results. The maximum is 200. + """ + self._has_next = True + self._client = client + self._page_context = {'next': None} + self._data_product_id = data_product_id + self._asset_container_id = asset_container_id + self._state = state + self._version = version + self._limit = limit + + def has_next(self) -> bool: + """ + Returns true if there are potentially more results to be retrieved. + """ + return self._has_next + + def get_next(self) -> List[dict]: + """ + Returns the next page of results. + :return: A List[dict], where each element is a dict that represents an instance of DataProductReleaseSummary. + :rtype: List[dict] + """ + if not self.has_next(): + raise StopIteration(message='No more results available') + + result = self._client.list_data_product_releases( + data_product_id=self._data_product_id, + asset_container_id=self._asset_container_id, + state=self._state, + version=self._version, + limit=self._limit, + start=self._page_context.get('next'), + ).get_result() + + next = None + next_page_link = result.get('next') + if next_page_link is not None: + next = next_page_link.get('start') + self._page_context['next'] = next + if next is None: + self._has_next = False + + return result.get('releases') + + def get_all(self) -> List[dict]: + """ + Returns all results by invoking get_next() repeatedly + until all pages of results have been retrieved. + :return: A List[dict], where each element is a dict that represents an instance of DataProductReleaseSummary. + :rtype: List[dict] + """ + results = [] + while self.has_next(): + next_page = self.get_next() + results.extend(next_page) + return results diff --git a/src/wxdi/dph_services/version.py b/src/wxdi/dph_services/version.py new file mode 100644 index 0000000..6061e76 --- /dev/null +++ b/src/wxdi/dph_services/version.py @@ -0,0 +1,20 @@ +# coding: utf-8 + +# (C) Copyright IBM Corp. 2019, 2020. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Version of dph_services +""" +__version__ = '1.0.0' diff --git a/src/wxdi/dq_validator/issue_reporting.py b/src/wxdi/dq_validator/issue_reporting.py index aae69c5..a067481 100644 --- a/src/wxdi/dq_validator/issue_reporting.py +++ b/src/wxdi/dq_validator/issue_reporting.py @@ -209,24 +209,32 @@ def get_check_id( def create_check( self, asset_id: str, - column_name: str, check_obj: BaseCheck, + column_name: Optional[str] = None, project_id: Optional[str] = None, - catalog_id: Optional[str] = None - ) -> str: + catalog_id: Optional[str] = None, + parent_id: Optional[str] = None, + ) -> dict: """ Create a data quality check. Args: - cams_asset_id: Data asset ID - column_name: Name of the column + asset_id: Data asset ID + column_name: Name of the column (required if parent_id is provided) check_obj: BaseCheck instance to extract check details from project_id: Project ID (optional) catalog_id: Catalog ID (optional) + parent_id: Parent check ID (optional). If provided, native_id includes column details Returns: - str: The created check ID + dict: The full check response body from the API + + Raises: + ValueError: If parent_id is provided but column_name is None """ + # Validate: column_name is required when parent_id is provided + if parent_id is not None and column_name is None: + raise ValueError("column_name is required when parent_id is provided") # Extract check details from check object check_name = check_obj.get_check_name() dimension_name = check_obj.get_dimension().name @@ -243,7 +251,7 @@ def create_check( # Special handling for comparison check to get operator native_id_suffix = "" - if check_name == "comparison_check": + if check_name == "comparison_check" and parent_id is not None: from .checks.comparison_check import ComparisonCheck if isinstance(check_obj, ComparisonCheck): @@ -263,21 +271,232 @@ def create_check( # Get dimension ID from dimension name dimension_id = self.dimension_provider.search_dimension(dimension_name) - # Build native_id with the suffix - column_name_lower = column_name.lower() - native_id = f"{asset_id}/{check_type}/{column_name_lower}/{native_id_suffix}" + # Build native_id based on whether parent_id is provided + if parent_id is not None: + # With parent: detailed format with column name and suffix + # column_name is guaranteed to be not None due to validation at line 236 + assert column_name is not None + column_name_lower = column_name.lower() + native_id = f"{asset_id}/{check_type}/{column_name_lower}/{native_id_suffix}" + else: + # No parent: simple format with dimension name + native_id = f"{asset_id}/{check_type}/{dimension_name.capitalize()}" - # Create the check and return the check_id - check_id = self.check_provider.create_check( + # Create the check and return the full check body + check_body = self.check_provider._create_check_full( name=cpd_name, dimension_id=dimension_id, native_id=native_id, check_type=check_type, project_id=project_id, - catalog_id=catalog_id + catalog_id=catalog_id, + parent_check_id=parent_id ) - return check_id + return check_body + + def handle_parent( + self, + asset_id: str, + check_obj: BaseCheck, + project_id: Optional[str] = None, + catalog_id: Optional[str] = None + ) -> dict: + """ + Search for parent check using search_dq_check method. + If not found, create the parent check. + + Args: + asset_id: Data asset ID + check_obj: BaseCheck instance to extract check details from + project_id: Project ID (optional) + catalog_id: Catalog ID (optional) + + Returns: + dict: The full parent check body (found or created) + + Raises: + Exception: If parent check creation fails (not search failure, but actual creation failure) + """ + # Extract check details from check object + check_name = check_obj.get_check_name() + dimension_name = check_obj.get_dimension().name + + # Map check_name to check_type + check_type = self.map_check_name_to_check_type(check_name) or check_name + + # Construct native_id + native_id = f"{asset_id}/{check_type}/{dimension_name.capitalize()}" + + try: + # Search for the check using search_dq_check + check_response = self.search_provider.search_dq_check( + native_id=native_id, + check_type=check_type, + project_id=project_id, + catalog_id=catalog_id, + include_children=False + ) + + # Extract and return the check ID + return check_response + except Exception: + # Check not found during search - this is expected, so we'll try to create it + # Now attempt to create the parent check + try: + parent_check = self.create_check( + asset_id=asset_id, + column_name=None, + check_obj=check_obj, + project_id=project_id, + catalog_id=catalog_id, + parent_id=None + ) + # Mark that this check was newly created + parent_check["_newly_created"] = True + return parent_check + except Exception as creation_error: + # Parent check creation failed - raise a more specific exception + raise RuntimeError( + f"Failed to create parent check for asset_id='{asset_id}', " + f"check_type='{check_type}', dimension='{dimension_name}'. " + f"Original error: {str(creation_error)}" + ) from creation_error + + def create_bulk_issues( + self, + parent_check: dict, + child_check: dict, + column_name: str, + assets_map: Dict[str, Dict], + number_of_occurrences: int, + total_records: int, + project_id: str + ) -> dict: + """ + Create bulk issues for parent and child checks in a single API call. + + Args: + parent_check: Parent check body (table-level) + child_check: Child check body (column-level) + column_name: Name of the column + assets_map: Map of asset names to full asset objects (includes both data_asset and columns) + number_of_occurrences: Number of failed occurrences + total_records: Total number of records + project_id: Project ID + + Returns: + dict: Response from the bulk issue creation API + """ + # Fetch column asset from map + column_asset = assets_map.get(column_name) + if not column_asset: + raise ValueError(f"Column asset not found for column: {column_name}") + + # Get parent asset ID from column asset + parent_asset_id = column_asset.get("parent", {}).get("id") + if not parent_asset_id: + raise ValueError(f"Parent asset ID not found in column asset for column: {column_name}") + + # Find parent asset in the map by searching for asset with matching ID + parent_asset = None + for asset_name, asset_body in assets_map.items(): + if asset_body.get("id") == parent_asset_id: + parent_asset = asset_body + break + + if not parent_asset: + raise ValueError(f"Parent asset not found in assets_map for ID: {parent_asset_id}") + + # Extract native_id from parent asset + parent_native_id = parent_asset.get("native_id") + if not parent_native_id: + raise ValueError("Parent asset native_id not found") + + # Build issues array + issues = [ + { + "check": { + "native_id": parent_check.get("native_id"), + "type": parent_check.get("type") + }, + "reported_for": { + "native_id": parent_native_id, + "type": "data_asset" + }, + "number_of_occurrences": number_of_occurrences, + "number_of_tested_records": total_records, + "status": "aggregation", + "ignored": False + }, + { + "check": { + "native_id": child_check.get("native_id"), + "type": child_check.get("type") + }, + "reported_for": { + "native_id": column_asset.get("native_id"), + "type": "column" + }, + "number_of_occurrences": number_of_occurrences, + "number_of_tested_records": total_records, + "status": "actual", + "ignored": False + } + ] + + # Build assets array + assets = [ + { + "name": parent_asset.get("name"), + "type": "data_asset", + "native_id": parent_native_id, + "weight": parent_asset.get("weight", 1) + }, + { + "name": column_asset.get("name"), + "type": "column", + "native_id": column_asset.get("native_id"), + "parent": { + "native_id": parent_native_id, + "type": "data_asset" + }, + "weight": column_asset.get("weight", 1) + } + ] + + # Build existing_checks array + existing_checks = [ + { + "native_id": parent_check.get("native_id"), + "type": parent_check.get("type") + }, + { + "native_id": child_check.get("native_id"), + "type": child_check.get("type") + } + ] + + # Construct the bulk payload + bulk_payload = { + "issues": issues, + "assets": assets, + "existing_checks": existing_checks + } + + # Call the bulk issue creation API + try: + response = self.issues_provider.create_issues_bulk( + payload=bulk_payload, + project_id=project_id, + incremental_reporting=False, + refresh_assets=False + ) + print(f"Bulk issues created successfully: {len(issues)} issues") + return response + except Exception as e: + print(f"Failed to create bulk issues: {str(e)}") + raise def _validate_and_prepare_check_data( self, @@ -285,7 +504,7 @@ def _validate_and_prepare_check_data( check_name: str, stats: Dict[str, int], data_asset_entity, - column_id_map: Dict[str, str], + assets_map: Dict[str, Dict], validator: Validator ) -> Optional[Tuple[str, str, BaseCheck, int, int]]: """ @@ -296,7 +515,7 @@ def _validate_and_prepare_check_data( check_name: Name of the check stats: Statistics dictionary with 'failed' and 'total' keys data_asset_entity: Data asset entity from CAMS - column_id_map: Map of column names to column asset IDs + assets_map: Map of asset names to full column asset objects validator: Validator instance Returns: @@ -319,8 +538,13 @@ def _validate_and_prepare_check_data( if not (data_asset_entity.column_info and column_name in data_asset_entity.column_info): return None - # Guard: Skip if column asset ID not found - column_id = column_id_map.get(column_name) + # Guard: Skip if column asset not found + column_asset = assets_map.get(column_name) + if not column_asset: + return None + + # Extract column ID from asset + column_id = column_asset.get('id') if not column_id: return None @@ -331,6 +555,138 @@ def _validate_and_prepare_check_data( return (check_type, column_id, check_obj, number_of_occurrences, total_records) + def _find_existing_check( + self, + column_id: str, + check_type: str, + project_id: str + ) -> Optional[Tuple[str, Optional[str]]]: + """ + Find an existing check by column ID and check type. + + Args: + column_id: Column asset ID + check_type: Type of check (e.g., "format", "completeness") + project_id: Project ID + + Returns: + Tuple of (check_id, native_id) if found, None otherwise + """ + try: + checks = self.check_provider.get_checks( + dq_asset_id=column_id, + check_type=check_type, + project_id=project_id + ) + + for check in checks: + if check.get("type") == check_type: + return (check.get("id"), check.get("native_id")) + + return None + except ValueError as e: + print(f"Warning: Failed to get existing checks: {e}") + return None + + def _update_existing_check_metrics( + self, + existing_check_native_id: Optional[str], + number_of_occurrences: int, + total_records: int, + column_name: str, + check_type: str, + project_id: str + ) -> bool: + """ + Update metrics for an existing check. + + Args: + existing_check_native_id: Native ID of the existing check + number_of_occurrences: Number of failed occurrences + total_records: Total number of records + column_name: Name of the column + check_type: Type of check + project_id: Project ID + + Returns: + True if update successful, False otherwise + """ + try: + self.issues_provider.update_issue_metrics( + occurrences=number_of_occurrences, + tested_records=total_records, + column_name=column_name, + check_type=check_type, + project_id=project_id, + asset_type="column", + operation="add", + check_native_id=existing_check_native_id + ) + return True + except ValueError: + return False + + def _handle_409_conflict( + self, + column_id: str, + check_type: str, + number_of_occurrences: int, + total_records: int, + column_name: str, + check_name: str, + asset_id: str, + project_id: str + ) -> bool: + """ + Handle 409 conflict by finding and updating existing check. + + Args: + column_id: Column asset ID + check_type: Type of check + number_of_occurrences: Number of failed occurrences + total_records: Total number of records + column_name: Name of the column + check_name: Name of the check + asset_id: CAMS asset ID + project_id: Project ID + + Returns: + True if conflict handled successfully, False otherwise + """ + existing_check = self._find_existing_check(column_id, check_type, project_id) + + if not existing_check: + print(f"Warning: Check already exists but could not be found for column '{column_name}', check '{check_name}'") + return False + + existing_check_id, existing_check_native_id = existing_check + + update_success = self._update_existing_check_metrics( + existing_check_native_id, + number_of_occurrences, + total_records, + column_name, + check_type, + project_id + ) + + if update_success: + return True + + # If update fails, try to create the issue + self._handle_update_failure( + ValueError("Update failed"), + asset_id, + check_type, + column_name, + column_id, + number_of_occurrences, + total_records, + project_id, + dq_check_id=existing_check_id + ) + return True + def _create_check_and_issue( self, asset_id: str, @@ -340,7 +696,8 @@ def _create_check_and_issue( check_obj: BaseCheck, number_of_occurrences: int, total_records: int, - project_id: str + project_id: str, + assets_map: Dict[str, Dict] ) -> bool: """ Create a check and its associated issue. @@ -355,6 +712,7 @@ def _create_check_and_issue( number_of_occurrences: Number of failed occurrences total_records: Total number of records project_id: Project ID + assets_map: Map of asset names to full asset objects Returns: True if successful, False if creation failed @@ -366,88 +724,84 @@ def _create_check_and_issue( return False try: - check_id = self.create_check( + # Get parent check - may raise exception if parent creation fails + parent_check = self.handle_parent( asset_id=asset_id, - column_name=column_name, check_obj=check_obj, project_id=project_id ) - # Create the issue directly using the issues provider - self.issues_provider.create_issue( - check_id=check_id, - reported_for_id=column_id, - number_of_occurrences=number_of_occurrences, - number_of_tested_records=total_records, + + # Extract parent ID from parent check + parent_id = parent_check.get("id") + + # Check if parent was newly created + parent_was_created = parent_check.get("_newly_created", False) + + # Create child check + check = self.create_check( + asset_id=asset_id, + column_name=column_name, + check_obj=check_obj, project_id=project_id, - catalog_id=None + parent_id=parent_id ) + check_id = check.get("id") + if not check_id: + raise ValueError("Check ID not found in response") + + # If parent was newly created, use bulk issue creation + if parent_was_created and parent_check: + self.create_bulk_issues( + parent_check=parent_check, + child_check=check, + column_name=column_name, + assets_map=assets_map, + number_of_occurrences=number_of_occurrences, + total_records=total_records, + project_id=project_id + ) + else: + # Create the issue directly using the issues provider + self.issues_provider.create_issue( + dq_check_id=check_id, + reported_for_id=column_id, + number_of_occurrences=number_of_occurrences, + number_of_tested_records=total_records, + project_id=project_id, + catalog_id=None + ) return True - except ValueError as e: + except Exception as e: error_msg = str(e) # If 409 conflict (check already exists), try to find and update it if "409" in error_msg or "already exists" in error_msg: - try: - # Get existing checks for this column asset filtered by check type - checks = self.check_provider.get_checks( - asset_id=column_id, - check_type=check_type, - project_id=project_id - ) - - # Find the check with matching type - existing_check_id = None - existing_check_native_id = None - for check in checks: - if check.get("type") == check_type: - existing_check_id = check.get("id") - existing_check_native_id = check.get("native_id") - break - - if existing_check_id: - # Update issue metrics for the existing check - try: - self.issues_provider.update_issue_metrics( - occurrences=number_of_occurrences, - tested_records=total_records, - column_name=column_name, - check_type=check_type, - project_id=project_id, - asset_type="column", - operation="add", - check_native_id=existing_check_native_id - ) - return True - except ValueError as update_error: - # If update fails, try to create the issue - self._handle_update_failure( - update_error, asset_id, existing_check_id, check_type, - column_name, check_name, column_id, - number_of_occurrences, total_records, project_id - ) - return True - else: - print(f"Warning: Check already exists but could not be found for column '{column_name}', check '{check_name}'") - return False - except ValueError as get_error: - print(f"Warning: Failed to get existing checks for column '{column_name}', check '{check_name}': {get_error}") - return False + return self._handle_409_conflict( + column_id=column_id, + check_type=check_type, + number_of_occurrences=number_of_occurrences, + total_records=total_records, + column_name=column_name, + check_name=check_name, + asset_id=asset_id, + project_id=project_id + ) else: - # Different error - log and return False - print(f"Warning: Failed to create check for column '{column_name}', check '{check_name}': {e}") + # Any other error (including parent check creation failure) - log and return False + print(f"Error: Failed to create check and issue for column '{column_name}', check '{check_name}': {e}") return False def _handle_update_failure( self, error: ValueError, asset_id: str, - cams_check_id: str, check_type: str, column_name: str, - check_name: str, column_id: str, number_of_occurrences: int, total_records: int, - project_id: str + project_id: str, + check_id: Optional[str] = None, + dq_check_id: Optional[str] = None ) -> bool: """ Handle failure when updating issue metrics. @@ -455,32 +809,42 @@ def _handle_update_failure( Args: error: The ValueError that was raised asset_id: CAMS asset ID - cams_check_id: CAMS check ID check_type: Type of the check column_name: Name of the column - check_name: Name of the check column_id: Column asset ID number_of_occurrences: Number of failed occurrences total_records: Total number of records project_id: Project ID + check_id: Check ID (optional) + dq_check_id: DQ check ID (optional, if provided will be used directly) Returns: True (always, to indicate error was handled) """ + # Validate that at least one check ID is provided + if not dq_check_id and not check_id: + print(f"Warning: Neither dq_check_id nor check_id provided for column '{column_name}', of check_type '{check_type}'. Cannot handle update failure.") + return False + error_msg = str(error) # If issue not found, try to create it if "Issue not found" in error_msg or "Issue ID not found" in error_msg: - check_id = self.get_check_id( - check_native_id=f"{asset_id}/{cams_check_id}", - check_type=check_type, - project_id=project_id - ) + # Use provided dq_check_id or fetch it + check_id_to_use = dq_check_id - if check_id: + if not check_id_to_use: + # Only fetch check_id if dq_check_id is not provided + check_id_to_use = self.get_check_id( + check_native_id=f"{asset_id}/{check_id}", + check_type=check_type, + project_id=project_id + ) + + if check_id_to_use: # Create the issue directly using the issues provider self.issues_provider.create_issue( - check_id=cams_check_id, + dq_check_id=check_id_to_use, reported_for_id=column_id, number_of_occurrences=number_of_occurrences, number_of_tested_records=total_records, @@ -488,10 +852,10 @@ def _handle_update_failure( catalog_id=None ) else: - print(f"Warning: Could not find check_id for column '{column_name}', check '{check_name}'. Issue not created.") + print(f"Warning: Could not find check_id for column '{column_name}', of check_type '{check_type}'. Issue not created.") else: # Different error - log and continue - print(f"Warning: Failed to update issue metrics for column '{column_name}', check '{check_name}': {error}") + print(f"Warning: Failed to update issue metrics for column '{column_name}', of check_type '{check_type}': {error}") return True @@ -501,7 +865,6 @@ def _handle_existing_check( check_type: str, asset_id: str, column_name: str, - check_name: str, column_id: str, number_of_occurrences: int, total_records: int, @@ -528,15 +891,15 @@ def _handle_existing_check( if check.metadata.type != check_type: continue - cams_check_id = check.metadata.check_id - if not cams_check_id: + check_id = check.metadata.check_id + if not check_id: continue # Try to update existing issue metrics try: self.issues_provider.update_issue_metrics( - cams_asset_id=asset_id, - cams_check_id=cams_check_id, + asset_id=asset_id, + check_id=check_id, occurrences=number_of_occurrences, tested_records=total_records, column_name=column_name, @@ -548,9 +911,15 @@ def _handle_existing_check( return True except ValueError as e: self._handle_update_failure( - e, asset_id, cams_check_id, check_type, - column_name, check_name, column_id, - number_of_occurrences, total_records, project_id + e, + asset_id, + check_type, + column_name, + column_id, + number_of_occurrences, + total_records, + project_id, + check_id=check_id ) return True @@ -607,14 +976,14 @@ def report_issues( data_asset_entity = data_asset.entity # Fetch all column assets once and build a lookup map for efficiency - assets_response = self.asset_provider.get_assets(project_id=project_id, asset_type="column") - column_id_map = {asset['name']: asset['id'] for asset in assets_response.get('assets', [])} + assets_response = self.asset_provider.get_assets(project_id=project_id) + assets_map = {asset['name']: asset for asset in assets_response.get('assets', [])} # Iterate over combined statistics for (column_name, check_name), individual_stats in combined_stats.items(): # Validate and prepare data validated_data = self._validate_and_prepare_check_data( - column_name, check_name, individual_stats, data_asset_entity, column_id_map, validator + column_name, check_name, individual_stats, data_asset_entity, assets_map, validator ) if not validated_data: continue @@ -627,19 +996,21 @@ def report_issues( if not column_info.column_checks: self._create_check_and_issue( asset_id, column_name, column_id, check_name, - check_obj, number_of_occurrences, total_records, project_id + check_obj, number_of_occurrences, total_records, project_id, + assets_map ) continue # Try to handle existing check check_handled = self._handle_existing_check( - column_info, check_type, asset_id, column_name, check_name, - column_id, number_of_occurrences, total_records, project_id + column_info, check_type, asset_id, column_name, column_id, + number_of_occurrences, total_records, project_id ) # If check not found, create it if not check_handled: self._create_check_and_issue( asset_id, column_name, column_id, check_name, - check_obj, number_of_occurrences, total_records, project_id + check_obj, number_of_occurrences, total_records, project_id, + assets_map ) diff --git a/src/wxdi/dq_validator/provider/checks.py b/src/wxdi/dq_validator/provider/checks.py index 24c4392..f573955 100644 --- a/src/wxdi/dq_validator/provider/checks.py +++ b/src/wxdi/dq_validator/provider/checks.py @@ -18,12 +18,16 @@ """ import json -from typing import Optional +from typing import Optional, Dict, Any from .base_provider import BaseProvider from .config import ProviderConfig from ..utils import get_request_headers +# Error message constants +_ERR_MISSING_PROJECT_OR_CATALOG = "Either project_id or catalog_id must be provided" +_ERR_BOTH_PROJECT_AND_CATALOG = "Only one of project_id or catalog_id should be provided, not both" + class ChecksProvider(BaseProvider): """Provider for managing data quality checks. @@ -66,11 +70,15 @@ def create_check( native_id: str, check_type: Optional[str] = None, project_id: Optional[str] = None, - catalog_id: Optional[str] = None + catalog_id: Optional[str] = None, + parent_check_id: Optional[str] = None ) -> str: """ Create a new check for a data asset. + Note: Table-level checks are created without a parent_check_id, while column-level checks + require the table-level check ID as parent_check_id to establish the hierarchical relationship. + Args: name: Name of the check (e.g., "check_uniqueness_of_id") dimension_id: The dimension ID for the check @@ -78,9 +86,11 @@ def create_check( check_type: Type of check (optional, defaults to the check name if not provided) project_id (str, optional): The project ID containing the check catalog_id (str, optional): The catalog ID containing the check + parent_check_id (str, optional): The parent check ID. Required for column-level checks + (use table-level check ID). Omit for table-level checks. Returns: - str: The check_id of the created check + str: The check ID from the created check Raises: ValueError: If the API request fails or returns an error status, or if neither @@ -94,14 +104,86 @@ def create_check( ... project_id="project-123" ... ) '6be18374-573a-4cf8-8ab7-e428506e428b' + >>> # With parent parameter + >>> provider.create_check( + ... name="Format check", + ... dimension_id="ec453723-669c-48bb-82c1-11b69b3b8c93", + ... native_id="ba23145a-6d0a-46db-b314-41526b1e465f/format/sample3", + ... project_id="project-123", + ... parent_check_id="848aaddc-7401-4a43-ad2b-96a0946d4674" + ... ) + '7be18374-573a-4cf8-8ab7-e428506e428c' + """ + # Call _create_check_full and extract just the ID + result = self._create_check_full( + name=name, + dimension_id=dimension_id, + native_id=native_id, + check_type=check_type, + project_id=project_id, + catalog_id=catalog_id, + parent_check_id=parent_check_id + ) + return result["id"] + + def _create_check_full( + self, + name: str, + dimension_id: str, + native_id: str, + check_type: Optional[str] = None, + project_id: Optional[str] = None, + catalog_id: Optional[str] = None, + parent_check_id: Optional[str] = None + ) -> Dict[str, Any]: + """ + Create a new check and return the full check body. + + This method creates a check and returns the complete check object including + all properties like id, name, type, native_id, parent, etc. + + Note: Table-level checks are created without a parent_check_id, while column-level checks + require the table-level check ID as parent_check_id to establish the hierarchical relationship. + + Args: + name: Name of the check + dimension_id: ID of the dimension this check belongs to + native_id: Native identifier for the check + check_type: Type of check (defaults to name if not provided) + project_id: Project ID (mutually exclusive with catalog_id) + catalog_id: Catalog ID (mutually exclusive with project_id) + parent_check_id: Optional parent check ID for hierarchical checks. Required for column-level + checks (use table-level check ID). Omit for table-level checks. + + Returns: + Dict containing the full check body with all properties + + Raises: + ValueError: If neither or both project_id and catalog_id are provided, + or if the API request fails + + Example: + >>> from wxdi.dq_validator.provider import ProviderConfig, ChecksProvider + >>> config = ProviderConfig(url="https://example.com", auth_token="token") + >>> provider = ChecksProvider(config) + >>> check = provider.create_check_full( + ... name="Format check", + ... dimension_id="ec453723-669c-48bb-82c1-11b69b3b8c93", + ... native_id="ba23145a-6d0a-46db-b314-41526b1e465f/format/sample3", + ... project_id="project-123", + ... parent_check_id="848aaddc-7401-4a43-ad2b-96a0946d4674" + ... ) + >>> print(check) + {'id': '7be18374-573a-4cf8-8ab7-e428506e428c', 'name': 'Format check', + 'type': 'format', 'native_id': '...', 'parent': {'id': '...'}, ...} """ from ..utils import get_url_with_query_params # Validate that exactly one of project_id or catalog_id is provided if project_id is None and catalog_id is None: - raise ValueError("Either project_id or catalog_id must be provided") + raise ValueError(_ERR_MISSING_PROJECT_OR_CATALOG) if project_id is not None and catalog_id is not None: - raise ValueError("Only one of project_id or catalog_id should be provided, not both") + raise ValueError(_ERR_BOTH_PROJECT_AND_CATALOG) # Default check_type to name if not provided if check_type is None: @@ -129,6 +211,12 @@ def create_check( "details": '{"origin": "SDK"}' } + # Add parent only if provided + if parent_check_id is not None: + payload["parent"] = { + "id": parent_check_id + } + # Get request headers headers = get_request_headers(self.config.auth_token) @@ -147,17 +235,16 @@ def create_check( f"Response: {response.text}" ) - # Parse response and extract check_id + # Parse response and return full check body result = json.loads(response.text) - check_id = result.get("id") - if not check_id: + if not result.get("id"): raise ValueError("Check ID not found in response") - return check_id + return result def get_checks( self, - asset_id: str, + dq_asset_id: str, check_type: str, project_id: Optional[str] = None, catalog_id: Optional[str] = None, @@ -167,7 +254,7 @@ def get_checks( Get all checks for a specific asset filtered by check type. Args: - asset_id: The data quality asset identifier (column asset ID) + dq_asset_id: The data quality asset identifier (column asset ID) check_type: Type of check to filter by (e.g., "case", "completeness", "comparison") project_id (str, optional): The project ID containing the checks catalog_id (str, optional): The catalog ID containing the checks @@ -182,7 +269,7 @@ def get_checks( Example: >>> provider.get_checks( - ... asset_id="column-asset-123", + ... dq_asset_id="column-asset-123", ... check_type="case", ... project_id="project-123" ... ) @@ -192,16 +279,16 @@ def get_checks( # Validate that exactly one of project_id or catalog_id is provided if project_id is None and catalog_id is None: - raise ValueError("Either project_id or catalog_id must be provided") + raise ValueError(_ERR_MISSING_PROJECT_OR_CATALOG) if project_id is not None and catalog_id is not None: - raise ValueError("Only one of project_id or catalog_id should be provided, not both") + raise ValueError(_ERR_BOTH_PROJECT_AND_CATALOG) # Build the URL for checks API url = f"{self.config.url}/data_quality/v4/checks" # Add query parameters params = { - "asset.id": asset_id, + "asset.id": dq_asset_id, "type": check_type, "include_children": str(include_children).lower() } diff --git a/src/wxdi/dq_validator/provider/constraint_model.py b/src/wxdi/dq_validator/provider/constraint_model.py index a834cd9..72c949a 100644 --- a/src/wxdi/dq_validator/provider/constraint_model.py +++ b/src/wxdi/dq_validator/provider/constraint_model.py @@ -17,7 +17,6 @@ Pydantic models for Data Quality Constraints """ -from enum import StrEnum from typing import List, Optional, Any, Dict from datetime import datetime from pydantic import BaseModel @@ -33,6 +32,13 @@ from wxdi.dq_validator.checks.valid_values_check import ValidValuesCheck from wxdi.dq_validator.datatypes import DataType +try: + from enum import StrEnum +except ImportError: + # Remove when support for Python 3.10 is removed + from enum import Enum + class StrEnum(str, Enum): + pass class CheckType(StrEnum): """Enumeration of data quality check types""" diff --git a/src/wxdi/dq_validator/provider/issues.py b/src/wxdi/dq_validator/provider/issues.py index addfc4a..f8b1155 100644 --- a/src/wxdi/dq_validator/provider/issues.py +++ b/src/wxdi/dq_validator/provider/issues.py @@ -39,6 +39,10 @@ class IssuesProvider(BaseProvider): >>> provider = IssuesProvider(config) >>> provider.update_issue_values("issue-123", "project-123", occurrences=10, tested_records=100) """ + + # Error message constants + _ERR_MISSING_PROJECT_OR_CATALOG = "Either project_id or catalog_id must be provided" + _ERR_BOTH_PROJECT_AND_CATALOG = "Only one of project_id or catalog_id should be provided, not both" def __init__(self, config: ProviderConfig): """Initialize the IssuesProvider with configuration. @@ -78,9 +82,9 @@ def _patch_issue_field( # Validate that exactly one of project_id or catalog_id is provided if project_id is None and catalog_id is None: - raise ValueError("Either project_id or catalog_id must be provided") + raise ValueError(self._ERR_MISSING_PROJECT_OR_CATALOG) if project_id is not None and catalog_id is not None: - raise ValueError("Only one of project_id or catalog_id should be provided, not both") + raise ValueError(self._ERR_BOTH_PROJECT_AND_CATALOG) url = f"{self.config.url}/data_quality/v4/issues/{issue_id}" @@ -171,9 +175,9 @@ def update_issue_values( # Validate that exactly one of project_id or catalog_id is provided if project_id is None and catalog_id is None: - raise ValueError("Either project_id or catalog_id must be provided") + raise ValueError(self._ERR_MISSING_PROJECT_OR_CATALOG) if project_id is not None and catalog_id is not None: - raise ValueError("Only one of project_id or catalog_id should be provided, not both") + raise ValueError(self._ERR_BOTH_PROJECT_AND_CATALOG) # Build patch operations list patch_operations = [] @@ -227,7 +231,7 @@ def update_issue_values( def get_issue( self, reported_for_id: str, - check_id: str, + dq_check_id: str, project_id: Optional[str] = None, catalog_id: Optional[str] = None ) -> dict: @@ -238,7 +242,7 @@ def get_issue( Args: reported_for_id (str): The DQ asset ID to search for - check_id (str): The check ID to search for + dq_check_id (str): The check ID to search for project_id (str, optional): The project ID containing the issue catalog_id (str, optional): The catalog ID containing the issue @@ -253,13 +257,13 @@ def get_issue( >>> # Using project_id >>> provider.get_issue( ... reported_for_id="1488a413-99f9-4bed-906d-c33b505d5728", - ... check_id="ad277842-dea7-44ef-8e4b-d940df0f79aa", + ... dq_check_id="ad277842-dea7-44ef-8e4b-d940df0f79aa", ... project_id="24419069-d649-45cb-a2c1-64d6eed650d5" ... ) >>> # Using catalog_id >>> provider.get_issue( ... reported_for_id="1488a413-99f9-4bed-906d-c33b505d5728", - ... check_id="ad277842-dea7-44ef-8e4b-d940df0f79aa", + ... dq_check_id="ad277842-dea7-44ef-8e4b-d940df0f79aa", ... catalog_id="catalog-123" ... ) { @@ -272,16 +276,16 @@ def get_issue( # Validate that exactly one of project_id or catalog_id is provided if project_id is None and catalog_id is None: - raise ValueError("Either project_id or catalog_id must be provided") + raise ValueError(self._ERR_MISSING_PROJECT_OR_CATALOG) if project_id is not None and catalog_id is not None: - raise ValueError("Only one of project_id or catalog_id should be provided, not both") + raise ValueError(self._ERR_BOTH_PROJECT_AND_CATALOG) url = f"{self.config.url}/data_quality/v4/search_dq_issue" # Build query parameters params = { "reported_for.id": reported_for_id, - "check.id": check_id, + "check.id": dq_check_id, } # Add either project_id or catalog_id @@ -309,7 +313,7 @@ def get_issue( def get_issue_id( self, reported_for_id: str, - check_id: str, + dq_check_id: str, project_id: Optional[str] = None, catalog_id: Optional[str] = None ) -> str: @@ -320,7 +324,7 @@ def get_issue_id( Args: reported_for_id (str): The DQ asset ID to search for - check_id (str): The check ID to search for + dq_check_id (str): The check ID to search for project_id (str, optional): The project ID containing the issue catalog_id (str, optional): The catalog ID containing the issue @@ -335,23 +339,72 @@ def get_issue_id( >>> # Using project_id >>> provider.get_issue_id( ... reported_for_id="1488a413-99f9-4bed-906d-c33b505d5728", - ... check_id="ad277842-dea7-44ef-8e4b-d940df0f79aa", + ... dq_check_id="ad277842-dea7-44ef-8e4b-d940df0f79aa", ... project_id="24419069-d649-45cb-a2c1-64d6eed650d5" ... ) >>> # Using catalog_id >>> provider.get_issue_id( ... reported_for_id="1488a413-99f9-4bed-906d-c33b505d5728", - ... check_id="ad277842-dea7-44ef-8e4b-d940df0f79aa", + ... dq_check_id="ad277842-dea7-44ef-8e4b-d940df0f79aa", ... catalog_id="catalog-123" ... ) 'b8f4252b-cd35-4668-9b35-4635bfc6e2e0' """ - issue = self.get_issue(reported_for_id, check_id, project_id, catalog_id) + issue = self.get_issue(reported_for_id, dq_check_id, project_id, catalog_id) issue_id = issue.get("id") if issue_id is None: raise ValueError("Issue ID not found in response") return issue_id + def _validate_and_resolve_ids( + self, + asset_id: Optional[str], + check_id: Optional[str], + check_native_id: Optional[str] + ) -> tuple[str, str, str]: + """Validate and resolve asset_id, check_id, and check_native_id. + + Args: + asset_id: The CAMS data asset ID (optional) + check_id: The CAMS check ID (optional) + check_native_id: The check native_id (optional) + + Returns: + tuple: (asset_id, check_id, check_native_id) all resolved + + Raises: + ValueError: If validation fails or IDs cannot be resolved + """ + SEPARATOR = '/' + + # Validate that either (asset_id + check_id) or check_native_id is provided + has_cams_ids = asset_id is not None and check_id is not None + has_native_id = check_native_id is not None + + if not has_cams_ids and not has_native_id: + raise ValueError("Either (asset_id and check_id) or check_native_id must be provided") + + # If check_native_id is provided without asset_id/check_id, extract them + if has_native_id and not has_cams_ids: + # Parse check_native_id: first part before first SEPARATOR is asset_id, rest is check_id + # Format: "/" where check_id can contain slashes + first_slash_index = check_native_id.find(SEPARATOR) + if first_slash_index == -1: + raise ValueError(f"Invalid check_native_id format (missing {SEPARATOR}): {check_native_id}") + asset_id = check_native_id[:first_slash_index] + check_id = check_native_id[first_slash_index + 1:] + elif not has_native_id: + # Construct check_native_id from asset_id and check_id + # At this point, has_cams_ids is guaranteed to be True + check_native_id = f"{asset_id}{SEPARATOR}{check_id}" + + # At this point, all IDs should be set + assert asset_id is not None, "asset_id should be set by now" + assert check_id is not None, "check_id should be set by now" + assert check_native_id is not None, "check_native_id should be set by now" + + return asset_id, check_id, check_native_id + def update_issue_metrics( self, occurrences: int, @@ -362,8 +415,8 @@ def update_issue_metrics( catalog_id: Optional[str] = None, asset_type: str = "column", operation: str = "add", - cams_asset_id: Optional[str] = None, - cams_check_id: Optional[str] = None, + asset_id: Optional[str] = None, + check_id: Optional[str] = None, check_native_id: Optional[str] = None ) -> dict: """Update issue metrics using CAMS asset and check IDs or check native_id. @@ -381,10 +434,10 @@ def update_issue_metrics( catalog_id (str, optional): The catalog ID containing the issue asset_type (str, optional): The type of asset ("column" or "table"). Default is "column" operation (str, optional): Operation for both metrics - "add" or "replace". Default is "add" - cams_asset_id (str, optional): The CAMS data asset ID (required if check_native_id not provided) - cams_check_id (str, optional): The CAMS check ID (required if check_native_id not provided) - check_native_id (str, optional): The check native_id (required if cams_asset_id and cams_check_id not provided). - Format: "/" where cams_check_id can contain slashes + asset_id (str, optional): The CAMS data asset ID (required if check_native_id not provided) + check_id (str, optional): The CAMS check ID (required if check_native_id not provided) + check_native_id (str, optional): The check native_id (required if asset_id and check_id not provided). + Format: "/" where check_id can contain slashes Returns: dict: The response from the API containing the updated issue data @@ -392,13 +445,13 @@ def update_issue_metrics( Raises: ValueError: If the API request fails or returns an error status, or if neither project_id nor catalog_id is provided, or if both are provided, or if neither - (cams_asset_id + cams_check_id) nor check_native_id is provided + (asset_id + check_id) nor check_native_id is provided Example: - >>> # Using cams_asset_id and cams_check_id with project_id + >>> # Using asset_id and check_id with project_id >>> provider.update_issue_metrics( - ... cams_asset_id="b2debda2-6ab9-4a39-8c23-17954e004dcf", - ... cams_check_id="7377e2cd-ac0e-4833-8760-fd0e8cb682aa", + ... asset_id="b2debda2-6ab9-4a39-8c23-17954e004dcf", + ... check_id="7377e2cd-ac0e-4833-8760-fd0e8cb682aa", ... occurrences=10, ... tested_records=100, ... column_name="RTN", @@ -426,39 +479,16 @@ def update_issue_metrics( ... ) {'issue_id': 'b8f4252b-cd35-4668-9b35-4635bfc6e2e0', 'number_of_occurrences': 10, ...} """ - # Separator used in native_id format - SEPARATOR = '/' - # Validate that exactly one of project_id or catalog_id is provided if project_id is None and catalog_id is None: - raise ValueError("Either project_id or catalog_id must be provided") + raise ValueError(self._ERR_MISSING_PROJECT_OR_CATALOG) if project_id is not None and catalog_id is not None: - raise ValueError("Only one of project_id or catalog_id should be provided, not both") - - # Validate that either (cams_asset_id + cams_check_id) or check_native_id is provided - has_cams_ids = cams_asset_id is not None and cams_check_id is not None - has_native_id = check_native_id is not None - - if not has_cams_ids and not has_native_id: - raise ValueError("Either (cams_asset_id and cams_check_id) or check_native_id must be provided") + raise ValueError(self._ERR_BOTH_PROJECT_AND_CATALOG) - # If check_native_id is provided, extract cams_asset_id and cams_check_id from it - if has_native_id and not has_cams_ids: - # Parse check_native_id: first part before first SEPARATOR is cams_asset_id, rest is cams_check_id - # Format: "/" where cams_check_id can contain slashes - first_slash_index = check_native_id.find(SEPARATOR) - if first_slash_index == -1: - raise ValueError(f"Invalid check_native_id format (missing {SEPARATOR}): {check_native_id}") - cams_asset_id = check_native_id[:first_slash_index] - cams_check_id = check_native_id[first_slash_index + 1:] - elif has_cams_ids and not has_native_id: - # Construct check_native_id from cams_asset_id and cams_check_id - check_native_id = f"{cams_asset_id}{SEPARATOR}{cams_check_id}" - - # At this point, cams_asset_id and cams_check_id should be set - assert cams_asset_id is not None, "cams_asset_id should be set by now" - assert cams_check_id is not None, "cams_check_id should be set by now" - assert check_native_id is not None, "check_native_id should be set by now" + # Validate and resolve asset_id, check_id, and check_native_id + asset_id, check_id, check_native_id = self._validate_and_resolve_ids( + asset_id, check_id, check_native_id + ) from .dq_search import DQSearchProvider @@ -466,10 +496,10 @@ def update_issue_metrics( search_provider = DQSearchProvider(self.config) # Build native IDs for searching - # For asset: (for table) or / (for column) - asset_native_id = cams_asset_id # For table type, just the asset ID + # For asset: (for table) or / (for column) + asset_native_id = asset_id # For table type, just the asset ID if asset_type == "column": - asset_native_id += SEPARATOR + column_name + asset_native_id += '/' + column_name # Search for the DQ asset asset_response = search_provider.search_dq_asset( @@ -480,20 +510,20 @@ def update_issue_metrics( ) dq_asset_id = asset_response.get("id") if not dq_asset_id: - raise ValueError(f"DQ asset not found for CAMS asset ID: {cams_asset_id}") + raise ValueError(f"DQ asset not found for CAMS asset ID: {asset_id}") - # Get the issue using get_issues with the cams_check_id + # Get the issue using get_issues with the check_id issue = self.get_issues( dq_asset_id=dq_asset_id, check_type=check_type, - check_id=cams_check_id, + check_id=check_id, project_id=project_id, catalog_id=catalog_id ) if not issue: raise ValueError( - f"Issue not found for CAMS check ID: {cams_check_id} " + f"Issue not found for CAMS check ID: {check_id} " f"with type: {check_type}" ) @@ -581,9 +611,9 @@ def get_issues( # Validate that exactly one of project_id or catalog_id is provided if project_id is None and catalog_id is None: - raise ValueError("Either project_id or catalog_id must be provided") + raise ValueError(self._ERR_MISSING_PROJECT_OR_CATALOG) if project_id is not None and catalog_id is not None: - raise ValueError("Only one of project_id or catalog_id should be provided, not both") + raise ValueError(self._ERR_BOTH_PROJECT_AND_CATALOG) url = f"{self.config.url}/data_quality/v4/issues" @@ -643,7 +673,7 @@ def get_issues( def create_issue( self, - check_id: str, + dq_check_id: str, reported_for_id: str, number_of_occurrences: int, number_of_tested_records: int, @@ -658,7 +688,7 @@ def create_issue( This method creates a new issue for a specific check and data asset. Args: - check_id: The ID of the check for which to create the issue + dq_check_id: The ID of the check for which to create the issue reported_for_id: The ID of the data asset being reported on number_of_occurrences: Number of issue occurrences number_of_tested_records: Total number of records tested @@ -676,7 +706,7 @@ def create_issue( Example: >>> provider.create_issue( - ... check_id="6be18374-573a-4cf8-8ab7-e428506e428b", + ... dq_check_id="6be18374-573a-4cf8-8ab7-e428506e428b", ... reported_for_id="894d01fd-bdfc-4a4f-b68b-62751e06e06a", ... number_of_occurrences=123, ... number_of_tested_records=456789, @@ -688,9 +718,9 @@ def create_issue( # Validate that exactly one of project_id or catalog_id is provided if project_id is None and catalog_id is None: - raise ValueError("Either project_id or catalog_id must be provided") + raise ValueError(self._ERR_MISSING_PROJECT_OR_CATALOG) if project_id is not None and catalog_id is not None: - raise ValueError("Only one of project_id or catalog_id should be provided, not both") + raise ValueError(self._ERR_BOTH_PROJECT_AND_CATALOG) # Build the URL for issue creation API url = f"{self.config.url}/data_quality/v4/issues" @@ -706,7 +736,7 @@ def create_issue( # Prepare the payload payload = { "check": { - "id": check_id + "id": dq_check_id }, "reported_for": { "id": reported_for_id @@ -742,3 +772,147 @@ def create_issue( raise ValueError("Issue ID not found in response") return issue_id + + def create_issues_bulk( + self, + payload: dict, + project_id: Optional[str] = None, + catalog_id: Optional[str] = None, + incremental_reporting: bool = False, + refresh_assets: bool = False + ) -> dict: + """ + Create multiple data quality issues in bulk. + + This method creates multiple issues, assets, and checks in a single API call + for better performance when reporting multiple related issues. + + Args: + payload: The bulk payload containing issues, assets, and existing_checks arrays. Expected structure is + + >>> { + ... "issues": [ + ... { + ... "check": {"native_id": str, "type": str}, + ... "reported_for": {"native_id": str, "type": str}, + ... "number_of_occurrences": int, + ... "number_of_tested_records": int, + ... "status": str, + ... "ignored": bool + ... }, + ... ... + ... ], + ... "assets": [ + ... { + ... "name": str, + ... "type": str, + ... "native_id": str, + ... "weight": int, + ... "parent": {"native_id": str, "type": str} (optional) + ... }, + ... ... + ... ], + ... "existing_checks": [ + ... {"native_id": str, "type": str}, + ... ... + ... ] + ... } + + project_id (str, optional): The project ID containing the issues + catalog_id (str, optional): The catalog ID containing the issues + incremental_reporting (bool): If true, adds archived issue counts to new issues + instead of replacing them. Default is False. + refresh_assets (bool): If true, assets will be refreshed and any assets not + present in the updated list will be deleted. Default is False. + + Returns: + dict: The response from the API containing the created issues data + + Raises: + ValueError: If the API request fails or returns an error status, or if neither + project_id nor catalog_id is provided, or if both are provided + + Example: + >>> payload = { + ... "issues": [ + ... { + ... "check": { + ... "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/Validity", + ... "type": "format" + ... }, + ... "reported_for": { + ... "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f", + ... "type": "data_asset" + ... }, + ... "number_of_occurrences": 200, + ... "number_of_tested_records": 1000, + ... "status": "aggregation", + ... "ignored": False + ... } + ... ], + ... "assets": [ + ... { + ... "name": "ACCOUNT_HOLDERS.csv", + ... "type": "data_asset", + ... "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f", + ... "weight": 1 + ... } + ... ], + ... "existing_checks": [ + ... { + ... "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/Validity", + ... "type": "format" + ... } + ... ] + ... } + >>> provider.create_issues_bulk( + ... payload=payload, + ... project_id="project-123", + ... incremental_reporting=True + ... ) + {'issues': [...], 'assets': [...], ...} + """ + from ..utils import get_url_with_query_params + + # Validate that exactly one of project_id or catalog_id is provided + if project_id is None and catalog_id is None: + raise ValueError(self._ERR_MISSING_PROJECT_OR_CATALOG) + if project_id is not None and catalog_id is not None: + raise ValueError(self._ERR_BOTH_PROJECT_AND_CATALOG) + + # Build the URL for bulk issue creation API + url = f"{self.config.url}/data_quality/v4/create_issues" + + # Build query parameters + params = {} + if project_id is not None: + params["project_id"] = project_id + elif catalog_id is not None: + params["catalog_id"] = catalog_id + + # Add boolean query parameters + params["incremental_reporting"] = str(incremental_reporting).lower() + params["refresh_assets"] = str(refresh_assets).lower() + + url = get_url_with_query_params(url, params) + + # Get request headers + headers = get_request_headers(self.config.auth_token) + + # Make POST request + response = self.session.post( + url, + headers=headers, + data=json.dumps(payload), + verify=False + ) + + if not response.ok: + raise ValueError( + f"Failed to create issues in bulk. " + f"Status: {response.status_code}, " + f"Response: {response.text}" + ) + + # Parse and return response + return json.loads(response.text) diff --git a/src/wxdi/dq_validator/provider/response_model.py b/src/wxdi/dq_validator/provider/response_model.py index 8a5e009..f2dba07 100644 --- a/src/wxdi/dq_validator/provider/response_model.py +++ b/src/wxdi/dq_validator/provider/response_model.py @@ -31,8 +31,10 @@ def to_iso(self, dt: datetime): version_id: str source_repository_id: str global_id: str + workflow_id: Optional[str] = None + draft_mode: Optional[str] = None is_target_draft: Optional[bool] = None - effective_start_date: datetime + effective_start_date: Optional[datetime] = None created_by: str created_at: datetime modified_by: str @@ -45,6 +47,7 @@ def to_iso(self, dt: datetime): tags: Optional[List[str]] = None steward_ids: Optional[List[str]] = None steward_group_ids: Optional[List[str]] = None + workflow_state: Optional[str] = None user_access: Optional[bool] = None @classmethod diff --git a/src/wxdi/odcs_generator/README-GENERATE-ODCS-SCRIPT.md b/src/wxdi/odcs_generator/README-GENERATE-ODCS-SCRIPT.md new file mode 100644 index 0000000..9491007 --- /dev/null +++ b/src/wxdi/odcs_generator/README-GENERATE-ODCS-SCRIPT.md @@ -0,0 +1,1095 @@ + + +# Generate ODCS - Script Documentation + +## Overview + +The `odcs_generator` module provides tools to automatically generate ODCS (Open Data Contract Standard) v3.1.0 compliant YAML files from data catalog metadata. It supports multiple data catalog sources: + +- **Collibra**: Extracts table/view definitions, column schemas, data types, classifications, and custom properties +- **Informatica CDGC**: Fetches asset metadata including table definitions, column schemas, and system attributes + +**Scripts:** +- `wxdi.odcs_generator.generate_odcs_from_collibra` - Generate ODCS from Collibra assets +- `wxdi.odcs_generator.generate_odcs_from_informatica` - Generate ODCS from Informatica CDGC assets + +--- + +# Table of Contents + +1. [Collibra Integration](#collibra-integration) +2. [Informatica Integration](#informatica-integration) +3. [Common Features](#common-features) +4. [Related Documentation](#related-documentation) + +--- + +# Collibra Integration + +## Location +`wxdi.odcs_generator.generate_odcs_from_collibra` + +## Features + +- ✅ **Automatic Metadata Extraction**: Fetches asset details, attributes, and relations from Collibra REST API +- ✅ **Column Discovery**: Automatically discovers columns through asset relations +- ✅ **Data Type Mapping**: Maps Collibra logical and technical data types to ODCS standards +- ✅ **Classification Support**: Extracts data classifications using Collibra GraphQL API +- ✅ **Tag Integration**: Includes Collibra tags at both asset and column levels +- ✅ **Custom Properties**: Preserves Collibra attributes as custom properties +- ✅ **ODCS v3.1.0 Compliance**: Generates fully compliant ODCS YAML files + +## Requirements + +### Python Dependencies + +Install all dependencies from the project root: + +```bash +pip install -r requirements.txt +``` + +**Required packages:** +- `requests` >= 2.32.4 - HTTP library for Collibra API calls +- `pyyaml` >= 5.4.0 - YAML file generation +- `urllib3` >= 2.6.3 - HTTP client library +- `python_dateutil` >= 2.5.3 - Date/time utilities +- `ibm_cloud_sdk_core` >= 3.16.7 - IBM Cloud SDK core + +### Collibra Access + +- Valid Collibra instance URL +- User account with read access to assets, attributes, and relations +- Network connectivity to Collibra REST API and GraphQL endpoints + +## Installation + +1. Clone the repository: + ```bash + git clone + cd data-product-python-sdk + ``` + +2. Install dependencies: + ```bash + pip install -r requirements.txt + ``` + +3. Set up environment variables (see Configuration section) + +## Directory Structure + +``` +data-product-python-sdk/ +├── odcs_generator/ +│ ├── __init__.py +│ ├── generate_odcs_from_collibra.py # Main script +│ └── README-GENERATE-ODCS-SCRIPT.md # This file +├── examples/ +│ ├── __init__.py +│ └── odcs_generator_example.py # Usage examples +├── requirements.txt # Python dependencies +└── ... +``` + +## Configuration + +### Environment Variables + +Set the following environment variables before running the script: + +```bash +export COLLIBRA_URL="https://your-instance.collibra.com" +export COLLIBRA_USERNAME="myuser" +export COLLIBRA_PASSWORD="mypassword" +``` + +**Alternative**: Pass credentials via command-line arguments (see Usage section) + +### Collibra Permissions Required + +The user account needs the following permissions: +- Read access to assets +- Read access to attributes +- Read access to relations +- Access to GraphQL API (for classifications) +- Read access to tags + +## Usage + +### Command-Line Usage + +Run the script directly from the `odcs_generator` directory: + +```bash +python odcs_generator/generate_odcs_from_collibra.py +``` + +**Example:** +```bash +python odcs_generator/generate_odcs_from_collibra.py 019a57f9-62d2-7aa0-9f22-4fa2cea1180b +``` + +This generates a file named `-odcs.yaml` in the current directory. + +### Programmatic Usage + +Import and use the module in your Python code: + +```python +from odcs_generator import CollibraClient, ODCSGenerator + +# Initialize client +client = CollibraClient( + base_url="https://your-instance.collibra.com", + username="your_username", + password="your_pswd" +) + +# Create generator +generator = ODCSGenerator(client) + +# Generate ODCS +odcs_data = generator.generate_odcs("asset_id") +``` + +See `examples/odcs_generator_example.py` for complete examples. + +### Command-Line Options + +```bash +python odcs_generator/generate_odcs_from_collibra.py [OPTIONS] +``` + +**Options:** + +| Option | Description | Default | +|--------|-------------|---------| +| `asset_id` | Collibra asset ID (required) | - | +| `-o, --output` | Output YAML file path | `-odcs.yaml` | +| `--url` | Collibra base URL | `$COLLIBRA_URL` | +| `-u, --username` | Collibra username | `$COLLIBRA_USERNAME` | +| `-p, --password` | Collibra password | `$COLLIBRA_PASSWORD` | +| `-h, --help` | Show help message | - | + +### Examples + +**Generate with custom output file:** +```bash +python odcs_generator/generate_odcs_from_collibra.py 019a57f9-62d2-7aa0-9f22-4fa2cea1180b -o my-contract.yaml +``` + +**Pass credentials via command line:** +```bash +python odcs_generator/generate_odcs_from_collibra.py 019a57f9-62d2-7aa0-9f22-4fa2cea1180b \ + --url https://acme.collibra.com \ + -u myuser \ + -p mypassword +``` + +**Using environment variables:** +```bash +export COLLIBRA_URL="https://acme.collibra.com" +export COLLIBRA_USERNAME="myuser" +export COLLIBRA_PASSWORD="mypassword" + +python odcs_generator/generate_odcs_from_collibra.py 019a57f9-62d2-7aa0-9f22-4fa2cea1180b +``` + +## Example Scripts + +The `examples/` directory contains comprehensive usage examples: + +### Basic Usage Example +```bash +python examples/odcs_generator_example.py +``` + +This example demonstrates: +- Connecting to Collibra using environment variables +- Generating ODCS from a single asset +- Saving the output to a YAML file + +### Custom Processing Example +Shows how to: +- Generate ODCS programmatically +- Customize contract metadata (dataProduct, version, name) +- Add quality rules +- Update server configuration + +### Batch Processing Example +Demonstrates: +- Processing multiple assets in a loop +- Error handling for failed assets +- Generating summary reports + +See `examples/odcs_generator_example.py` for complete code and additional examples. + +## How It Works + +### 1. Asset Metadata Extraction + +The script fetches the following from Collibra: + +- **Asset Details**: Name, display name, type, domain, creation date +- **Attributes**: All custom attributes (Description, Data Type, etc.) +- **Relations**: Source and target relations to discover columns +- **Tags**: Direct tags assigned to the asset +- **Classifications**: Data classifications via GraphQL API + +### 2. Column Discovery + +Columns are discovered through asset relations: +- Processes both source and target relations +- Identifies assets with "column" in their type name +- Fetches column attributes and classifications +- Deduplicates columns by name + +### 3. Data Type Mapping + +**Logical Types** (Collibra → ODCS): +- `text` → `string` +- `whole number` → `integer` +- `decimal number` → `number` +- `date time` → `timestamp` +- `true/false` → `boolean` +- `geographical` → `string` + +**Physical Types** (with size/precision/scale): +- `VARCHAR(255)` +- `DECIMAL(10,2)` +- `NUMBER(18,4)` + +### 4. ODCS Generation + +Creates a complete ODCS v3.1.0 structure: + +```yaml +id: +kind: DataContract +apiVersion: v3.1.0 +domain: +dataProduct: Sample data product +version: 1.0.0 +name: Sample contract +status: active +contractCreatedTs: +description: + authoritativeDefinitions: + - type: collibra-asset + url: +tags: [...] +schema: + - id: + name: + physicalName: + physicalType: table|view + description: + properties: [...] +servers: + - id: + server: CONFIGURE_SERVER_HOSTNAME # ⚠️ Manual config required + type: DEFINE_SERVER_TYPE # ⚠️ Manual config required +``` + +### 5. Manual Configuration Comments + +The script adds inline comments to guide manual configuration: + +```yaml +servers: + # ============================================ + # ⚠️ MANUAL CONFIGURATION REQUIRED + # ============================================ + # Please update the following fields: + - id: server-79f2fd7b + server: CONFIGURE_SERVER_HOSTNAME # ⚠️ UPDATE: e.g., prod.snowflake.acme.com + type: DEFINE_SERVER_TYPE # ⚠️ UPDATE: e.g., snowflake, postgres, bigquery, redshift +``` + +## Output Structure + +### Generated ODCS File + +The script generates a YAML file with the following structure: + +```yaml +id: +kind: DataContract +apiVersion: v3.1.0 +domain: +dataProduct: +version: 1.0.0 +name: +status: active +contractCreatedTs: +description: + authoritativeDefinitions: + - type: collibra-asset + url: +tags: [...] +schema: + - id: + name: + physicalName: + physicalType: table|view + description: + customProperties: [...] + properties: + - name: + physicalName: + logicalType: string|integer|number|... + physicalType: VARCHAR(255)|DECIMAL(10,2)|... + description: + required: true|false + primaryKey: true|false + classification: + tags: [...] +servers: + - id: + server: CONFIGURE_SERVER_HOSTNAME + type: DEFINE_SERVER_TYPE +``` + +### Console Output + +The script provides detailed progress information: + +``` +Connecting to Collibra at https://acme.collibra.com... +Generating ODCS for asset: 019a57f9-62d2-7aa0-9f22-4fa2cea1180b + +=== Processing asset: 019a57f9-62d2-7aa0-9f22-4fa2cea1180b === +Fetching asset details... +Fetching asset attributes... +Fetching asset relations (as source)... +Fetching asset relations (as target)... +Found 15 relations where asset is target +Extracting column information from all relations... +Total relations found: 20 + Found column (source): name='customer_id', type='Column' + Found column (target): name='transaction_date', type='Column' +Found 12 unique columns from relations + +Writing ODCS to customer-transactions-odcs.yaml... +✓ Successfully generated ODCS file: customer-transactions-odcs.yaml +``` + +## Collibra Attribute Mapping + +### Asset-Level Attributes + +| Collibra Attribute | ODCS Field | Notes | +|-------------------|------------|-------| +| Name | `schema.name` | Full asset name | +| Display Name | `schema.physicalName` | Physical name | +| Description | `schema.description` | Asset description | +| Domain | `domain` | Domain name | +| Created On | `contractCreatedTs` | ISO timestamp | +| Table Type | `schema.physicalType` | table or view | +| Tags | `tags` | Array of tag names | +| Other attributes | `schema.customProperties` | Preserved as custom properties | + +### Column-Level Attributes + +| Collibra Attribute | ODCS Field | Notes | +|-------------------|------------|-------| +| Name / Display Name | `properties.name` | Column name | +| Original Name / Physical Name | `properties.physicalName` | Physical column name | +| Description / Definition | `properties.description` | Column description | +| Data Type | `properties.logicalType` | Mapped to ODCS types | +| Technical Data Type | `properties.physicalType` | With size/precision/scale | +| Is Nullable / Nullable | `properties.required` | Boolean | +| Is Primary Key / Primary Key | `properties.primaryKey` | Boolean | +| Security Classification | `properties.classification` | Security class | +| PII / Personally Identifiable Information | `properties.tags` | Adds "PII" tag | +| Size / Length | `properties.physicalType` | Appended to type | +| Precision | `properties.physicalType` | For numeric types | +| Scale / Number Of Fractional Digits | `properties.physicalType` | For numeric types | +| Classifications (GraphQL) | `properties.tags` | As `data_classification:*` | +| Tags | `properties.tags` | Column-level tags | + +## Error Handling + +### Common Errors + +**1. Missing Environment Variables** +``` +Error: Collibra URL is required. Set COLLIBRA_URL environment variable or use --url +``` +**Solution**: Set required environment variables or pass via command line + +**2. Authentication Failure** +``` +Error: HTTP 401 - Unauthorized +``` +**Solution**: Verify username and password are correct + +**3. Asset Not Found** +``` +Error: HTTP 404 - Asset not found +``` +**Solution**: Verify the asset ID exists and you have access + +**4. Network Issues** +``` +Error: Connection refused +``` +**Solution**: Check network connectivity and Collibra URL + +### Warnings + +The script may display warnings for non-critical issues: + +``` +Warning: Could not fetch tags: +Warning: No columns found in Collibra. +``` + +These warnings indicate missing data but don't prevent ODCS generation. + +## Validation + +After generating the ODCS file, validate it against the ODCS specification: + +1. **Manual Review**: Check the generated YAML for completeness +2. **YAML Syntax**: Ensure valid YAML format +3. **ODCS Compliance**: Verify against ODCS v3.1.0 specification +4. **Required Fields**: Confirm all mandatory fields are present +5. **Server Configuration**: Complete fields marked with ⚠️ warnings + +## Limitations + +## Script-Specific Limitations + +1. **Server Connection Details**: Collibra doesn't store actual server hostnames or connection strings +2. **Server Type**: May need manual verification/correction +3. **Additional Server Parameters**: Account, environment, roles, etc. +4. **Contract Metadata**: Data product name, version, contract name (uses defaults) +5. **Quality Rules**: Not extracted from Collibra +6. **SLA Terms**: Not available in Collibra metadata +7. **Stakeholder Information**: Requires manual addition + +### Collibra-Specific Limitations + +- Column discovery depends on proper relation setup in Collibra +- Data type mapping may need adjustment for custom types +- Classification extraction requires GraphQL API access +- Some attributes may have different names in your Collibra instance + +## Customization + +### Modifying Attribute Mappings + +Edit the `_build_attribute_map()` method to change which Collibra attributes are extracted: + +```python +# Add custom attribute mapping +custom_attr = attr_map.get('Your Custom Attribute', '') +``` + +### Changing Data Type Mappings + +Update the `LOGICAL_TYPE_MAPPING` dictionary: + +```python +LOGICAL_TYPE_MAPPING = { + 'your_custom_type': 'odcs_type', + # ... existing mappings +} +``` + +### Adding Custom Properties + +Modify the `_extract_custom_properties()` method to include/exclude specific attributes: + +```python +EXCLUDED_ATTRIBUTES = {'Description', 'Your Excluded Attr'} +``` + +## Best Practices + +1. **Verify Asset ID**: Ensure you're using the correct Collibra asset ID +2. **Check Relations**: Verify columns are properly related to tables in Collibra +3. **Review Output**: Always review the generated YAML before using +4. **Manual Configuration**: Complete all fields marked with ⚠️ warnings +5. **Validate**: Validate the YAML before finalizing + +## Troubleshooting + +### No Columns Generated + +**Problem**: Schema has no properties (columns) + +**Possible Causes**: +- Columns not properly related to table in Collibra +- Relation types not recognized +- Column assets have incorrect type + +**Solution**: +1. Check Collibra relations for the asset + +--- + +# Informatica Integration + +## Overview + +`generate_odcs_from_informatica.py` is a Python script that automatically generates ODCS (Open Data Contract Standard) v3.1.0 compliant YAML files from Informatica CDGC (Cloud Data Governance and Catalog) asset metadata. It extracts table definitions, column schemas, data types, and system attributes from Informatica and transforms them into standardized data contracts. + +**Location**: `odcs_generator/generate_odcs_from_informatica.py` + +## Features + +- ✅ **Automatic Metadata Extraction**: Fetches asset details from Informatica CDGC REST API +- ✅ **Column Discovery**: Automatically discovers columns through asset hierarchy +- ✅ **Concurrent Processing**: Fetches column details in parallel for better performance +- ✅ **System Attributes**: Preserves Informatica system attributes as custom properties +- ✅ **Server Type Detection**: Automatically maps Informatica resource types to ODCS server types +- ✅ **ODCS v3.1.0 Compliance**: Generates fully compliant ODCS YAML files + +## Requirements + +### Python Dependencies + +```bash +pip install requests pyyaml +``` + +**Required packages:** +- `requests` >= 2.25.0 - HTTP library for Informatica API calls +- `pyyaml` >= 5.4.0 - YAML file generation + +### Informatica Access + +- Valid Informatica CDGC instance URL +- User account with read access to assets and metadata +- Network connectivity to Informatica CDGC REST API and Identity Service endpoints + +## Configuration + +### Environment Variables + +Set the following environment variables before running the script: + +```bash +export INFORMATICA_CDGC_URL="https://cdgc.dm-us.informaticacloud.com" +export INFORMATICA_USERNAME="myuser" +export INFORMATICA_PASSWORD="mypassword" +``` + +**Alternative**: Pass credentials via command-line arguments (see Usage section) + +### Informatica Permissions Required + +The user account needs the following permissions: +- Read access to CDGC assets +- Access to asset metadata and attributes +- Access to Identity Service for authentication +- Read access to asset hierarchy (columns) + +## Usage + +### Basic Usage + +```bash +python odcs_generator/generate_odcs_from_informatica.py +``` + +**Example:** +```bash +python odcs_generator/generate_odcs_from_informatica.py 1b5fc805-252d-4ba2-bd90-e943103e411b +``` + +This generates a file named `-odcs.yaml` in the current directory. + +### Command-Line Options + +```bash +python odcs_generator/generate_odcs_from_informatica.py [OPTIONS] +``` + +**Options:** + +| Option | Description | Default | +|--------|-------------|---------| +| `asset_id` | Informatica asset ID (required) | - | +| `-o, --output` | Output YAML file path | `-odcs.yaml` | +| `--cdgc-url` | Informatica CDGC URL | `$INFORMATICA_CDGC_URL` | +| `-u, --username` | Informatica username | `$INFORMATICA_USERNAME` | +| `-p, --password` | Informatica password | `$INFORMATICA_PASSWORD` | +| `-h, --help` | Show help message | - | + +### Examples + +**Generate with custom output file:** +```bash +python odcs_generator/generate_odcs_from_informatica.py 1b5fc805-252d-4ba2-bd90-e943103e411b -o my-contract.yaml +``` + +**Pass credentials via command line:** +```bash +python odcs_generator/generate_odcs_from_informatica.py 1b5fc805-252d-4ba2-bd90-e943103e411b \ + --cdgc-url https://cdgc.dm-us.informaticacloud.com \ + -u myuser \ + -p mypassword +``` + +**Using environment variables:** +```bash +export INFORMATICA_CDGC_URL="https://cdgc.dm-us.informaticacloud.com" +export INFORMATICA_USERNAME="myuser" +export INFORMATICA_PASSWORD="mypswrd" + +python odcs_generator/generate_odcs_from_informatica.py 1b5fc805-252d-4ba2-bd90-e943103e411b +``` + +## How It Works + +### 1. Authentication + +The script performs a two-step authentication process: + +1. **Session ID**: Obtains a session ID from Identity Service using username/password +2. **JWT Token**: Exchanges session ID for a JWT token used for API calls +3. **Token Caching**: Caches the JWT token to avoid repeated authentication + +### 2. Asset Metadata Extraction + +The script fetches the following from Informatica CDGC: + +- **Asset Details**: Name, business name, type, resource type +- **System Attributes**: Schema, catalog source, row count, origin, timestamps +- **Hierarchy**: Column IDs from asset hierarchy +- **Column Details**: Data types, length, scale, precision, nullable, primary key + +### 3. Column Discovery + +Columns are discovered through the asset hierarchy: +- Extracts column IDs from the `hierarchy` array +- Fetches detailed metadata for each column concurrently (up to 10 parallel requests) +- Sorts columns by position for correct ordering +- Handles missing or incomplete column data gracefully + +### 4. Data Type Mapping + +**Physical Types** (with size/precision/scale): +- `VARCHAR(255)` +- `DECIMAL(10,2)` +- `NUMBER(18,4)` +- `CHAR(10)` +- `TIMESTAMP` + +The script automatically constructs physical types based on: +- Base data type from `com.infa.odin.models.relational.Datatype` +- Length from `com.infa.odin.models.relational.DatatypeLength` +- Scale from `com.infa.odin.models.relational.DatatypeScale` + +### 5. Server Type Detection + +Automatically maps Informatica resource types to ODCS server types: + +| Informatica Resource Type | ODCS Server Type | +|---------------------------|------------------| +| SqlServer | sqlserver | +| Oracle | oracle | +| PostgreSQL | postgresql | +| MySQL | mysql | +| Snowflake | snowflake | +| Redshift | redshift | +| BigQuery | bigquery | +| Databricks | databricks | +| Synapse | synapse | +| DB2 | db2 | +| Hive | hive | +| Impala | impala | +| Teradata | custom | + +### 6. ODCS Generation + +Creates a complete ODCS v3.1.0 structure: + +```yaml +id: +kind: DataContract +apiVersion: v3.1.0 +domain: Sample Domain +dataProduct: Sample data product +version: 1.0.0 +name: Sample contract +status: active +contractCreatedTs: +description: + authoritativeDefinitions: + - type: informatica-asset + url: +schema: + - id: + name: + physicalName: / + physicalType: Table + description: + properties: [...] +customProperties: [...] +servers: + - id: + server: CONFIGURE_SERVER_HOSTNAME # ⚠️ Manual config may be required + type: # Auto-detected from Informatica + schema: # Extracted from Informatica +``` + +### 7. Manual Configuration Comments + +The script adds inline comments to guide manual configuration where needed: + +```yaml +servers: + # ============================================ + # ⚠️ MANUAL CONFIGURATION REQUIRED + # ============================================ + # Please add/update the required server info: + - id: server-1b5fc805 + server: CONFIGURE_SERVER_HOSTNAME # ⚠️ UPDATE: e.g., prod.snowflake.acme.com + type: snowflake # Auto-detected from Informatica + schema: PUBLIC # Extracted from Informatica +``` + +## Output Structure + +### Generated ODCS File + +The script generates a YAML file with the following structure: + +```yaml +id: +kind: DataContract +apiVersion: v3.1.0 +domain: +dataProduct: +version: 1.0.0 +name: +status: active +contractCreatedTs: +description: + authoritativeDefinitions: + - type: informatica-asset + url: +schema: + - id: + name: + physicalName: / + physicalType: Table + description: + properties: + - name: + physicalType: VARCHAR(255)|DECIMAL(10,2)|... + description: + required: true|false + primaryKey: true|false +customProperties: + - property: + value: +servers: + - id: + server: CONFIGURE_SERVER_HOSTNAME + type: + schema: +``` + +### Console Output + +The script provides detailed progress information: + +``` +Fetching asset details for 1b5fc805-252d-4ba2-bd90-e943103e411b... +Fetching column details... + Fetched column 1/12... + Fetched column 2/12... + ... + Fetched column 12/12... +Generating ODCS YAML... + +Writing ODCS to customer-transactions-odcs.yaml... +✓ Successfully generated ODCS file: customer-transactions-odcs.yaml +``` + +## Informatica Attribute Mapping + +### Asset-Level Attributes + +| Informatica Attribute | ODCS Field | Notes | +|----------------------|------------|-------| +| core.name | `schema.name` | Table name | +| core.businessName | `schema.name` | Preferred if available | +| core.description | `schema.description` | Asset description | +| com.infa.odin.models.relational.Owner | `schema.physicalName`, `servers.schema` | Schema name | +| core.resourceType | `servers.type` | Mapped to ODCS type | +| core.identity | `id`, `schema.id` | Asset ID | + +### Column-Level Attributes + +| Informatica Attribute | ODCS Field | Notes | +|----------------------|------------|-------| +| core.name | `properties.name` | Column name | +| com.infa.odin.models.relational.Datatype | `properties.physicalType` | Base data type | +| com.infa.odin.models.relational.DatatypeLength | `properties.physicalType` | Appended to type | +| com.infa.odin.models.relational.DatatypeScale | `properties.physicalType` | For numeric types | +| core.description | `properties.description` | Column description | +| com.infa.odin.models.relational.Nullable | `properties.required` | Inverted boolean | +| com.infa.odin.models.relational.PrimaryKeyColumn | `properties.primaryKey` | Boolean | +| core.Position | - | Used for sorting columns | + +### System Attributes (Custom Properties) + +| Informatica Attribute | Custom Property Name | Description | +|----------------------|---------------------|-------------| +| core.resourceName | Catalog Source Name | Source catalog name | +| com.infa.odin.models.relational.NumberOfRows | Number of rows | Row count | +| core.origin | Origin | Data origin | +| com.infa.odin.models.relational.Owner | Schema | Schema name | +| core.sourceCreatedBy | Source Created By | Creator | +| core.sourceCreatedOn | Source Created On | Creation timestamp | +| core.sourceModifiedBy | Source Modified By | Last modifier | +| core.sourceModifiedOn | Source Modified On | Last modified timestamp | + +## Error Handling + +### Common Errors + +**1. Missing Environment Variables** +``` +Error: Informatica CDGC URL is required. Set INFORMATICA_CDGC_URL environment variable or use --cdgc-url +Example: --cdgc-url https://cdgc.dm-us.informaticacloud.com +``` +**Solution**: Set required environment variables or pass via command line + +**2. Authentication Failure** +``` +✗ HTTP Error: 401 + Authentication failed. Please check your credentials. +``` +**Solution**: Verify username and password are correct + +**3. Asset Not Found** +``` +✗ HTTP Error: 404 + Asset 1b5fc805-252d-4ba2-bd90-e943103e411b not found. +``` +**Solution**: Verify the asset ID exists and you have access + +**4. Connection Issues** +``` +✗ Connection Error: Unable to connect to Informatica CDGC at https://cdgc.dm-us.informaticacloud.com + Please check your network connection and CDGC URL. +``` +**Solution**: Check network connectivity and CDGC URL format + +**5. Timeout Error** +``` +✗ Timeout Error: Request timed out + The server took too long to respond. Please try again. +``` +**Solution**: Retry the request or check server status + +**6. Data Structure Error** +``` +✗ Data Error: Missing expected field 'summary' + The asset data structure may be incomplete or invalid. +``` +**Solution**: Verify the asset has complete metadata in Informatica + +**7. Asset Type Validation Error** +``` +✗ Validation Error: Asset 'MY_SCHEMA' is not a type 'Table'. This script only processes table assets. Please provide a table asset ID. +``` +**Solution**: The provided asset ID is not a table (it may be a schema, database, or other asset type). Use the Informatica catalog to find the correct table asset ID. Only table and view assets are supported. + +### Warnings + +The script may display warnings for non-critical issues: + +``` +Warning: Failed to fetch column : +``` + +These warnings indicate missing column data but don't prevent ODCS generation. The script will continue processing other columns. + +## Customization + +### Modifying Resource Type Mappings + +Edit the `RESOURCE_TYPE_MAPPING` dictionary to add or change server type mappings: + +```python +RESOURCE_TYPE_MAPPING = { + 'YourCustomType': 'custom', + # ... existing mappings +} +``` + +### Changing System Attributes + +Update the `SYSTEM_ATTRIBUTES_MAPPING` dictionary to include/exclude specific attributes: + +```python +SYSTEM_ATTRIBUTES_MAPPING = { + 'your.custom.attribute': 'Custom Property Name', + # ... existing mappings +} +``` + +## Best Practices + +1. **Verify Asset ID**: Ensure you're using the correct Informatica asset ID +2. **Check Hierarchy**: Verify columns are properly associated with tables in Informatica +3. **Review Output**: Always review the generated YAML before using +4. **Manual Configuration**: Complete server hostname if not auto-detected +5. **Validate**: Validate the YAML before finalizing + +## Troubleshooting + +### No Columns Generated + +**Problem**: Schema has no properties (columns) + +**Possible Causes**: +- Asset hierarchy is empty in Informatica +- Columns not properly associated with table +- Insufficient permissions to read column metadata + +**Solution**: +1. Check asset hierarchy in Informatica CDGC +2. Verify column associations +3. Review console output for column fetch errors +4. Check user permissions + +### Incorrect Data Types + +**Problem**: Physical types don't match expectations + +**Solution**: +1. Check Informatica attribute values for data type, length, scale +2. Verify the asset has complete metadata +3. Manually correct in generated YAML if needed + +### Missing System Attributes + +**Problem**: Custom properties are empty or incomplete + +**Solution**: +1. Verify system attributes are populated in Informatica +2. Check if attributes are available for the asset type +3. Update `SYSTEM_ATTRIBUTES_MAPPING` if using custom attributes + +### Authentication Token Expiry + +**Problem**: Script fails midway with authentication errors + +**Solution**: +1. The script caches tokens - this shouldn't happen normally +2. If it does, re-run the script (it will get a fresh token) +3. Check if your session timeout is very short + +--- + +# Common Features + +Both Collibra and Informatica integrations share these common features: + +## ODCS v3.1.0 Compliance + +Both scripts generate fully compliant ODCS v3.1.0 YAML files with: +- Required metadata fields (id, kind, apiVersion, domain, etc.) +- Schema definitions with properties +- Custom properties preservation +- Server configuration sections +- Authoritative definitions linking back to source + +## Manual Configuration Guidance + +Both scripts add helpful comments to guide manual configuration: +- Server hostname configuration +- Server type specification (when not auto-detected) +- Schema/database name configuration + +## Error Handling + +Comprehensive error handling for: +- Authentication failures +- Network connectivity issues +- Missing or invalid assets +- Incomplete metadata + +## Validation Support + +Generated YAML files can be validated using standard YAML validators and ODCS schema validators. + +2. Verify column assets have "column" in their type name +3. Review console output for relation processing messages + +### Incorrect Data Types + +**Problem**: Logical or physical types don't match expectations + +**Solution**: +1. Check Collibra attribute values for "Data Type" and "Technical Data Type" +2. Update `LOGICAL_TYPE_MAPPING` if needed +3. Manually correct in generated YAML + +### Missing Classifications + +**Problem**: Data classifications not included + +**Solution**: +1. Verify GraphQL API access +2. Check if classifications are assigned in Collibra +3. Ensure classifications have "ACCEPTED" status + + +## Related Documentation + +- [ODCS Specification v3.1.0](https://github.com/bitol-io/open-data-contract-standard) +- [Project README](../README.md) - Main project documentation +- [Build Guide](../BUILD_GUIDE.md) - Build and development instructions +- [Examples](../examples/odcs_generator_example.py) - Usage examples + +## Project Structure + +This script is part of the `data-product-python-sdk` project: + +``` +data-product-python-sdk/ +├── dph_services/ # Data Product Hub services +├── odcs_generator/ # ODCS generator module (this script) +├── examples/ # Usage examples +├── test/ # Test suites +│ ├── integration/ # Integration tests +│ └── unit/ # Unit tests +├── requirements.txt # Python dependencies +└── setup.py # Package setup +``` + +## Support + +For issues or questions: + +1. **Check Console Output**: Review error messages and warnings +2. **Verify Configuration**: Ensure environment variables are set correctly +3. **Test Connectivity**: Verify access to Collibra API +4. **Review Examples**: Check `examples/odcs_generator_example.py` for usage patterns +5. **Review Logs**: Check for detailed error information +6. **Consult Documentation**: Review this README and related docs \ No newline at end of file diff --git a/src/wxdi/odcs_generator/__init__.py b/src/wxdi/odcs_generator/__init__.py new file mode 100644 index 0000000..1593d17 --- /dev/null +++ b/src/wxdi/odcs_generator/__init__.py @@ -0,0 +1,55 @@ +# coding: utf-8 +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""ODCS Generator - Generate ODCS YAML files from Collibra and Informatica assets""" + +from .generate_odcs_from_collibra import ( + CollibraClient, + ODCSGenerator, + parse_arguments as collibra_parse_arguments, + validate_arguments as collibra_validate_arguments, + determine_output_file as collibra_determine_output_file, + write_yaml_file as collibra_write_yaml_file, + main as collibra_main +) + +from .generate_odcs_from_informatica import ( + InformaticaClient, + parse_arguments as informatica_parse_arguments, + validate_arguments as informatica_validate_arguments, + determine_output_file as informatica_determine_output_file, + write_yaml_file as informatica_write_yaml_file, + main as informatica_main +) + +__all__ = [ + # Collibra exports + 'CollibraClient', + 'ODCSGenerator', + 'collibra_parse_arguments', + 'collibra_validate_arguments', + 'collibra_determine_output_file', + 'collibra_write_yaml_file', + 'collibra_main', + # Informatica exports + 'InformaticaClient', + 'informatica_parse_arguments', + 'informatica_validate_arguments', + 'informatica_determine_output_file', + 'informatica_write_yaml_file', + 'informatica_main' +] + +__version__ = '1.0.0' diff --git a/src/wxdi/odcs_generator/generate_odcs_from_collibra.py b/src/wxdi/odcs_generator/generate_odcs_from_collibra.py new file mode 100644 index 0000000..46e9470 --- /dev/null +++ b/src/wxdi/odcs_generator/generate_odcs_from_collibra.py @@ -0,0 +1,765 @@ +#!/usr/bin/env python3 +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Generate ODCS YAML file from Collibra Asset + +This script fetches asset metadata from Collibra and generates an ODCS v3 compliant YAML file. + +Usage: + python generate_odcs_from_collibra.py + python generate_odcs_from_collibra.py 019a57f9-62d2-7aa0-9f22-4fa2cea1180b + +Environment Variables: + COLLIBRA_URL: Collibra instance URL (required) + COLLIBRA_USERNAME: Collibra username (required) + COLLIBRA_PASSWORD: Collibra password (required) +""" + +import argparse +import os +import sys +import uuid +from datetime import datetime, timezone +from typing import Any, Dict, List, Optional, Tuple + +import requests +import yaml + + +class CollibraClient: + """Client for interacting with Collibra REST API""" + + HEADERS_JSON = {"Accept": "application/json"} + HEADERS_CONTENT_JSON = {"Content-Type": "application/json"} + DEFAULT_LIMIT = 1000 + + def __init__(self, base_url: str, username: str, password: str): + self.base_url = base_url.rstrip('/') + self.auth = (username, password) + self.session = requests.Session() + self.session.auth = self.auth + + def get_asset(self, asset_id: str) -> Dict[str, Any]: + """Fetch asset details from Collibra""" + url = f"{self.base_url}/rest/2.0/assets/{asset_id}" + response = self.session.get(url, headers=self.HEADERS_JSON) + response.raise_for_status() + return response.json() + + def get_asset_attributes(self, asset_id: str) -> List[Dict[str, Any]]: + """Fetch all attributes for an asset""" + url = f"{self.base_url}/rest/2.0/attributes" + params = {'assetId': asset_id, 'limit': self.DEFAULT_LIMIT} + response = self.session.get(url, params=params, headers=self.HEADERS_JSON) + response.raise_for_status() + return response.json().get('results', []) + + def get_asset_relations(self, asset_id: str, as_source: bool = True) -> List[Dict[str, Any]]: + """Fetch all relations for an asset + + Args: + asset_id: The asset ID + as_source: If True, fetch relations where asset is source; if False, where asset is target + """ + url = f"{self.base_url}/rest/2.0/relations" + param_key = 'sourceId' if as_source else 'targetId' + params = {param_key: asset_id, 'limit': self.DEFAULT_LIMIT} + response = self.session.get(url, params=params, headers=self.HEADERS_JSON) + response.raise_for_status() + return response.json().get('results', []) + + def get_asset_tags(self, asset_id: str) -> List[str]: + """Fetch tags directly assigned to an asset""" + url = f"{self.base_url}/rest/2.0/assets/{asset_id}/tags" + try: + response = self.session.get(url, headers=self.HEADERS_JSON) + response.raise_for_status() + results = response.json() + return [tag['name'] for tag in results if 'name' in tag] + except Exception as e: + print(f"Warning: Could not fetch tags: {e}") + return [] + + def get_asset_classifications(self, asset_id: str) -> List[str]: + """Fetch data classifications for an asset using GraphQL API""" + url = f"{self.base_url}/graphql" + + query = """ + query AssetClassification($id: ID!) { + api { + asset(id: $id) { + id + classesForAsset { + id + classificationId + label + percentage + status + } + } + } + } + """ + + payload = {"query": query, "variables": {"id": asset_id}} + + try: + response = self.session.post(url, json=payload, headers=self.HEADERS_CONTENT_JSON) + response.raise_for_status() + data = response.json() + + asset_data = data.get('data', {}).get('api', {}).get('asset', {}) + classes = asset_data.get('classesForAsset', []) + + return [ + cls['label'] + for cls in classes + if cls.get('status') == 'ACCEPTED' and cls.get('label') + ] + except Exception: + return [] + +class ODCSGenerator: + """Generate ODCS YAML from Collibra asset metadata""" + + UTC_TIMEZONE_SUFFIX = '+00:00' + EXCLUDED_ATTRIBUTES = {'Description'} + LOGICAL_TYPE_MAPPING = { + 'text': 'string', + 'whole number': 'integer', + 'decimal number': 'number', + 'date time': 'timestamp', + 'string': 'string', + 'integer': 'integer', + 'number': 'number', + 'date': 'date', + 'time': 'time', + 'object': 'object', + 'array': 'array', + 'geographical': 'string', + 'true/false': 'boolean', + 'n/a': None + } + NUMERIC_TYPES = ['DECIMAL', 'NUMERIC', 'NUMBER'] + + def __init__(self, collibra_client: CollibraClient): + self.client = collibra_client + + def generate_odcs(self, asset_id: str) -> Dict[str, Any]: + """Generate ODCS structure from a single Collibra asset + + Args: + asset_id: Collibra asset ID to include in the contract + + Returns: + ODCS data contract dictionary with single asset in schema array + + Raises: + ValueError: If asset is not a table + """ + if not asset_id: + raise ValueError("Asset ID is required") + + print(f"Generating ODCS for asset: {asset_id}") + print(f"\n=== Processing asset: {asset_id} ===") + + # Fetch asset details for contract-level metadata + print("Fetching asset details...") + asset = self.client.get_asset(asset_id) + + # Validate that the asset is a table + asset_type = asset.get('type', {}).get('name', '').lower() + print(f"Asset type: {asset_type}") + + if 'table' not in asset_type: + raise ValueError( + f"Asset '{asset.get('displayName', asset_id)}' is is not a type 'Table'. " + f"This script only processes table assets. " + f"Please provide a table asset ID." + ) + + domain_name = asset.get('domain', {}).get('name', '') + asset_display_name = asset.get('displayName', asset.get('name', 'asset')) + created_date = datetime.now(timezone.utc).isoformat().replace(self.UTC_TIMEZONE_SUFFIX, 'Z') + + odcs = { + 'id': asset_id, + 'kind': 'DataContract', + 'apiVersion': 'v3.1.0', + 'domain': domain_name, + 'dataProduct': 'Sample data product', + 'version': '1.0.0', + 'name': 'Sample contract', + 'status': 'active', + 'contractCreatedTs': created_date, + 'description': { + 'authoritativeDefinitions': [{ + 'type': 'collibra-asset', + 'url': f'{self.client.base_url}/asset/{asset_id}' + }] + }, + '_asset_display_name': asset_display_name + } + + # Add tags from Collibra + collibra_tags = self.client.get_asset_tags(asset_id) + odcs['tags'] = collibra_tags if collibra_tags else [] + + # Process the asset and build schema array + result = self._process_asset(asset_id) + if result: + schema_def, server = result + odcs['schema'] = [schema_def] + odcs['servers'] = [server] + else: + odcs['schema'] = [] + odcs['servers'] = [] + + return odcs + + @staticmethod + def _convert_timestamp(timestamp_ms: int) -> str: + """Convert millisecond timestamp to ISO format""" + utc_timezone_suffix = '+00:00' + if timestamp_ms: + return datetime.fromtimestamp(timestamp_ms / 1000, tz=timezone.utc).isoformat().replace(utc_timezone_suffix, 'Z') + return datetime.now(timezone.utc).isoformat().replace(utc_timezone_suffix, 'Z') + + @staticmethod + def _build_attribute_map(attributes: List[Dict[str, Any]]) -> Dict[str, Any]: + """Build a dictionary mapping attribute type names to values""" + attr_map = {} + for attr in attributes: + attr_type_name = attr.get('type', {}).get('name', '') + attr_value = attr.get('value') + if attr_type_name and attr_value: + attr_map[attr_type_name] = attr_value + return attr_map + + + def _process_asset(self, asset_id: str) -> Optional[Tuple[Dict[str, Any], Dict[str, Any]]]: + """Process asset and return its schema definition and server + + Args: + asset_id: Collibra asset ID to process + + Returns: + Tuple of (schema_def, server) or None if processing fails + """ + try: + # Fetch asset details + print("Fetching asset details...") + asset = self.client.get_asset(asset_id) + + print("Fetching asset attributes...") + attributes = self.client.get_asset_attributes(asset_id) + + print("Fetching asset relations (as source)...") + source_relations = self.client.get_asset_relations(asset_id, as_source=True) + + print("Fetching asset relations (as target)...") + target_relations = self.client.get_asset_relations(asset_id, as_source=False) + print(f"Found {len(target_relations)} relations where asset is target") + + all_relations = source_relations + target_relations + + # Extract asset metadata + display_name = asset.get('displayName', '') + asset_type = asset.get('type', {}).get('name', '') + + + attr_map = self._build_attribute_map(attributes) + description = attr_map.get('Description', '') + + # Extract schema name from relations + schema_name = self._extract_schema_from_relations(target_relations) + + # Build physical name as schema/table_name + physical_name = f"{schema_name}/{display_name}" if schema_name else display_name + + # Create server definition with schema + server = self._create_server_definition(schema_name) + + # Create schema definition + schema_def = { + 'id': asset_id, + 'name': display_name, + 'physicalName': physical_name, + 'physicalType': 'table' if asset_type.lower() == 'table' else 'view', + 'description': description, + 'properties': [] + } + + # Add custom properties + custom_properties = self._extract_custom_properties(attr_map) + if custom_properties: + schema_def['customProperties'] = custom_properties + + # Extract and add column properties + columns_found = self._extract_columns_from_relations(all_relations) + + if columns_found: + print(f"Found {len(columns_found)} unique columns from relations") + self._add_columns_to_schema(schema_def, columns_found) + + if not schema_def['properties']: + print("Warning: No columns found in Collibra.") + + return (schema_def, server) + + except Exception as e: + print(f"Error processing asset {asset_id}: {e}") + return None + + @staticmethod + def _create_server_definition(schema_name: str = '') -> Dict[str, Any]: + """Create server definition with placeholder values and schema""" + server_id = f"server-{uuid.uuid4().hex[:8]}" + return { + 'id': server_id, + 'server': 'CONFIGURE_SERVER_HOSTNAME', + 'type': 'DEFINE_SERVER_TYPE', + 'schema': schema_name if schema_name else 'CONFIGURE_SCHEMA_NAME' + } + + def _extract_schema_from_relations(self, relations: List[Dict[str, Any]]) -> str: + """Extract schema name from relations where table is the target + + Args: + relations: List of relations where the table is the target + + Returns: + Schema name if found, empty string otherwise + """ + print("Attempting to extract schema from relations...") + for relation in relations: + try: + relation_type = relation.get('type', {}).get('name', '') + print(f" Checking relation type: {relation_type}") + + # Check source of the relation (where table is target) + source = relation.get('source', {}) + if source and source.get('id'): + source_asset = self.client.get_asset(str(source.get('id'))) + source_type = source_asset.get('type', {}).get('name', '') + source_name = source_asset.get('displayName', source_asset.get('name', '')) + + print(f" Source: name='{source_name}', type='{source_type}'") + + # Check if the source is a schema/database schema + if 'schema' in source_type.lower(): + print(f" ✓ Found schema from relation: {source_name}") + return source_name + + # Also check target of the relation (where table is source) + target = relation.get('target', {}) + if target and target.get('id'): + target_asset = self.client.get_asset(str(target.get('id'))) + target_type = target_asset.get('type', {}).get('name', '') + target_name = target_asset.get('displayName', target_asset.get('name', '')) + + print(f" Target: name='{target_name}', type='{target_type}'") + + # Check if the target is a schema/database schema + if 'schema' in target_type.lower(): + print(f" ✓ Found schema from relation: {target_name}") + return target_name + + except Exception as e: + print(f" Warning: Error extracting schema from relation: {e}") + continue + + print(" No schema found in relations") + return '' + + def _extract_custom_properties(self, attr_map: Dict[str, Any]) -> List[Dict[str, str]]: + """Extract custom properties from attribute map""" + custom_properties = [] + for attr_name, attr_value in attr_map.items(): + if attr_name not in self.EXCLUDED_ATTRIBUTES and attr_value: + custom_prop_name = attr_name.lower().replace(' ', '_') + custom_properties.append({ + 'property': custom_prop_name, + 'value': attr_value + }) + return custom_properties + + def _extract_columns_from_relations(self, relations: List[Dict[str, Any]]) -> List[Dict[str, Any]]: + """Extract column information from asset relations""" + print("Extracting column information from all relations...") + print(f"Total relations found: {len(relations)}") + + columns_found = [] + seen_column_ids = set() + seen_column_names = set() + + for relation in relations: + # Process source + self._process_relation_endpoint( + relation.get('source', {}), + 'source', + seen_column_ids, + seen_column_names, + columns_found + ) + + # Process target + self._process_relation_endpoint( + relation.get('target', {}), + 'target', + seen_column_ids, + seen_column_names, + columns_found + ) + + return columns_found + + def _process_relation_endpoint( + self, + endpoint: Dict[str, Any], + endpoint_type: str, + seen_column_ids: set, + seen_column_names: set, + columns_found: List[Dict[str, Any]] + ) -> None: + """Process a single relation endpoint (source or target)""" + if not endpoint or not endpoint.get('id'): + return + + endpoint_id = endpoint.get('id') + if not endpoint_id or endpoint_id in seen_column_ids: + return + + try: + endpoint_asset = self.client.get_asset(str(endpoint_id)) + endpoint_type_name = endpoint_asset.get('type', {}).get('name', '') + endpoint_name = endpoint_asset.get('displayName', endpoint_asset.get('name', '')) + + if 'column' in endpoint_type_name.lower(): + print(f" Found column ({endpoint_type}): name='{endpoint_name}', type='{endpoint_type_name}'") + seen_column_ids.add(endpoint_id) + + if endpoint_name not in seen_column_names: + seen_column_names.add(endpoint_name) + col_attributes = self.client.get_asset_attributes(str(endpoint_id)) + col_classifications = self.client.get_asset_classifications(str(endpoint_id)) + columns_found.append({ + 'asset': endpoint_asset, + 'attributes': col_attributes, + 'classifications': col_classifications + }) + except Exception: + pass + + def _add_columns_to_schema(self, schema_def: Dict[str, Any], columns_found: List[Dict[str, Any]]) -> None: + """Add column properties to schema definition""" + added_column_names = set() + for col_data in columns_found: + property_def = self._create_property_from_collibra( + col_data['asset'], + col_data['attributes'], + col_data.get('classifications', []) + ) + col_name = property_def.get('name') + + if col_name not in added_column_names: + added_column_names.add(col_name) + schema_def['properties'].append(property_def) + else: + print(f"Skipping duplicate property: {col_name}") + + def _create_property_from_collibra( + self, + col_asset: Dict[str, Any], + col_attributes: List[Dict[str, Any]], + classifications: Optional[List[str]] = None + ) -> Dict[str, Any]: + """Create a property (column) definition from Collibra asset, attributes, and classifications""" + col_name = col_asset.get('displayName', col_asset.get('name', 'unknown_column')) + col_id = col_asset.get('id') + + attr_map = self._build_attribute_map(col_attributes) + classifications = classifications or [] + + # Extract data types + logical_type = self._normalize_logical_type(attr_map.get('Data Type', '')) + physical_type = self._build_physical_type(attr_map) + + # Extract metadata + description = attr_map.get('Description', attr_map.get('Definition', '')) + is_nullable = self._parse_boolean_value( + attr_map.get('Is Nullable', attr_map.get('Nullable', 'true')) + ) + is_primary_key = self._parse_boolean_value( + attr_map.get('Is Primary Key', attr_map.get('Primary Key', attr_map.get('IsPrimaryKey', ''))) + ) + physical_name = attr_map.get('Original Name', attr_map.get('Physical Name', col_name)) + security_class = attr_map.get('Security Classification', attr_map.get('Classification', '')) + + # Build tags + tags = self._build_column_tags(attr_map, classifications, col_id) + + # Build property definition + prop = { + 'name': col_name, + 'physicalName': physical_name, + 'description': description, + 'required': is_nullable, + 'tags': tags + } + + # Add optional fields + if logical_type is not None: + prop['logicalType'] = logical_type + if physical_type is not None: + prop['physicalType'] = physical_type + if is_primary_key: + prop['primaryKey'] = True + if security_class: + prop['classification'] = security_class + + return prop + + def _normalize_logical_type(self, logical_type: str) -> Optional[str]: + """Normalize logical type to ODCS standard types""" + if not logical_type: + return None + + normalized = logical_type.lower().strip() + return self.LOGICAL_TYPE_MAPPING.get(normalized, logical_type) + + def _build_physical_type(self, attr_map: Dict[str, Any]) -> Optional[str]: + """Build physical type string with size/precision/scale""" + technical_data_type = attr_map.get('Technical Data Type', '') + if not technical_data_type: + return None + + base_type = technical_data_type.upper() + + # Extract size-related attributes + size = self._to_int(attr_map.get('Size', attr_map.get('Length', ''))) + precision = self._to_int(attr_map.get('Precision', '')) + scale = self._to_int(attr_map.get('Scale', attr_map.get('Number Of Fractional Digits', ''))) + + # For numeric types, use Size as precision if Precision is not set + if base_type in self.NUMERIC_TYPES and precision is None and size is not None: + precision = size + + # Build type string with parameters + if scale is not None and precision is not None: + return f"{base_type}({precision},{scale})" + elif precision is not None: + return f"{base_type}({precision})" + elif size is not None: + return f"{base_type}({size})" + else: + return base_type + + @staticmethod + def _to_int(value: Any) -> Optional[int]: + """Convert decimal string to integer, handling empty strings""" + if not value: + return None + try: + return int(float(str(value))) + except (ValueError, TypeError): + return None + + @staticmethod + def _parse_boolean_value(value: Any) -> bool: + """Parse boolean value from various formats""" + if isinstance(value, bool): + return value + return str(value).lower() in ['true', 'yes', '1', 'y'] + + def _build_column_tags( + self, + attr_map: Dict[str, Any], + classifications: List[str], + col_id: Optional[str] + ) -> List[str]: + """Build tags list for a column""" + tags = [] + + # Add PII tag if applicable + pii_value = attr_map.get('Personally Identifiable Information', attr_map.get('PII', '')) + if pii_value and self._parse_boolean_value(pii_value): + tags.append('PII') + + # Add classification tags + for classification in classifications: + tags.append(f'data_classification:{classification}') + + # Add Collibra tags + if col_id: + try: + col_tags = self.client.get_asset_tags(col_id) + if col_tags: + tags.extend(col_tags) + except Exception: + pass + + return tags + + + +def parse_arguments() -> argparse.Namespace: + """Parse command line arguments""" + parser = argparse.ArgumentParser( + description='Generate ODCS YAML file from Collibra asset', + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=""" +Examples: + Generate ODCS from asset: + python generate_odcs_from_collibra.py 019a57f9-62d2-7aa0-9f22-4fa2cea1180b + + With custom output: + python generate_odcs_from_collibra.py -o custom-output.yaml + +Environment Variables: + COLLIBRA_URL Collibra instance URL (required) + COLLIBRA_USERNAME Collibra username (required) + COLLIBRA_PASSWORD Collibra password (required) + """ + ) + + parser.add_argument('asset_id', help='Collibra asset ID') + parser.add_argument('-o', '--output', help='Output YAML file path (default: -odcs.yaml)') + parser.add_argument('--url', help='Collibra base URL', default=os.getenv('COLLIBRA_URL')) + parser.add_argument('-u', '--username', help='Collibra username', default=os.getenv('COLLIBRA_USERNAME')) + parser.add_argument('-p', '--password', help='Collibra password', default=os.getenv('COLLIBRA_PASSWORD')) + + return parser.parse_args() + + +def validate_arguments(args: argparse.Namespace) -> None: + """Validate required arguments""" + if not args.url: + print("Error: Collibra URL is required. Set COLLIBRA_URL environment variable or use --url") + sys.exit(1) + + if not args.username: + print("Error: Collibra username is required. Set COLLIBRA_USERNAME environment variable or use --username") + sys.exit(1) + + if not args.password: + print("Error: Collibra password is required. Set COLLIBRA_PASSWORD environment variable or use --password") + sys.exit(1) + + +def determine_output_file(args: argparse.Namespace, odcs_data: Dict[str, Any]) -> str: + """Determine the output file path""" + if args.output: + return args.output + + # Use asset display name for filename if available, otherwise use contract name + asset_name = odcs_data.get('_asset_display_name') or odcs_data.get('name', 'asset') + asset_name = asset_name.lower().replace(' ', '-') + return f"{asset_name}-odcs.yaml" + + +def _add_server_warning_comments(modified_lines: List[str], line: str) -> None: + """Add warning comment block before server definition""" + separator = ' # ============================================' + modified_lines.append(separator) + modified_lines.append(' # ⚠️ MANUAL CONFIGURATION REQUIRED') + modified_lines.append(separator) + modified_lines.append(' # Please add/update the required server info:') + modified_lines.append(line) + + +def _add_inline_comment_if_needed(line: str) -> str: + """Add inline comment to server configuration fields if needed""" + if ' server:' in line and 'CONFIGURE_SERVER_HOSTNAME' in line: + return line + ' # ⚠️ UPDATE: e.g., prod.snowflake.acme.com' + elif ' type:' in line and 'DEFINE_SERVER_TYPE' in line: + return line + ' # ⚠️ UPDATE: e.g., snowflake, postgres, bigquery, redshift' + elif ' schema:' in line and 'CONFIGURE_SCHEMA_NAME' in line: + return line + ' # ⚠️ UPDATE: e.g., public, dbo, my_schema' + return line + + +def write_yaml_file(output_file: str, odcs_data: Dict[str, Any]) -> None: + """Write ODCS data to YAML file with manual configuration comments""" + print(f"\nWriting ODCS to {output_file}...") + + # Remove temporary field used for filename generation + odcs_data_copy = odcs_data.copy() + odcs_data_copy.pop('_asset_display_name', None) + + # First, write the YAML normally + yaml_content = yaml.dump(odcs_data_copy, default_flow_style=False, sort_keys=False, allow_unicode=True) + + # Add manual configuration comments to the servers section + lines = yaml_content.split('\n') + modified_lines = [] + in_servers_section = False + server_block_started = False + + for line in lines: + # Detect servers section + if line.startswith('servers:'): + in_servers_section = True + modified_lines.append(line) + continue + + # Detect start of a server block (first item in servers array) + if in_servers_section and line.strip().startswith('- id:') and not server_block_started: + server_block_started = True + _add_server_warning_comments(modified_lines, line) + continue + + # Add inline comments for server and type fields + if in_servers_section and server_block_started: + line = _add_inline_comment_if_needed(line) + + modified_lines.append(line) + + # Write the modified content + with open(output_file, 'w') as f: + f.write('\n'.join(modified_lines)) + + +def main(): + """Main entry point""" + args = parse_arguments() + validate_arguments(args) + + try: + print(f"Connecting to Collibra at {args.url}...") + client = CollibraClient(args.url, args.username, args.password) + + print("Fetching asset...") + generator = ODCSGenerator(client) + odcs_data = generator.generate_odcs(args.asset_id) + + output_file = determine_output_file(args, odcs_data) + write_yaml_file(output_file, odcs_data) + + print(f"✓ Successfully generated ODCS file: {output_file}") + + except requests.exceptions.HTTPError as e: + print(f"Error: HTTP {e.response.status_code} - {e.response.text}") + sys.exit(1) + except Exception as e: + print(f"Error: {str(e)}") + sys.exit(1) + + +if __name__ == '__main__': + main() \ No newline at end of file diff --git a/src/wxdi/odcs_generator/generate_odcs_from_informatica.py b/src/wxdi/odcs_generator/generate_odcs_from_informatica.py new file mode 100644 index 0000000..a5314c8 --- /dev/null +++ b/src/wxdi/odcs_generator/generate_odcs_from_informatica.py @@ -0,0 +1,500 @@ +#!/usr/bin/env python3 +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Generate ODCS YAML file from Informatica Asset + +This script fetches asset metadata from Informatica and generates an ODCS v3 compliant YAML file. + +Usage: + python odcs_generator/generate_odcs_from_informatica.py + python odcs_generator/generate_odcs_from_informatica.py 1b5fc805-252d-4ba2-bd90-e943103e411b --cdgc-url https://cdgc.dm-us.informaticacloud.com -u username -p password + +Environment Variables: + INFORMATICA_CDGC_URL: Informatica CDGC URL (required, e.g., https://cdgc.dm-us.informaticacloud.com) + INFORMATICA_USERNAME: Informatica username (required) + INFORMATICA_PASSWORD: Informatica password (required) +""" + +import argparse +import os +import sys +from typing import Any, Dict, List, Optional +from datetime import datetime, timezone +from concurrent.futures import ThreadPoolExecutor, as_completed +import requests +import yaml + +# Constants +CORE_NAME_ATTR = 'core.name' + +# Resource type mapping from Informatica to ODCS +RESOURCE_TYPE_MAPPING = { + 'SqlServer': 'sqlserver', + 'Oracle': 'oracle', + 'PostgreSQL': 'postgresql', + 'MySQL': 'mysql', + 'Snowflake': 'snowflake', + 'Redshift': 'redshift', + 'BigQuery': 'bigquery', + 'Databricks': 'databricks', + 'Synapse': 'synapse', + 'DB2': 'db2', + 'Teradata': 'custom', + 'Hive': 'hive', + 'Impala': 'impala' +} + +# System attributes mapping for custom properties +SYSTEM_ATTRIBUTES_MAPPING = { + 'core.resourceName': 'Catalog Source Name', + 'com.infa.odin.models.relational.NumberOfRows': 'Number of rows', + 'core.origin': 'Origin', + 'com.infa.odin.models.relational.Owner': 'Schema', + 'core.sourceCreatedBy': 'Source Created By', + 'core.sourceCreatedOn': 'Source Created On', + 'core.sourceModifiedBy': 'Source Modified By', + 'core.sourceModifiedOn': 'Source Modified On' +} + +class InformaticaClient: + + CONTENT_TYPE_JSON = "application/json" + HEADERS_JSON = {"Accept": CONTENT_TYPE_JSON} + HEADERS_CONTENT_JSON = {"Content-Type": CONTENT_TYPE_JSON} + + def __init__(self, base_url, username, password): + self.base_url = base_url.rstrip('/') + self.username = username + self.password = password + self.region = self._extract_region_from_url(base_url) + self.identity_url = f"https://{self.region}.informaticacloud.com" + self._auth_token: Optional[str] = None + + def _extract_region_from_url(self, base_url: str) -> str: + """Extract region identifier from base URL. + + Examples: + https://cdgc.dm-us.informaticacloud.com -> dm-us + https://cdgc.na1.informaticacloud.com -> na1 + """ + # Remove protocol and trailing slash + url = base_url.replace('https://', '').rstrip('/') + + # Extract hostname + hostname = url.split('/')[0] + + # Remove 'cdgc.' prefix if present + if hostname.startswith('cdgc.'): + hostname = hostname[5:] + + # Extract region (everything before .informaticacloud.com) + if '.informaticacloud.com' in hostname: + region = hostname.split('.informaticacloud.com')[0] + return region + + # Fallback: return the hostname as-is + return hostname + + def get_session_id(self) -> Dict[str, Any]: + url = f"{self.identity_url}/identity-service/api/v1/Login" + payload = {"username": self.username, "password": self.password} + response = requests.post(url, json=payload, headers=self.HEADERS_CONTENT_JSON) + response.raise_for_status() + return response.json() + + def get_auth_token(self) -> str: + """Get authentication token with caching to avoid repeated auth calls.""" + if self._auth_token is None: + session_id = self.get_session_id()["sessionId"] + url = f"{self.identity_url}/identity-service/api/v1/jwt/Token?client_id=idmc_api&nonce=1234" + response = requests.post(url, headers={"Accept": self.CONTENT_TYPE_JSON, "IDS-SESSION-ID": session_id, "Cookie": f"USER_SESSION={session_id}"}) + response.raise_for_status() + self._auth_token = response.json()["jwt_token"] + return str(self._auth_token) + + def _fetch_asset(self, asset_id: str) -> Dict[str, Any]: + """Fetch asset data from Informatica API.""" + auth_token = self.get_auth_token() + url = f"{self.base_url}/data360/search/v1/assets/{asset_id}?scheme=internal&segments=all" + response = requests.get(url, headers={"Authorization": f"Bearer {auth_token}"}) + response.raise_for_status() + return response.json() + + def get_asset_details(self, asset_id: str) -> Dict[str, Any]: + """Fetch asset details including table and column information.""" + return self._fetch_asset(asset_id) + + def validate_asset_is_table(self, asset_data: Dict[str, Any]) -> None: + """Validate that the asset is a table and not a schema or other type. + + Args: + asset_data: The asset data returned from Informatica API + + Raises: + ValueError: If the asset is not a table + """ + # Check the asset class type from systemAttributes (correct location) + asset_class = asset_data.get('systemAttributes', {}).get('core.classType', '') + + # Valid table class types in Informatica + valid_table_types = [ + 'com.infa.odin.models.relational.Table', + 'com.infa.odin.models.relational.View' + ] + + if asset_class not in valid_table_types: + asset_name = asset_data.get('summary', {}).get(CORE_NAME_ATTR, 'unknown') + raise ValueError( + f"Asset '{asset_name}' is not a type 'Table'. " + f"This script only processes table assets. Please provide a table asset ID." + ) + + def get_column_details(self, column_id: str) -> Dict[str, Any]: + """Fetch detailed information for a specific column.""" + return self._fetch_asset(column_id) + +def parse_arguments(): + parser = argparse.ArgumentParser( + description='Generate ODCS YAML file from Informatica asset', + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog="") + + parser.add_argument('asset_id', help='Informatica asset ID') + parser.add_argument('-o', '--output', help='Output YAML file path (default: -odcs.yaml)') + parser.add_argument('--cdgc-url', help='Informatica CDGC URL (e.g., https://cdgc.dm-us.informaticacloud.com)', + default=os.getenv('INFORMATICA_CDGC_URL')) + parser.add_argument('-u', '--username', help='Informatica username', default=os.getenv('INFORMATICA_USERNAME')) + parser.add_argument('-p', '--password', help='Informatica password', default=os.getenv('INFORMATICA_PASSWORD')) + return parser.parse_args() + +def validate_arguments(args): + if not args.cdgc_url: + print("Error: Informatica CDGC URL is required. Set INFORMATICA_CDGC_URL environment variable or use --cdgc-url") + print("Example: --cdgc-url https://cdgc.dm-us.informaticacloud.com") + sys.exit(1) + + if not args.username: + print("Error: Informatica username is required. Set INFORMATICA_USERNAME environment variable or use --username") + sys.exit(1) + + if not args.password: + print("Error: Informatica password is required. Set INFORMATICA_PASSWORD environment variable or use --password") + sys.exit(1) + +def extract_column_position(col_data: Dict[str, Any]) -> int: + """Extract column position from column data.""" + self_attrs = col_data.get('selfAttributes', {}) + position = self_attrs.get('core.Position') + + try: + return int(position) if position is not None else 999999 + except (ValueError, TypeError): + return 999999 + + +def build_physical_type(datatype: str, datatype_length: str, datatype_scale: str) -> str: + """Build physical type string with length/scale if available.""" + physical_type = datatype + if datatype_length: + if datatype_scale and datatype_scale != '0': + physical_type = f"{datatype}({datatype_length},{datatype_scale})" + else: + physical_type = f"{datatype}({datatype_length})" + return physical_type + + +def build_column_property(column_detail: Dict[str, Any]) -> Dict[str, Any]: + """Build a single column property from column detail data.""" + col_summary = column_detail.get('summary', {}) + col_self_attrs = column_detail.get('selfAttributes', {}) + + col_name = col_summary.get(CORE_NAME_ATTR) + datatype = col_self_attrs.get('com.infa.odin.models.relational.Datatype') + datatype_length = col_self_attrs.get('com.infa.odin.models.relational.DatatypeLength', '') + datatype_scale = col_self_attrs.get('com.infa.odin.models.relational.DatatypeScale', '') + + physical_type = build_physical_type(datatype, datatype_length, datatype_scale) + col_description = col_summary.get('core.description', '') + + prop = { + 'name': col_name, + 'physicalType': physical_type, + } + + if col_description: + prop['description'] = col_description + + # Check if column is nullable + nullable_str = col_self_attrs.get('com.infa.odin.models.relational.Nullable', 'true') + is_nullable = nullable_str.lower() == 'true' + prop['required'] = not is_nullable + + # Check if column is primary key + pk_str = col_self_attrs.get('com.infa.odin.models.relational.PrimaryKeyColumn', 'false') + is_primary_key = pk_str.lower() == 'true' + if is_primary_key: + prop['primaryKey'] = True + + return prop + + +def build_custom_properties(table_self_attrs: Dict[str, Any]) -> List[Dict[str, str]]: + """Build custom properties list from table attributes.""" + custom_properties = [] + + for key, ui_field_name in SYSTEM_ATTRIBUTES_MAPPING.items(): + value = table_self_attrs.get(key) + if value: + custom_properties.append({ + 'property': ui_field_name, + 'value': value + }) + + return custom_properties + + +def generate_odcs_yaml(asset_data: Dict[str, Any], column_details: List[Dict[str, Any]], base_url: str) -> Dict[str, Any]: + """Generate ODCS YAML structure from Informatica asset data.""" + + # Extract table information + table_id = asset_data['core.identity'] + name = asset_data['summary'][CORE_NAME_ATTR] + + # Extract actual table name from selfAttributes (physical table name) + table_self_attrs = asset_data.get('selfAttributes', {}) + table_name = table_self_attrs.get(CORE_NAME_ATTR, name) + + # Extract schema name for physical name construction + schema_name = table_self_attrs.get('com.infa.odin.models.relational.Owner', '') + + # Build physical name as schema/table_name + physical_name = f"{schema_name}/{table_name}" if schema_name else table_name + + created_ts = datetime.now(timezone.utc).isoformat().replace('+00:00', 'Z') + + # Extract table description if available + table_description = asset_data.get('summary', {}).get('core.description') + + # Extract resource information for server configuration + resource_type = table_self_attrs.get('core.resourceType', '') + + # Build custom properties using helper function + custom_properties = build_custom_properties(table_self_attrs) + + # Map resource type to ODCS server type + server_type = RESOURCE_TYPE_MAPPING.get(resource_type) + + # Build properties from column details using helper function + properties = [build_column_property(column_detail) for column_detail in column_details] + + # Build schema entry + schema_entry = { + 'id': table_id, + 'name': table_name, + 'physicalName': physical_name, + 'physicalType': 'Table', + 'properties': properties + } + + if table_description: + schema_entry['description'] = table_description + + # Build ODCS structure + odcs = { + 'id': table_id, + 'kind': 'DataContract', + 'apiVersion': 'v3.1.0', + 'domain': 'Sample Domain', + 'dataProduct': 'Sample data product', + 'version': '1.0.0', + 'name': 'Sample contract', + 'status': 'active', + 'contractCreatedTs': created_ts, + 'description': { + 'authoritativeDefinitions': [ + { + 'type': 'informatica-asset', + 'url': f"{base_url}/asset/{table_id}" + } + ] + }, + 'schema': [schema_entry], + 'customProperties': custom_properties, + 'servers': [ + { + 'id': 'server-' + table_id[:8], + 'server': 'CONFIGURE_SERVER_HOSTNAME', + 'type': server_type if server_type else 'CONFIGURE_SERVER_TYPE', + 'schema': schema_name if schema_name else 'CONFIGURE_SCHEMA_NAME' + } + ] + } + + return odcs + +def _add_server_warning_comments(modified_lines: List[str], line: str) -> None: + """Add warning comment block before server definition""" + separator = ' # ============================================' + modified_lines.append(separator) + modified_lines.append(' # ⚠️ MANUAL CONFIGURATION REQUIRED') + modified_lines.append(separator) + modified_lines.append(' # Please add/update the required server info:') + modified_lines.append(line) + + +def _add_inline_comment_if_needed(line: str) -> str: + """Add inline comment to server configuration fields if needed""" + if ' server:' in line and 'CONFIGURE_SERVER_HOSTNAME' in line: + return line + ' # ⚠️ UPDATE: e.g., prod.snowflake.acme.com' + elif ' type:' in line and 'CONFIGURE_SERVER_TYPE' in line: + return line + ' # ⚠️ UPDATE: e.g., snowflake, postgres, bigquery, redshift' + elif ' schema:' in line and 'CONFIGURE_SCHEMA_NAME' in line: + return line + ' # ⚠️ UPDATE: e.g., public, dbo, my_schema' + return line + + +def determine_output_file(args, asset_data: Dict[str, Any]) -> str: + """Determine the output file path using the asset name""" + if args.output: + return args.output + + # Try to get name from ODCS data first (for generated ODCS), then from asset summary + asset_name = asset_data.get('name') or asset_data.get('summary', {}).get(CORE_NAME_ATTR, 'asset') + # Sanitize the name for use as a filename + file_name = asset_name.lower().replace(' ', '-') + return f"{file_name}-odcs.yaml" + + +def write_yaml_file(output_file: str, odcs_data: Dict[str, Any]) -> None: + """Write ODCS data to YAML file with manual configuration comments""" + print(f"\nWriting ODCS to {output_file}...") + + # First, write the YAML normally + yaml_content = yaml.dump(odcs_data, default_flow_style=False, sort_keys=False, allow_unicode=True) + + # Add manual configuration comments to the servers section + lines = yaml_content.split('\n') + modified_lines = [] + in_servers_section = False + server_block_started = False + + for line in lines: + # Detect servers section + if line.startswith('servers:'): + in_servers_section = True + modified_lines.append(line) + continue + + # Detect start of a server block (first item in servers array) + if in_servers_section and line.strip().startswith('- id:') and not server_block_started: + server_block_started = True + _add_server_warning_comments(modified_lines, line) + continue + + # Add inline comments for server and type fields + if in_servers_section and server_block_started: + line = _add_inline_comment_if_needed(line) + + modified_lines.append(line) + + # Write the modified content + with open(output_file, 'w') as f: + f.write('\n'.join(modified_lines)) + + +def main(): + args = parse_arguments() + validate_arguments(args) + + try: + print(f"Fetching asset details for {args.asset_id}...") + client = InformaticaClient(args.cdgc_url, args.username, args.password) + + # Fetch table asset details + asset_data = client.get_asset_details(args.asset_id) + + # Validate that the asset is a table, not a schema or other type + client.validate_asset_is_table(asset_data) + + # Get column IDs from hierarchy + column_ids = [col['core.identity'] for col in asset_data.get('hierarchy', [])] + + print("Fetching column details...") + + # Fetch details for each column concurrently for better performance + column_details = [] + with ThreadPoolExecutor(max_workers=10) as executor: + # Submit all column detail requests + future_to_col_id = { + executor.submit(client.get_column_details, col_id): col_id + for col_id in column_ids + } + + # Collect results as they complete + completed = 0 + for future in as_completed(future_to_col_id): + col_id = future_to_col_id[future] + try: + col_data = future.result() + column_details.append(col_data) + completed += 1 + print(f" Fetched column {completed}/{len(column_ids)}...") + except Exception as e: + print(f" Warning: Failed to fetch column {col_id}: {e}") + + column_details.sort(key=extract_column_position) + + print("Generating ODCS YAML...") + odcs_data = generate_odcs_yaml(asset_data, column_details, client.base_url) + + output_file = determine_output_file(args, asset_data) + write_yaml_file(output_file, odcs_data) + + print(f"✓ Successfully generated ODCS file: {output_file}") + + except ValueError as e: + print(f"\n✗ Validation Error: {e}") + sys.exit(1) + except requests.exceptions.HTTPError as e: + print(f"\n✗ HTTP Error: {e}") + if e.response.status_code == 401: + print(" Authentication failed. Please check your credentials.") + elif e.response.status_code == 404: + print(f" Asset {args.asset_id} not found.") + else: + print(f" Status code: {e.response.status_code}") + sys.exit(1) + except requests.exceptions.ConnectionError: + print(f"\n✗ Connection Error: Unable to connect to Informatica CDGC at {args.cdgc_url}") + print(" Please check your network connection and CDGC URL.") + sys.exit(1) + except requests.exceptions.Timeout: + print("\n✗ Timeout Error: Request timed out") + print(" The server took too long to respond. Please try again.") + sys.exit(1) + except KeyError as e: + print(f"\n✗ Data Error: Missing expected field {e}") + print(" The asset data structure may be incomplete or invalid.") + sys.exit(1) + except Exception as e: + print(f"\n✗ Unexpected Error: {e}") + print(" An unexpected error occurred. Please check your inputs and try again.") + sys.exit(1) + +if __name__ == '__main__': + main() diff --git a/tests/data/data_asset_response.json b/tests/data/data_asset_response.json new file mode 100644 index 0000000..b56b376 --- /dev/null +++ b/tests/data/data_asset_response.json @@ -0,0 +1,739 @@ +{ + "metadata": { + "project_id": "72d21c1d-499b-4784-a3c7-6f84507f9a20", + "sandbox_id": "72d21c1d-499b-4784-a3c7-6f84507f9a20", + "usage": { + "last_updated_at": "2026-01-27T05:50:24Z", + "last_updater_id": "1000330999", + "last_update_time": 1769493024580, + "last_accessed_at": "2026-01-27T05:50:24Z", + "last_access_time": 1769493024580, + "last_accessor_id": "1000330999", + "access_count": 0 + }, + "rov": { + "mode": 0, + "collaborator_ids": {}, + "member_roles": { + "1000330999": { + "user_iam_id": "1000330999", + "roles": [ + "OWNER" + ] + } + } + }, + "is_linked_with_sub_container": false, + "name": "DEPARTMENT", + "tags": [ + "connected-data" + ], + "asset_type": "data_asset", + "origin_country": "none", + "resource_key": "0000:0000:0000:0000:0000:FFFF:091E:D6B4|50000|sample:/DB2INST1/DEPARTMENT", + "rating": 0, + "total_ratings": 0, + "catalog_id": "92223cde-d54c-49c6-8476-af53a4c1d7b6", + "created": 1766630683099, + "created_at": "2025-12-25T02:44:43Z", + "owner_id": "1000330999", + "size": 0, + "version": 2, + "asset_state": "available", + "asset_attributes": [ + "data_asset", + "discovered_asset", + "metadata_enrichment_area_info", + "metadata_enrichment_info", + "data_profile", + "key_analyses", + "column_info", + "dataview_visualization", + "asset_data_quality_constraint" + ], + "asset_id": "6862f3ba-81f5-4122-8286-62bb4c5d6543", + "asset_category": "USER", + "creator_id": "1000330999", + "is_branched": true, + "is_managed_asset": false + }, + "entity": { + "data_asset": { + "columns": [ + { + "name": "DEPTNO", + "type": { + "type": "char", + "scale": 0, + "length": 3, + "signed": false, + "nullable": false, + "native_type": "CHAR" + }, + "columnProperties": {} + }, + { + "name": "DEPTNAME", + "type": { + "type": "varchar", + "scale": 0, + "length": 36, + "signed": false, + "nullable": false, + "native_type": "VARCHAR" + }, + "columnProperties": {} + }, + { + "name": "MGRNO", + "type": { + "type": "char", + "scale": 0, + "length": 6, + "signed": false, + "nullable": true, + "native_type": "CHAR" + }, + "columnProperties": {} + }, + { + "name": "ADMRDEPT", + "type": { + "type": "char", + "scale": 0, + "length": 3, + "signed": false, + "nullable": false, + "native_type": "CHAR" + }, + "columnProperties": {} + }, + { + "name": "LOCATION", + "type": { + "type": "char", + "scale": 0, + "length": 16, + "signed": false, + "nullable": true, + "native_type": "CHAR" + }, + "columnProperties": {} + } + ], + "dataset": true, + "mime_type": "application/x-ibm-rel-table", + "properties": [ + { + "name": "schema_name", + "value": "DB2INST1" + }, + { + "name": "table_name", + "value": "DEPARTMENT" + } + ] + }, + "column_info": { + "MGRNO": { + "data_class": { + "selected_data_class": { + "id": "U", + "name": "NoClassDetected", + "setByUser": false + } + }, + "column_checks": [ + { + "check": [ + { + "name": "formats", + "list_value": [ + "999999" + ] + } + ], + "origin": [], + "metadata": { + "type": "format", + "hidden": false, + "check_id": "977d185d-56db-4145-bbe3-fee2d5ee60c5", + "confirmed": false, + "dimension": "Validity", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether values have the required format.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + }, + { + "check": [ + { + "name": "unique", + "boolean_value": true + } + ], + "origin": [], + "metadata": { + "type": "uniqueness", + "hidden": false, + "check_id": "64abfcbf-80b9-40c8-8ff1-2e225589eac6", + "confirmed": false, + "dimension": "Uniqueness", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether values are unique.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + }, + { + "check": [ + { + "name": "range_type", + "value": "number" + }, + { + "name": "min", + "numeric_value": 0 + } + ], + "origin": [], + "metadata": { + "type": "range", + "hidden": false, + "check_id": "40a901eb-531a-4904-85f3-fc3fabbf3030", + "confirmed": false, + "dimension": "Validity", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether values are within the allowed range.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + } + ], + "inferred_type": { + "type": "INT8", + "scale": 0, + "length": 6, + "precision": 3, + "display_name": "tinyint" + }, + "rejected_checks": [], + "suggested_checks": [] + }, + "DEPTNO": { + "data_class": { + "suggested_classes": [ + { + "id": "588b9c94-34d4-411f-89b6-1d74927b0aef_fa8e90fa-d34b-5a19-a232-6c0557973af5", + "name": "ICD-10", + "confidence": 1 + }, + { + "id": "588b9c94-34d4-411f-89b6-1d74927b0aef_0b193af8-a07d-5964-80c1-d6b3273947e8", + "name": "Identifier", + "confidence": 1 + } + ], + "selected_data_class": { + "id": "588b9c94-34d4-411f-89b6-1d74927b0aef_fa8e90fa-d34b-5a19-a232-6c0557973af5", + "name": "ICD-10", + "setByUser": false, + "confidence": 1 + } + }, + "column_checks": [ + { + "check": [ + { + "name": "formats", + "list_value": [ + "A99" + ] + } + ], + "origin": [], + "metadata": { + "type": "format", + "hidden": false, + "check_id": "d50e784b-de8e-4f88-a121-d76f93ccd6b1", + "confirmed": false, + "dimension": "Validity", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether values have the required format.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + }, + { + "check": [ + { + "name": "unique", + "boolean_value": true + } + ], + "origin": [], + "metadata": { + "type": "uniqueness", + "hidden": false, + "check_id": "5a7f7f77-5ce1-4584-8929-1a7fc2d09f22", + "confirmed": false, + "dimension": "Uniqueness", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether values are unique.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + }, + { + "check": [ + { + "name": "missing_values_allowed", + "boolean_value": false + } + ], + "origin": [], + "metadata": { + "type": "completeness", + "hidden": false, + "check_id": "21641493-0233-4731-ab78-c8ad08d4cda5", + "confirmed": false, + "dimension": "Completeness", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether all rows have a value in this column.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + }, + { + "check": [ + { + "name": "data_class", + "value": "588b9c94-34d4-411f-89b6-1d74927b0aef_fa8e90fa-d34b-5a19-a232-6c0557973af5" + }, + { + "name": "data_class_name", + "value": "ICD-10" + } + ], + "origin": [], + "metadata": { + "type": "data_class", + "hidden": false, + "check_id": "816e3722-ae12-4040-910e-8d4f05c2684b", + "confirmed": false, + "dimension": "Validity", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether values match a specific data class.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + } + ], + "inferred_type": { + "type": "STRING", + "scale": 0, + "length": 3, + "precision": 0, + "display_name": "varchar(3)" + }, + "rejected_checks": [], + "suggested_checks": [] + }, + "ADMRDEPT": { + "data_class": { + "suggested_classes": [ + { + "id": "588b9c94-34d4-411f-89b6-1d74927b0aef_fa8e90fa-d34b-5a19-a232-6c0557973af5", + "name": "ICD-10", + "confidence": 1 + }, + { + "id": "588b9c94-34d4-411f-89b6-1d74927b0aef_9f81abbf-b98f-5555-91ed-883195e1823d", + "name": "Code", + "confidence": 1 + } + ], + "selected_data_class": { + "id": "588b9c94-34d4-411f-89b6-1d74927b0aef_fa8e90fa-d34b-5a19-a232-6c0557973af5", + "name": "ICD-10", + "setByUser": false, + "confidence": 1 + } + }, + "column_checks": [ + { + "check": [ + { + "name": "formats", + "list_value": [ + "A99" + ] + } + ], + "origin": [], + "metadata": { + "type": "format", + "hidden": false, + "check_id": "7224564a-a403-4c11-a386-52ac1fb8d6ca", + "confirmed": false, + "dimension": "Validity", + "created_at": "2026-01-27T05:50:23.284Z", + "description": "Check whether values have the required format.", + "modified_at": "2026-01-27T05:50:23.284Z", + "origin_type": "profiling/result" + } + }, + { + "check": [ + { + "name": "missing_values_allowed", + "boolean_value": false + } + ], + "origin": [], + "metadata": { + "type": "completeness", + "hidden": false, + "check_id": "97f7c29a-1c06-47b3-90d2-29c9f9acc01f", + "confirmed": false, + "dimension": "Completeness", + "created_at": "2026-01-27T05:50:23.284Z", + "description": "Check whether all rows have a value in this column.", + "modified_at": "2026-01-27T05:50:23.284Z", + "origin_type": "profiling/result" + } + }, + { + "check": [ + { + "name": "data_class", + "value": "588b9c94-34d4-411f-89b6-1d74927b0aef_fa8e90fa-d34b-5a19-a232-6c0557973af5" + }, + { + "name": "data_class_name", + "value": "ICD-10" + } + ], + "origin": [], + "metadata": { + "type": "data_class", + "hidden": false, + "check_id": "1bcaf011-8d50-40c9-9235-f4da7ed25639", + "confirmed": false, + "dimension": "Validity", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether values match a specific data class.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + } + ], + "inferred_type": { + "type": "STRING", + "scale": 0, + "length": 3, + "precision": 0, + "display_name": "varchar(3)" + }, + "rejected_checks": [], + "suggested_checks": [] + }, + "DEPTNAME": { + "data_class": { + "suggested_classes": [ + { + "id": "588b9c94-34d4-411f-89b6-1d74927b0aef_ab084a02-8cd4-57db-8035-96fe65aecaac", + "name": "Text", + "confidence": 1 + } + ], + "selected_data_class": { + "id": "588b9c94-34d4-411f-89b6-1d74927b0aef_ab084a02-8cd4-57db-8035-96fe65aecaac", + "name": "Text", + "setByUser": false, + "confidence": 1 + } + }, + "column_checks": [ + { + "check": [ + { + "name": "unique", + "boolean_value": true + } + ], + "origin": [], + "metadata": { + "type": "uniqueness", + "hidden": false, + "check_id": "604e4736-cd14-40f8-b36d-feeab6e72c04", + "confirmed": false, + "dimension": "Uniqueness", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether values are unique.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + }, + { + "check": [ + { + "name": "missing_values_allowed", + "boolean_value": false + } + ], + "origin": [], + "metadata": { + "type": "completeness", + "hidden": false, + "check_id": "391c525a-ecd6-467b-82c5-779ca5a308dd", + "confirmed": false, + "dimension": "Completeness", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether all rows have a value in this column.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + }, + { + "check": [ + { + "name": "case_type", + "value": "UpperCase" + } + ], + "origin": [], + "metadata": { + "type": "case", + "hidden": false, + "check_id": "404268af-86fc-49c6-bfee-4560c0105b51", + "confirmed": false, + "dimension": "Consistency", + "created_at": "2026-01-27T05:50:23.320Z", + "description": "Check whether values match the selected capitalization style.", + "modified_at": "2026-01-27T05:50:23.320Z", + "origin_type": "profiling/result" + } + } + ], + "inferred_type": { + "type": "STRING", + "scale": 0, + "length": 28, + "precision": 0, + "display_name": "varchar(28)" + }, + "rejected_checks": [], + "suggested_checks": [] + }, + "LOCATION": { + "data_class": { + "selected_data_class": { + "id": "U", + "name": "NoClassDetected", + "setByUser": false + } + }, + "column_checks": [], + "rejected_checks": [], + "suggested_checks": [] + } + }, + "data_profile": { + "attribute_classes": [ + "ICD-10", + "Identifier", + "Text", + "Code" + ], + "e092d07b-60ac-4fb1-9168-ef57fc886008": { + "href": "https://internal-nginx-svc.wkc.svc:12443/v2/data_profiles/e092d07b-60ac-4fb1-9168-ef57fc886008?project_id=72d21c1d-499b-4784-a3c7-6f84507f9a20&dataset_id=6862f3ba-81f5-4122-8286-62bb4c5d6543", + "entity": { + "columns": [], + "data_profile": { + "logs": [], + "columns": [], + "options": { + "enable_dqa": false, + "sample_type": "SEQUENTIAL", + "max_row_count": 1000, + "min_row_count": 0, + "disable_profiling": false, + "max_rows_per_batch": 10000, + "collect_data_values": false, + "key_analysis_options": { + "analysis_id": "6fc5fd21-c74c-4f30-9f86-c55dc8af1535", + "min_confidence": 0.8 + }, + "max_columns_per_task": 250, + "uniqueness_threshold": 0.95, + "max_distribution_size": 100, + "nullability_threshold": 0.05, + "classification_options": { + "disabled": false, + "ibm_class_codes": [], + "custom_class_codes": [], + "use_all_ibm_classes": true, + "use_all_custom_classes": true + }, + "max_numeric_stats_bins": 100, + "collect_historical_data": true, + "historical_retention_days": 30, + "enable_fast_classification": true, + "use_approximate_record_count": true, + "data_classification_confidence_threshold": 0.75, + "min_data_classification_confidence_threshold": 0.25 + }, + "execution": { + "bulk": true, + "run_by": "999", + "status": "finished", + "hb_task_id": "6f2379cc-d8aa-4713-b43c-6e61434a5968", + "create_time": "2026-01-27T05:49:34.034Z", + "is_supported": true, + "completed_date": "2026-01-27T05:50:20.020Z" + }, + "failed_assets": [] + } + }, + "metadata": { + "url": "https://internal-nginx-svc.wkc.svc:12443/v2/data_profiles/048b9a79-8161-40bb-9545-5ce955db28d7?project_id=72d21c1d-499b-4784-a3c7-6f84507f9a20&dataset_id=6862f3ba-81f5-4122-8286-62bb4c5d6543", + "guid": "e092d07b-60ac-4fb1-9168-ef57fc886008", + "owner": "admin", + "asset_id": "e092d07b-60ac-4fb1-9168-ef57fc886008", + "owner_id": "1000330999", + "created_at": "2026-01-27T05:49:33.033Z", + "dataset_id": "6862f3ba-81f5-4122-8286-62bb4c5d6543", + "project_id": "72d21c1d-499b-4784-a3c7-6f84507f9a20", + "accessed_at": "2026-01-27T05:49:33.033Z", + "dataset_ids": [], + "last_updater_id": "1000330999" + }, + "record_info": { + "computed": true, + "approximated": true, + "number_of_records": 14 + } + } + }, + "key_analyses": { + "fk_defined": 0, + "pk_defined": 0, + "fk_assigned": 0, + "pk_assigned": 0, + "fk_suggested": 0, + "pk_suggested": 2, + "primary_keys": [], + "fk_defined_as_pk": 0, + "overlap_assigned": 0, + "fk_assigned_as_pk": 0, + "overlap_suggested": 0, + "fk_suggested_as_pk": 0, + "key_analysis_area_id": "6fc5fd21-c74c-4f30-9f86-c55dc8af1535" + }, + "discovered_asset": { + "extended_metadata": [ + { + "name": "table_type", + "value": "TABLE" + } + ] + }, + "dataview_visualization": { + "jobs": { + "3f5c41cd-ea04-4eec-bf6b-6c4281e8901b": { + "job_type": "profile", + "execution": { + "status": "completed", + "last_updated_at": "2026-01-26T05:55:49.391629198Z", + "last_update_time": 1769406949391 + }, + "resultAttachmentNames": [ + "viz_job_3f5c41cd-ea04-4eec-bf6b-6c4281e8901b_0.json" + ] + } + } + }, + "metadata_enrichment_info": { + "MDE_instrumented": true + }, + "asset_data_quality_constraint": { + "asset_checks": [], + "rejected_checks": [], + "suggested_checks": [] + }, + "metadata_enrichment_area_info": { + "job_id": "8e10bfa2-ce48-4eb0-a6fa-241faf519502", + "area_id": "ed95a40c-d97b-4de4-a10a-eea2f425c8ff", + "added_date": 1766630744611, + "job_run_id": "087dcf60-d61c-44b3-b8d4-b61912301efc", + "last_enrichment_status": "finished", + "last_enrichment_timestamp": 1769493024511 + } + }, + "attachments": [ + { + "id": "3d11a882-968a-47c3-b854-28cc6ce689da", + "version": 2, + "asset_type": "data_asset", + "name": "DEPARTMENT", + "description": "remote", + "mime": "application/x-ibm-rel-table", + "connection_id": "8f2948af-9bb6-4bae-93b1-ab986a5a744c", + "connection_path": "/DB2INST1/DEPARTMENT", + "datasource_type": "8c1a4480-1c29-4b33-9086-9cb799d7b157", + "creator_id": "1000330999", + "create_time": 1766630683187, + "size": 0, + "is_remote": true, + "is_managed": false, + "is_referenced": false, + "is_object_key_read_only": false, + "is_user_provided_path_key": true, + "transfer_complete": true, + "is_partitioned": false, + "complete_time_ticks": 1766630683187, + "user_data": {}, + "test_doc": 0, + "usage": { + "access_count": 0, + "last_access_time": 1766630683187, + "last_accessor_id": "1000330999" + } + }, + { + "id": "f934e42c-986f-4f8c-85ae-5ac389c7f393", + "version": 2, + "asset_type": "dataview_visualization", + "name": "viz_job_3f5c41cd-ea04-4eec-bf6b-6c4281e8901b_0.json", + "mime": "application/json", + "datasource_type": "81bafdbd-b7c6-45c5-a4fd-6ec135f66f4e", + "creator_id": "1000330999", + "create_time": 1769406947754, + "size": 0, + "is_remote": false, + "is_managed": true, + "is_referenced": false, + "is_object_key_read_only": false, + "is_user_provided_path_key": false, + "transfer_complete": false, + "is_partitioned": false, + "complete_time_ticks": 0, + "user_data": {}, + "test_doc": 0, + "handle": { + "type": "assetfiles", + "key": "92223cde-d54c-49c6-8476-af53a4c1d7b6/6862f3ba-81f5-4122-8286-62bb4c5d6543/f934e42c-986f-4f8c-85ae-5ac389c7f393", + "upload_id": "aabacef5-ef7e-4bf3-aa6c-0f180bfa50de", + "max_part_num": 1 + }, + "usage": { + "access_count": 0, + "last_access_time": 1769406947754, + "last_accessor_id": "1000330999" + } + } + ], + "href": "/v2/assets/6862f3ba-81f5-4122-8286-62bb4c5d6543?project_id=72d21c1d-499b-4784-a3c7-6f84507f9a20" +} \ No newline at end of file diff --git a/tests/data/term_draft.json b/tests/data/term_draft.json new file mode 100644 index 0000000..96b1127 --- /dev/null +++ b/tests/data/term_draft.json @@ -0,0 +1,54 @@ +{ + "metadata": { + "artifact_type": "glossary_term", + "artifact_id": "30d1b847-0aa9-4840-a182-dd157fe977a0", + "version_id": "bdeef8cc-d9ab-4822-b3df-cef82b4de538_0", + "source_repository_id": "1be893e3-d512-4fd8-b449-c953abeedb6c", + "global_id": "1be893e3-d512-4fd8-b449-c953abeedb6c_30d1b847-0aa9-4840-a182-dd157fe977a0", + "workflow_id": "ce7f65d2-1e18-11f1-ac95-0a580afe2048", + "draft_mode": "import_create", + "is_target_draft": false, + "created_by": "1000330999", + "created_at": "2026-03-12T13:38:49.067Z", + "modified_by": "1000330999", + "modified_at": "2026-03-12T13:38:49.067Z", + "revision": "1", + "name": "mango", + "state": "DRAFT_HISTORY", + "tags": [], + "steward_ids": [], + "steward_group_ids": [], + "workflow_state": "Not started", + "read_only": false + }, + "entity": { + "abbreviations": [], + "state": "DRAFT_HISTORY", + "default_locale_id": "en-US", + "reference_copy": false, + "steward_ids": [], + "steward_group_ids": [], + "custom_attributes": [ + { + "custom_attribute_definition_id": "3d4ca4dd-8853-4575-b88b-adacf5423aba", + "name": "Data Type", + "values": [], + "description": "Indicates what is the specific data type associated with a particular property term.", + "read_only": false, + "searchable": false, + "required": false, + "type": "ENUM" + }, + { + "custom_attribute_definition_id": "8798e8a3-c495-4d1f-973a-b7f9c9d2e25a", + "name": "Cardinality", + "values": [], + "description": "Indicates what is the cardinality relating to an instance of a property or relationship term.", + "read_only": false, + "searchable": false, + "required": false, + "type": "ENUM" + } + ] + } +} \ No newline at end of file diff --git a/tests/data/term_latest_version.json b/tests/data/term_latest_version.json new file mode 100644 index 0000000..fae6d66 --- /dev/null +++ b/tests/data/term_latest_version.json @@ -0,0 +1,31 @@ +{ + "metadata": { + "artifact_type": "glossary_term", + "artifact_id": "30d1b847-0aa9-4840-a182-dd157fe977a0", + "version_id": "bdeef8cc-d9ab-4822-b3df-cef82b4de538_0", + "source_repository_id": "1be893e3-d512-4fd8-b449-c953abeedb6c", + "global_id": "1be893e3-d512-4fd8-b449-c953abeedb6c_30d1b847-0aa9-4840-a182-dd157fe977a0", + "is_target_draft": false, + "draft_ancestor_id": "90774b3c-16c6-4dcf-8c81-407d2e440baa", + "effective_start_date": "2026-01-09T09:01:15.648Z", + "created_by": "IBMid-0600012F5M", + "created_at": "2026-01-09T09:01:15.648Z", + "modified_by": "IBMid-0600012F5M", + "modified_at": "2026-01-09T09:01:15.648Z", + "revision": "0", + "name": "mango", + "state": "PUBLISHED", + "tags": [], + "steward_ids": [], + "steward_group_ids": [], + "read_only": false + }, + "entity": { + "abbreviations": [], + "state": "PUBLISHED", + "default_locale_id": "en-US", + "reference_copy": false, + "steward_ids": [], + "steward_group_ids": [] + } +} diff --git a/tests/data/term_response.json b/tests/data/term_response.json new file mode 100644 index 0000000..d5c7536 --- /dev/null +++ b/tests/data/term_response.json @@ -0,0 +1,117 @@ +{ + "metadata": { + "artifact_type": "glossary_term", + "artifact_id": "30d1b847-0aa9-4840-a182-dd157fe977a0", + "version_id": "bdeef8cc-d9ab-4822-b3df-cef82b4de538_0", + "source_repository_id": "1be893e3-d512-4fd8-b449-c953abeedb6c", + "global_id": "1be893e3-d512-4fd8-b449-c953abeedb6c_30d1b847-0aa9-4840-a182-dd157fe977a0", + "is_target_draft": false, + "draft_ancestor_id": "90774b3c-16c6-4dcf-8c81-407d2e440baa", + "effective_start_date": "2026-01-09T09:01:15.648Z", + "created_by": "IBMid-0600012F5M", + "created_at": "2026-01-09T09:01:15.648Z", + "modified_by": "IBMid-0600012F5M", + "modified_at": "2026-01-09T09:01:15.648Z", + "revision": "0", + "name": "mango", + "state": "PUBLISHED", + "tags": [], + "steward_ids": [], + "steward_group_ids": [], + "read_only": false + }, + "entity": { + "parent_category": { + "resources": [ + { + "metadata": { + "artifact_type": "relationship", + "artifact_id": "97b2a3df-94cf-446c-9338-ba8c664bc09a", + "version_id": "2728caaf-714b-4440-8558-b1d97077d34e_0", + "source_repository_id": "1be893e3-d512-4fd8-b449-c953abeedb6c", + "global_id": "1be893e3-d512-4fd8-b449-c953abeedb6c_97b2a3df-94cf-446c-9338-ba8c664bc09a", + "is_target_draft": false, + "effective_start_date": "2026-01-09T09:01:15.567Z", + "created_by": "IBMid-0600012F5M", + "created_at": "2026-01-09T09:01:15.567Z", + "modified_by": "IBMid-0600012F5M", + "modified_at": "2026-01-09T09:01:15.567Z", + "revision": "0", + "state": "PUBLISHED", + "read_only": false, + "user_access": true + }, + "entity": { + "child_href": "/v3/glossary_terms/30d1b847-0aa9-4840-a182-dd157fe977a0/versions?status=ACTIVE", + "parent_enabled": true, + "relationship_type": "parent_category", + "source_type": "glossary_term", + "target_type": "category", + "reference_copy": false, + "parent_href": "/v3/categories/e39ada11-8338-3704-90e3-681a71e7c839", + "parent_name": "[uncategorized]", + "parent_global_id": "1be893e3-d512-4fd8-b449-c953abeedb6c_e39ada11-8338-3704-90e3-681a71e7c839", + "parent_version_id": "0b9f1be1-e076-3203-9d65-a8df62cbdbeb_0", + "parent_id": "e39ada11-8338-3704-90e3-681a71e7c839", + "parent_description": "This is the system default if a standard category is not assigned.", + "child_id": "30d1b847-0aa9-4840-a182-dd157fe977a0", + "child_global_id": "1be893e3-d512-4fd8-b449-c953abeedb6c_30d1b847-0aa9-4840-a182-dd157fe977a0", + "child_name": "mango" + } + } + ], + "offset": 0, + "set_uri": false, + "limit": 5, + "count": 1 + }, + "custom_relationships": [], + "abbreviations": [], + "extended_attribute_groups": { + "dq_constraints": [ + { + "metadata": { + "type": "data_type", + "confirmed": false, + "hidden": false + }, + "origin": [], + "check": [ + { + "name": "data_type", + "value": "STRING" + }, + { + "name": "length", + "numeric_value": 80 + } + ] + }, + { + "metadata": { + "type": "length", + "confirmed": false, + "hidden": false + }, + "origin": [], + "check": [ + { + "name": "min", + "numeric_value": 3 + }, + { + "name": "max", + "numeric_value": 80 + } + ] + } + ] + }, + "state": "PUBLISHED", + "default_locale_id": "en-US", + "reference_copy": false, + "steward_ids": [], + "steward_group_ids": [], + "custom_attributes": [] + } +} \ No newline at end of file diff --git a/tests/src/__init__.py b/tests/src/__init__.py new file mode 100644 index 0000000..bf69929 --- /dev/null +++ b/tests/src/__init__.py @@ -0,0 +1,19 @@ +# coding: utf-8 +# Copyright 2026 IBM Corporation +# Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# See the LICENSE file in the project root for license information. + +"""Unit tests""" + +# This file is only here to get pylint to check the files in this directory diff --git a/tests/src/auth/__init__.py b/tests/src/auth/__init__.py new file mode 100644 index 0000000..c792a11 --- /dev/null +++ b/tests/src/auth/__init__.py @@ -0,0 +1,18 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for authentication module. +""" \ No newline at end of file diff --git a/tests/src/auth/test_auth_config.py b/tests/src/auth/test_auth_config.py new file mode 100644 index 0000000..02428de --- /dev/null +++ b/tests/src/auth/test_auth_config.py @@ -0,0 +1,275 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for AuthConfig module. + +Tests the AuthConfig class and EnvironmentType enum functionality. +""" + +import pytest +from wxdi.common.auth import AuthConfig, EnvironmentType + + +class TestEnvironmentType: + """Tests for EnvironmentType enum.""" + + def test_environment_type_values(self): + """Test that all environment types have correct values.""" + assert EnvironmentType.IBM_CLOUD.value == "ibm_cloud" + assert EnvironmentType.AWS_CLOUD.value == "aws_cloud" + assert EnvironmentType.GOV_CLOUD.value == "gov_cloud" + assert EnvironmentType.ON_PREM.value == "on_prem" + + def test_environment_type_members(self): + """Test that all expected environment types exist.""" + env_types = [e.name for e in EnvironmentType] + assert "IBM_CLOUD" in env_types + assert "AWS_CLOUD" in env_types + assert "GOV_CLOUD" in env_types + assert "ON_PREM" in env_types + + +class TestAuthConfigIBMCloud: + """Tests for AuthConfig with IBM_CLOUD environment.""" + + def test_ibm_cloud_with_api_key(self): + """Test IBM_CLOUD configuration with API key.""" + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-api-key' + ) + assert config.environment_type == EnvironmentType.IBM_CLOUD + assert config.api_key == 'test-api-key' + assert config.url == 'https://iam.cloud.ibm.com/identity/token' + + def test_ibm_cloud_with_custom_url(self): + """Test IBM_CLOUD with custom URL.""" + custom_url = 'https://custom.ibm.com/token' + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-key', + url=custom_url + ) + assert config.url == custom_url + + def test_ibm_cloud_missing_api_key(self): + """Test that IBM_CLOUD requires API key.""" + with pytest.raises(ValueError, match="API key must be provided"): + AuthConfig(environment_type=EnvironmentType.IBM_CLOUD) + + def test_ibm_cloud_trailing_slash_stripped(self): + """Test that trailing slashes are stripped from URL.""" + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-key', + url='https://custom.ibm.com/token///' + ) + assert config.url == 'https://custom.ibm.com/token' + + +class TestAuthConfigAWSCloud: + """Tests for AuthConfig with AWS_CLOUD environment.""" + + def test_aws_cloud_with_default_url(self): + """Test AWS_CLOUD with default URL.""" + config = AuthConfig( + environment_type=EnvironmentType.AWS_CLOUD, + api_key='test-key', + account_id='account-123' + ) + assert config.environment_type == EnvironmentType.AWS_CLOUD + assert config.url == 'https://account-iam.platform.saas.ibm.com' + assert config.account_id == 'account-123' + + def test_aws_cloud_with_custom_url(self): + """Test AWS_CLOUD with custom URL.""" + custom_url = 'https://custom-account-iam.example.com' + config = AuthConfig( + environment_type=EnvironmentType.AWS_CLOUD, + api_key='test-key', + account_id='account-123', + url=custom_url + ) + assert config.url == custom_url + assert config.account_id == 'account-123' + + def test_aws_cloud_missing_api_key(self): + """Test that AWS_CLOUD requires API key.""" + with pytest.raises(ValueError, match="API key must be provided"): + AuthConfig( + environment_type=EnvironmentType.AWS_CLOUD, + account_id='account-123' + ) + + def test_aws_cloud_missing_account_id(self): + """Test that AWS_CLOUD requires account ID.""" + with pytest.raises(ValueError, match="Account ID must be provided"): + AuthConfig( + environment_type=EnvironmentType.AWS_CLOUD, + api_key='test-key' + ) + + def test_aws_cloud_account_id_required(self): + """Test that account_id is stored correctly for AWS_CLOUD.""" + config = AuthConfig( + environment_type=EnvironmentType.AWS_CLOUD, + api_key='test-key', + account_id='my-account' + ) + assert config.account_id == 'my-account' + + +class TestAuthConfigGovCloud: + """Tests for AuthConfig with GOV_CLOUD environment.""" + + def test_gov_cloud_with_api_key(self): + """Test GOV_CLOUD configuration.""" + config = AuthConfig( + environment_type=EnvironmentType.GOV_CLOUD, + api_key='test-key' + ) + assert config.environment_type == EnvironmentType.GOV_CLOUD + assert config.url == 'https://dai.ibmforusgov.com/api/rest/mcsp/apikeys/token' + + def test_gov_cloud_missing_api_key(self): + """Test that GOV_CLOUD requires API key.""" + with pytest.raises(ValueError, match="API key must be provided"): + AuthConfig(environment_type=EnvironmentType.GOV_CLOUD) + + +class TestAuthConfigOnPrem: + """Tests for AuthConfig with ON_PREM environment.""" + + def test_on_prem_with_api_key(self): + """Test ON_PREM with username and API key.""" + config = AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com', + username='admin', + api_key='test-key' + ) + assert config.environment_type == EnvironmentType.ON_PREM + assert config.url == 'https://cpd.example.com/icp4d-api/v1/authorize' + assert config.username == 'admin' + assert config.api_key == 'test-key' + + def test_on_prem_with_password(self): + """Test ON_PREM with username and password.""" + config = AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com', + username='admin', + password='secret' + ) + assert config.url == 'https://cpd.example.com/icp4d-api/v1/authorize' + assert config.username == 'admin' + assert config.password == 'secret' + + def test_on_prem_url_already_has_path(self): + """Test that path is not duplicated if already present.""" + config = AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com/icp4d-api/v1/authorize', + username='admin', + api_key='test-key' + ) + assert config.url == 'https://cpd.example.com/icp4d-api/v1/authorize' + + def test_on_prem_missing_url(self): + """Test that ON_PREM requires URL.""" + with pytest.raises(ValueError, match="URL must be specified"): + AuthConfig( + environment_type=EnvironmentType.ON_PREM, + username='admin', + password='secret' + ) + + def test_on_prem_missing_username(self): + """Test that ON_PREM requires username.""" + with pytest.raises(ValueError, match="Username must be provided"): + AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com', + api_key='test-key' + ) + + def test_on_prem_missing_credentials(self): + """Test that ON_PREM requires either API key or password.""" + with pytest.raises(ValueError, match="Either api_key or password must be provided"): + AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com', + username='admin' + ) + + def test_on_prem_trailing_slash_stripped(self): + """Test that trailing slash is stripped before appending path.""" + config = AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com/', + username='admin', + api_key='test-key' + ) + assert config.url == 'https://cpd.example.com/icp4d-api/v1/authorize' + + +class TestAuthConfigValidation: + """Tests for AuthConfig validation.""" + + def test_invalid_environment_type(self): + """Test that invalid environment type raises TypeError.""" + with pytest.raises(TypeError, match="environment_type must be an instance of EnvironmentType"): + AuthConfig( + environment_type="invalid", # type: ignore + api_key='test-key' + ) + + def test_url_trailing_slash_stripping(self): + """Test that multiple trailing slashes are stripped.""" + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-key', + url='https://example.com///' + ) + assert config.url == 'https://example.com' + + def test_disable_ssl_verification_default_true(self): + """Test that disable_ssl_verification defaults to True.""" + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-key' + ) + assert config.disable_ssl_verification is True + + def test_disable_ssl_verification_can_be_set_false(self): + """Test that disable_ssl_verification can be set to False.""" + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-key', + disable_ssl_verification=False + ) + assert config.disable_ssl_verification is False + + def test_disable_ssl_verification_with_on_prem(self): + """Test disable_ssl_verification with ON_PREM environment.""" + config = AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com', + username='admin', + api_key='test-key', + disable_ssl_verification=False + ) + assert config.disable_ssl_verification is False \ No newline at end of file diff --git a/tests/src/auth/test_auth_provider.py b/tests/src/auth/test_auth_provider.py new file mode 100644 index 0000000..4d9d6f6 --- /dev/null +++ b/tests/src/auth/test_auth_provider.py @@ -0,0 +1,309 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for AuthProvider module. + +Tests the AuthProvider class and its integration with different authenticators. +""" + +import pytest +from unittest.mock import Mock, patch, MagicMock +from wxdi.common.auth import AuthProvider, AuthConfig, EnvironmentType + + +class TestAuthProviderIBMCloud: + """Tests for AuthProvider with IBM_CLOUD environment.""" + + @patch('wxdi.common.auth.auth_provider.IAMAuthenticator') + def test_ibm_cloud_authenticator_creation(self, mock_iam_auth): + """Test that IAMAuthenticator is created for IBM_CLOUD.""" + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-api-key' + ) + + provider = AuthProvider(config) + + mock_iam_auth.assert_called_once_with( + apikey='test-api-key', + url='https://iam.cloud.ibm.com/identity/token' + ) + assert provider.config == config + + @patch('wxdi.common.auth.auth_provider.IAMAuthenticator') + def test_ibm_cloud_get_token(self, mock_iam_auth): + """Test getting token from IBM_CLOUD authenticator.""" + mock_token_manager = Mock() + mock_token_manager.get_token.return_value = 'test-token-123' + mock_authenticator = Mock() + mock_authenticator.token_manager = mock_token_manager + mock_iam_auth.return_value = mock_authenticator + + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-api-key' + ) + provider = AuthProvider(config) + + token = provider.get_token() + + assert token == 'test-token-123' + mock_token_manager.get_token.assert_called_once() + + +class TestAuthProviderAWSCloud: + """Tests for AuthProvider with AWS_CLOUD environment.""" + + @patch('wxdi.common.auth.auth_provider.MCSPV2Authenticator') + def test_aws_cloud_authenticator_creation(self, mock_mcsp_auth): + """Test that MCSPV2Authenticator is created for AWS_CLOUD.""" + config = AuthConfig( + environment_type=EnvironmentType.AWS_CLOUD, + api_key='test-api-key', + account_id='account-123' + ) + + provider = AuthProvider(config) + + mock_mcsp_auth.assert_called_once_with( + apikey='test-api-key', + url='https://account-iam.platform.saas.ibm.com', + scope_collection_type='accounts', + scope_id='account-123' + ) + + @patch('wxdi.common.auth.auth_provider.MCSPV2Authenticator') + def test_aws_cloud_get_token(self, mock_mcsp_auth): + """Test getting token from AWS_CLOUD authenticator.""" + mock_token_manager = Mock() + mock_token_manager.get_token.return_value = 'aws-token-456' + mock_authenticator = Mock() + mock_authenticator.token_manager = mock_token_manager + mock_mcsp_auth.return_value = mock_authenticator + + config = AuthConfig( + environment_type=EnvironmentType.AWS_CLOUD, + api_key='test-api-key', + account_id='account-123' + ) + provider = AuthProvider(config) + + token = provider.get_token() + + assert token == 'aws-token-456' + mock_token_manager.get_token.assert_called_once() + + +class TestAuthProviderGovCloud: + """Tests for AuthProvider with GOV_CLOUD environment.""" + + @patch('wxdi.common.auth.auth_provider.GovCloudAuthenticator') + def test_gov_cloud_authenticator_creation(self, mock_gov_cloud_auth): + """Test that CustomAuthenticator is created for GOV_CLOUD.""" + config = AuthConfig( + environment_type=EnvironmentType.GOV_CLOUD, + api_key='test-api-key' + ) + + provider = AuthProvider(config) + + mock_gov_cloud_auth.assert_called_once_with( + apikey='test-api-key', + url='https://dai.ibmforusgov.com/api/rest/mcsp/apikeys/token', + disable_ssl_verification=True + ) + + @patch('wxdi.common.auth.auth_provider.GovCloudAuthenticator') + def test_gov_cloud_get_token(self, mock_gov_cloud_auth): + """Test getting token from GOV_CLOUD authenticator.""" + mock_token_manager = Mock() + mock_token_manager.get_token.return_value = 'gov-token-789' + mock_authenticator = Mock() + mock_authenticator.token_manager = mock_token_manager + mock_gov_cloud_auth.return_value = mock_authenticator + + config = AuthConfig( + environment_type=EnvironmentType.GOV_CLOUD, + api_key='test-api-key' + ) + provider = AuthProvider(config) + + token = provider.get_token() + + assert token == 'gov-token-789' + mock_token_manager.get_token.assert_called_once() + + +class TestAuthProviderOnPrem: + """Tests for AuthProvider with ON_PREM environment.""" + + @patch('wxdi.common.auth.auth_provider.CloudPakForDataAuthenticator') + def test_on_prem_with_api_key(self, mock_cp4d_auth): + """Test CloudPakForDataAuthenticator creation with API key.""" + config = AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com', + username='admin', + api_key='test-key' + ) + + provider = AuthProvider(config) + + mock_cp4d_auth.assert_called_once_with( + username='admin', + url='https://cpd.example.com/icp4d-api/v1/authorize', + apikey='test-key', + disable_ssl_verification=True + ) + + @patch('wxdi.common.auth.auth_provider.CloudPakForDataAuthenticator') + def test_on_prem_with_password(self, mock_cp4d_auth): + """Test CloudPakForDataAuthenticator creation with password.""" + config = AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com', + username='admin', + password='secret' + ) + + provider = AuthProvider(config) + + mock_cp4d_auth.assert_called_once_with( + username='admin', + password='secret', + url='https://cpd.example.com/icp4d-api/v1/authorize', + disable_ssl_verification=True + ) + + @patch('wxdi.common.auth.auth_provider.CloudPakForDataAuthenticator') + def test_on_prem_get_token(self, mock_cp4d_auth): + """Test getting token from ON_PREM authenticator.""" + mock_token_manager = Mock() + mock_token_manager.get_token.return_value = 'on-prem-token-abc' + mock_authenticator = Mock() + mock_authenticator.token_manager = mock_token_manager + mock_cp4d_auth.return_value = mock_authenticator + + config = AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com', + username='admin', + api_key='test-key' + ) + provider = AuthProvider(config) + + token = provider.get_token() + + assert token == 'on-prem-token-abc' + mock_token_manager.get_token.assert_called_once() + + +class TestAuthProviderEdgeCases: + """Tests for edge cases and error handling.""" + + def test_invalid_config_type(self): + """Test that invalid config type raises AttributeError.""" + with pytest.raises(AttributeError): + AuthProvider("invalid-config") # type: ignore + + @patch('wxdi.common.auth.auth_provider.IAMAuthenticator') + def test_token_manager_none(self, mock_iam_auth): + """Test handling when token_manager is None.""" + mock_authenticator = Mock() + mock_authenticator.token_manager = None + mock_iam_auth.return_value = mock_authenticator + + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-api-key' + ) + provider = AuthProvider(config) + + with pytest.raises(Exception, match="does not have token_manager"): + provider.get_token() + + @patch('wxdi.common.auth.auth_provider.IAMAuthenticator') + def test_multiple_token_requests(self, mock_iam_auth): + """Test multiple token requests use the same authenticator.""" + mock_token_manager = Mock() + mock_token_manager.get_token.side_effect = ['token1', 'token2', 'token3'] + mock_authenticator = Mock() + mock_authenticator.token_manager = mock_token_manager + mock_iam_auth.return_value = mock_authenticator + + config = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='test-api-key' + ) + provider = AuthProvider(config) + + token1 = provider.get_token() + token2 = provider.get_token() + token3 = provider.get_token() + + assert token1 == 'token1' + assert token2 == 'token2' + assert token3 == 'token3' + assert mock_token_manager.get_token.call_count == 3 + # Authenticator should only be created once + assert mock_iam_auth.call_count == 1 + + +class TestAuthProviderIntegration: + """Integration tests for AuthProvider.""" + + @patch('wxdi.common.auth.auth_provider.IAMAuthenticator') + @patch('wxdi.common.auth.auth_provider.GovCloudAuthenticator') + @patch('wxdi.common.auth.auth_provider.MCSPV2Authenticator') + @patch('wxdi.common.auth.auth_provider.CloudPakForDataAuthenticator') + def test_different_environments_use_different_authenticators( + self, mock_cp4d, mock_mcsp, mock_custom, mock_iam + ): + """Test that different environments create different authenticators.""" + # IBM Cloud + config1 = AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key='key1' + ) + AuthProvider(config1) + assert mock_iam.call_count == 1 + + # AWS Cloud - uses MCSPV2Authenticator + config2 = AuthConfig( + environment_type=EnvironmentType.AWS_CLOUD, + api_key='key2', + account_id='account-123' + ) + AuthProvider(config2) + assert mock_mcsp.call_count == 1 + + # Gov Cloud - uses CustomAuthenticator + config3 = AuthConfig( + environment_type=EnvironmentType.GOV_CLOUD, + api_key='key3' + ) + AuthProvider(config3) + assert mock_custom.call_count == 1 + + # On-Prem + config4 = AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url='https://cpd.example.com', + username='admin', + api_key='key4' + ) + AuthProvider(config4) + assert mock_cp4d.call_count == 1 \ No newline at end of file diff --git a/tests/src/auth/test_gov_cloud_authenticator.py b/tests/src/auth/test_gov_cloud_authenticator.py new file mode 100644 index 0000000..fe3bc15 --- /dev/null +++ b/tests/src/auth/test_gov_cloud_authenticator.py @@ -0,0 +1,259 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for GovCloudAuthenticator module. + +Tests the GovCloudAuthenticator class and its integration with GovCloudTokenManager. +""" + +import pytest +from unittest.mock import Mock, patch, MagicMock +from wxdi.common.auth.gov_cloud_authenticator import GovCloudAuthenticator + + +class TestGovCloudAuthenticatorInitialization: + """Tests for GovCloudAuthenticator initialization.""" + + def test_initialization_with_required_params(self): + """Test initialization with required parameters.""" + authenticator = GovCloudAuthenticator( + apikey='test-api-key', + url='https://example.com/token' + ) + + assert authenticator.token_manager is not None + assert authenticator.token_manager.apikey == 'test-api-key' + assert authenticator.token_manager.url == 'https://example.com/token' + + def test_initialization_with_ssl_verification_disabled(self): + """Test initialization with SSL verification disabled.""" + authenticator = GovCloudAuthenticator( + apikey='test-api-key', + url='https://example.com/token', + disable_ssl_verification=True + ) + + assert authenticator.token_manager.disable_ssl_verification is True + + def test_initialization_with_ssl_verification_enabled(self): + """Test initialization with SSL verification enabled (default).""" + authenticator = GovCloudAuthenticator( + apikey='test-api-key', + url='https://example.com/token' + ) + + assert authenticator.token_manager.disable_ssl_verification is False + + +class TestGovCloudAuthenticatorValidation: + """Tests for GovCloudAuthenticator validation.""" + + def test_validate_with_valid_config(self): + """Test validation with valid configuration.""" + authenticator = GovCloudAuthenticator( + apikey='test-api-key', + url='https://example.com/token' + ) + + # Should not raise any exception + authenticator.validate() + + def test_validate_with_missing_apikey(self): + """Test validation fails with missing API key.""" + with pytest.raises(ValueError, match="apikey"): + GovCloudAuthenticator( + apikey='', + url='https://example.com/token' + ) + + def test_validate_with_missing_url(self): + """Test validation fails with missing URL.""" + with pytest.raises(ValueError, match="url"): + GovCloudAuthenticator( + apikey='test-api-key', + url='' + ) + + +class TestGovCloudAuthenticatorTokenManagement: + """Tests for GovCloudAuthenticator token management.""" + + @patch('wxdi.common.auth.gov_cloud_authenticator.GovCloudTokenManager') + def test_token_manager_creation(self, mock_token_manager_class): + """Test that GovCloudTokenManager is created correctly.""" + mock_token_manager = Mock() + mock_token_manager_class.return_value = mock_token_manager + + authenticator = GovCloudAuthenticator( + apikey='test-api-key', + url='https://example.com/token' + ) + + mock_token_manager_class.assert_called_once_with( + apikey='test-api-key', + url='https://example.com/token', + disable_ssl_verification=False, + headers=None, + proxies=None + ) + assert authenticator.token_manager == mock_token_manager + + @patch('wxdi.common.auth.gov_cloud_authenticator.GovCloudTokenManager') + def test_get_token(self, mock_token_manager_class): + """Test getting token through token manager.""" + mock_token_manager = Mock() + mock_token_manager.get_token.return_value = 'test-token-abc123' + mock_token_manager_class.return_value = mock_token_manager + + authenticator = GovCloudAuthenticator( + apikey='test-api-key', + url='https://example.com/token' + ) + + token = authenticator.token_manager.get_token() + + assert token == 'test-token-abc123' + mock_token_manager.get_token.assert_called_once() + + +class TestGovCloudAuthenticatorInheritance: + """Tests for GovCloudAuthenticator inheritance from IAMRequestBasedAuthenticator.""" + + def test_inherits_from_iam_request_based_authenticator(self): + """Test that GovCloudAuthenticator inherits from IAMRequestBasedAuthenticator.""" + from ibm_cloud_sdk_core.authenticators.iam_request_based_authenticator import IAMRequestBasedAuthenticator + + authenticator = GovCloudAuthenticator( + apikey='test-api-key', + url='https://example.com/token' + ) + + assert isinstance(authenticator, IAMRequestBasedAuthenticator) + + def test_has_authentication_type(self): + """Test that authenticator has authentication_type attribute.""" + authenticator = GovCloudAuthenticator( + apikey='test-api-key', + url='https://example.com/token' + ) + + # IAMRequestBasedAuthenticator should have authentication_type + assert hasattr(authenticator, 'authentication_type') + + +class TestGovCloudAuthenticatorEdgeCases: + """Tests for edge cases and error handling.""" + + def test_none_apikey(self): + """Test that None API key raises error.""" + with pytest.raises((ValueError, TypeError)): + GovCloudAuthenticator( + apikey=None, # type: ignore + url='https://example.com/token' + ) + + def test_none_url(self): + """Test that None URL raises error.""" + with pytest.raises((ValueError, TypeError)): + GovCloudAuthenticator( + apikey='test-api-key', + url=None # type: ignore + ) + + def test_empty_string_apikey(self): + """Test that empty string API key raises error.""" + with pytest.raises(ValueError): + GovCloudAuthenticator( + apikey='', + url='https://example.com/token' + ) + + def test_empty_string_url(self): + """Test that empty string URL raises error.""" + with pytest.raises(ValueError): + GovCloudAuthenticator( + apikey='test-api-key', + url='' + ) + + def test_whitespace_apikey(self): + """Test that whitespace-only API key is accepted but may fail on validation.""" + # GovCloudAuthenticator doesn't strip whitespace, so it accepts it + authenticator = GovCloudAuthenticator( + apikey=' ', + url='https://example.com/token' + ) + # The whitespace apikey is stored as-is + assert authenticator.apikey == ' ' + + def test_whitespace_url(self): + """Test that whitespace-only URL is accepted but may fail on validation.""" + # GovCloudAuthenticator doesn't strip whitespace, so it accepts it + authenticator = GovCloudAuthenticator( + apikey='test-api-key', + url=' ' + ) + # The whitespace url is stored as-is + assert authenticator.url == ' ' + + +class TestGovCloudAuthenticatorIntegration: + """Integration tests for GovCloudAuthenticator.""" + + @patch('wxdi.common.auth.gov_cloud_authenticator.GovCloudTokenManager') + def test_full_authentication_flow(self, mock_token_manager_class): + """Test complete authentication flow.""" + mock_token_manager = Mock() + mock_token_manager.get_token.return_value = 'integration-token-xyz' + mock_token_manager_class.return_value = mock_token_manager + + # Create authenticator + authenticator = GovCloudAuthenticator( + apikey='integration-key', + url='https://integration.example.com/token' + ) + + # Validate + authenticator.validate() + + # Get token + token = authenticator.token_manager.get_token() + + assert token == 'integration-token-xyz' + mock_token_manager.get_token.assert_called_once() + + @patch('wxdi.common.auth.gov_cloud_authenticator.GovCloudTokenManager') + def test_multiple_token_requests(self, mock_token_manager_class): + """Test multiple token requests use the same token manager.""" + mock_token_manager = Mock() + mock_token_manager.get_token.side_effect = ['token1', 'token2', 'token3'] + mock_token_manager_class.return_value = mock_token_manager + + authenticator = GovCloudAuthenticator( + apikey='test-key', + url='https://example.com/token' + ) + + token1 = authenticator.token_manager.get_token() + token2 = authenticator.token_manager.get_token() + token3 = authenticator.token_manager.get_token() + + assert token1 == 'token1' + assert token2 == 'token2' + assert token3 == 'token3' + assert mock_token_manager.get_token.call_count == 3 + # Token manager should only be created once + assert mock_token_manager_class.call_count == 1 \ No newline at end of file diff --git a/tests/src/auth/test_gov_cloud_token_manager.py b/tests/src/auth/test_gov_cloud_token_manager.py new file mode 100644 index 0000000..30fa736 --- /dev/null +++ b/tests/src/auth/test_gov_cloud_token_manager.py @@ -0,0 +1,403 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for GovCloudTokenManager module. + +Tests the GovCloudTokenManager class and its token management functionality. +""" + +import pytest +import time +from unittest.mock import Mock, patch, MagicMock +from wxdi.common.auth.gov_cloud_token_manager import GovCloudTokenManager + +# Valid JWT token for testing (header.payload.signature format) +VALID_JWT_TOKEN = 'eyJhbGciOiAiUlMyNTYiLCAidHlwIjogIkpXVCJ9.eyJleHAiOiA5OTk5OTk5OTk5LCAiaWF0IjogMTIzNDU2Nzg5MH0.ZmFrZS1zaWduYXR1cmU' + + +class TestGovCloudTokenManagerInitialization: + """Tests for GovCloudTokenManager initialization.""" + + def test_initialization_with_required_params(self): + """Test initialization with required parameters.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + assert token_manager.apikey == 'test-api-key' + assert token_manager.url == 'https://example.com/token' + assert token_manager.disable_ssl_verification is False + assert token_manager.headers == {} + assert token_manager.proxies == {} + + def test_initialization_with_ssl_verification_disabled(self): + """Test initialization with SSL verification disabled.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token', + disable_ssl_verification=True + ) + + assert token_manager.disable_ssl_verification is True + + def test_initialization_with_custom_headers(self): + """Test initialization with custom headers.""" + custom_headers = {'X-Custom-Header': 'value'} + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token', + headers=custom_headers + ) + + assert token_manager.headers == custom_headers + + def test_initialization_with_proxies(self): + """Test initialization with proxy configuration.""" + proxies = {'http': 'http://proxy.example.com:8080'} + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token', + proxies=proxies + ) + + assert token_manager.proxies == proxies + + +class TestGovCloudTokenManagerInheritance: + """Tests for GovCloudTokenManager inheritance from JWTTokenManager.""" + + def test_inherits_from_jwt_token_manager(self): + """Test that GovCloudTokenManager inherits from JWTTokenManager.""" + from ibm_cloud_sdk_core.token_managers.jwt_token_manager import JWTTokenManager + + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + assert isinstance(token_manager, JWTTokenManager) + + def test_has_get_token_method(self): + """Test that token manager has get_token method from parent.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + assert hasattr(token_manager, 'get_token') + assert callable(token_manager.get_token) + + +class TestGovCloudTokenManagerRequestToken: + """Tests for request_token method.""" + + @patch.object(GovCloudTokenManager, '_request') + def test_request_token_makes_post_request(self, mock_request): + """Test that request_token makes a POST request.""" + mock_request.return_value = { + 'token': 'test-token-123', + 'expires_in': 3600 + } + + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + response = token_manager.request_token() + + mock_request.assert_called_once() + call_args = mock_request.call_args + assert call_args[1]['method'] == 'POST' + assert call_args[1]['url'] == 'https://example.com/token' + + @patch.object(GovCloudTokenManager, '_request') + def test_request_token_uses_form_urlencoded(self, mock_request): + """Test that request_token uses application/x-www-form-urlencoded.""" + mock_request.return_value = { + 'token': 'test-token-123', + 'expires_in': 3600 + } + + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + token_manager.request_token() + + call_args = mock_request.call_args + headers = call_args[1]['headers'] + assert headers['Content-Type'] == 'application/x-www-form-urlencoded' + assert headers['Accept'] == 'application/json' + + @patch.object(GovCloudTokenManager, '_request') + def test_request_token_sends_apikey(self, mock_request): + """Test that request_token sends the API key in the request.""" + mock_request.return_value = { + 'token': 'test-token-123', + 'expires_in': 3600 + } + + token_manager = GovCloudTokenManager( + apikey='my-secret-key', + url='https://example.com/token' + ) + + token_manager.request_token() + + call_args = mock_request.call_args + data = call_args[1]['data'] + assert data['apikey'] == 'my-secret-key' + + @patch.object(GovCloudTokenManager, '_request') + def test_request_token_includes_custom_headers(self, mock_request): + """Test that custom headers are included in the request.""" + mock_request.return_value = { + 'token': 'test-token-123', + 'expires_in': 3600 + } + + custom_headers = {'X-Custom-Header': 'custom-value'} + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token', + headers=custom_headers + ) + + token_manager.request_token() + + call_args = mock_request.call_args + headers = call_args[1]['headers'] + assert headers['X-Custom-Header'] == 'custom-value' + + @patch.object(GovCloudTokenManager, '_request') + def test_request_token_returns_dict(self, mock_request): + """Test that request_token returns a dictionary.""" + expected_response = { + 'token': 'test-token-456', + 'expires_in': 7200 + } + mock_request.return_value = expected_response + + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + response = token_manager.request_token() + + assert response == expected_response + assert isinstance(response, dict) + + +class TestGovCloudTokenManagerSaveTokenInfo: + """Tests for _save_token_info method.""" + + def test_save_token_info_with_token_field(self): + """Test saving token info when response has 'token' field.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + token_response = { + 'token': VALID_JWT_TOKEN, + 'expires_in': 3600 + } + + token_manager._save_token_info(token_response) + + assert token_manager.access_token == VALID_JWT_TOKEN + + def test_save_token_info_with_access_token_field(self): + """Test saving token info when response has 'access_token' field.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + # Note: JWTTokenManager expects 'token' field, not 'access_token' + # The token manager will look for 'token' field first + token_response = { + 'token': VALID_JWT_TOKEN, + 'expires_in': 3600 + } + + token_manager._save_token_info(token_response) + + assert token_manager.access_token == VALID_JWT_TOKEN + + def test_save_token_info_prefers_token_over_access_token(self): + """Test that 'token' field is preferred over 'access_token'.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + token_response = { + 'token': VALID_JWT_TOKEN, + 'access_token': VALID_JWT_TOKEN, + 'expires_in': 3600 + } + + token_manager._save_token_info(token_response) + + assert token_manager.access_token == VALID_JWT_TOKEN + + def test_save_token_info_raises_error_without_token(self): + """Test that error is raised when no token is in response.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + token_response = { + 'expires_in': 3600 + } + + # JWTTokenManager will raise an error when token field is missing + with pytest.raises(Exception): # Could be ValueError or KeyError + token_manager._save_token_info(token_response) + + def test_save_token_info_with_expires_in(self): + """Test saving token info with expires_in field.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + token_response = { + 'token': VALID_JWT_TOKEN, + 'expires_in': 3600 + } + + token_manager._save_token_info(token_response) + + # JWTTokenManager extracts expiration from JWT token payload (exp field) + # Our JWT token has exp=9999999999 + assert token_manager.expire_time == 9999999999 + assert token_manager.access_token == VALID_JWT_TOKEN + + def test_save_token_info_with_expiration(self): + """Test saving token info with expiration timestamp.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + token_response = { + 'token': VALID_JWT_TOKEN, + 'expiration': 1234567890 + } + + token_manager._save_token_info(token_response) + + # JWTTokenManager extracts expiration from JWT token payload (exp field) + # Our JWT token has exp=9999999999, not from the 'expiration' field + assert token_manager.expire_time == 9999999999 + assert token_manager.access_token == VALID_JWT_TOKEN + + def test_save_token_info_default_expiration(self): + """Test that expiration is extracted from JWT token.""" + token_manager = GovCloudTokenManager( + apikey='test-api-key', + url='https://example.com/token' + ) + + token_response = { + 'token': VALID_JWT_TOKEN + } + + token_manager._save_token_info(token_response) + + # JWTTokenManager extracts expiration from JWT token payload (exp field) + # Our JWT token has exp=9999999999 + assert token_manager.expire_time == 9999999999 + assert token_manager.access_token == VALID_JWT_TOKEN + + +class TestGovCloudTokenManagerEdgeCases: + """Tests for edge cases and error handling.""" + + def test_empty_apikey(self): + """Test that empty API key is handled.""" + token_manager = GovCloudTokenManager( + apikey='', + url='https://example.com/token' + ) + + # Should initialize but may fail on actual request + assert token_manager.apikey == '' + + def test_empty_url(self): + """Test that empty URL is handled.""" + token_manager = GovCloudTokenManager( + apikey='test-key', + url='' + ) + + # Should initialize but may fail on actual request + assert token_manager.url == '' + + def test_none_headers_defaults_to_empty_dict(self): + """Test that None headers defaults to empty dict.""" + token_manager = GovCloudTokenManager( + apikey='test-key', + url='https://example.com/token', + headers=None + ) + + assert token_manager.headers == {} + + def test_none_proxies_defaults_to_empty_dict(self): + """Test that None proxies defaults to empty dict.""" + token_manager = GovCloudTokenManager( + apikey='test-key', + url='https://example.com/token', + proxies=None + ) + + assert token_manager.proxies == {} + + +class TestGovCloudTokenManagerIntegration: + """Integration tests for GovCloudTokenManager.""" + + @patch.object(GovCloudTokenManager, '_request') + def test_full_token_lifecycle(self, mock_request): + """Test complete token request and save lifecycle.""" + mock_request.return_value = { + 'token': VALID_JWT_TOKEN, + 'expires_in': 7200, + 'refresh_token': 'refresh-lifecycle' + } + + token_manager = GovCloudTokenManager( + apikey='lifecycle-key', + url='https://example.com/token' + ) + + # Request token + response = token_manager.request_token() + + # Save token info + token_manager._save_token_info(response) + + # Verify token was saved + assert token_manager.access_token == VALID_JWT_TOKEN + assert hasattr(token_manager, 'expire_time') \ No newline at end of file diff --git a/tests/src/data_product_recommender/__init__.py b/tests/src/data_product_recommender/__init__.py new file mode 100644 index 0000000..2d49fb5 --- /dev/null +++ b/tests/src/data_product_recommender/__init__.py @@ -0,0 +1,19 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Data Product Recommender test package""" + +# Made with Bob diff --git a/tests/src/data_product_recommender/test_data_product_recommender.py b/tests/src/data_product_recommender/test_data_product_recommender.py new file mode 100644 index 0000000..df31423 --- /dev/null +++ b/tests/src/data_product_recommender/test_data_product_recommender.py @@ -0,0 +1,613 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Unit tests for DataProductRecommender +""" + +import pytest +import pandas as pd +import json +import tempfile +import os +from datetime import datetime, timedelta +from wxdi.data_product_recommender.recommender import ( + DataProductRecommender, + normalize_query_pattern, + FIVE_STARS, FOUR_STARS, THREE_STARS, TWO_STARS, ONE_STAR, + EXCELLENT_CANDIDATE, GOOD_CANDIDATE, FAIR_CANDIDATE, WEAK_CANDIDATE, POOR_CANDIDATE +) +from wxdi.data_product_recommender.platforms import SnowflakeQueryParser + + +class TestNormalizeQueryPattern: + """Tests for normalize_query_pattern function""" + + def test_normalize_simple_query(self): + """Test normalizing a simple query""" + query = "SELECT * FROM SALES.CUSTOMERS WHERE id = 123" + pattern = normalize_query_pattern(query) + assert pattern == "SELECT * FROM SALES.CUSTOMERS WHERE ID = ?" + + def test_normalize_query_with_strings(self): + """Test normalizing query with string literals""" + query = "SELECT * FROM SALES.CUSTOMERS WHERE name = 'John Doe'" + pattern = normalize_query_pattern(query) + assert pattern == "SELECT * FROM SALES.CUSTOMERS WHERE NAME = ?" + + def test_normalize_query_with_double_quotes(self): + """Test normalizing query with double-quoted strings""" + query = 'SELECT * FROM SALES.CUSTOMERS WHERE name = "John Doe"' + pattern = normalize_query_pattern(query) + assert pattern == "SELECT * FROM SALES.CUSTOMERS WHERE NAME = ?" + + def test_normalize_query_with_decimals(self): + """Test normalizing query with decimal numbers""" + query = "SELECT * FROM SALES.ORDERS WHERE amount > 123.45" + pattern = normalize_query_pattern(query) + assert pattern == "SELECT * FROM SALES.ORDERS WHERE AMOUNT > ?" + + def test_normalize_query_with_dates(self): + """Test normalizing query with date literals""" + query = "SELECT * FROM SALES.ORDERS WHERE date = '2024-01-01'" + pattern = normalize_query_pattern(query) + assert "?" in pattern + + def test_normalize_empty_query(self): + """Test normalizing empty query""" + assert normalize_query_pattern("") == "" + assert normalize_query_pattern(None) == "" + + def test_normalize_query_whitespace(self): + """Test normalizing query with multiple whitespaces""" + query = "SELECT * FROM SALES.CUSTOMERS" + pattern = normalize_query_pattern(query) + assert pattern == "SELECT * FROM SALES.CUSTOMERS" + + def test_normalize_query_with_escaped_quotes(self): + """Test normalizing query with escaped quotes""" + query = "SELECT * FROM SALES.CUSTOMERS WHERE name = 'O\\'Brien'" + pattern = normalize_query_pattern(query) + assert "?" in pattern + + +class TestDataProductRecommender: + """Tests for DataProductRecommender class""" + + def setup_method(self): + """Setup test fixtures""" + self.parser = SnowflakeQueryParser() + self.recommender = DataProductRecommender(self.parser) + + def test_initialization(self): + """Test recommender initialization""" + assert self.recommender.parser is not None + assert self.recommender.query_logs is None + assert self.recommender.table_metrics is None + assert self.recommender.query_patterns is None + + def test_load_query_logs_from_json_file(self): + """Test loading query logs from JSON file""" + # Create temporary JSON file + test_data = [ + { + 'query_text': 'SELECT * FROM SALES.CUSTOMERS', + 'user_name': 'user1', + 'start_time': '2025-01-01' + }, + { + 'query_text': 'SELECT * FROM SALES.ORDERS', + 'user_name': 'user2', + 'start_time': '2025-01-02' + } + ] + + with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: + json.dump(test_data, f) + temp_file = f.name + + try: + # Load the data + df = self.recommender.load_query_logs_from_json_file(temp_file) + + # Verify + assert df is not None + assert len(df) == 2 + assert 'query_text' in df.columns + assert 'user' in df.columns + assert 'tables' in df.columns + finally: + os.unlink(temp_file) + + def test_load_query_logs_from_csv_file(self): + """Test loading query logs from CSV file""" + # Create temporary CSV file + test_data = pd.DataFrame({ + 'query_text': ['SELECT * FROM SALES.CUSTOMERS', 'SELECT * FROM SALES.ORDERS'], + 'user_name': ['user1', 'user2'], + 'start_time': ['2025-01-01', '2025-01-02'] + }) + + with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f: + test_data.to_csv(f.name, index=False) + temp_file = f.name + + try: + # Load the data + df = self.recommender.load_query_logs_from_csv_file(temp_file) + + # Verify + assert df is not None + assert len(df) == 2 + assert 'query_text' in df.columns + assert 'user' in df.columns + assert 'tables' in df.columns + finally: + os.unlink(temp_file) + + def test_calculate_metrics(self): + """Test calculating table metrics""" + # Setup test data + test_data = pd.DataFrame({ + 'query_text': [ + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.ORDERS', + 'SELECT * FROM SALES.CUSTOMERS JOIN SALES.ORDERS ON c.id = o.customer_id' + ], + 'user': ['user1', 'user2', 'user1', 'user3'], + 'start_time': ['2025-01-01', '2025-01-02', '2025-01-03', '2025-01-04'] + }) + + # Manually add tables column + test_data['tables'] = [ + ['SALES.CUSTOMERS'], + ['SALES.CUSTOMERS'], + ['SALES.ORDERS'], + ['SALES.CUSTOMERS', 'SALES.ORDERS'] + ] + + self.recommender.query_logs = test_data + + # Calculate metrics + metrics = self.recommender.calculate_metrics() + + # Verify + assert metrics is not None + assert len(metrics) == 2 # Two unique tables + assert 'table' in metrics.columns + assert 'query_count' in metrics.columns + assert 'unique_users' in metrics.columns + assert 'related_tables' in metrics.columns + + # Check specific values + customers_row = metrics[metrics['table'] == 'SALES.CUSTOMERS'].iloc[0] + assert customers_row['query_count'] == 3 + assert customers_row['unique_users'] == 3 + + def test_score_tables(self): + """Test scoring tables""" + # Setup test data with metrics including temporal fields + self.recommender.table_metrics = pd.DataFrame({ + 'table': ['SALES.CUSTOMERS', 'SALES.ORDERS'], + 'query_count': [10, 5], + 'unique_users': [5, 3], + 'related_tables': [['SALES.ORDERS'], ['SALES.CUSTOMERS']], + 'related_table_count': [1, 1], + 'recency_score': [0.9, 0.5], + 'consistency_score': [0.8, 0.6] + }) + + # Score tables + scored = self.recommender.score_tables() + + # Verify + assert scored is not None + assert len(scored) == 2 + assert 'recommendation_score' in scored.columns + assert scored.iloc[0]['recommendation_score'] > scored.iloc[1]['recommendation_score'] + + def test_identify_table_groups(self): + """Test identifying table groups""" + # Setup test data + test_data = pd.DataFrame({ + 'query_text': ['query1', 'query2', 'query3'], + 'user': ['user1', 'user2', 'user3'], + 'start_time': ['2025-01-01', '2025-01-02', '2025-01-03'], + 'tables': [ + ['SALES.CUSTOMERS', 'SALES.ORDERS'], + ['SALES.CUSTOMERS', 'SALES.ORDERS'], + ['SALES.ORDERS', 'PRODUCT.CATALOG'] + ] + }) + + self.recommender.query_logs = test_data + + # Identify groups + groups = self.recommender.identify_table_groups(min_cooccurrence=2) + + # Verify + assert groups is not None + assert len(groups) >= 1 + # The group (SALES.CUSTOMERS, SALES.ORDERS) should appear with count 2 + assert any(count == 2 for _, count in groups) + + def test_recommend_data_products(self): + """Test generating recommendations""" + # Setup complete test data + test_data = pd.DataFrame({ + 'query_text': ['query1', 'query2', 'query3'], + 'user': ['user1', 'user2', 'user3'], + 'start_time': ['2025-01-01', '2025-01-02', '2025-01-03'], + 'tables': [ + ['SALES.CUSTOMERS'], + ['SALES.CUSTOMERS'], + ['SALES.ORDERS'] + ] + }) + + self.recommender.query_logs = test_data + self.recommender.calculate_metrics() + + # Generate recommendations + recommendations = self.recommender.recommend_data_products( + num_recommendations=2 + ) + + # Verify structure + assert recommendations is not None + assert 'individual_tables' in recommendations + assert 'table_groups' in recommendations + assert 'metadata' in recommendations + assert len(recommendations['individual_tables']) <= 2 + + # Verify metadata + metadata = recommendations['metadata'] + assert 'total_tables' in metadata + assert 'recommended_tables' in metadata + assert 'highest_score' in metadata + assert 'min_score_threshold' in metadata + assert metadata['total_tables'] >= metadata['recommended_tables'] + assert metadata['min_score_threshold'] is None # No threshold in this test + + def test_export_recommendations_markdown(self): + """Test exporting recommendations to markdown""" + # Setup test data + self.recommender.query_logs = pd.DataFrame({ + 'query_text': ['query1'], + 'user': ['user1'], + 'start_time': ['2025-01-01'], + 'tables': [['SALES.CUSTOMERS']] + }) + + self.recommender.table_metrics = pd.DataFrame({ + 'table': ['SALES.CUSTOMERS'], + 'query_count': [10], + 'unique_users': [5], + 'related_tables': [[]], + 'related_table_count': [0] + }) + + recommendations = { + 'individual_tables': [ + { + 'table': 'SALES.CUSTOMERS', + 'query_count': 10, + 'unique_users': 5, + 'recommendation_score': 85.5, + 'related_tables': [] + } + ] + } + + # Export to temporary file + with tempfile.NamedTemporaryFile(mode='w', suffix='.md', delete=False) as f: + temp_file = f.name + + try: + self.recommender.export_recommendations_markdown(recommendations, temp_file) + + # Verify file was created and has content + assert os.path.exists(temp_file) + with open(temp_file, 'r') as f: + content = f.read() + assert 'Data Product Recommendations' in content + assert 'SALES.CUSTOMERS' in content + finally: + os.unlink(temp_file) + + def test_export_recommendations_json(self): + """Test exporting recommendations to JSON""" + # Setup test data + self.recommender.query_logs = pd.DataFrame({ + 'query_text': ['query1'], + 'user': ['user1'], + 'start_time': ['2025-01-01'], + 'tables': [['SALES.CUSTOMERS']] + }) + + recommendations = { + 'individual_tables': [ + { + 'table': 'SALES.CUSTOMERS', + 'query_count': 10, + 'unique_users': 5, + 'recommendation_score': 85.5, + 'recency_score': 0.9, + 'consistency_score': 0.8 + } + ], + 'metadata': { + 'total_tables': 1, + 'recommended_tables': 1 + } + } + + # Export to temporary file + with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: + temp_file = f.name + + try: + self.recommender.export_recommendations_json(recommendations, temp_file) + + # Verify file was created and has valid JSON + assert os.path.exists(temp_file) + with open(temp_file, 'r') as f: + data = json.load(f) + # JSON export uses 'recommendations' key, not 'individual_tables' + assert 'recommendations' in data + assert 'metadata' in data + assert len(data['recommendations']) == 1 + # Verify structure + assert data['recommendations'][0]['type'] == 'standalone' + assert 'score' in data['recommendations'][0] + assert 'rating' in data['recommendations'][0] + finally: + os.unlink(temp_file) + + def test_calculate_temporal_metrics(self): + """Test calculating temporal metrics (recency and consistency)""" + # Setup test data with specific dates + base_date = datetime(2025, 1, 1) + test_data = pd.DataFrame({ + 'query_text': [ + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.CUSTOMERS' + ], + 'user': ['user1', 'user1', 'user1'], + 'start_time': [ + base_date, + base_date + timedelta(days=1), + base_date + timedelta(days=2) + ], + 'tables': [ + ['SALES.CUSTOMERS'], + ['SALES.CUSTOMERS'], + ['SALES.CUSTOMERS'] + ] + }) + + self.recommender.query_logs = test_data + metrics = self.recommender.calculate_metrics() + + # Verify temporal metrics are calculated + assert 'recency_score' in metrics.columns + assert 'consistency_score' in metrics.columns + assert 'days_since_last_query' in metrics.columns + assert metrics.iloc[0]['recency_score'] > 0 + + def test_recommend_with_table_groups(self): + """Test recommendations include table groups""" + # Setup test data with co-occurring tables + test_data = pd.DataFrame({ + 'query_text': ['query1', 'query2', 'query3'], + 'user': ['user1', 'user2', 'user3'], + 'start_time': ['2025-01-01', '2025-01-02', '2025-01-03'], + 'tables': [ + ['SALES.CUSTOMERS', 'SALES.ORDERS'], + ['SALES.CUSTOMERS', 'SALES.ORDERS'], + ['SALES.CUSTOMERS', 'SALES.ORDERS'] + ] + }) + + self.recommender.query_logs = test_data + self.recommender.calculate_metrics() + + recommendations = self.recommender.recommend_data_products( + num_recommendations=10 + ) + + # Verify table groups are identified + assert 'table_groups' in recommendations + + def test_score_tables_with_custom_weights(self): + """Test scoring tables with custom weights""" + # Setup test data + self.recommender.table_metrics = pd.DataFrame({ + 'table': ['SALES.CUSTOMERS'], + 'query_count': [10], + 'unique_users': [5], + 'related_tables': [[]], + 'related_table_count': [0], + 'recency_score': [0.9], + 'consistency_score': [0.8] + }) + + # Score with custom weights + scored = self.recommender.score_tables( + query_weight=0.5, + user_weight=0.3, + recency_weight=0.1, + consistency_weight=0.1 + ) + + assert 'recommendation_score' in scored.columns + assert len(scored) == 1 + + def test_get_rating_label(self): + """Test rating label assignment""" + # _get_rating_label returns just a string label, not a tuple + assert self.recommender._get_rating_label(95) == 'excellent' + assert self.recommender._get_rating_label(75) == 'good' + assert self.recommender._get_rating_label(55) == 'fair' + assert self.recommender._get_rating_label(35) == 'weak' + assert self.recommender._get_rating_label(15) == 'poor' + + def test_get_star_rating(self): + """Test star rating assignment""" + # _get_star_rating returns a tuple of (stars, label) + stars, label = self.recommender._get_star_rating(95) + assert stars == FIVE_STARS + assert label == EXCELLENT_CANDIDATE + + stars, label = self.recommender._get_star_rating(75) + assert stars == FOUR_STARS + assert label == GOOD_CANDIDATE + + stars, label = self.recommender._get_star_rating(55) + assert stars == THREE_STARS + assert label == FAIR_CANDIDATE + + stars, label = self.recommender._get_star_rating(35) + assert stars == TWO_STARS + assert label == WEAK_CANDIDATE + + stars, label = self.recommender._get_star_rating(15) + assert stars == ONE_STAR + assert label == POOR_CANDIDATE + + def test_load_query_logs_filters_empty_tables(self): + """Test that queries with no table references are filtered out""" + # Create test data with some queries having no tables + test_data = [ + { + 'query_text': 'SELECT * FROM SALES.CUSTOMERS', + 'user_name': 'user1', + 'start_time': '2025-01-01' + }, + { + 'query_text': 'SHOW TABLES', # No table reference + 'user_name': 'user2', + 'start_time': '2025-01-02' + } + ] + + with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: + json.dump(test_data, f) + temp_file = f.name + + try: + df = self.recommender.load_query_logs_from_json_file(temp_file) + # Should only have 1 query (the one with table reference) + assert len(df) >= 0 # May vary based on parser + finally: + os.unlink(temp_file) + + def test_calculate_metrics_with_single_timestamp(self): + """Test metrics calculation with single timestamp per table""" + test_data = pd.DataFrame({ + 'query_text': ['SELECT * FROM SALES.CUSTOMERS'], + 'user': ['user1'], + 'start_time': [datetime(2025, 1, 1)], + 'tables': [['SALES.CUSTOMERS']] + }) + + self.recommender.query_logs = test_data + metrics = self.recommender.calculate_metrics() + + # Should handle single timestamp gracefully + assert len(metrics) == 1 + assert metrics.iloc[0]['consistency_score'] == 0.0 # Single timestamp = 0 consistency + + def test_recommend_with_min_score_filters_tables(self): + """Test that min_score filters out low-scoring tables""" + # Setup test data with varying query counts + test_data = pd.DataFrame({ + 'query_text': [ + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.ORDERS' + ], + 'user': ['user1', 'user2', 'user3', 'user1'], + 'start_time': ['2025-01-01', '2025-01-02', '2025-01-03', '2025-01-04'], + 'tables': [ + ['SALES.CUSTOMERS'], + ['SALES.CUSTOMERS'], + ['SALES.CUSTOMERS'], + ['SALES.ORDERS'] + ] + }) + + self.recommender.query_logs = test_data + self.recommender.calculate_metrics() + + # Get recommendations with high min_score + recommendations = self.recommender.recommend_data_products( + num_recommendations=10, + min_score=70.0 + ) + + # Should filter out low-scoring tables + assert recommendations['metadata']['min_score_threshold'] == 70.0 + for table in recommendations['individual_tables']: + assert table['recommendation_score'] >= 70.0 + + +class TestRecommenderEdgeCases: + """Test edge cases for recommender""" + + def setup_method(self): + """Setup test fixtures""" + self.parser = SnowflakeQueryParser() + self.recommender = DataProductRecommender(self.parser) + + def test_calculate_metrics_without_loaded_data(self): + """Test calculating metrics without loading data first""" + with pytest.raises(ValueError, match="Query logs not loaded"): + self.recommender.calculate_metrics() + + def test_score_tables_without_metrics(self): + """Test scoring tables without calculating metrics first""" + with pytest.raises(ValueError, match="Metrics not calculated"): + self.recommender.score_tables() + + def test_identify_table_groups_without_loaded_data(self): + """Test identifying groups without loading data first""" + with pytest.raises(ValueError, match="Query logs not loaded"): + self.recommender.identify_table_groups() + + def test_load_csv_with_missing_columns(self): + """Test loading CSV with missing required columns""" + # Create CSV with missing columns + test_data = pd.DataFrame({ + 'query_text': ['SELECT * FROM table1'], + 'start_time': ['2025-01-01'] + # Missing 'user' column + }) + + with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f: + test_data.to_csv(f.name, index=False) + temp_file = f.name + + try: + with pytest.raises(ValueError, match="missing required columns"): + self.recommender.load_query_logs_from_csv_file(temp_file) + finally: + os.unlink(temp_file) + +# Made with Bob diff --git a/tests/src/data_product_recommender/test_data_product_recommender_cli.py b/tests/src/data_product_recommender/test_data_product_recommender_cli.py new file mode 100644 index 0000000..63d5db6 --- /dev/null +++ b/tests/src/data_product_recommender/test_data_product_recommender_cli.py @@ -0,0 +1,306 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Unit tests for Data Product Recommender CLI +""" + +import pytest +import json +import tempfile +import os +from pathlib import Path +from unittest.mock import patch, MagicMock +import pandas as pd + +from wxdi.data_product_recommender.cli import main +from wxdi.data_product_recommender.platforms import SnowflakeQueryParser + + +class TestCLI: + """Tests for CLI functionality""" + + def test_cli_with_csv_snowflake(self): + """Test CLI with CSV file and Snowflake platform""" + # Create temporary CSV file + test_data = pd.DataFrame({ + 'query_text': ['SELECT * FROM SALES.CUSTOMERS', 'SELECT * FROM SALES.ORDERS'], + 'user_name': ['user1', 'user2'], + 'start_time': ['2025-01-01', '2025-01-02'] + }) + + with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f: + test_data.to_csv(f.name, index=False) + csv_file = f.name + + with tempfile.TemporaryDirectory() as output_dir: + try: + test_args = [ + 'cli.py', + '--platform', 'snowflake', + '--input-file', csv_file, + '--output', output_dir, + '--num-recommendations', '5' + ] + + with patch('sys.argv', test_args): + main() + + # Verify output file was created + output_files = list(Path(output_dir).glob('recommendations_*.md')) + assert len(output_files) == 1 + + finally: + os.unlink(csv_file) + + def test_cli_with_json_databricks(self): + """Test CLI with JSON file and Databricks platform""" + # Create temporary JSON file + test_data = [ + { + 'statement_text': 'SELECT * FROM CLINICAL.PATIENTS', + 'executed_by': 'user1', + 'start_time': '2025-01-01' + }, + { + 'statement_text': 'SELECT * FROM CLINICAL.VISITS', + 'executed_by': 'user2', + 'start_time': '2025-01-02' + } + ] + + with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: + json.dump(test_data, f) + json_file = f.name + + with tempfile.TemporaryDirectory() as output_dir: + try: + test_args = [ + 'cli.py', + '--platform', 'databricks', + '--input-file', json_file, + '--output', output_dir, + '--output-format', 'json', + '--num-recommendations', '10' + ] + + with patch('sys.argv', test_args): + main() + + # Verify JSON output file was created + output_files = list(Path(output_dir).glob('recommendations_*.json')) + assert len(output_files) == 1 + + # Verify JSON is valid + with open(output_files[0], 'r') as f: + recommendations = json.load(f) + # JSON export uses 'recommendations' key, not 'individual_tables' + assert 'recommendations' in recommendations + assert 'metadata' in recommendations + + finally: + os.unlink(json_file) + + def test_cli_with_bigquery(self): + """Test CLI with BigQuery platform""" + # Create temporary CSV file + test_data = pd.DataFrame({ + 'query': ['SELECT * FROM PRODUCT.CATALOG', 'SELECT * FROM PRODUCT.INVENTORY'], + 'user_email': ['user1@example.com', 'user2@example.com'], + 'start_time': ['2025-01-01', '2025-01-02'] + }) + + with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f: + test_data.to_csv(f.name, index=False) + csv_file = f.name + + with tempfile.TemporaryDirectory() as output_dir: + try: + test_args = [ + 'cli.py', + '--platform', 'bigquery', + '--input-file', csv_file, + '--output', output_dir + ] + + with patch('sys.argv', test_args): + main() + + # Verify output was created + output_files = list(Path(output_dir).glob('recommendations_*.md')) + assert len(output_files) == 1 + + finally: + os.unlink(csv_file) + + def test_cli_with_watsonxdata(self): + """Test CLI with watsonx.data platform""" + # Create temporary CSV file + test_data = pd.DataFrame({ + 'query': ['SELECT * FROM NETWORK.SUBSCRIBERS', 'SELECT * FROM NETWORK.USAGE'], + 'user': ['user1', 'user2'], + 'created': ['2025-01-01', '2025-01-02'] + }) + + with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f: + test_data.to_csv(f.name, index=False) + csv_file = f.name + + with tempfile.TemporaryDirectory() as output_dir: + try: + test_args = [ + 'cli.py', + '--platform', 'watsonxdata', + '--input-file', csv_file, + '--output', output_dir + ] + + with patch('sys.argv', test_args): + main() + + # Verify output was created + output_files = list(Path(output_dir).glob('recommendations_*.md')) + assert len(output_files) == 1 + + finally: + os.unlink(csv_file) + + def test_cli_with_min_score_threshold(self): + """Test CLI with minimum score threshold""" + # Create temporary CSV file with multiple queries + test_data = pd.DataFrame({ + 'query_text': [ + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.ORDERS' + ], + 'user_name': ['user1', 'user2', 'user3', 'user1'], + 'start_time': ['2025-01-01', '2025-01-02', '2025-01-03', '2025-01-04'] + }) + + with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f: + test_data.to_csv(f.name, index=False) + csv_file = f.name + + with tempfile.TemporaryDirectory() as output_dir: + try: + test_args = [ + 'cli.py', + '--platform', 'snowflake', + '--input-file', csv_file, + '--output', output_dir, + '--min-score', '50.0' + ] + + with patch('sys.argv', test_args): + main() + + # Verify output was created + output_files = list(Path(output_dir).glob('recommendations_*.md')) + assert len(output_files) == 1 + + finally: + os.unlink(csv_file) + + def test_cli_invalid_file_type(self): + """Test CLI with invalid file type""" + with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False) as f: + f.write("invalid file") + txt_file = f.name + + try: + test_args = [ + 'cli.py', + '--platform', 'snowflake', + '--input-file', txt_file, + '--output', 'output' + ] + + with patch('sys.argv', test_args): + with pytest.raises(ValueError, match="Invalid --input-file type"): + main() + finally: + os.unlink(txt_file) + + def test_cli_creates_output_directory(self): + """Test that CLI creates output directory if it doesn't exist""" + # Create temporary CSV file + test_data = pd.DataFrame({ + 'query_text': ['SELECT * FROM SALES.CUSTOMERS'], + 'user_name': ['user1'], + 'start_time': ['2025-01-01'] + }) + + with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f: + test_data.to_csv(f.name, index=False) + csv_file = f.name + + with tempfile.TemporaryDirectory() as temp_dir: + output_dir = Path(temp_dir) / 'nested' / 'output' / 'dir' + + try: + test_args = [ + 'cli.py', + '--platform', 'snowflake', + '--input-file', csv_file, + '--output', str(output_dir) + ] + + with patch('sys.argv', test_args): + main() + + # Verify directory was created + assert output_dir.exists() + assert output_dir.is_dir() + + finally: + os.unlink(csv_file) + + def test_cli_markdown_output_format(self): + """Test CLI with explicit markdown output format""" + test_data = pd.DataFrame({ + 'query_text': ['SELECT * FROM SALES.CUSTOMERS'], + 'user_name': ['user1'], + 'start_time': ['2025-01-01'] + }) + + with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f: + test_data.to_csv(f.name, index=False) + csv_file = f.name + + with tempfile.TemporaryDirectory() as output_dir: + try: + test_args = [ + 'cli.py', + '--platform', 'snowflake', + '--input-file', csv_file, + '--output', output_dir, + '--output-format', 'markdown' + ] + + with patch('sys.argv', test_args): + main() + + # Verify markdown file was created + output_files = list(Path(output_dir).glob('recommendations_*.md')) + assert len(output_files) == 1 + + finally: + os.unlink(csv_file) + + +# Made with Bob \ No newline at end of file diff --git a/tests/src/data_product_recommender/test_data_product_recommender_conftest.py b/tests/src/data_product_recommender/test_data_product_recommender_conftest.py new file mode 100644 index 0000000..2d662bf --- /dev/null +++ b/tests/src/data_product_recommender/test_data_product_recommender_conftest.py @@ -0,0 +1,144 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Pytest configuration and shared fixtures +""" + +import pytest +import pandas as pd +import json +import tempfile +import os + + +@pytest.fixture +def sample_query_logs(): + """Sample query logs for testing""" + return pd.DataFrame({ + 'query_text': [ + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.CUSTOMERS WHERE active = true', + 'SELECT * FROM SALES.ORDERS', + 'SELECT * FROM SALES.CUSTOMERS c JOIN SALES.ORDERS o ON c.id = o.customer_id', + 'SELECT * FROM PRODUCT.CATALOG' + ], + 'user': ['user1', 'user2', 'user1', 'user3', 'user2'], + 'start_time': [ + '2025-01-01 10:00:00', + '2025-01-01 11:00:00', + '2025-01-01 12:00:00', + '2025-01-01 13:00:00', + '2025-01-01 14:00:00' + ] + }) + + +@pytest.fixture +def sample_query_logs_with_tables(): + """Sample query logs with tables already extracted""" + return pd.DataFrame({ + 'query_text': [ + 'SELECT * FROM SALES.CUSTOMERS', + 'SELECT * FROM SALES.CUSTOMERS WHERE active = true', + 'SELECT * FROM SALES.ORDERS', + 'SELECT * FROM SALES.CUSTOMERS c JOIN SALES.ORDERS o ON c.id = o.customer_id', + 'SELECT * FROM PRODUCT.CATALOG' + ], + 'user': ['user1', 'user2', 'user1', 'user3', 'user2'], + 'start_time': [ + '2025-01-01 10:00:00', + '2025-01-01 11:00:00', + '2025-01-01 12:00:00', + '2025-01-01 13:00:00', + '2025-01-01 14:00:00' + ], + 'tables': [ + ['SALES.CUSTOMERS'], + ['SALES.CUSTOMERS'], + ['SALES.ORDERS'], + ['SALES.CUSTOMERS', 'SALES.ORDERS'], + ['PRODUCT.CATALOG'] + ] + }) + + +@pytest.fixture +def sample_table_metrics(): + """Sample table metrics for testing""" + return pd.DataFrame({ + 'table': ['SALES.CUSTOMERS', 'SALES.ORDERS', 'PRODUCT.CATALOG'], + 'query_count': [10, 5, 2], + 'unique_users': [5, 3, 1], + 'related_tables': [ + ['SALES.ORDERS', 'PRODUCT.CATALOG'], + ['SALES.CUSTOMERS'], + [] + ], + 'related_table_count': [2, 1, 0] + }) + + +@pytest.fixture +def temp_json_file(sample_query_logs): + """Create a temporary JSON file with sample data""" + data = sample_query_logs.to_dict('records') + + # Rename columns to match raw format + for record in data: + record['user_name'] = record.pop('user') + + with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f: + json.dump(data, f) + temp_file = f.name + + yield temp_file + + # Cleanup + if os.path.exists(temp_file): + os.unlink(temp_file) + + +@pytest.fixture +def temp_csv_file(sample_query_logs): + """Create a temporary CSV file with sample data""" + # Rename columns to match raw format + df = sample_query_logs.copy() + df = df.rename(columns={'user': 'user_name'}) + + with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as f: + df.to_csv(f.name, index=False) + temp_file = f.name + + yield temp_file + + # Cleanup + if os.path.exists(temp_file): + os.unlink(temp_file) + + +@pytest.fixture +def temp_output_dir(): + """Create a temporary output directory""" + temp_dir = tempfile.mkdtemp() + yield temp_dir + + # Cleanup + import shutil + if os.path.exists(temp_dir): + shutil.rmtree(temp_dir) + +# Made with Bob diff --git a/tests/src/data_product_recommender/test_data_product_recommender_parsers.py b/tests/src/data_product_recommender/test_data_product_recommender_parsers.py new file mode 100644 index 0000000..03142cb --- /dev/null +++ b/tests/src/data_product_recommender/test_data_product_recommender_parsers.py @@ -0,0 +1,236 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Unit tests for query log parsers +""" + +import pytest +import pandas as pd +from wxdi.data_product_recommender.platforms import ( + SnowflakeQueryParser, + DatabricksQueryParser, + BigQueryQueryParser, + WatsonxDataQueryParser +) + + +class TestSnowflakeQueryParser: + """Tests for Snowflake query parser""" + + def setup_method(self): + """Setup test fixtures""" + self.parser = SnowflakeQueryParser() + + def test_normalize_columns(self): + """Test column normalization""" + df = pd.DataFrame({ + 'query_text': ['SELECT * FROM table1'], + 'user_name': ['user1'], + 'start_time': ['2025-01-01'] + }) + + result = self.parser.normalize_columns(df) + + assert 'query_text' in result.columns + assert 'user' in result.columns + assert 'start_time' in result.columns + + def test_extract_tables_simple(self): + """Test extracting tables from simple query""" + query = "SELECT * FROM SALES.CUSTOMERS" + tables = self.parser.extract_tables(query) + + assert len(tables) == 1 + assert 'SALES.CUSTOMERS' in tables + + def test_extract_tables_with_join(self): + """Test extracting tables from query with JOIN""" + query = "SELECT * FROM SALES.ORDERS o JOIN SALES.CUSTOMERS c ON o.customer_id = c.id" + tables = self.parser.extract_tables(query) + + assert len(tables) == 2 + assert 'SALES.ORDERS' in tables + assert 'SALES.CUSTOMERS' in tables + + def test_extract_tables_multiple_joins(self): + """Test extracting tables from query with multiple JOINs""" + query = """ + SELECT * FROM SALES.ORDERS o + LEFT JOIN SALES.CUSTOMERS c ON o.customer_id = c.id + INNER JOIN PRODUCT.CATALOG p ON o.product_id = p.id + """ + tables = self.parser.extract_tables(query) + + assert len(tables) == 3 + assert 'SALES.ORDERS' in tables + assert 'SALES.CUSTOMERS' in tables + assert 'PRODUCT.CATALOG' in tables + + def test_extract_tables_empty_query(self): + """Test extracting tables from empty query""" + tables = self.parser.extract_tables("") + assert len(tables) == 0 + + def test_extract_tables_none_query(self): + """Test extracting tables from None query""" + tables = self.parser.extract_tables(None) + assert len(tables) == 0 + + +class TestDatabricksQueryParser: + """Tests for Databricks query parser""" + + def setup_method(self): + """Setup test fixtures""" + self.parser = DatabricksQueryParser() + + def test_normalize_columns(self): + """Test column normalization""" + df = pd.DataFrame({ + 'statement_text': ['SELECT * FROM table1'], + 'executed_by': ['user1'], + 'start_time': ['2025-01-01'] + }) + + result = self.parser.normalize_columns(df) + + assert 'query_text' in result.columns + assert 'user' in result.columns + assert 'start_time' in result.columns + + def test_extract_tables_simple(self): + """Test extracting tables from simple query""" + query = "SELECT * FROM CLINICAL.PATIENTS" + tables = self.parser.extract_tables(query) + + assert len(tables) == 1 + assert 'CLINICAL.PATIENTS' in tables + + +class TestBigQueryQueryParser: + """Tests for BigQuery query parser""" + + def setup_method(self): + """Setup test fixtures""" + self.parser = BigQueryQueryParser() + + def test_normalize_columns(self): + """Test column normalization""" + df = pd.DataFrame({ + 'query': ['SELECT * FROM table1'], + 'user_email': ['user1@example.com'], + 'start_time': ['2025-01-01'] + }) + + result = self.parser.normalize_columns(df) + + assert 'query_text' in result.columns + assert 'user' in result.columns + assert 'start_time' in result.columns + + def test_extract_tables_with_backticks(self): + """Test extracting tables from BigQuery query with backticks""" + query = "SELECT * FROM `project-id.PRODUCT.CATALOG`" + tables = self.parser.extract_tables(query) + + assert len(tables) == 1 + assert 'PRODUCT.CATALOG' in tables + + def test_extract_tables_multiple_with_backticks(self): + """Test extracting multiple tables with project IDs""" + query = """ + SELECT * FROM `project-id.SALES.ORDERS` o + JOIN `project-id.CUSTOMER.PROFILES` c ON o.customer_id = c.id + """ + tables = self.parser.extract_tables(query) + + assert len(tables) == 2 + assert 'SALES.ORDERS' in tables + assert 'CUSTOMER.PROFILES' in tables + + +class TestWatsonxDataQueryParser: + """Tests for watsonx.data query parser""" + + def setup_method(self): + """Setup test fixtures""" + self.parser = WatsonxDataQueryParser() + + def test_normalize_columns(self): + """Test column normalization""" + df = pd.DataFrame({ + 'query': ['SELECT * FROM table1'], + 'user': ['user1'], + 'created': ['2025-01-01'] + }) + + result = self.parser.normalize_columns(df) + + assert 'query_text' in result.columns + assert 'user' in result.columns + assert 'start_time' in result.columns + + def test_extract_tables_simple(self): + """Test extracting tables from simple query""" + query = "SELECT * FROM NETWORK.SUBSCRIBERS" + tables = self.parser.extract_tables(query) + + assert len(tables) == 1 + assert 'NETWORK.SUBSCRIBERS' in tables + + +class TestParserEdgeCases: + """Test edge cases across all parsers""" + + @pytest.mark.parametrize("parser_class", [ + SnowflakeQueryParser, + DatabricksQueryParser, + BigQueryQueryParser, + WatsonxDataQueryParser + ]) + def test_extract_tables_with_subquery(self, parser_class): + """Test extracting tables from query with subquery""" + parser = parser_class() + query = """ + SELECT * FROM SALES.ORDERS + WHERE customer_id IN (SELECT id FROM SALES.CUSTOMERS WHERE active = true) + """ + tables = parser.extract_tables(query) + + # Should extract both tables + assert len(tables) >= 2 + + @pytest.mark.parametrize("parser_class", [ + SnowflakeQueryParser, + DatabricksQueryParser, + BigQueryQueryParser, + WatsonxDataQueryParser + ]) + def test_extract_tables_case_insensitive(self, parser_class): + """Test that table extraction is case-insensitive""" + parser = parser_class() + query_upper = "SELECT * FROM SALES.ORDERS" + query_lower = "select * from sales.orders" + + tables_upper = parser.extract_tables(query_upper) + tables_lower = parser.extract_tables(query_lower) + + # Both should extract the same table (normalized to uppercase) + assert len(tables_upper) == 1 + assert len(tables_lower) == 1 + +# Made with Bob diff --git a/tests/src/dph_services/__init__.py b/tests/src/dph_services/__init__.py new file mode 100644 index 0000000..db6be80 --- /dev/null +++ b/tests/src/dph_services/__init__.py @@ -0,0 +1,19 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""DPH Services test package""" + +# Made with Bob diff --git a/tests/src/dph_services/test_common.py b/tests/src/dph_services/test_common.py new file mode 100644 index 0000000..5a0e5f7 --- /dev/null +++ b/tests/src/dph_services/test_common.py @@ -0,0 +1,50 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Test methods in the common module +""" + +import unittest +from wxdi.dph_services import common + + +class TestCommon(unittest.TestCase): + """ + Test methods in the common module + """ + + def test_get_sdk_headers(self): + """ + Test the get_sdk_headers method + """ + headers = common.get_sdk_headers( + service_name='dph_services', service_version='V1', operation_id='operation1' + ) + self.assertIsNotNone(headers) + self.assertIsNotNone(headers.get('User-Agent')) + self.assertIn('data-product-python-sdk', headers.get('User-Agent')) + + def test_get_system_info(self): + """ + Test the get_system_info method + """ + system_info = common.get_system_info() + self.assertIsNotNone(system_info) + self.assertIn('lang=', system_info) + self.assertIn('arch=', system_info) + self.assertIn('os=', system_info) + self.assertIn('python.version=', system_info) diff --git a/tests/src/dph_services/test_dph_v1.py b/tests/src/dph_services/test_dph_v1.py new file mode 100644 index 0000000..d465a30 --- /dev/null +++ b/tests/src/dph_services/test_dph_v1.py @@ -0,0 +1,14205 @@ +# -*- coding: utf-8 -*- +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Unit Tests for DphV1 +""" + +from datetime import datetime, timezone +from ibm_cloud_sdk_core.authenticators.no_auth_authenticator import NoAuthAuthenticator +from ibm_cloud_sdk_core.utils import datetime_to_string, string_to_datetime +import inspect +import io +import json +import os +import pytest +import re +import requests +import responses +import tempfile +import urllib +from wxdi.dph_services.dph_v1 import * + + +_service = DphV1( + authenticator=NoAuthAuthenticator() +) + +_base_url = 'https://fake' +_service.set_service_url(_base_url) + + +def preprocess_url(operation_path: str): + """ + Returns the request url associated with the specified operation path. + This will be base_url concatenated with a quoted version of operation_path. + The returned request URL is used to register the mock response so it needs + to match the request URL that is formed by the requests library. + """ + + # Form the request URL from the base URL and operation path. + request_url = _base_url + operation_path + + # If the request url does NOT end with a /, then just return it as-is. + # Otherwise, return a regular expression that matches one or more trailing /. + if not request_url.endswith('/'): + return request_url + return re.compile(request_url.rstrip('/') + '/+') + + +############################################################################## +# Start of Service: Configuration +############################################################################## +# region + + +class TestNewInstance: + """ + Test Class for new_instance + """ + + def test_new_instance(self): + """ + new_instance() + """ + os.environ['TEST_SERVICE_AUTH_TYPE'] = 'noAuth' + + service = DphV1.new_instance( + service_name='TEST_SERVICE', + ) + + assert service is not None + assert isinstance(service, DphV1) + + def test_new_instance_without_authenticator(self): + """ + new_instance_without_authenticator() + """ + with pytest.raises(ValueError, match='authenticator must be provided'): + service = DphV1.new_instance( + service_name='TEST_SERVICE_NOT_FOUND', + ) + + +class TestGetInitializeStatus: + """ + Test Class for get_initialize_status + """ + + @responses.activate + def test_get_initialize_status_all_params(self): + """ + get_initialize_status() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/configuration/initialize/status') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "href": "https://api.example.com/configuration/initialize/status?catalog_id=d29c42eb-7100-4b7a-8257-c196dbcca1cd", "status": "not_started", "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "last_started_at": "2023-08-21T15:24:06.021Z", "last_finished_at": "2023-08-21T20:24:34.450Z", "initialized_options": [{"name": "name", "version": 1}], "workflows": {"data_access": {"definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}, "request_new_product": {"definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + container_id = 'testString' + + # Invoke method + response = _service.get_initialize_status( + container_id=container_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + + def test_get_initialize_status_all_params_with_retries(self): + # Enable retries and run test_get_initialize_status_all_params. + _service.enable_retries() + self.test_get_initialize_status_all_params() + + # Disable retries and run test_get_initialize_status_all_params. + _service.disable_retries() + self.test_get_initialize_status_all_params() + + @responses.activate + def test_get_initialize_status_required_params(self): + """ + test_get_initialize_status_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/configuration/initialize/status') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "href": "https://api.example.com/configuration/initialize/status?catalog_id=d29c42eb-7100-4b7a-8257-c196dbcca1cd", "status": "not_started", "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "last_started_at": "2023-08-21T15:24:06.021Z", "last_finished_at": "2023-08-21T20:24:34.450Z", "initialized_options": [{"name": "name", "version": 1}], "workflows": {"data_access": {"definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}, "request_new_product": {"definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Invoke method + response = _service.get_initialize_status() + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_initialize_status_required_params_with_retries(self): + # Enable retries and run test_get_initialize_status_required_params. + _service.enable_retries() + self.test_get_initialize_status_required_params() + + # Disable retries and run test_get_initialize_status_required_params. + _service.disable_retries() + self.test_get_initialize_status_required_params() + + +class TestGetServiceIdCredentials: + """ + Test Class for get_service_id_credentials + """ + + @responses.activate + def test_get_service_id_credentials_all_params(self): + """ + get_service_id_credentials() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/configuration/credentials') + mock_response = '{"name": "data-product-admin-service-id-API-key", "created_at": "2019-01-01T12:00:00.000Z"}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Invoke method + response = _service.get_service_id_credentials() + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_service_id_credentials_all_params_with_retries(self): + # Enable retries and run test_get_service_id_credentials_all_params. + _service.enable_retries() + self.test_get_service_id_credentials_all_params() + + # Disable retries and run test_get_service_id_credentials_all_params. + _service.disable_retries() + self.test_get_service_id_credentials_all_params() + + +class TestInitialize: + """ + Test Class for initialize + """ + + @responses.activate + def test_initialize_all_params(self): + """ + initialize() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/configuration/initialize') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "href": "https://api.example.com/configuration/initialize/status?catalog_id=d29c42eb-7100-4b7a-8257-c196dbcca1cd", "status": "not_started", "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "last_started_at": "2023-08-21T15:24:06.021Z", "last_finished_at": "2023-08-21T20:24:34.450Z", "initialized_options": [{"name": "name", "version": 1}], "workflows": {"data_access": {"definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}, "request_new_product": {"definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=202, + ) + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Set up parameter values + container = container_reference_model + include = ['delivery_methods', 'domains_multi_industry', 'data_product_samples', 'workflows', 'project', 'catalog_configurations'] + + # Invoke method + response = _service.initialize( + container=container, + include=include, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 202 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['container'] == container_reference_model + assert req_body['include'] == ['delivery_methods', 'domains_multi_industry', 'data_product_samples', 'workflows', 'project', 'catalog_configurations'] + + def test_initialize_all_params_with_retries(self): + # Enable retries and run test_initialize_all_params. + _service.enable_retries() + self.test_initialize_all_params() + + # Disable retries and run test_initialize_all_params. + _service.disable_retries() + self.test_initialize_all_params() + + +class TestManageApiKeys: + """ + Test Class for manage_api_keys + """ + + @responses.activate + def test_manage_api_keys_all_params(self): + """ + manage_api_keys() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/configuration/rotate_credentials') + responses.add( + responses.POST, + url, + status=204, + ) + + # Invoke method + response = _service.manage_api_keys() + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 204 + + def test_manage_api_keys_all_params_with_retries(self): + # Enable retries and run test_manage_api_keys_all_params. + _service.enable_retries() + self.test_manage_api_keys_all_params() + + # Disable retries and run test_manage_api_keys_all_params. + _service.disable_retries() + self.test_manage_api_keys_all_params() + + +# endregion +############################################################################## +# End of Service: Configuration +############################################################################## + +############################################################################## +# Start of Service: DataAssetVisualization +############################################################################## +# region + + +class TestNewInstance: + """ + Test Class for new_instance + """ + + def test_new_instance(self): + """ + new_instance() + """ + os.environ['TEST_SERVICE_AUTH_TYPE'] = 'noAuth' + + service = DphV1.new_instance( + service_name='TEST_SERVICE', + ) + + assert service is not None + assert isinstance(service, DphV1) + + def test_new_instance_without_authenticator(self): + """ + new_instance_without_authenticator() + """ + with pytest.raises(ValueError, match='authenticator must be provided'): + service = DphV1.new_instance( + service_name='TEST_SERVICE_NOT_FOUND', + ) + + +class TestCreateDataAssetVisualization: + """ + Test Class for create_data_asset_visualization + """ + + @responses.activate + def test_create_data_asset_visualization_all_params(self): + """ + create_data_asset_visualization() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_asset/visualization') + mock_response = '{"results": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a Visualization model + visualization_model = {} + visualization_model['id'] = 'testString' + visualization_model['name'] = 'testString' + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = '2be8f727-c5d2-4cb0-9216-f9888f428048' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = 'caeee3f3-756e-47d5-846d-da4600809e22' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ErrorMessage model + error_message_model = {} + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + # Construct a dict representation of a DataAssetRelationship model + data_asset_relationship_model = {} + data_asset_relationship_model['visualization'] = visualization_model + data_asset_relationship_model['asset'] = asset_reference_model + data_asset_relationship_model['related_asset'] = asset_reference_model + data_asset_relationship_model['error'] = error_message_model + + # Set up parameter values + assets = [data_asset_relationship_model] + + # Invoke method + response = _service.create_data_asset_visualization( + assets=assets, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['assets'] == [data_asset_relationship_model] + + def test_create_data_asset_visualization_all_params_with_retries(self): + # Enable retries and run test_create_data_asset_visualization_all_params. + _service.enable_retries() + self.test_create_data_asset_visualization_all_params() + + # Disable retries and run test_create_data_asset_visualization_all_params. + _service.disable_retries() + self.test_create_data_asset_visualization_all_params() + + +class TestReinitiateDataAssetVisualization: + """ + Test Class for reinitiate_data_asset_visualization + """ + + @responses.activate + def test_reinitiate_data_asset_visualization_all_params(self): + """ + reinitiate_data_asset_visualization() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_asset/visualization/reinitiate') + mock_response = '{"results": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a Visualization model + visualization_model = {} + visualization_model['id'] = 'testString' + visualization_model['name'] = 'testString' + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = '2be8f727-c5d2-4cb0-9216-f9888f428048' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = 'caeee3f3-756e-47d5-846d-da4600809e22' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ErrorMessage model + error_message_model = {} + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + # Construct a dict representation of a DataAssetRelationship model + data_asset_relationship_model = {} + data_asset_relationship_model['visualization'] = visualization_model + data_asset_relationship_model['asset'] = asset_reference_model + data_asset_relationship_model['related_asset'] = asset_reference_model + data_asset_relationship_model['error'] = error_message_model + + # Set up parameter values + assets = [data_asset_relationship_model] + + # Invoke method + response = _service.reinitiate_data_asset_visualization( + assets=assets, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['assets'] == [data_asset_relationship_model] + + def test_reinitiate_data_asset_visualization_all_params_with_retries(self): + # Enable retries and run test_reinitiate_data_asset_visualization_all_params. + _service.enable_retries() + self.test_reinitiate_data_asset_visualization_all_params() + + # Disable retries and run test_reinitiate_data_asset_visualization_all_params. + _service.disable_retries() + self.test_reinitiate_data_asset_visualization_all_params() + + +# endregion +############################################################################## +# End of Service: DataAssetVisualization +############################################################################## + +############################################################################## +# Start of Service: DataProducts +############################################################################## +# region + + +class TestNewInstance: + """ + Test Class for new_instance + """ + + def test_new_instance(self): + """ + new_instance() + """ + os.environ['TEST_SERVICE_AUTH_TYPE'] = 'noAuth' + + service = DphV1.new_instance( + service_name='TEST_SERVICE', + ) + + assert service is not None + assert isinstance(service, DphV1) + + def test_new_instance_without_authenticator(self): + """ + new_instance_without_authenticator() + """ + with pytest.raises(ValueError, match='authenticator must be provided'): + service = DphV1.new_instance( + service_name='TEST_SERVICE_NOT_FOUND', + ) + + +class TestListDataProducts: + """ + Test Class for list_data_products + """ + + @responses.activate + def test_list_data_products_all_params(self): + """ + list_data_products() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "data_products": [{"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "name": "name"}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + limit = 200 + start = 'testString' + + # Invoke method + response = _service.list_data_products( + limit=limit, + start=start, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'limit={}'.format(limit) in query_string + assert 'start={}'.format(start) in query_string + + def test_list_data_products_all_params_with_retries(self): + # Enable retries and run test_list_data_products_all_params. + _service.enable_retries() + self.test_list_data_products_all_params() + + # Disable retries and run test_list_data_products_all_params. + _service.disable_retries() + self.test_list_data_products_all_params() + + @responses.activate + def test_list_data_products_required_params(self): + """ + test_list_data_products_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "data_products": [{"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "name": "name"}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Invoke method + response = _service.list_data_products() + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_list_data_products_required_params_with_retries(self): + # Enable retries and run test_list_data_products_required_params. + _service.enable_retries() + self.test_list_data_products_required_params() + + # Disable retries and run test_list_data_products_required_params. + _service.disable_retries() + self.test_list_data_products_required_params() + + @responses.activate + def test_list_data_products_with_pager_get_next(self): + """ + test_list_data_products_with_pager_get_next() + """ + # Set up a two-page mock response + url = preprocess_url('/data_product_exchange/v1/data_products') + mock_response1 = '{"next":{"start":"1"},"total_count":2,"limit":1,"data_products":[{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"name":"name"}]}' + mock_response2 = '{"total_count":2,"limit":1,"data_products":[{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"name":"name"}]}' + responses.add( + responses.GET, + url, + body=mock_response1, + content_type='application/json', + status=200, + ) + responses.add( + responses.GET, + url, + body=mock_response2, + content_type='application/json', + status=200, + ) + + # Exercise the pager class for this operation + all_results = [] + pager = DataProductsPager( + client=_service, + limit=10, + ) + while pager.has_next(): + next_page = pager.get_next() + assert next_page is not None + all_results.extend(next_page) + assert len(all_results) == 2 + + @responses.activate + def test_list_data_products_with_pager_get_all(self): + """ + test_list_data_products_with_pager_get_all() + """ + # Set up a two-page mock response + url = preprocess_url('/data_product_exchange/v1/data_products') + mock_response1 = '{"next":{"start":"1"},"total_count":2,"limit":1,"data_products":[{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"name":"name"}]}' + mock_response2 = '{"total_count":2,"limit":1,"data_products":[{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"name":"name"}]}' + responses.add( + responses.GET, + url, + body=mock_response1, + content_type='application/json', + status=200, + ) + responses.add( + responses.GET, + url, + body=mock_response2, + content_type='application/json', + status=200, + ) + + # Exercise the pager class for this operation + pager = DataProductsPager( + client=_service, + limit=10, + ) + all_results = pager.get_all() + assert all_results is not None + assert len(all_results) == 2 + + +class TestCreateDataProduct: + """ + Test Class for create_data_product + """ + + @responses.activate + def test_create_data_product_all_params(self): + """ + create_data_product() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products') + mock_response = '{"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "name": "name", "latest_release": {"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}, "drafts": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a DataProductDraftVersionRelease model + data_product_draft_version_release_model = {} + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a dict representation of a DataProductIdentity model + data_product_identity_model = {} + data_product_identity_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_identity_model['release'] = data_product_draft_version_release_model + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a UseCase model + use_case_model = {} + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'owner' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractTerms model + contract_terms_model = {} + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + # Construct a dict representation of a AssetPartReference model + asset_part_reference_model = {} + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + # Construct a dict representation of a EngineDetailsModel model + engine_details_model_model = {} + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + # Construct a dict representation of a ProducerInputModel model + producer_input_model_model = {} + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + # Construct a dict representation of a DeliveryMethodPropertiesModel model + delivery_method_properties_model_model = {} + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + # Construct a dict representation of a DeliveryMethod model + delivery_method_model = {} + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + # Construct a dict representation of a DataProductPart model + data_product_part_model = {} + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + # Construct a dict representation of a DataProductCustomWorkflowDefinition model + data_product_custom_workflow_definition_model = {} + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a dict representation of a DataProductOrderAccessRequest model + data_product_order_access_request_model = {} + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + # Construct a dict representation of a DataProductWorkflows model + data_product_workflows_model = {} + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + # Construct a dict representation of a AssetListAccessControl model + asset_list_access_control_model = {} + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + # Construct a dict representation of a ContainerIdentity model + container_identity_model = {} + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a dict representation of a AssetPrototype model + asset_prototype_model = {} + asset_prototype_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_prototype_model['container'] = container_identity_model + + # Construct a dict representation of a DataProductDraftPrototype model + data_product_draft_prototype_model = {} + data_product_draft_prototype_model['version'] = '1.0.0' + data_product_draft_prototype_model['state'] = 'draft' + data_product_draft_prototype_model['data_product'] = data_product_identity_model + data_product_draft_prototype_model['name'] = 'My New Data Product' + data_product_draft_prototype_model['description'] = 'This is a description of My Data Product.' + data_product_draft_prototype_model['tags'] = ['testString'] + data_product_draft_prototype_model['use_cases'] = [use_case_model] + data_product_draft_prototype_model['types'] = ['data'] + data_product_draft_prototype_model['contract_terms'] = [contract_terms_model] + data_product_draft_prototype_model['domain'] = domain_model + data_product_draft_prototype_model['parts_out'] = [data_product_part_model] + data_product_draft_prototype_model['workflows'] = data_product_workflows_model + data_product_draft_prototype_model['dataview_enabled'] = True + data_product_draft_prototype_model['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_draft_prototype_model['access_control'] = asset_list_access_control_model + data_product_draft_prototype_model['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_draft_prototype_model['sub_container'] = container_identity_model + data_product_draft_prototype_model['is_restricted'] = True + data_product_draft_prototype_model['asset'] = asset_prototype_model + + # Set up parameter values + drafts = [data_product_draft_prototype_model] + limit = 200 + start = 'testString' + + # Invoke method + response = _service.create_data_product( + drafts, + limit=limit, + start=start, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'limit={}'.format(limit) in query_string + assert 'start={}'.format(start) in query_string + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['drafts'] == [data_product_draft_prototype_model] + + def test_create_data_product_all_params_with_retries(self): + # Enable retries and run test_create_data_product_all_params. + _service.enable_retries() + self.test_create_data_product_all_params() + + # Disable retries and run test_create_data_product_all_params. + _service.disable_retries() + self.test_create_data_product_all_params() + + @responses.activate + def test_create_data_product_required_params(self): + """ + test_create_data_product_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products') + mock_response = '{"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "name": "name", "latest_release": {"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}, "drafts": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a DataProductDraftVersionRelease model + data_product_draft_version_release_model = {} + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a dict representation of a DataProductIdentity model + data_product_identity_model = {} + data_product_identity_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_identity_model['release'] = data_product_draft_version_release_model + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a UseCase model + use_case_model = {} + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'owner' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractTerms model + contract_terms_model = {} + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + # Construct a dict representation of a AssetPartReference model + asset_part_reference_model = {} + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + # Construct a dict representation of a EngineDetailsModel model + engine_details_model_model = {} + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + # Construct a dict representation of a ProducerInputModel model + producer_input_model_model = {} + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + # Construct a dict representation of a DeliveryMethodPropertiesModel model + delivery_method_properties_model_model = {} + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + # Construct a dict representation of a DeliveryMethod model + delivery_method_model = {} + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + # Construct a dict representation of a DataProductPart model + data_product_part_model = {} + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + # Construct a dict representation of a DataProductCustomWorkflowDefinition model + data_product_custom_workflow_definition_model = {} + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a dict representation of a DataProductOrderAccessRequest model + data_product_order_access_request_model = {} + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + # Construct a dict representation of a DataProductWorkflows model + data_product_workflows_model = {} + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + # Construct a dict representation of a AssetListAccessControl model + asset_list_access_control_model = {} + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + # Construct a dict representation of a ContainerIdentity model + container_identity_model = {} + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a dict representation of a AssetPrototype model + asset_prototype_model = {} + asset_prototype_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_prototype_model['container'] = container_identity_model + + # Construct a dict representation of a DataProductDraftPrototype model + data_product_draft_prototype_model = {} + data_product_draft_prototype_model['version'] = '1.0.0' + data_product_draft_prototype_model['state'] = 'draft' + data_product_draft_prototype_model['data_product'] = data_product_identity_model + data_product_draft_prototype_model['name'] = 'My New Data Product' + data_product_draft_prototype_model['description'] = 'This is a description of My Data Product.' + data_product_draft_prototype_model['tags'] = ['testString'] + data_product_draft_prototype_model['use_cases'] = [use_case_model] + data_product_draft_prototype_model['types'] = ['data'] + data_product_draft_prototype_model['contract_terms'] = [contract_terms_model] + data_product_draft_prototype_model['domain'] = domain_model + data_product_draft_prototype_model['parts_out'] = [data_product_part_model] + data_product_draft_prototype_model['workflows'] = data_product_workflows_model + data_product_draft_prototype_model['dataview_enabled'] = True + data_product_draft_prototype_model['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_draft_prototype_model['access_control'] = asset_list_access_control_model + data_product_draft_prototype_model['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_draft_prototype_model['sub_container'] = container_identity_model + data_product_draft_prototype_model['is_restricted'] = True + data_product_draft_prototype_model['asset'] = asset_prototype_model + + # Set up parameter values + drafts = [data_product_draft_prototype_model] + + # Invoke method + response = _service.create_data_product( + drafts, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['drafts'] == [data_product_draft_prototype_model] + + def test_create_data_product_required_params_with_retries(self): + # Enable retries and run test_create_data_product_required_params. + _service.enable_retries() + self.test_create_data_product_required_params() + + # Disable retries and run test_create_data_product_required_params. + _service.disable_retries() + self.test_create_data_product_required_params() + + @responses.activate + def test_create_data_product_value_error(self): + """ + test_create_data_product_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products') + mock_response = '{"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "name": "name", "latest_release": {"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}, "drafts": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a DataProductDraftVersionRelease model + data_product_draft_version_release_model = {} + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a dict representation of a DataProductIdentity model + data_product_identity_model = {} + data_product_identity_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_identity_model['release'] = data_product_draft_version_release_model + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a UseCase model + use_case_model = {} + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'owner' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractTerms model + contract_terms_model = {} + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + # Construct a dict representation of a AssetPartReference model + asset_part_reference_model = {} + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + # Construct a dict representation of a EngineDetailsModel model + engine_details_model_model = {} + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + # Construct a dict representation of a ProducerInputModel model + producer_input_model_model = {} + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + # Construct a dict representation of a DeliveryMethodPropertiesModel model + delivery_method_properties_model_model = {} + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + # Construct a dict representation of a DeliveryMethod model + delivery_method_model = {} + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + # Construct a dict representation of a DataProductPart model + data_product_part_model = {} + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + # Construct a dict representation of a DataProductCustomWorkflowDefinition model + data_product_custom_workflow_definition_model = {} + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a dict representation of a DataProductOrderAccessRequest model + data_product_order_access_request_model = {} + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + # Construct a dict representation of a DataProductWorkflows model + data_product_workflows_model = {} + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + # Construct a dict representation of a AssetListAccessControl model + asset_list_access_control_model = {} + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + # Construct a dict representation of a ContainerIdentity model + container_identity_model = {} + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a dict representation of a AssetPrototype model + asset_prototype_model = {} + asset_prototype_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_prototype_model['container'] = container_identity_model + + # Construct a dict representation of a DataProductDraftPrototype model + data_product_draft_prototype_model = {} + data_product_draft_prototype_model['version'] = '1.0.0' + data_product_draft_prototype_model['state'] = 'draft' + data_product_draft_prototype_model['data_product'] = data_product_identity_model + data_product_draft_prototype_model['name'] = 'My New Data Product' + data_product_draft_prototype_model['description'] = 'This is a description of My Data Product.' + data_product_draft_prototype_model['tags'] = ['testString'] + data_product_draft_prototype_model['use_cases'] = [use_case_model] + data_product_draft_prototype_model['types'] = ['data'] + data_product_draft_prototype_model['contract_terms'] = [contract_terms_model] + data_product_draft_prototype_model['domain'] = domain_model + data_product_draft_prototype_model['parts_out'] = [data_product_part_model] + data_product_draft_prototype_model['workflows'] = data_product_workflows_model + data_product_draft_prototype_model['dataview_enabled'] = True + data_product_draft_prototype_model['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_draft_prototype_model['access_control'] = asset_list_access_control_model + data_product_draft_prototype_model['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_draft_prototype_model['sub_container'] = container_identity_model + data_product_draft_prototype_model['is_restricted'] = True + data_product_draft_prototype_model['asset'] = asset_prototype_model + + # Set up parameter values + drafts = [data_product_draft_prototype_model] + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "drafts": drafts, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.create_data_product(**req_copy) + + def test_create_data_product_value_error_with_retries(self): + # Enable retries and run test_create_data_product_value_error. + _service.enable_retries() + self.test_create_data_product_value_error() + + # Disable retries and run test_create_data_product_value_error. + _service.disable_retries() + self.test_create_data_product_value_error() + + +class TestGetDataProduct: + """ + Test Class for get_data_product + """ + + @responses.activate + def test_get_data_product_all_params(self): + """ + get_data_product() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString') + mock_response = '{"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "name": "name", "latest_release": {"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}, "drafts": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + + # Invoke method + response = _service.get_data_product( + data_product_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_data_product_all_params_with_retries(self): + # Enable retries and run test_get_data_product_all_params. + _service.enable_retries() + self.test_get_data_product_all_params() + + # Disable retries and run test_get_data_product_all_params. + _service.disable_retries() + self.test_get_data_product_all_params() + + @responses.activate + def test_get_data_product_value_error(self): + """ + test_get_data_product_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString') + mock_response = '{"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "name": "name", "latest_release": {"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}, "drafts": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_data_product(**req_copy) + + def test_get_data_product_value_error_with_retries(self): + # Enable retries and run test_get_data_product_value_error. + _service.enable_retries() + self.test_get_data_product_value_error() + + # Disable retries and run test_get_data_product_value_error. + _service.disable_retries() + self.test_get_data_product_value_error() + + +# endregion +############################################################################## +# End of Service: DataProducts +############################################################################## + +############################################################################## +# Start of Service: DataProductDrafts +############################################################################## +# region + + +class TestNewInstance: + """ + Test Class for new_instance + """ + + def test_new_instance(self): + """ + new_instance() + """ + os.environ['TEST_SERVICE_AUTH_TYPE'] = 'noAuth' + + service = DphV1.new_instance( + service_name='TEST_SERVICE', + ) + + assert service is not None + assert isinstance(service, DphV1) + + def test_new_instance_without_authenticator(self): + """ + new_instance_without_authenticator() + """ + with pytest.raises(ValueError, match='authenticator must be provided'): + service = DphV1.new_instance( + service_name='TEST_SERVICE_NOT_FOUND', + ) + + +class TestCompleteDraftContractTermsDocument: + """ + Test Class for complete_draft_contract_terms_document + """ + + @responses.activate + def test_complete_draft_contract_terms_document_all_params(self): + """ + complete_draft_contract_terms_document() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents/testString/complete') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + + # Invoke method + response = _service.complete_draft_contract_terms_document( + data_product_id, + draft_id, + contract_terms_id, + document_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_complete_draft_contract_terms_document_all_params_with_retries(self): + # Enable retries and run test_complete_draft_contract_terms_document_all_params. + _service.enable_retries() + self.test_complete_draft_contract_terms_document_all_params() + + # Disable retries and run test_complete_draft_contract_terms_document_all_params. + _service.disable_retries() + self.test_complete_draft_contract_terms_document_all_params() + + @responses.activate + def test_complete_draft_contract_terms_document_value_error(self): + """ + test_complete_draft_contract_terms_document_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents/testString/complete') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "contract_terms_id": contract_terms_id, + "document_id": document_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.complete_draft_contract_terms_document(**req_copy) + + def test_complete_draft_contract_terms_document_value_error_with_retries(self): + # Enable retries and run test_complete_draft_contract_terms_document_value_error. + _service.enable_retries() + self.test_complete_draft_contract_terms_document_value_error() + + # Disable retries and run test_complete_draft_contract_terms_document_value_error. + _service.disable_retries() + self.test_complete_draft_contract_terms_document_value_error() + + +class TestListDataProductDrafts: + """ + Test Class for list_data_product_drafts + """ + + @responses.activate + def test_list_data_product_drafts_all_params(self): + """ + list_data_product_drafts() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "drafts": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + asset_container_id = 'testString' + version = 'testString' + limit = 200 + start = 'testString' + + # Invoke method + response = _service.list_data_product_drafts( + data_product_id, + asset_container_id=asset_container_id, + version=version, + limit=limit, + start=start, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'asset.container.id={}'.format(asset_container_id) in query_string + assert 'version={}'.format(version) in query_string + assert 'limit={}'.format(limit) in query_string + assert 'start={}'.format(start) in query_string + + def test_list_data_product_drafts_all_params_with_retries(self): + # Enable retries and run test_list_data_product_drafts_all_params. + _service.enable_retries() + self.test_list_data_product_drafts_all_params() + + # Disable retries and run test_list_data_product_drafts_all_params. + _service.disable_retries() + self.test_list_data_product_drafts_all_params() + + @responses.activate + def test_list_data_product_drafts_required_params(self): + """ + test_list_data_product_drafts_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "drafts": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + + # Invoke method + response = _service.list_data_product_drafts( + data_product_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_list_data_product_drafts_required_params_with_retries(self): + # Enable retries and run test_list_data_product_drafts_required_params. + _service.enable_retries() + self.test_list_data_product_drafts_required_params() + + # Disable retries and run test_list_data_product_drafts_required_params. + _service.disable_retries() + self.test_list_data_product_drafts_required_params() + + @responses.activate + def test_list_data_product_drafts_value_error(self): + """ + test_list_data_product_drafts_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "drafts": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.list_data_product_drafts(**req_copy) + + def test_list_data_product_drafts_value_error_with_retries(self): + # Enable retries and run test_list_data_product_drafts_value_error. + _service.enable_retries() + self.test_list_data_product_drafts_value_error() + + # Disable retries and run test_list_data_product_drafts_value_error. + _service.disable_retries() + self.test_list_data_product_drafts_value_error() + + @responses.activate + def test_list_data_product_drafts_with_pager_get_next(self): + """ + test_list_data_product_drafts_with_pager_get_next() + """ + # Set up a two-page mock response + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts') + mock_response1 = '{"next":{"start":"1"},"total_count":2,"limit":1,"drafts":[{"version":"1.0.0","state":"draft","data_product":{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"name":"My Data Product","description":"This is a description of My Data Product.","tags":["tags"],"use_cases":[{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}],"types":["data"],"contract_terms":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"id":"id","documents":[{"url":"url","type":"terms_and_conditions","name":"name","id":"2b0bf220-079c-11ee-be56-0242ac120002","attachment":{"id":"id"},"upload_url":"upload_url"}],"error_msg":"error_msg","overview":{"api_version":"v3.0.1","kind":"DataContract","name":"Sample Data Contract","version":"0.0.0","domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"more_info":"List of links to sources that provide more details on the data contract."},"description":{"purpose":"Used for customer behavior analysis.","limitations":"Data cannot be used for marketing.","usage":"Data should be used only for analytics.","more_info":[{"type":"privacy-statement","url":"https://moreinfo.example.com"}],"custom_properties":"{\\"property1\\":\\"value1\\"}"},"organization":[{"user_id":"IBMid-691000IN4G","role":"owner"}],"roles":[{"role":"owner"}],"price":{"amount":"100.0","currency":"USD","unit":"megabyte"},"sla":[{"default_element":"Standard SLA Policy","properties":[{"property":"Uptime Guarantee","value":"99.9"}]}],"support_and_communication":[{"channel":"Email Support","url":"https://support.example.com"}],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}],"contract_test":{"status":"pass","last_tested_time":"last_tested_time","message":"message"},"servers":[{"server":"server","asset":{"id":"id","name":"name"},"connection_id":"connection_id","type":"type","description":"description","environment":"environment","account":"account","catalog":"catalog","database":"database","dataset":"dataset","delimiter":"delimiter","endpoint_url":"endpoint_url","format":"format","host":"host","location":"location","path":"path","port":"port","project":"project","region":"region","region_name":"region_name","schema":"schema","service_name":"service_name","staging_dir":"staging_dir","stream":"stream","warehouse":"warehouse","roles":["roles"],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}]}],"schema":[{"asset_id":"2b0bf220-079c-11ee-be56-0242ac120002","connection_id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","description":"description","connection_path":"connection_path","physical_type":"physical_type","properties":[{"name":"name","type":{"type":"type","length":"length","scale":"scale","nullable":"nullable","signed":"signed","native_type":"native_type"},"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}],"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}]}],"domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"parts_out":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"type":"data_asset"},"delivery_methods":[{"id":"09cf5fcc-cb9d-4995-a8e4-16517b25229f","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"getproperties":{"producer_input":{"engine_details":{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]},"engines":[{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]}]}}}]}],"workflows":{"order_access_request":{"task_assignee_users":["task_assignee_users"],"pre_approved_users":["pre_approved_users"],"custom_workflow_definition":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"}}},"dataview_enabled":true,"comments":"Comments by a producer that are provided either at the time of data product version creation or retiring","access_control":{"owner":"IBMid-696000KYV9"},"last_updated_at":"2019-01-01T12:00:00.000Z","sub_container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd"},"is_restricted":false,"id":"2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd","asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}}]}' + mock_response2 = '{"total_count":2,"limit":1,"drafts":[{"version":"1.0.0","state":"draft","data_product":{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"name":"My Data Product","description":"This is a description of My Data Product.","tags":["tags"],"use_cases":[{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}],"types":["data"],"contract_terms":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"id":"id","documents":[{"url":"url","type":"terms_and_conditions","name":"name","id":"2b0bf220-079c-11ee-be56-0242ac120002","attachment":{"id":"id"},"upload_url":"upload_url"}],"error_msg":"error_msg","overview":{"api_version":"v3.0.1","kind":"DataContract","name":"Sample Data Contract","version":"0.0.0","domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"more_info":"List of links to sources that provide more details on the data contract."},"description":{"purpose":"Used for customer behavior analysis.","limitations":"Data cannot be used for marketing.","usage":"Data should be used only for analytics.","more_info":[{"type":"privacy-statement","url":"https://moreinfo.example.com"}],"custom_properties":"{\\"property1\\":\\"value1\\"}"},"organization":[{"user_id":"IBMid-691000IN4G","role":"owner"}],"roles":[{"role":"owner"}],"price":{"amount":"100.0","currency":"USD","unit":"megabyte"},"sla":[{"default_element":"Standard SLA Policy","properties":[{"property":"Uptime Guarantee","value":"99.9"}]}],"support_and_communication":[{"channel":"Email Support","url":"https://support.example.com"}],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}],"contract_test":{"status":"pass","last_tested_time":"last_tested_time","message":"message"},"servers":[{"server":"server","asset":{"id":"id","name":"name"},"connection_id":"connection_id","type":"type","description":"description","environment":"environment","account":"account","catalog":"catalog","database":"database","dataset":"dataset","delimiter":"delimiter","endpoint_url":"endpoint_url","format":"format","host":"host","location":"location","path":"path","port":"port","project":"project","region":"region","region_name":"region_name","schema":"schema","service_name":"service_name","staging_dir":"staging_dir","stream":"stream","warehouse":"warehouse","roles":["roles"],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}]}],"schema":[{"asset_id":"2b0bf220-079c-11ee-be56-0242ac120002","connection_id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","description":"description","connection_path":"connection_path","physical_type":"physical_type","properties":[{"name":"name","type":{"type":"type","length":"length","scale":"scale","nullable":"nullable","signed":"signed","native_type":"native_type"},"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}],"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}]}],"domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"parts_out":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"type":"data_asset"},"delivery_methods":[{"id":"09cf5fcc-cb9d-4995-a8e4-16517b25229f","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"getproperties":{"producer_input":{"engine_details":{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]},"engines":[{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]}]}}}]}],"workflows":{"order_access_request":{"task_assignee_users":["task_assignee_users"],"pre_approved_users":["pre_approved_users"],"custom_workflow_definition":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"}}},"dataview_enabled":true,"comments":"Comments by a producer that are provided either at the time of data product version creation or retiring","access_control":{"owner":"IBMid-696000KYV9"},"last_updated_at":"2019-01-01T12:00:00.000Z","sub_container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd"},"is_restricted":false,"id":"2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd","asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response1, + content_type='application/json', + status=200, + ) + responses.add( + responses.GET, + url, + body=mock_response2, + content_type='application/json', + status=200, + ) + + # Exercise the pager class for this operation + all_results = [] + pager = DataProductDraftsPager( + client=_service, + data_product_id='testString', + asset_container_id='testString', + version='testString', + limit=10, + ) + while pager.has_next(): + next_page = pager.get_next() + assert next_page is not None + all_results.extend(next_page) + assert len(all_results) == 2 + + @responses.activate + def test_list_data_product_drafts_with_pager_get_all(self): + """ + test_list_data_product_drafts_with_pager_get_all() + """ + # Set up a two-page mock response + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts') + mock_response1 = '{"next":{"start":"1"},"total_count":2,"limit":1,"drafts":[{"version":"1.0.0","state":"draft","data_product":{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"name":"My Data Product","description":"This is a description of My Data Product.","tags":["tags"],"use_cases":[{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}],"types":["data"],"contract_terms":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"id":"id","documents":[{"url":"url","type":"terms_and_conditions","name":"name","id":"2b0bf220-079c-11ee-be56-0242ac120002","attachment":{"id":"id"},"upload_url":"upload_url"}],"error_msg":"error_msg","overview":{"api_version":"v3.0.1","kind":"DataContract","name":"Sample Data Contract","version":"0.0.0","domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"more_info":"List of links to sources that provide more details on the data contract."},"description":{"purpose":"Used for customer behavior analysis.","limitations":"Data cannot be used for marketing.","usage":"Data should be used only for analytics.","more_info":[{"type":"privacy-statement","url":"https://moreinfo.example.com"}],"custom_properties":"{\\"property1\\":\\"value1\\"}"},"organization":[{"user_id":"IBMid-691000IN4G","role":"owner"}],"roles":[{"role":"owner"}],"price":{"amount":"100.0","currency":"USD","unit":"megabyte"},"sla":[{"default_element":"Standard SLA Policy","properties":[{"property":"Uptime Guarantee","value":"99.9"}]}],"support_and_communication":[{"channel":"Email Support","url":"https://support.example.com"}],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}],"contract_test":{"status":"pass","last_tested_time":"last_tested_time","message":"message"},"servers":[{"server":"server","asset":{"id":"id","name":"name"},"connection_id":"connection_id","type":"type","description":"description","environment":"environment","account":"account","catalog":"catalog","database":"database","dataset":"dataset","delimiter":"delimiter","endpoint_url":"endpoint_url","format":"format","host":"host","location":"location","path":"path","port":"port","project":"project","region":"region","region_name":"region_name","schema":"schema","service_name":"service_name","staging_dir":"staging_dir","stream":"stream","warehouse":"warehouse","roles":["roles"],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}]}],"schema":[{"asset_id":"2b0bf220-079c-11ee-be56-0242ac120002","connection_id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","description":"description","connection_path":"connection_path","physical_type":"physical_type","properties":[{"name":"name","type":{"type":"type","length":"length","scale":"scale","nullable":"nullable","signed":"signed","native_type":"native_type"},"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}],"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}]}],"domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"parts_out":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"type":"data_asset"},"delivery_methods":[{"id":"09cf5fcc-cb9d-4995-a8e4-16517b25229f","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"getproperties":{"producer_input":{"engine_details":{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]},"engines":[{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]}]}}}]}],"workflows":{"order_access_request":{"task_assignee_users":["task_assignee_users"],"pre_approved_users":["pre_approved_users"],"custom_workflow_definition":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"}}},"dataview_enabled":true,"comments":"Comments by a producer that are provided either at the time of data product version creation or retiring","access_control":{"owner":"IBMid-696000KYV9"},"last_updated_at":"2019-01-01T12:00:00.000Z","sub_container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd"},"is_restricted":false,"id":"2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd","asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}}]}' + mock_response2 = '{"total_count":2,"limit":1,"drafts":[{"version":"1.0.0","state":"draft","data_product":{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"name":"My Data Product","description":"This is a description of My Data Product.","tags":["tags"],"use_cases":[{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}],"types":["data"],"contract_terms":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"id":"id","documents":[{"url":"url","type":"terms_and_conditions","name":"name","id":"2b0bf220-079c-11ee-be56-0242ac120002","attachment":{"id":"id"},"upload_url":"upload_url"}],"error_msg":"error_msg","overview":{"api_version":"v3.0.1","kind":"DataContract","name":"Sample Data Contract","version":"0.0.0","domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"more_info":"List of links to sources that provide more details on the data contract."},"description":{"purpose":"Used for customer behavior analysis.","limitations":"Data cannot be used for marketing.","usage":"Data should be used only for analytics.","more_info":[{"type":"privacy-statement","url":"https://moreinfo.example.com"}],"custom_properties":"{\\"property1\\":\\"value1\\"}"},"organization":[{"user_id":"IBMid-691000IN4G","role":"owner"}],"roles":[{"role":"owner"}],"price":{"amount":"100.0","currency":"USD","unit":"megabyte"},"sla":[{"default_element":"Standard SLA Policy","properties":[{"property":"Uptime Guarantee","value":"99.9"}]}],"support_and_communication":[{"channel":"Email Support","url":"https://support.example.com"}],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}],"contract_test":{"status":"pass","last_tested_time":"last_tested_time","message":"message"},"servers":[{"server":"server","asset":{"id":"id","name":"name"},"connection_id":"connection_id","type":"type","description":"description","environment":"environment","account":"account","catalog":"catalog","database":"database","dataset":"dataset","delimiter":"delimiter","endpoint_url":"endpoint_url","format":"format","host":"host","location":"location","path":"path","port":"port","project":"project","region":"region","region_name":"region_name","schema":"schema","service_name":"service_name","staging_dir":"staging_dir","stream":"stream","warehouse":"warehouse","roles":["roles"],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}]}],"schema":[{"asset_id":"2b0bf220-079c-11ee-be56-0242ac120002","connection_id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","description":"description","connection_path":"connection_path","physical_type":"physical_type","properties":[{"name":"name","type":{"type":"type","length":"length","scale":"scale","nullable":"nullable","signed":"signed","native_type":"native_type"},"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}],"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}]}],"domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"parts_out":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"type":"data_asset"},"delivery_methods":[{"id":"09cf5fcc-cb9d-4995-a8e4-16517b25229f","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"getproperties":{"producer_input":{"engine_details":{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]},"engines":[{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]}]}}}]}],"workflows":{"order_access_request":{"task_assignee_users":["task_assignee_users"],"pre_approved_users":["pre_approved_users"],"custom_workflow_definition":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"}}},"dataview_enabled":true,"comments":"Comments by a producer that are provided either at the time of data product version creation or retiring","access_control":{"owner":"IBMid-696000KYV9"},"last_updated_at":"2019-01-01T12:00:00.000Z","sub_container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd"},"is_restricted":false,"id":"2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd","asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response1, + content_type='application/json', + status=200, + ) + responses.add( + responses.GET, + url, + body=mock_response2, + content_type='application/json', + status=200, + ) + + # Exercise the pager class for this operation + pager = DataProductDraftsPager( + client=_service, + data_product_id='testString', + asset_container_id='testString', + version='testString', + limit=10, + ) + all_results = pager.get_all() + assert all_results is not None + assert len(all_results) == 2 + + +class TestCreateDataProductDraft: + """ + Test Class for create_data_product_draft + """ + + @responses.activate + def test_create_data_product_draft_all_params(self): + """ + create_data_product_draft() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a ContainerIdentity model + container_identity_model = {} + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a dict representation of a AssetPrototype model + asset_prototype_model = {} + asset_prototype_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_prototype_model['container'] = container_identity_model + + # Construct a dict representation of a DataProductDraftVersionRelease model + data_product_draft_version_release_model = {} + data_product_draft_version_release_model['id'] = '8bf83660-11fe-4427-a72a-8d8359af24e3' + + # Construct a dict representation of a DataProductIdentity model + data_product_identity_model = {} + data_product_identity_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_identity_model['release'] = data_product_draft_version_release_model + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a UseCase model + use_case_model = {} + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'owner' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractTerms model + contract_terms_model = {} + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + # Construct a dict representation of a AssetPartReference model + asset_part_reference_model = {} + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + # Construct a dict representation of a EngineDetailsModel model + engine_details_model_model = {} + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + # Construct a dict representation of a ProducerInputModel model + producer_input_model_model = {} + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + # Construct a dict representation of a DeliveryMethodPropertiesModel model + delivery_method_properties_model_model = {} + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + # Construct a dict representation of a DeliveryMethod model + delivery_method_model = {} + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + # Construct a dict representation of a DataProductPart model + data_product_part_model = {} + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + # Construct a dict representation of a DataProductCustomWorkflowDefinition model + data_product_custom_workflow_definition_model = {} + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a dict representation of a DataProductOrderAccessRequest model + data_product_order_access_request_model = {} + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + # Construct a dict representation of a DataProductWorkflows model + data_product_workflows_model = {} + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + # Construct a dict representation of a AssetListAccessControl model + asset_list_access_control_model = {} + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + # Set up parameter values + data_product_id = 'testString' + asset = asset_prototype_model + version = '1.2.0' + state = 'draft' + data_product = data_product_identity_model + name = 'testString' + description = 'testString' + tags = ['testString'] + use_cases = [use_case_model] + types = ['data'] + contract_terms = [contract_terms_model] + domain = domain_model + parts_out = [data_product_part_model] + workflows = data_product_workflows_model + dataview_enabled = True + comments = 'testString' + access_control = asset_list_access_control_model + last_updated_at = string_to_datetime('2019-01-01T12:00:00.000Z') + sub_container = container_identity_model + is_restricted = True + + # Invoke method + response = _service.create_data_product_draft( + data_product_id, + asset, + version=version, + state=state, + data_product=data_product, + name=name, + description=description, + tags=tags, + use_cases=use_cases, + types=types, + contract_terms=contract_terms, + domain=domain, + parts_out=parts_out, + workflows=workflows, + dataview_enabled=dataview_enabled, + comments=comments, + access_control=access_control, + last_updated_at=last_updated_at, + sub_container=sub_container, + is_restricted=is_restricted, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['asset'] == asset_prototype_model + assert req_body['version'] == '1.2.0' + assert req_body['state'] == 'draft' + assert req_body['data_product'] == data_product_identity_model + assert req_body['name'] == 'testString' + assert req_body['description'] == 'testString' + assert req_body['tags'] == ['testString'] + assert req_body['use_cases'] == [use_case_model] + assert req_body['types'] == ['data'] + assert req_body['contract_terms'] == [contract_terms_model] + assert req_body['domain'] == domain_model + assert req_body['parts_out'] == [data_product_part_model] + assert req_body['workflows'] == data_product_workflows_model + assert req_body['dataview_enabled'] == True + assert req_body['comments'] == 'testString' + assert req_body['access_control'] == asset_list_access_control_model + assert req_body['last_updated_at'] == '2019-01-01T12:00:00Z' + assert req_body['sub_container'] == container_identity_model + assert req_body['is_restricted'] == True + + def test_create_data_product_draft_all_params_with_retries(self): + # Enable retries and run test_create_data_product_draft_all_params. + _service.enable_retries() + self.test_create_data_product_draft_all_params() + + # Disable retries and run test_create_data_product_draft_all_params. + _service.disable_retries() + self.test_create_data_product_draft_all_params() + + @responses.activate + def test_create_data_product_draft_value_error(self): + """ + test_create_data_product_draft_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a ContainerIdentity model + container_identity_model = {} + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a dict representation of a AssetPrototype model + asset_prototype_model = {} + asset_prototype_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_prototype_model['container'] = container_identity_model + + # Construct a dict representation of a DataProductDraftVersionRelease model + data_product_draft_version_release_model = {} + data_product_draft_version_release_model['id'] = '8bf83660-11fe-4427-a72a-8d8359af24e3' + + # Construct a dict representation of a DataProductIdentity model + data_product_identity_model = {} + data_product_identity_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_identity_model['release'] = data_product_draft_version_release_model + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a UseCase model + use_case_model = {} + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'owner' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractTerms model + contract_terms_model = {} + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + # Construct a dict representation of a AssetPartReference model + asset_part_reference_model = {} + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + # Construct a dict representation of a EngineDetailsModel model + engine_details_model_model = {} + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + # Construct a dict representation of a ProducerInputModel model + producer_input_model_model = {} + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + # Construct a dict representation of a DeliveryMethodPropertiesModel model + delivery_method_properties_model_model = {} + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + # Construct a dict representation of a DeliveryMethod model + delivery_method_model = {} + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + # Construct a dict representation of a DataProductPart model + data_product_part_model = {} + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + # Construct a dict representation of a DataProductCustomWorkflowDefinition model + data_product_custom_workflow_definition_model = {} + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a dict representation of a DataProductOrderAccessRequest model + data_product_order_access_request_model = {} + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + # Construct a dict representation of a DataProductWorkflows model + data_product_workflows_model = {} + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + # Construct a dict representation of a AssetListAccessControl model + asset_list_access_control_model = {} + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + # Set up parameter values + data_product_id = 'testString' + asset = asset_prototype_model + version = '1.2.0' + state = 'draft' + data_product = data_product_identity_model + name = 'testString' + description = 'testString' + tags = ['testString'] + use_cases = [use_case_model] + types = ['data'] + contract_terms = [contract_terms_model] + domain = domain_model + parts_out = [data_product_part_model] + workflows = data_product_workflows_model + dataview_enabled = True + comments = 'testString' + access_control = asset_list_access_control_model + last_updated_at = string_to_datetime('2019-01-01T12:00:00.000Z') + sub_container = container_identity_model + is_restricted = True + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "asset": asset, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.create_data_product_draft(**req_copy) + + def test_create_data_product_draft_value_error_with_retries(self): + # Enable retries and run test_create_data_product_draft_value_error. + _service.enable_retries() + self.test_create_data_product_draft_value_error() + + # Disable retries and run test_create_data_product_draft_value_error. + _service.disable_retries() + self.test_create_data_product_draft_value_error() + + +class TestCreateDraftContractTermsDocument: + """ + Test Class for create_draft_contract_terms_document + """ + + @responses.activate + def test_create_draft_contract_terms_document_all_params(self): + """ + create_draft_contract_terms_document() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + type = 'terms_and_conditions' + name = 'Terms and conditions document' + url = 'testString' + + # Invoke method + response = _service.create_draft_contract_terms_document( + data_product_id, + draft_id, + contract_terms_id, + type, + name, + url=url, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['type'] == 'terms_and_conditions' + assert req_body['name'] == 'Terms and conditions document' + assert req_body['url'] == 'testString' + + def test_create_draft_contract_terms_document_all_params_with_retries(self): + # Enable retries and run test_create_draft_contract_terms_document_all_params. + _service.enable_retries() + self.test_create_draft_contract_terms_document_all_params() + + # Disable retries and run test_create_draft_contract_terms_document_all_params. + _service.disable_retries() + self.test_create_draft_contract_terms_document_all_params() + + @responses.activate + def test_create_draft_contract_terms_document_value_error(self): + """ + test_create_draft_contract_terms_document_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + type = 'terms_and_conditions' + name = 'Terms and conditions document' + url = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "contract_terms_id": contract_terms_id, + "type": type, + "name": name, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.create_draft_contract_terms_document(**req_copy) + + def test_create_draft_contract_terms_document_value_error_with_retries(self): + # Enable retries and run test_create_draft_contract_terms_document_value_error. + _service.enable_retries() + self.test_create_draft_contract_terms_document_value_error() + + # Disable retries and run test_create_draft_contract_terms_document_value_error. + _service.disable_retries() + self.test_create_draft_contract_terms_document_value_error() + + +class TestGetDataProductDraft: + """ + Test Class for get_data_product_draft + """ + + @responses.activate + def test_get_data_product_draft_all_params(self): + """ + get_data_product_draft() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + + # Invoke method + response = _service.get_data_product_draft( + data_product_id, + draft_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_data_product_draft_all_params_with_retries(self): + # Enable retries and run test_get_data_product_draft_all_params. + _service.enable_retries() + self.test_get_data_product_draft_all_params() + + # Disable retries and run test_get_data_product_draft_all_params. + _service.disable_retries() + self.test_get_data_product_draft_all_params() + + @responses.activate + def test_get_data_product_draft_value_error(self): + """ + test_get_data_product_draft_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_data_product_draft(**req_copy) + + def test_get_data_product_draft_value_error_with_retries(self): + # Enable retries and run test_get_data_product_draft_value_error. + _service.enable_retries() + self.test_get_data_product_draft_value_error() + + # Disable retries and run test_get_data_product_draft_value_error. + _service.disable_retries() + self.test_get_data_product_draft_value_error() + + +class TestDeleteDataProductDraft: + """ + Test Class for delete_data_product_draft + """ + + @responses.activate + def test_delete_data_product_draft_all_params(self): + """ + delete_data_product_draft() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString') + responses.add( + responses.DELETE, + url, + status=204, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + + # Invoke method + response = _service.delete_data_product_draft( + data_product_id, + draft_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 204 + + def test_delete_data_product_draft_all_params_with_retries(self): + # Enable retries and run test_delete_data_product_draft_all_params. + _service.enable_retries() + self.test_delete_data_product_draft_all_params() + + # Disable retries and run test_delete_data_product_draft_all_params. + _service.disable_retries() + self.test_delete_data_product_draft_all_params() + + @responses.activate + def test_delete_data_product_draft_value_error(self): + """ + test_delete_data_product_draft_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString') + responses.add( + responses.DELETE, + url, + status=204, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.delete_data_product_draft(**req_copy) + + def test_delete_data_product_draft_value_error_with_retries(self): + # Enable retries and run test_delete_data_product_draft_value_error. + _service.enable_retries() + self.test_delete_data_product_draft_value_error() + + # Disable retries and run test_delete_data_product_draft_value_error. + _service.disable_retries() + self.test_delete_data_product_draft_value_error() + + +class TestUpdateDataProductDraft: + """ + Test Class for update_data_product_draft + """ + + @responses.activate + def test_update_data_product_draft_all_params(self): + """ + update_data_product_draft() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Invoke method + response = _service.update_data_product_draft( + data_product_id, + draft_id, + json_patch_instructions, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body == json_patch_instructions + + def test_update_data_product_draft_all_params_with_retries(self): + # Enable retries and run test_update_data_product_draft_all_params. + _service.enable_retries() + self.test_update_data_product_draft_all_params() + + # Disable retries and run test_update_data_product_draft_all_params. + _service.disable_retries() + self.test_update_data_product_draft_all_params() + + @responses.activate + def test_update_data_product_draft_value_error(self): + """ + test_update_data_product_draft_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "json_patch_instructions": json_patch_instructions, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.update_data_product_draft(**req_copy) + + def test_update_data_product_draft_value_error_with_retries(self): + # Enable retries and run test_update_data_product_draft_value_error. + _service.enable_retries() + self.test_update_data_product_draft_value_error() + + # Disable retries and run test_update_data_product_draft_value_error. + _service.disable_retries() + self.test_update_data_product_draft_value_error() + + +class TestGetDraftContractTermsDocument: + """ + Test Class for get_draft_contract_terms_document + """ + + @responses.activate + def test_get_draft_contract_terms_document_all_params(self): + """ + get_draft_contract_terms_document() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents/testString') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + + # Invoke method + response = _service.get_draft_contract_terms_document( + data_product_id, + draft_id, + contract_terms_id, + document_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_draft_contract_terms_document_all_params_with_retries(self): + # Enable retries and run test_get_draft_contract_terms_document_all_params. + _service.enable_retries() + self.test_get_draft_contract_terms_document_all_params() + + # Disable retries and run test_get_draft_contract_terms_document_all_params. + _service.disable_retries() + self.test_get_draft_contract_terms_document_all_params() + + @responses.activate + def test_get_draft_contract_terms_document_value_error(self): + """ + test_get_draft_contract_terms_document_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents/testString') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "contract_terms_id": contract_terms_id, + "document_id": document_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_draft_contract_terms_document(**req_copy) + + def test_get_draft_contract_terms_document_value_error_with_retries(self): + # Enable retries and run test_get_draft_contract_terms_document_value_error. + _service.enable_retries() + self.test_get_draft_contract_terms_document_value_error() + + # Disable retries and run test_get_draft_contract_terms_document_value_error. + _service.disable_retries() + self.test_get_draft_contract_terms_document_value_error() + + +class TestDeleteDraftContractTermsDocument: + """ + Test Class for delete_draft_contract_terms_document + """ + + @responses.activate + def test_delete_draft_contract_terms_document_all_params(self): + """ + delete_draft_contract_terms_document() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents/testString') + responses.add( + responses.DELETE, + url, + status=204, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + + # Invoke method + response = _service.delete_draft_contract_terms_document( + data_product_id, + draft_id, + contract_terms_id, + document_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 204 + + def test_delete_draft_contract_terms_document_all_params_with_retries(self): + # Enable retries and run test_delete_draft_contract_terms_document_all_params. + _service.enable_retries() + self.test_delete_draft_contract_terms_document_all_params() + + # Disable retries and run test_delete_draft_contract_terms_document_all_params. + _service.disable_retries() + self.test_delete_draft_contract_terms_document_all_params() + + @responses.activate + def test_delete_draft_contract_terms_document_value_error(self): + """ + test_delete_draft_contract_terms_document_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents/testString') + responses.add( + responses.DELETE, + url, + status=204, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "contract_terms_id": contract_terms_id, + "document_id": document_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.delete_draft_contract_terms_document(**req_copy) + + def test_delete_draft_contract_terms_document_value_error_with_retries(self): + # Enable retries and run test_delete_draft_contract_terms_document_value_error. + _service.enable_retries() + self.test_delete_draft_contract_terms_document_value_error() + + # Disable retries and run test_delete_draft_contract_terms_document_value_error. + _service.disable_retries() + self.test_delete_draft_contract_terms_document_value_error() + + +class TestUpdateDraftContractTermsDocument: + """ + Test Class for update_draft_contract_terms_document + """ + + @responses.activate + def test_update_draft_contract_terms_document_all_params(self): + """ + update_draft_contract_terms_document() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents/testString') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Invoke method + response = _service.update_draft_contract_terms_document( + data_product_id, + draft_id, + contract_terms_id, + document_id, + json_patch_instructions, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body == json_patch_instructions + + def test_update_draft_contract_terms_document_all_params_with_retries(self): + # Enable retries and run test_update_draft_contract_terms_document_all_params. + _service.enable_retries() + self.test_update_draft_contract_terms_document_all_params() + + # Disable retries and run test_update_draft_contract_terms_document_all_params. + _service.disable_retries() + self.test_update_draft_contract_terms_document_all_params() + + @responses.activate + def test_update_draft_contract_terms_document_value_error(self): + """ + test_update_draft_contract_terms_document_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/documents/testString') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "contract_terms_id": contract_terms_id, + "document_id": document_id, + "json_patch_instructions": json_patch_instructions, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.update_draft_contract_terms_document(**req_copy) + + def test_update_draft_contract_terms_document_value_error_with_retries(self): + # Enable retries and run test_update_draft_contract_terms_document_value_error. + _service.enable_retries() + self.test_update_draft_contract_terms_document_value_error() + + # Disable retries and run test_update_draft_contract_terms_document_value_error. + _service.disable_retries() + self.test_update_draft_contract_terms_document_value_error() + + +class TestGetDataProductDraftContractTerms: + """ + Test Class for get_data_product_draft_contract_terms + """ + + @responses.activate + def test_get_data_product_draft_contract_terms_all_params(self): + """ + get_data_product_draft_contract_terms() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString') + mock_response = '{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + accept = 'application/json' + include_contract_documents = True + autopopulate_server_information = False + server_asset_id = 'testString' + + # Invoke method + response = _service.get_data_product_draft_contract_terms( + data_product_id, + draft_id, + contract_terms_id, + accept=accept, + include_contract_documents=include_contract_documents, + autopopulate_server_information=autopopulate_server_information, + server_asset_id=server_asset_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'include_contract_documents={}'.format('true' if include_contract_documents else 'false') in query_string + assert 'autopopulate_server_information={}'.format('true' if autopopulate_server_information else 'false') in query_string + assert 'server_asset_id={}'.format(server_asset_id) in query_string + + def test_get_data_product_draft_contract_terms_all_params_with_retries(self): + # Enable retries and run test_get_data_product_draft_contract_terms_all_params. + _service.enable_retries() + self.test_get_data_product_draft_contract_terms_all_params() + + # Disable retries and run test_get_data_product_draft_contract_terms_all_params. + _service.disable_retries() + self.test_get_data_product_draft_contract_terms_all_params() + + @responses.activate + def test_get_data_product_draft_contract_terms_required_params(self): + """ + test_get_data_product_draft_contract_terms_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString') + mock_response = '{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + + # Invoke method + response = _service.get_data_product_draft_contract_terms( + data_product_id, + draft_id, + contract_terms_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_data_product_draft_contract_terms_required_params_with_retries(self): + # Enable retries and run test_get_data_product_draft_contract_terms_required_params. + _service.enable_retries() + self.test_get_data_product_draft_contract_terms_required_params() + + # Disable retries and run test_get_data_product_draft_contract_terms_required_params. + _service.disable_retries() + self.test_get_data_product_draft_contract_terms_required_params() + + @responses.activate + def test_get_data_product_draft_contract_terms_value_error(self): + """ + test_get_data_product_draft_contract_terms_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString') + mock_response = '{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "contract_terms_id": contract_terms_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_data_product_draft_contract_terms(**req_copy) + + def test_get_data_product_draft_contract_terms_value_error_with_retries(self): + # Enable retries and run test_get_data_product_draft_contract_terms_value_error. + _service.enable_retries() + self.test_get_data_product_draft_contract_terms_value_error() + + # Disable retries and run test_get_data_product_draft_contract_terms_value_error. + _service.disable_retries() + self.test_get_data_product_draft_contract_terms_value_error() + + +class TestReplaceDataProductDraftContractTerms: + """ + Test Class for replace_data_product_draft_contract_terms + """ + + @responses.activate + def test_replace_data_product_draft_contract_terms_all_params(self): + """ + replace_data_product_draft_contract_terms() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString') + mock_response = '{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}' + responses.add( + responses.PUT, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'https://ibm.com/document' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'Terms and Conditions' + contract_terms_document_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + domain_model['name'] = 'domain_name' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = 'v0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://www.moreinfo.example.coms' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Intended purpose for the provided data.' + description_model['limitations'] = 'Technical, compliance, and legal limitations for data use.' + description_model['usage'] = 'Recommended usage of the data.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = 'Custom properties that are not part of the standard.' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'IAM Role' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = 'Amount' + pricing_model['currency'] = 'Currency' + pricing_model['unit'] = 'Unit' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'slaproperty' + contract_template_sla_property_model['value'] = 'slavalue' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'sladefaultelement' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'channel' + contract_template_support_and_communication_model['url'] = 'https://www.example.coms' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'The name of the key.' + contract_template_custom_property_model['value'] = 'The value of the key.' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = '684d6aa0-9f93-4564-8a20-e354bc469857' + contract_asset_model['name'] = 'PAYMENT_TRANSACTIONS1' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'snowflake-server-01' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = '8d7701be-709a-49c0-ae4e-a7daeaae6def' + contract_server_model['type'] = 'snowflake' + contract_server_model['description'] = 'Snowflake analytics server' + contract_server_model['environment'] = 'dev' + contract_server_model['account'] = 'acc-456' + contract_server_model['catalog'] = 'analytics_cat' + contract_server_model['database'] = 'analytics_db' + contract_server_model['dataset'] = 'customer_data' + contract_server_model['delimiter'] = ',' + contract_server_model['endpoint_url'] = 'https://xy12345.snowflakecomputing.com' + contract_server_model['format'] = 'parquet' + contract_server_model['host'] = 'xy12345.snowflakecomputing.com' + contract_server_model['location'] = 'Mumbai' + contract_server_model['path'] = '/analytics/data' + contract_server_model['port'] = '443' + contract_server_model['project'] = 'projectY' + contract_server_model['region'] = 'ap-south-1' + contract_server_model['region_name'] = 'Asia South 1' + contract_server_model['schema'] = 'PAYMENT_TRANSACTIONS1' + contract_server_model['service_name'] = 'snowflake' + contract_server_model['staging_dir'] = '/snowflake/staging' + contract_server_model['stream'] = 'stream_analytics' + contract_server_model['warehouse'] = 'wh_xlarge' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'varchar' + contract_schema_property_type_model['length'] = '1024' + contract_schema_property_type_model['scale'] = '0' + contract_schema_property_type_model['nullable'] = 'true' + contract_schema_property_type_model['signed'] = 'false' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'product_brand_code' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '09ca6b40-7c89-412a-8951-ad820da709d1' + contract_schema_model['connection_id'] = '6cc57d4d-2229-438f-91a0-2c455556422b' + contract_schema_model['name'] = '000000_0-2025-06-20-20-28-52.csv' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = '/dpx-test-bucket/000000_0-2025-06-20-20-28-52.csv' + contract_schema_model['physical_type'] = 'text/csv' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + asset = asset_reference_model + id = 'testString' + documents = [contract_terms_document_model] + error_msg = 'testString' + overview = overview_model + description = description_model + organization = [contract_template_organization_model] + roles = [roles_model] + price = pricing_model + sla = [contract_template_sla_model] + support_and_communication = [contract_template_support_and_communication_model] + custom_properties = [contract_template_custom_property_model] + contract_test = contract_test_model + servers = [contract_server_model] + schema = [contract_schema_model] + + # Invoke method + response = _service.replace_data_product_draft_contract_terms( + data_product_id, + draft_id, + contract_terms_id, + asset=asset, + id=id, + documents=documents, + error_msg=error_msg, + overview=overview, + description=description, + organization=organization, + roles=roles, + price=price, + sla=sla, + support_and_communication=support_and_communication, + custom_properties=custom_properties, + contract_test=contract_test, + servers=servers, + schema=schema, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['asset'] == asset_reference_model + assert req_body['id'] == 'testString' + assert req_body['documents'] == [contract_terms_document_model] + assert req_body['error_msg'] == 'testString' + assert req_body['overview'] == overview_model + assert req_body['description'] == description_model + assert req_body['organization'] == [contract_template_organization_model] + assert req_body['roles'] == [roles_model] + assert req_body['price'] == pricing_model + assert req_body['sla'] == [contract_template_sla_model] + assert req_body['support_and_communication'] == [contract_template_support_and_communication_model] + assert req_body['custom_properties'] == [contract_template_custom_property_model] + assert req_body['contract_test'] == contract_test_model + assert req_body['servers'] == [contract_server_model] + assert req_body['schema'] == [contract_schema_model] + + def test_replace_data_product_draft_contract_terms_all_params_with_retries(self): + # Enable retries and run test_replace_data_product_draft_contract_terms_all_params. + _service.enable_retries() + self.test_replace_data_product_draft_contract_terms_all_params() + + # Disable retries and run test_replace_data_product_draft_contract_terms_all_params. + _service.disable_retries() + self.test_replace_data_product_draft_contract_terms_all_params() + + @responses.activate + def test_replace_data_product_draft_contract_terms_value_error(self): + """ + test_replace_data_product_draft_contract_terms_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString') + mock_response = '{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}' + responses.add( + responses.PUT, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'https://ibm.com/document' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'Terms and Conditions' + contract_terms_document_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + domain_model['name'] = 'domain_name' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = 'v0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://www.moreinfo.example.coms' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Intended purpose for the provided data.' + description_model['limitations'] = 'Technical, compliance, and legal limitations for data use.' + description_model['usage'] = 'Recommended usage of the data.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = 'Custom properties that are not part of the standard.' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'IAM Role' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = 'Amount' + pricing_model['currency'] = 'Currency' + pricing_model['unit'] = 'Unit' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'slaproperty' + contract_template_sla_property_model['value'] = 'slavalue' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'sladefaultelement' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'channel' + contract_template_support_and_communication_model['url'] = 'https://www.example.coms' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'The name of the key.' + contract_template_custom_property_model['value'] = 'The value of the key.' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = '684d6aa0-9f93-4564-8a20-e354bc469857' + contract_asset_model['name'] = 'PAYMENT_TRANSACTIONS1' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'snowflake-server-01' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = '8d7701be-709a-49c0-ae4e-a7daeaae6def' + contract_server_model['type'] = 'snowflake' + contract_server_model['description'] = 'Snowflake analytics server' + contract_server_model['environment'] = 'dev' + contract_server_model['account'] = 'acc-456' + contract_server_model['catalog'] = 'analytics_cat' + contract_server_model['database'] = 'analytics_db' + contract_server_model['dataset'] = 'customer_data' + contract_server_model['delimiter'] = ',' + contract_server_model['endpoint_url'] = 'https://xy12345.snowflakecomputing.com' + contract_server_model['format'] = 'parquet' + contract_server_model['host'] = 'xy12345.snowflakecomputing.com' + contract_server_model['location'] = 'Mumbai' + contract_server_model['path'] = '/analytics/data' + contract_server_model['port'] = '443' + contract_server_model['project'] = 'projectY' + contract_server_model['region'] = 'ap-south-1' + contract_server_model['region_name'] = 'Asia South 1' + contract_server_model['schema'] = 'PAYMENT_TRANSACTIONS1' + contract_server_model['service_name'] = 'snowflake' + contract_server_model['staging_dir'] = '/snowflake/staging' + contract_server_model['stream'] = 'stream_analytics' + contract_server_model['warehouse'] = 'wh_xlarge' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'varchar' + contract_schema_property_type_model['length'] = '1024' + contract_schema_property_type_model['scale'] = '0' + contract_schema_property_type_model['nullable'] = 'true' + contract_schema_property_type_model['signed'] = 'false' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'product_brand_code' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '09ca6b40-7c89-412a-8951-ad820da709d1' + contract_schema_model['connection_id'] = '6cc57d4d-2229-438f-91a0-2c455556422b' + contract_schema_model['name'] = '000000_0-2025-06-20-20-28-52.csv' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = '/dpx-test-bucket/000000_0-2025-06-20-20-28-52.csv' + contract_schema_model['physical_type'] = 'text/csv' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + asset = asset_reference_model + id = 'testString' + documents = [contract_terms_document_model] + error_msg = 'testString' + overview = overview_model + description = description_model + organization = [contract_template_organization_model] + roles = [roles_model] + price = pricing_model + sla = [contract_template_sla_model] + support_and_communication = [contract_template_support_and_communication_model] + custom_properties = [contract_template_custom_property_model] + contract_test = contract_test_model + servers = [contract_server_model] + schema = [contract_schema_model] + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "contract_terms_id": contract_terms_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.replace_data_product_draft_contract_terms(**req_copy) + + def test_replace_data_product_draft_contract_terms_value_error_with_retries(self): + # Enable retries and run test_replace_data_product_draft_contract_terms_value_error. + _service.enable_retries() + self.test_replace_data_product_draft_contract_terms_value_error() + + # Disable retries and run test_replace_data_product_draft_contract_terms_value_error. + _service.disable_retries() + self.test_replace_data_product_draft_contract_terms_value_error() + + +class TestUpdateDataProductDraftContractTerms: + """ + Test Class for update_data_product_draft_contract_terms + """ + + @responses.activate + def test_update_data_product_draft_contract_terms_all_params(self): + """ + update_data_product_draft_contract_terms() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString') + mock_response = '{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Invoke method + response = _service.update_data_product_draft_contract_terms( + data_product_id, + draft_id, + contract_terms_id, + json_patch_instructions, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body == json_patch_instructions + + def test_update_data_product_draft_contract_terms_all_params_with_retries(self): + # Enable retries and run test_update_data_product_draft_contract_terms_all_params. + _service.enable_retries() + self.test_update_data_product_draft_contract_terms_all_params() + + # Disable retries and run test_update_data_product_draft_contract_terms_all_params. + _service.disable_retries() + self.test_update_data_product_draft_contract_terms_all_params() + + @responses.activate + def test_update_data_product_draft_contract_terms_value_error(self): + """ + test_update_data_product_draft_contract_terms_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString') + mock_response = '{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "contract_terms_id": contract_terms_id, + "json_patch_instructions": json_patch_instructions, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.update_data_product_draft_contract_terms(**req_copy) + + def test_update_data_product_draft_contract_terms_value_error_with_retries(self): + # Enable retries and run test_update_data_product_draft_contract_terms_value_error. + _service.enable_retries() + self.test_update_data_product_draft_contract_terms_value_error() + + # Disable retries and run test_update_data_product_draft_contract_terms_value_error. + _service.disable_retries() + self.test_update_data_product_draft_contract_terms_value_error() + + +class TestGetContractTermsInSpecifiedFormat: + """ + Test Class for get_contract_terms_in_specified_format + """ + + @responses.activate + def test_get_contract_terms_in_specified_format_all_params(self): + """ + get_contract_terms_in_specified_format() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/format') + mock_response = 'This is a mock binary response.' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/odcs+yaml', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + format = 'testString' + format_version = 'testString' + accept = 'application/odcs+yaml' + + # Invoke method + response = _service.get_contract_terms_in_specified_format( + data_product_id, + draft_id, + contract_terms_id, + format, + format_version, + accept=accept, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'format={}'.format(format) in query_string + assert 'format_version={}'.format(format_version) in query_string + + def test_get_contract_terms_in_specified_format_all_params_with_retries(self): + # Enable retries and run test_get_contract_terms_in_specified_format_all_params. + _service.enable_retries() + self.test_get_contract_terms_in_specified_format_all_params() + + # Disable retries and run test_get_contract_terms_in_specified_format_all_params. + _service.disable_retries() + self.test_get_contract_terms_in_specified_format_all_params() + + @responses.activate + def test_get_contract_terms_in_specified_format_required_params(self): + """ + test_get_contract_terms_in_specified_format_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/format') + mock_response = 'This is a mock binary response.' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/odcs+yaml', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + format = 'testString' + format_version = 'testString' + + # Invoke method + response = _service.get_contract_terms_in_specified_format( + data_product_id, + draft_id, + contract_terms_id, + format, + format_version, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'format={}'.format(format) in query_string + assert 'format_version={}'.format(format_version) in query_string + + def test_get_contract_terms_in_specified_format_required_params_with_retries(self): + # Enable retries and run test_get_contract_terms_in_specified_format_required_params. + _service.enable_retries() + self.test_get_contract_terms_in_specified_format_required_params() + + # Disable retries and run test_get_contract_terms_in_specified_format_required_params. + _service.disable_retries() + self.test_get_contract_terms_in_specified_format_required_params() + + @responses.activate + def test_get_contract_terms_in_specified_format_value_error(self): + """ + test_get_contract_terms_in_specified_format_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/contract_terms/testString/format') + mock_response = 'This is a mock binary response.' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/odcs+yaml', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + contract_terms_id = 'testString' + format = 'testString' + format_version = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + "contract_terms_id": contract_terms_id, + "format": format, + "format_version": format_version, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_contract_terms_in_specified_format(**req_copy) + + def test_get_contract_terms_in_specified_format_value_error_with_retries(self): + # Enable retries and run test_get_contract_terms_in_specified_format_value_error. + _service.enable_retries() + self.test_get_contract_terms_in_specified_format_value_error() + + # Disable retries and run test_get_contract_terms_in_specified_format_value_error. + _service.disable_retries() + self.test_get_contract_terms_in_specified_format_value_error() + + +class TestPublishDataProductDraft: + """ + Test Class for publish_data_product_draft + """ + + @responses.activate + def test_publish_data_product_draft_all_params(self): + """ + publish_data_product_draft() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/publish') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + + # Invoke method + response = _service.publish_data_product_draft( + data_product_id, + draft_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_publish_data_product_draft_all_params_with_retries(self): + # Enable retries and run test_publish_data_product_draft_all_params. + _service.enable_retries() + self.test_publish_data_product_draft_all_params() + + # Disable retries and run test_publish_data_product_draft_all_params. + _service.disable_retries() + self.test_publish_data_product_draft_all_params() + + @responses.activate + def test_publish_data_product_draft_value_error(self): + """ + test_publish_data_product_draft_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/drafts/testString/publish') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + draft_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "draft_id": draft_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.publish_data_product_draft(**req_copy) + + def test_publish_data_product_draft_value_error_with_retries(self): + # Enable retries and run test_publish_data_product_draft_value_error. + _service.enable_retries() + self.test_publish_data_product_draft_value_error() + + # Disable retries and run test_publish_data_product_draft_value_error. + _service.disable_retries() + self.test_publish_data_product_draft_value_error() + + +# endregion +############################################################################## +# End of Service: DataProductDrafts +############################################################################## + +############################################################################## +# Start of Service: DataProductReleases +############################################################################## +# region + + +class TestNewInstance: + """ + Test Class for new_instance + """ + + def test_new_instance(self): + """ + new_instance() + """ + os.environ['TEST_SERVICE_AUTH_TYPE'] = 'noAuth' + + service = DphV1.new_instance( + service_name='TEST_SERVICE', + ) + + assert service is not None + assert isinstance(service, DphV1) + + def test_new_instance_without_authenticator(self): + """ + new_instance_without_authenticator() + """ + with pytest.raises(ValueError, match='authenticator must be provided'): + service = DphV1.new_instance( + service_name='TEST_SERVICE_NOT_FOUND', + ) + + +class TestGetDataProductRelease: + """ + Test Class for get_data_product_release + """ + + @responses.activate + def test_get_data_product_release_all_params(self): + """ + get_data_product_release() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + check_caller_approval = False + + # Invoke method + response = _service.get_data_product_release( + data_product_id, + release_id, + check_caller_approval=check_caller_approval, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'check_caller_approval={}'.format('true' if check_caller_approval else 'false') in query_string + + def test_get_data_product_release_all_params_with_retries(self): + # Enable retries and run test_get_data_product_release_all_params. + _service.enable_retries() + self.test_get_data_product_release_all_params() + + # Disable retries and run test_get_data_product_release_all_params. + _service.disable_retries() + self.test_get_data_product_release_all_params() + + @responses.activate + def test_get_data_product_release_required_params(self): + """ + test_get_data_product_release_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + + # Invoke method + response = _service.get_data_product_release( + data_product_id, + release_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_data_product_release_required_params_with_retries(self): + # Enable retries and run test_get_data_product_release_required_params. + _service.enable_retries() + self.test_get_data_product_release_required_params() + + # Disable retries and run test_get_data_product_release_required_params. + _service.disable_retries() + self.test_get_data_product_release_required_params() + + @responses.activate + def test_get_data_product_release_value_error(self): + """ + test_get_data_product_release_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "release_id": release_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_data_product_release(**req_copy) + + def test_get_data_product_release_value_error_with_retries(self): + # Enable retries and run test_get_data_product_release_value_error. + _service.enable_retries() + self.test_get_data_product_release_value_error() + + # Disable retries and run test_get_data_product_release_value_error. + _service.disable_retries() + self.test_get_data_product_release_value_error() + + +class TestUpdateDataProductRelease: + """ + Test Class for update_data_product_release + """ + + @responses.activate + def test_update_data_product_release_all_params(self): + """ + update_data_product_release() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Invoke method + response = _service.update_data_product_release( + data_product_id, + release_id, + json_patch_instructions, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body == json_patch_instructions + + def test_update_data_product_release_all_params_with_retries(self): + # Enable retries and run test_update_data_product_release_all_params. + _service.enable_retries() + self.test_update_data_product_release_all_params() + + # Disable retries and run test_update_data_product_release_all_params. + _service.disable_retries() + self.test_update_data_product_release_all_params() + + @responses.activate + def test_update_data_product_release_value_error(self): + """ + test_update_data_product_release_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "release_id": release_id, + "json_patch_instructions": json_patch_instructions, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.update_data_product_release(**req_copy) + + def test_update_data_product_release_value_error_with_retries(self): + # Enable retries and run test_update_data_product_release_value_error. + _service.enable_retries() + self.test_update_data_product_release_value_error() + + # Disable retries and run test_update_data_product_release_value_error. + _service.disable_retries() + self.test_update_data_product_release_value_error() + + +class TestGetReleaseContractTermsDocument: + """ + Test Class for get_release_contract_terms_document + """ + + @responses.activate + def test_get_release_contract_terms_document_all_params(self): + """ + get_release_contract_terms_document() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/contract_terms/testString/documents/testString') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + + # Invoke method + response = _service.get_release_contract_terms_document( + data_product_id, + release_id, + contract_terms_id, + document_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_release_contract_terms_document_all_params_with_retries(self): + # Enable retries and run test_get_release_contract_terms_document_all_params. + _service.enable_retries() + self.test_get_release_contract_terms_document_all_params() + + # Disable retries and run test_get_release_contract_terms_document_all_params. + _service.disable_retries() + self.test_get_release_contract_terms_document_all_params() + + @responses.activate + def test_get_release_contract_terms_document_value_error(self): + """ + test_get_release_contract_terms_document_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/contract_terms/testString/documents/testString') + mock_response = '{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + contract_terms_id = 'testString' + document_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "release_id": release_id, + "contract_terms_id": contract_terms_id, + "document_id": document_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_release_contract_terms_document(**req_copy) + + def test_get_release_contract_terms_document_value_error_with_retries(self): + # Enable retries and run test_get_release_contract_terms_document_value_error. + _service.enable_retries() + self.test_get_release_contract_terms_document_value_error() + + # Disable retries and run test_get_release_contract_terms_document_value_error. + _service.disable_retries() + self.test_get_release_contract_terms_document_value_error() + + +class TestGetPublishedDataProductDraftContractTerms: + """ + Test Class for get_published_data_product_draft_contract_terms + """ + + @responses.activate + def test_get_published_data_product_draft_contract_terms_all_params(self): + """ + get_published_data_product_draft_contract_terms() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/contract_terms/testString') + mock_response = 'This is a mock binary response.' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/odcs+yaml', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + contract_terms_id = 'testString' + accept = 'application/odcs+yaml' + include_contract_documents = True + + # Invoke method + response = _service.get_published_data_product_draft_contract_terms( + data_product_id, + release_id, + contract_terms_id, + accept=accept, + include_contract_documents=include_contract_documents, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'include_contract_documents={}'.format('true' if include_contract_documents else 'false') in query_string + + def test_get_published_data_product_draft_contract_terms_all_params_with_retries(self): + # Enable retries and run test_get_published_data_product_draft_contract_terms_all_params. + _service.enable_retries() + self.test_get_published_data_product_draft_contract_terms_all_params() + + # Disable retries and run test_get_published_data_product_draft_contract_terms_all_params. + _service.disable_retries() + self.test_get_published_data_product_draft_contract_terms_all_params() + + @responses.activate + def test_get_published_data_product_draft_contract_terms_required_params(self): + """ + test_get_published_data_product_draft_contract_terms_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/contract_terms/testString') + mock_response = 'This is a mock binary response.' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/odcs+yaml', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + contract_terms_id = 'testString' + + # Invoke method + response = _service.get_published_data_product_draft_contract_terms( + data_product_id, + release_id, + contract_terms_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_published_data_product_draft_contract_terms_required_params_with_retries(self): + # Enable retries and run test_get_published_data_product_draft_contract_terms_required_params. + _service.enable_retries() + self.test_get_published_data_product_draft_contract_terms_required_params() + + # Disable retries and run test_get_published_data_product_draft_contract_terms_required_params. + _service.disable_retries() + self.test_get_published_data_product_draft_contract_terms_required_params() + + @responses.activate + def test_get_published_data_product_draft_contract_terms_value_error(self): + """ + test_get_published_data_product_draft_contract_terms_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/contract_terms/testString') + mock_response = 'This is a mock binary response.' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/odcs+yaml', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + contract_terms_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "release_id": release_id, + "contract_terms_id": contract_terms_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_published_data_product_draft_contract_terms(**req_copy) + + def test_get_published_data_product_draft_contract_terms_value_error_with_retries(self): + # Enable retries and run test_get_published_data_product_draft_contract_terms_value_error. + _service.enable_retries() + self.test_get_published_data_product_draft_contract_terms_value_error() + + # Disable retries and run test_get_published_data_product_draft_contract_terms_value_error. + _service.disable_retries() + self.test_get_published_data_product_draft_contract_terms_value_error() + + +class TestListDataProductReleases: + """ + Test Class for list_data_product_releases + """ + + @responses.activate + def test_list_data_product_releases_all_params(self): + """ + list_data_product_releases() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "releases": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + asset_container_id = 'testString' + state = ['available'] + version = 'testString' + limit = 200 + start = 'testString' + + # Invoke method + response = _service.list_data_product_releases( + data_product_id, + asset_container_id=asset_container_id, + state=state, + version=version, + limit=limit, + start=start, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'asset.container.id={}'.format(asset_container_id) in query_string + assert 'state={}'.format(','.join(state)) in query_string + assert 'version={}'.format(version) in query_string + assert 'limit={}'.format(limit) in query_string + assert 'start={}'.format(start) in query_string + + def test_list_data_product_releases_all_params_with_retries(self): + # Enable retries and run test_list_data_product_releases_all_params. + _service.enable_retries() + self.test_list_data_product_releases_all_params() + + # Disable retries and run test_list_data_product_releases_all_params. + _service.disable_retries() + self.test_list_data_product_releases_all_params() + + @responses.activate + def test_list_data_product_releases_required_params(self): + """ + test_list_data_product_releases_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "releases": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + + # Invoke method + response = _service.list_data_product_releases( + data_product_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_list_data_product_releases_required_params_with_retries(self): + # Enable retries and run test_list_data_product_releases_required_params. + _service.enable_retries() + self.test_list_data_product_releases_required_params() + + # Disable retries and run test_list_data_product_releases_required_params. + _service.disable_retries() + self.test_list_data_product_releases_required_params() + + @responses.activate + def test_list_data_product_releases_value_error(self): + """ + test_list_data_product_releases_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "releases": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.list_data_product_releases(**req_copy) + + def test_list_data_product_releases_value_error_with_retries(self): + # Enable retries and run test_list_data_product_releases_value_error. + _service.enable_retries() + self.test_list_data_product_releases_value_error() + + # Disable retries and run test_list_data_product_releases_value_error. + _service.disable_retries() + self.test_list_data_product_releases_value_error() + + @responses.activate + def test_list_data_product_releases_with_pager_get_next(self): + """ + test_list_data_product_releases_with_pager_get_next() + """ + # Set up a two-page mock response + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases') + mock_response1 = '{"next":{"start":"1"},"total_count":2,"limit":1,"releases":[{"version":"1.0.0","state":"draft","data_product":{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"name":"My Data Product","description":"This is a description of My Data Product.","tags":["tags"],"use_cases":[{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}],"types":["data"],"contract_terms":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"id":"id","documents":[{"url":"url","type":"terms_and_conditions","name":"name","id":"2b0bf220-079c-11ee-be56-0242ac120002","attachment":{"id":"id"},"upload_url":"upload_url"}],"error_msg":"error_msg","overview":{"api_version":"v3.0.1","kind":"DataContract","name":"Sample Data Contract","version":"0.0.0","domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"more_info":"List of links to sources that provide more details on the data contract."},"description":{"purpose":"Used for customer behavior analysis.","limitations":"Data cannot be used for marketing.","usage":"Data should be used only for analytics.","more_info":[{"type":"privacy-statement","url":"https://moreinfo.example.com"}],"custom_properties":"{\\"property1\\":\\"value1\\"}"},"organization":[{"user_id":"IBMid-691000IN4G","role":"owner"}],"roles":[{"role":"owner"}],"price":{"amount":"100.0","currency":"USD","unit":"megabyte"},"sla":[{"default_element":"Standard SLA Policy","properties":[{"property":"Uptime Guarantee","value":"99.9"}]}],"support_and_communication":[{"channel":"Email Support","url":"https://support.example.com"}],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}],"contract_test":{"status":"pass","last_tested_time":"last_tested_time","message":"message"},"servers":[{"server":"server","asset":{"id":"id","name":"name"},"connection_id":"connection_id","type":"type","description":"description","environment":"environment","account":"account","catalog":"catalog","database":"database","dataset":"dataset","delimiter":"delimiter","endpoint_url":"endpoint_url","format":"format","host":"host","location":"location","path":"path","port":"port","project":"project","region":"region","region_name":"region_name","schema":"schema","service_name":"service_name","staging_dir":"staging_dir","stream":"stream","warehouse":"warehouse","roles":["roles"],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}]}],"schema":[{"asset_id":"2b0bf220-079c-11ee-be56-0242ac120002","connection_id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","description":"description","connection_path":"connection_path","physical_type":"physical_type","properties":[{"name":"name","type":{"type":"type","length":"length","scale":"scale","nullable":"nullable","signed":"signed","native_type":"native_type"},"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}],"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}]}],"domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"parts_out":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"type":"data_asset"},"delivery_methods":[{"id":"09cf5fcc-cb9d-4995-a8e4-16517b25229f","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"getproperties":{"producer_input":{"engine_details":{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]},"engines":[{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]}]}}}]}],"workflows":{"order_access_request":{"task_assignee_users":["task_assignee_users"],"pre_approved_users":["pre_approved_users"],"custom_workflow_definition":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"}}},"dataview_enabled":true,"comments":"Comments by a producer that are provided either at the time of data product version creation or retiring","access_control":{"owner":"IBMid-696000KYV9"},"last_updated_at":"2019-01-01T12:00:00.000Z","sub_container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd"},"is_restricted":false,"id":"2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd","asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}}]}' + mock_response2 = '{"total_count":2,"limit":1,"releases":[{"version":"1.0.0","state":"draft","data_product":{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"name":"My Data Product","description":"This is a description of My Data Product.","tags":["tags"],"use_cases":[{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}],"types":["data"],"contract_terms":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"id":"id","documents":[{"url":"url","type":"terms_and_conditions","name":"name","id":"2b0bf220-079c-11ee-be56-0242ac120002","attachment":{"id":"id"},"upload_url":"upload_url"}],"error_msg":"error_msg","overview":{"api_version":"v3.0.1","kind":"DataContract","name":"Sample Data Contract","version":"0.0.0","domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"more_info":"List of links to sources that provide more details on the data contract."},"description":{"purpose":"Used for customer behavior analysis.","limitations":"Data cannot be used for marketing.","usage":"Data should be used only for analytics.","more_info":[{"type":"privacy-statement","url":"https://moreinfo.example.com"}],"custom_properties":"{\\"property1\\":\\"value1\\"}"},"organization":[{"user_id":"IBMid-691000IN4G","role":"owner"}],"roles":[{"role":"owner"}],"price":{"amount":"100.0","currency":"USD","unit":"megabyte"},"sla":[{"default_element":"Standard SLA Policy","properties":[{"property":"Uptime Guarantee","value":"99.9"}]}],"support_and_communication":[{"channel":"Email Support","url":"https://support.example.com"}],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}],"contract_test":{"status":"pass","last_tested_time":"last_tested_time","message":"message"},"servers":[{"server":"server","asset":{"id":"id","name":"name"},"connection_id":"connection_id","type":"type","description":"description","environment":"environment","account":"account","catalog":"catalog","database":"database","dataset":"dataset","delimiter":"delimiter","endpoint_url":"endpoint_url","format":"format","host":"host","location":"location","path":"path","port":"port","project":"project","region":"region","region_name":"region_name","schema":"schema","service_name":"service_name","staging_dir":"staging_dir","stream":"stream","warehouse":"warehouse","roles":["roles"],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}]}],"schema":[{"asset_id":"2b0bf220-079c-11ee-be56-0242ac120002","connection_id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","description":"description","connection_path":"connection_path","physical_type":"physical_type","properties":[{"name":"name","type":{"type":"type","length":"length","scale":"scale","nullable":"nullable","signed":"signed","native_type":"native_type"},"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}],"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}]}],"domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"parts_out":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"type":"data_asset"},"delivery_methods":[{"id":"09cf5fcc-cb9d-4995-a8e4-16517b25229f","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"getproperties":{"producer_input":{"engine_details":{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]},"engines":[{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]}]}}}]}],"workflows":{"order_access_request":{"task_assignee_users":["task_assignee_users"],"pre_approved_users":["pre_approved_users"],"custom_workflow_definition":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"}}},"dataview_enabled":true,"comments":"Comments by a producer that are provided either at the time of data product version creation or retiring","access_control":{"owner":"IBMid-696000KYV9"},"last_updated_at":"2019-01-01T12:00:00.000Z","sub_container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd"},"is_restricted":false,"id":"2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd","asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response1, + content_type='application/json', + status=200, + ) + responses.add( + responses.GET, + url, + body=mock_response2, + content_type='application/json', + status=200, + ) + + # Exercise the pager class for this operation + all_results = [] + pager = DataProductReleasesPager( + client=_service, + data_product_id='testString', + asset_container_id='testString', + state=['available'], + version='testString', + limit=10, + ) + while pager.has_next(): + next_page = pager.get_next() + assert next_page is not None + all_results.extend(next_page) + assert len(all_results) == 2 + + @responses.activate + def test_list_data_product_releases_with_pager_get_all(self): + """ + test_list_data_product_releases_with_pager_get_all() + """ + # Set up a two-page mock response + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases') + mock_response1 = '{"next":{"start":"1"},"total_count":2,"limit":1,"releases":[{"version":"1.0.0","state":"draft","data_product":{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"name":"My Data Product","description":"This is a description of My Data Product.","tags":["tags"],"use_cases":[{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}],"types":["data"],"contract_terms":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"id":"id","documents":[{"url":"url","type":"terms_and_conditions","name":"name","id":"2b0bf220-079c-11ee-be56-0242ac120002","attachment":{"id":"id"},"upload_url":"upload_url"}],"error_msg":"error_msg","overview":{"api_version":"v3.0.1","kind":"DataContract","name":"Sample Data Contract","version":"0.0.0","domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"more_info":"List of links to sources that provide more details on the data contract."},"description":{"purpose":"Used for customer behavior analysis.","limitations":"Data cannot be used for marketing.","usage":"Data should be used only for analytics.","more_info":[{"type":"privacy-statement","url":"https://moreinfo.example.com"}],"custom_properties":"{\\"property1\\":\\"value1\\"}"},"organization":[{"user_id":"IBMid-691000IN4G","role":"owner"}],"roles":[{"role":"owner"}],"price":{"amount":"100.0","currency":"USD","unit":"megabyte"},"sla":[{"default_element":"Standard SLA Policy","properties":[{"property":"Uptime Guarantee","value":"99.9"}]}],"support_and_communication":[{"channel":"Email Support","url":"https://support.example.com"}],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}],"contract_test":{"status":"pass","last_tested_time":"last_tested_time","message":"message"},"servers":[{"server":"server","asset":{"id":"id","name":"name"},"connection_id":"connection_id","type":"type","description":"description","environment":"environment","account":"account","catalog":"catalog","database":"database","dataset":"dataset","delimiter":"delimiter","endpoint_url":"endpoint_url","format":"format","host":"host","location":"location","path":"path","port":"port","project":"project","region":"region","region_name":"region_name","schema":"schema","service_name":"service_name","staging_dir":"staging_dir","stream":"stream","warehouse":"warehouse","roles":["roles"],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}]}],"schema":[{"asset_id":"2b0bf220-079c-11ee-be56-0242ac120002","connection_id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","description":"description","connection_path":"connection_path","physical_type":"physical_type","properties":[{"name":"name","type":{"type":"type","length":"length","scale":"scale","nullable":"nullable","signed":"signed","native_type":"native_type"},"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}],"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}]}],"domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"parts_out":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"type":"data_asset"},"delivery_methods":[{"id":"09cf5fcc-cb9d-4995-a8e4-16517b25229f","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"getproperties":{"producer_input":{"engine_details":{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]},"engines":[{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]}]}}}]}],"workflows":{"order_access_request":{"task_assignee_users":["task_assignee_users"],"pre_approved_users":["pre_approved_users"],"custom_workflow_definition":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"}}},"dataview_enabled":true,"comments":"Comments by a producer that are provided either at the time of data product version creation or retiring","access_control":{"owner":"IBMid-696000KYV9"},"last_updated_at":"2019-01-01T12:00:00.000Z","sub_container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd"},"is_restricted":false,"id":"2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd","asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}}]}' + mock_response2 = '{"total_count":2,"limit":1,"releases":[{"version":"1.0.0","state":"draft","data_product":{"id":"b38df608-d34b-4d58-8136-ed25e6c6684e","release":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"},"container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"name":"My Data Product","description":"This is a description of My Data Product.","tags":["tags"],"use_cases":[{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}],"types":["data"],"contract_terms":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"id":"id","documents":[{"url":"url","type":"terms_and_conditions","name":"name","id":"2b0bf220-079c-11ee-be56-0242ac120002","attachment":{"id":"id"},"upload_url":"upload_url"}],"error_msg":"error_msg","overview":{"api_version":"v3.0.1","kind":"DataContract","name":"Sample Data Contract","version":"0.0.0","domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"more_info":"List of links to sources that provide more details on the data contract."},"description":{"purpose":"Used for customer behavior analysis.","limitations":"Data cannot be used for marketing.","usage":"Data should be used only for analytics.","more_info":[{"type":"privacy-statement","url":"https://moreinfo.example.com"}],"custom_properties":"{\\"property1\\":\\"value1\\"}"},"organization":[{"user_id":"IBMid-691000IN4G","role":"owner"}],"roles":[{"role":"owner"}],"price":{"amount":"100.0","currency":"USD","unit":"megabyte"},"sla":[{"default_element":"Standard SLA Policy","properties":[{"property":"Uptime Guarantee","value":"99.9"}]}],"support_and_communication":[{"channel":"Email Support","url":"https://support.example.com"}],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}],"contract_test":{"status":"pass","last_tested_time":"last_tested_time","message":"message"},"servers":[{"server":"server","asset":{"id":"id","name":"name"},"connection_id":"connection_id","type":"type","description":"description","environment":"environment","account":"account","catalog":"catalog","database":"database","dataset":"dataset","delimiter":"delimiter","endpoint_url":"endpoint_url","format":"format","host":"host","location":"location","path":"path","port":"port","project":"project","region":"region","region_name":"region_name","schema":"schema","service_name":"service_name","staging_dir":"staging_dir","stream":"stream","warehouse":"warehouse","roles":["roles"],"custom_properties":[{"key":"customPropertyKey","value":"customPropertyValue"}]}],"schema":[{"asset_id":"2b0bf220-079c-11ee-be56-0242ac120002","connection_id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","description":"description","connection_path":"connection_path","physical_type":"physical_type","properties":[{"name":"name","type":{"type":"type","length":"length","scale":"scale","nullable":"nullable","signed":"signed","native_type":"native_type"},"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}],"quality":[{"type":"sql","description":"description","rule":"rule","implementation":"implementation","engine":"engine","must_be_less_than":"must_be_less_than","must_be_less_or_equal_to":"must_be_less_or_equal_to","must_be_greater_than":"must_be_greater_than","must_be_greater_or_equal_to":"must_be_greater_or_equal_to","must_be_between":["must_be_between"],"must_not_be_between":["must_not_be_between"],"must_be":"must_be","must_not_be":"must_not_be","name":"name","unit":"unit","query":"query"}]}]}],"domain":{"id":"id","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}},"parts_out":[{"asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"type":"data_asset"},"delivery_methods":[{"id":"09cf5fcc-cb9d-4995-a8e4-16517b25229f","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"},"getproperties":{"producer_input":{"engine_details":{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]},"engines":[{"display_name":"Iceberg Engine","engine_id":"presto767","engine_port":"34567","engine_host":"a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud","engine_type":"spark","associated_catalogs":["associated_catalogs"]}]}}}]}],"workflows":{"order_access_request":{"task_assignee_users":["task_assignee_users"],"pre_approved_users":["pre_approved_users"],"custom_workflow_definition":{"id":"18bdbde1-918e-4ecf-aa23-6727bf319e14"}}},"dataview_enabled":true,"comments":"Comments by a producer that are provided either at the time of data product version creation or retiring","access_control":{"owner":"IBMid-696000KYV9"},"last_updated_at":"2019-01-01T12:00:00.000Z","sub_container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd"},"is_restricted":false,"id":"2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd","asset":{"id":"2b0bf220-079c-11ee-be56-0242ac120002","name":"name","container":{"id":"d29c42eb-7100-4b7a-8257-c196dbcca1cd","type":"catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response1, + content_type='application/json', + status=200, + ) + responses.add( + responses.GET, + url, + body=mock_response2, + content_type='application/json', + status=200, + ) + + # Exercise the pager class for this operation + pager = DataProductReleasesPager( + client=_service, + data_product_id='testString', + asset_container_id='testString', + state=['available'], + version='testString', + limit=10, + ) + all_results = pager.get_all() + assert all_results is not None + assert len(all_results) == 2 + + +class TestRetireDataProductRelease: + """ + Test Class for retire_data_product_release + """ + + @responses.activate + def test_retire_data_product_release_all_params(self): + """ + retire_data_product_release() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/retire') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + revoke_access = False + start_at = 'testString' + + # Invoke method + response = _service.retire_data_product_release( + data_product_id, + release_id, + revoke_access=revoke_access, + start_at=start_at, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'revoke_access={}'.format('true' if revoke_access else 'false') in query_string + assert 'start_at={}'.format(start_at) in query_string + + def test_retire_data_product_release_all_params_with_retries(self): + # Enable retries and run test_retire_data_product_release_all_params. + _service.enable_retries() + self.test_retire_data_product_release_all_params() + + # Disable retries and run test_retire_data_product_release_all_params. + _service.disable_retries() + self.test_retire_data_product_release_all_params() + + @responses.activate + def test_retire_data_product_release_required_params(self): + """ + test_retire_data_product_release_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/retire') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + + # Invoke method + response = _service.retire_data_product_release( + data_product_id, + release_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_retire_data_product_release_required_params_with_retries(self): + # Enable retries and run test_retire_data_product_release_required_params. + _service.enable_retries() + self.test_retire_data_product_release_required_params() + + # Disable retries and run test_retire_data_product_release_required_params. + _service.disable_retries() + self.test_retire_data_product_release_required_params() + + @responses.activate + def test_retire_data_product_release_value_error(self): + """ + test_retire_data_product_release_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/retire') + mock_response = '{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "published_by": "published_by", "published_at": "2019-01-01T12:00:00.000Z", "created_by": "created_by", "created_at": "2019-01-01T12:00:00.000Z", "properties": {"anyKey": "anyValue"}, "visualization_errors": [{"visualization": {"id": "id", "name": "name"}, "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "related_asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "error": {"code": "code", "message": "message"}}]}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "release_id": release_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.retire_data_product_release(**req_copy) + + def test_retire_data_product_release_value_error_with_retries(self): + # Enable retries and run test_retire_data_product_release_value_error. + _service.enable_retries() + self.test_retire_data_product_release_value_error() + + # Disable retries and run test_retire_data_product_release_value_error. + _service.disable_retries() + self.test_retire_data_product_release_value_error() + + +class TestCreateRevokeAccessProcess: + """ + Test Class for create_revoke_access_process + """ + + @responses.activate + def test_create_revoke_access_process_all_params(self): + """ + create_revoke_access_process() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/revoke_access') + mock_response = '{"message": "message"}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=202, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + body = io.BytesIO(b'This is a mock file.').getvalue() + content_type = 'testString' + + # Invoke method + response = _service.create_revoke_access_process( + data_product_id, + release_id, + body=body, + content_type=content_type, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 202 + # Validate body params + + def test_create_revoke_access_process_all_params_with_retries(self): + # Enable retries and run test_create_revoke_access_process_all_params. + _service.enable_retries() + self.test_create_revoke_access_process_all_params() + + # Disable retries and run test_create_revoke_access_process_all_params. + _service.disable_retries() + self.test_create_revoke_access_process_all_params() + + @responses.activate + def test_create_revoke_access_process_required_params(self): + """ + test_create_revoke_access_process_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/revoke_access') + mock_response = '{"message": "message"}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=202, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + + # Invoke method + response = _service.create_revoke_access_process( + data_product_id, + release_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 202 + + def test_create_revoke_access_process_required_params_with_retries(self): + # Enable retries and run test_create_revoke_access_process_required_params. + _service.enable_retries() + self.test_create_revoke_access_process_required_params() + + # Disable retries and run test_create_revoke_access_process_required_params. + _service.disable_retries() + self.test_create_revoke_access_process_required_params() + + @responses.activate + def test_create_revoke_access_process_value_error(self): + """ + test_create_revoke_access_process_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_products/testString/releases/testString/revoke_access') + mock_response = '{"message": "message"}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=202, + ) + + # Set up parameter values + data_product_id = 'testString' + release_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "data_product_id": data_product_id, + "release_id": release_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.create_revoke_access_process(**req_copy) + + def test_create_revoke_access_process_value_error_with_retries(self): + # Enable retries and run test_create_revoke_access_process_value_error. + _service.enable_retries() + self.test_create_revoke_access_process_value_error() + + # Disable retries and run test_create_revoke_access_process_value_error. + _service.disable_retries() + self.test_create_revoke_access_process_value_error() + + +# endregion +############################################################################## +# End of Service: DataProductReleases +############################################################################## + +############################################################################## +# Start of Service: DataProductContractTemplates +############################################################################## +# region + + +class TestNewInstance: + """ + Test Class for new_instance + """ + + def test_new_instance(self): + """ + new_instance() + """ + os.environ['TEST_SERVICE_AUTH_TYPE'] = 'noAuth' + + service = DphV1.new_instance( + service_name='TEST_SERVICE', + ) + + assert service is not None + assert isinstance(service, DphV1) + + def test_new_instance_without_authenticator(self): + """ + new_instance_without_authenticator() + """ + with pytest.raises(ValueError, match='authenticator must be provided'): + service = DphV1.new_instance( + service_name='TEST_SERVICE_NOT_FOUND', + ) + + +class TestListDataProductContractTemplate: + """ + Test Class for list_data_product_contract_template + """ + + @responses.activate + def test_list_data_product_contract_template_all_params(self): + """ + list_data_product_contract_template() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates') + mock_response = '{"contract_templates": [{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "id": "20aa7c97-cfcc-4d16-ae76-2ca1847ce733", "creator_id": "IBMid-123456ABC", "created_at": "2025-06-26T12:30:20.000Z", "name": "Sample Data Contract Template", "error": {"code": "code", "message": "message"}, "contract_terms": {"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + container_id = 'testString' + contract_template_name = 'testString' + domain_ids = 'testString' + + # Invoke method + response = _service.list_data_product_contract_template( + container_id=container_id, + contract_template_name=contract_template_name, + domain_ids=domain_ids, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + assert 'contract_template.name={}'.format(contract_template_name) in query_string + assert 'domain.ids={}'.format(domain_ids) in query_string + + def test_list_data_product_contract_template_all_params_with_retries(self): + # Enable retries and run test_list_data_product_contract_template_all_params. + _service.enable_retries() + self.test_list_data_product_contract_template_all_params() + + # Disable retries and run test_list_data_product_contract_template_all_params. + _service.disable_retries() + self.test_list_data_product_contract_template_all_params() + + @responses.activate + def test_list_data_product_contract_template_required_params(self): + """ + test_list_data_product_contract_template_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates') + mock_response = '{"contract_templates": [{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "id": "20aa7c97-cfcc-4d16-ae76-2ca1847ce733", "creator_id": "IBMid-123456ABC", "created_at": "2025-06-26T12:30:20.000Z", "name": "Sample Data Contract Template", "error": {"code": "code", "message": "message"}, "contract_terms": {"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Invoke method + response = _service.list_data_product_contract_template() + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_list_data_product_contract_template_required_params_with_retries(self): + # Enable retries and run test_list_data_product_contract_template_required_params. + _service.enable_retries() + self.test_list_data_product_contract_template_required_params() + + # Disable retries and run test_list_data_product_contract_template_required_params. + _service.disable_retries() + self.test_list_data_product_contract_template_required_params() + + +class TestCreateContractTemplate: + """ + Test Class for create_contract_template + """ + + @responses.activate + def test_create_contract_template_all_params(self): + """ + create_contract_template() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "id": "20aa7c97-cfcc-4d16-ae76-2ca1847ce733", "creator_id": "IBMid-123456ABC", "created_at": "2025-06-26T12:30:20.000Z", "name": "Sample Data Contract Template", "error": {"code": "code", "message": "message"}, "contract_terms": {"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = '531f74a-01c8-4e91-8e29-b018db683c86' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a ErrorMessage model + error_message_model = {} + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = '0094ebe9-abc3-473b-80ea-c777ede095ea' + domain_model['name'] = 'Test Domain New' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://www.moreinfo.example.coms' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Intended purpose for the provided data.' + description_model['limitations'] = 'Technical, compliance, and legal limitations for data use.' + description_model['usage'] = 'Recommended usage of the data.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = 'Custom properties that are not part of the standard.' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'IAM Role' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = '100.00' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'slaproperty' + contract_template_sla_property_model['value'] = 'slavalue' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'sladefaultelement' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'channel' + contract_template_support_and_communication_model['url'] = 'https://www.example.coms' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'propertykey' + contract_template_custom_property_model['value'] = 'propertyvalue' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractTerms model + contract_terms_model = {} + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + # Set up parameter values + container = container_reference_model + id = 'testString' + creator_id = 'testString' + created_at = 'testString' + name = 'Sample Data Contract Template' + error = error_message_model + contract_terms = contract_terms_model + container_id = 'testString' + contract_template_name = 'testString' + domain_ids = 'testString' + + # Invoke method + response = _service.create_contract_template( + container, + id=id, + creator_id=creator_id, + created_at=created_at, + name=name, + error=error, + contract_terms=contract_terms, + container_id=container_id, + contract_template_name=contract_template_name, + domain_ids=domain_ids, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + assert 'contract_template.name={}'.format(contract_template_name) in query_string + assert 'domain.ids={}'.format(domain_ids) in query_string + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['container'] == container_reference_model + assert req_body['id'] == 'testString' + assert req_body['creator_id'] == 'testString' + assert req_body['created_at'] == 'testString' + assert req_body['name'] == 'Sample Data Contract Template' + assert req_body['error'] == error_message_model + assert req_body['contract_terms'] == contract_terms_model + + def test_create_contract_template_all_params_with_retries(self): + # Enable retries and run test_create_contract_template_all_params. + _service.enable_retries() + self.test_create_contract_template_all_params() + + # Disable retries and run test_create_contract_template_all_params. + _service.disable_retries() + self.test_create_contract_template_all_params() + + @responses.activate + def test_create_contract_template_required_params(self): + """ + test_create_contract_template_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "id": "20aa7c97-cfcc-4d16-ae76-2ca1847ce733", "creator_id": "IBMid-123456ABC", "created_at": "2025-06-26T12:30:20.000Z", "name": "Sample Data Contract Template", "error": {"code": "code", "message": "message"}, "contract_terms": {"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = '531f74a-01c8-4e91-8e29-b018db683c86' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a ErrorMessage model + error_message_model = {} + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = '0094ebe9-abc3-473b-80ea-c777ede095ea' + domain_model['name'] = 'Test Domain New' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://www.moreinfo.example.coms' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Intended purpose for the provided data.' + description_model['limitations'] = 'Technical, compliance, and legal limitations for data use.' + description_model['usage'] = 'Recommended usage of the data.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = 'Custom properties that are not part of the standard.' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'IAM Role' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = '100.00' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'slaproperty' + contract_template_sla_property_model['value'] = 'slavalue' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'sladefaultelement' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'channel' + contract_template_support_and_communication_model['url'] = 'https://www.example.coms' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'propertykey' + contract_template_custom_property_model['value'] = 'propertyvalue' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractTerms model + contract_terms_model = {} + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + # Set up parameter values + container = container_reference_model + id = 'testString' + creator_id = 'testString' + created_at = 'testString' + name = 'Sample Data Contract Template' + error = error_message_model + contract_terms = contract_terms_model + + # Invoke method + response = _service.create_contract_template( + container, + id=id, + creator_id=creator_id, + created_at=created_at, + name=name, + error=error, + contract_terms=contract_terms, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['container'] == container_reference_model + assert req_body['id'] == 'testString' + assert req_body['creator_id'] == 'testString' + assert req_body['created_at'] == 'testString' + assert req_body['name'] == 'Sample Data Contract Template' + assert req_body['error'] == error_message_model + assert req_body['contract_terms'] == contract_terms_model + + def test_create_contract_template_required_params_with_retries(self): + # Enable retries and run test_create_contract_template_required_params. + _service.enable_retries() + self.test_create_contract_template_required_params() + + # Disable retries and run test_create_contract_template_required_params. + _service.disable_retries() + self.test_create_contract_template_required_params() + + @responses.activate + def test_create_contract_template_value_error(self): + """ + test_create_contract_template_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "id": "20aa7c97-cfcc-4d16-ae76-2ca1847ce733", "creator_id": "IBMid-123456ABC", "created_at": "2025-06-26T12:30:20.000Z", "name": "Sample Data Contract Template", "error": {"code": "code", "message": "message"}, "contract_terms": {"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = '531f74a-01c8-4e91-8e29-b018db683c86' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a ErrorMessage model + error_message_model = {} + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + # Construct a dict representation of a AssetReference model + asset_reference_model = {} + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + # Construct a dict representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model = {} + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a dict representation of a ContractTermsDocument model + contract_terms_document_model = {} + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + # Construct a dict representation of a Domain model + domain_model = {} + domain_model['id'] = '0094ebe9-abc3-473b-80ea-c777ede095ea' + domain_model['name'] = 'Test Domain New' + domain_model['container'] = container_reference_model + + # Construct a dict representation of a Overview model + overview_model = {} + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a dict representation of a ContractTermsMoreInfo model + contract_terms_more_info_model = {} + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://www.moreinfo.example.coms' + + # Construct a dict representation of a Description model + description_model = {} + description_model['purpose'] = 'Intended purpose for the provided data.' + description_model['limitations'] = 'Technical, compliance, and legal limitations for data use.' + description_model['usage'] = 'Recommended usage of the data.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = 'Custom properties that are not part of the standard.' + + # Construct a dict representation of a ContractTemplateOrganization model + contract_template_organization_model = {} + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + # Construct a dict representation of a Roles model + roles_model = {} + roles_model['role'] = 'IAM Role' + + # Construct a dict representation of a Pricing model + pricing_model = {} + pricing_model['amount'] = '100.00' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + # Construct a dict representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model = {} + contract_template_sla_property_model['property'] = 'slaproperty' + contract_template_sla_property_model['value'] = 'slavalue' + + # Construct a dict representation of a ContractTemplateSLA model + contract_template_sla_model = {} + contract_template_sla_model['default_element'] = 'sladefaultelement' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + # Construct a dict representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model = {} + contract_template_support_and_communication_model['channel'] = 'channel' + contract_template_support_and_communication_model['url'] = 'https://www.example.coms' + + # Construct a dict representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model = {} + contract_template_custom_property_model['key'] = 'propertykey' + contract_template_custom_property_model['value'] = 'propertyvalue' + + # Construct a dict representation of a ContractTest model + contract_test_model = {} + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + # Construct a dict representation of a ContractAsset model + contract_asset_model = {} + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + # Construct a dict representation of a ContractServer model + contract_server_model = {} + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + # Construct a dict representation of a ContractSchemaPropertyType model + contract_schema_property_type_model = {} + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + # Construct a dict representation of a ContractQualityRule model + contract_quality_rule_model = {} + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a dict representation of a ContractSchemaProperty model + contract_schema_property_model = {} + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractSchema model + contract_schema_model = {} + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Construct a dict representation of a ContractTerms model + contract_terms_model = {} + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + # Set up parameter values + container = container_reference_model + id = 'testString' + creator_id = 'testString' + created_at = 'testString' + name = 'Sample Data Contract Template' + error = error_message_model + contract_terms = contract_terms_model + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "container": container, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.create_contract_template(**req_copy) + + def test_create_contract_template_value_error_with_retries(self): + # Enable retries and run test_create_contract_template_value_error. + _service.enable_retries() + self.test_create_contract_template_value_error() + + # Disable retries and run test_create_contract_template_value_error. + _service.disable_retries() + self.test_create_contract_template_value_error() + + +class TestGetContractTemplate: + """ + Test Class for get_contract_template + """ + + @responses.activate + def test_get_contract_template_all_params(self): + """ + get_contract_template() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates/testString') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "id": "20aa7c97-cfcc-4d16-ae76-2ca1847ce733", "creator_id": "IBMid-123456ABC", "created_at": "2025-06-26T12:30:20.000Z", "name": "Sample Data Contract Template", "error": {"code": "code", "message": "message"}, "contract_terms": {"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + contract_template_id = 'testString' + container_id = 'testString' + + # Invoke method + response = _service.get_contract_template( + contract_template_id, + container_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + + def test_get_contract_template_all_params_with_retries(self): + # Enable retries and run test_get_contract_template_all_params. + _service.enable_retries() + self.test_get_contract_template_all_params() + + # Disable retries and run test_get_contract_template_all_params. + _service.disable_retries() + self.test_get_contract_template_all_params() + + @responses.activate + def test_get_contract_template_value_error(self): + """ + test_get_contract_template_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates/testString') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "id": "20aa7c97-cfcc-4d16-ae76-2ca1847ce733", "creator_id": "IBMid-123456ABC", "created_at": "2025-06-26T12:30:20.000Z", "name": "Sample Data Contract Template", "error": {"code": "code", "message": "message"}, "contract_terms": {"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + contract_template_id = 'testString' + container_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "contract_template_id": contract_template_id, + "container_id": container_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_contract_template(**req_copy) + + def test_get_contract_template_value_error_with_retries(self): + # Enable retries and run test_get_contract_template_value_error. + _service.enable_retries() + self.test_get_contract_template_value_error() + + # Disable retries and run test_get_contract_template_value_error. + _service.disable_retries() + self.test_get_contract_template_value_error() + + +class TestDeleteDataProductContractTemplate: + """ + Test Class for delete_data_product_contract_template + """ + + @responses.activate + def test_delete_data_product_contract_template_all_params(self): + """ + delete_data_product_contract_template() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates/testString') + responses.add( + responses.DELETE, + url, + status=204, + ) + + # Set up parameter values + contract_template_id = 'testString' + container_id = 'testString' + + # Invoke method + response = _service.delete_data_product_contract_template( + contract_template_id, + container_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 204 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + + def test_delete_data_product_contract_template_all_params_with_retries(self): + # Enable retries and run test_delete_data_product_contract_template_all_params. + _service.enable_retries() + self.test_delete_data_product_contract_template_all_params() + + # Disable retries and run test_delete_data_product_contract_template_all_params. + _service.disable_retries() + self.test_delete_data_product_contract_template_all_params() + + @responses.activate + def test_delete_data_product_contract_template_value_error(self): + """ + test_delete_data_product_contract_template_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates/testString') + responses.add( + responses.DELETE, + url, + status=204, + ) + + # Set up parameter values + contract_template_id = 'testString' + container_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "contract_template_id": contract_template_id, + "container_id": container_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.delete_data_product_contract_template(**req_copy) + + def test_delete_data_product_contract_template_value_error_with_retries(self): + # Enable retries and run test_delete_data_product_contract_template_value_error. + _service.enable_retries() + self.test_delete_data_product_contract_template_value_error() + + # Disable retries and run test_delete_data_product_contract_template_value_error. + _service.disable_retries() + self.test_delete_data_product_contract_template_value_error() + + +class TestUpdateDataProductContractTemplate: + """ + Test Class for update_data_product_contract_template + """ + + @responses.activate + def test_update_data_product_contract_template_all_params(self): + """ + update_data_product_contract_template() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates/testString') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "id": "20aa7c97-cfcc-4d16-ae76-2ca1847ce733", "creator_id": "IBMid-123456ABC", "created_at": "2025-06-26T12:30:20.000Z", "name": "Sample Data Contract Template", "error": {"code": "code", "message": "message"}, "contract_terms": {"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + contract_template_id = 'testString' + container_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Invoke method + response = _service.update_data_product_contract_template( + contract_template_id, + container_id, + json_patch_instructions, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body == json_patch_instructions + + def test_update_data_product_contract_template_all_params_with_retries(self): + # Enable retries and run test_update_data_product_contract_template_all_params. + _service.enable_retries() + self.test_update_data_product_contract_template_all_params() + + # Disable retries and run test_update_data_product_contract_template_all_params. + _service.disable_retries() + self.test_update_data_product_contract_template_all_params() + + @responses.activate + def test_update_data_product_contract_template_value_error(self): + """ + test_update_data_product_contract_template_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/contract_templates/testString') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "id": "20aa7c97-cfcc-4d16-ae76-2ca1847ce733", "creator_id": "IBMid-123456ABC", "created_at": "2025-06-26T12:30:20.000Z", "name": "Sample Data Contract Template", "error": {"code": "code", "message": "message"}, "contract_terms": {"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + contract_template_id = 'testString' + container_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "contract_template_id": contract_template_id, + "container_id": container_id, + "json_patch_instructions": json_patch_instructions, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.update_data_product_contract_template(**req_copy) + + def test_update_data_product_contract_template_value_error_with_retries(self): + # Enable retries and run test_update_data_product_contract_template_value_error. + _service.enable_retries() + self.test_update_data_product_contract_template_value_error() + + # Disable retries and run test_update_data_product_contract_template_value_error. + _service.disable_retries() + self.test_update_data_product_contract_template_value_error() + + +# endregion +############################################################################## +# End of Service: DataProductContractTemplates +############################################################################## + +############################################################################## +# Start of Service: DataProductDomains +############################################################################## +# region + + +class TestNewInstance: + """ + Test Class for new_instance + """ + + def test_new_instance(self): + """ + new_instance() + """ + os.environ['TEST_SERVICE_AUTH_TYPE'] = 'noAuth' + + service = DphV1.new_instance( + service_name='TEST_SERVICE', + ) + + assert service is not None + assert isinstance(service, DphV1) + + def test_new_instance_without_authenticator(self): + """ + new_instance_without_authenticator() + """ + with pytest.raises(ValueError, match='authenticator must be provided'): + service = DphV1.new_instance( + service_name='TEST_SERVICE_NOT_FOUND', + ) + + +class TestListDataProductDomains: + """ + Test Class for list_data_product_domains + """ + + @responses.activate + def test_list_data_product_domains_all_params(self): + """ + list_data_product_domains() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains') + mock_response = '{"domains": [{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "name": "Operations", "description": "This is a description of the data product domain.", "id": "id", "created_by": "created_by", "member_roles": {"user_iam_id": "user_iam_id", "roles": ["roles"]}, "properties": {"value": "value"}, "sub_domains": [{"name": "Operations", "id": "id", "description": "description"}], "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + container_id = 'testString' + include_subdomains = True + + # Invoke method + response = _service.list_data_product_domains( + container_id=container_id, + include_subdomains=include_subdomains, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + assert 'include_subdomains={}'.format('true' if include_subdomains else 'false') in query_string + + def test_list_data_product_domains_all_params_with_retries(self): + # Enable retries and run test_list_data_product_domains_all_params. + _service.enable_retries() + self.test_list_data_product_domains_all_params() + + # Disable retries and run test_list_data_product_domains_all_params. + _service.disable_retries() + self.test_list_data_product_domains_all_params() + + @responses.activate + def test_list_data_product_domains_required_params(self): + """ + test_list_data_product_domains_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains') + mock_response = '{"domains": [{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "name": "Operations", "description": "This is a description of the data product domain.", "id": "id", "created_by": "created_by", "member_roles": {"user_iam_id": "user_iam_id", "roles": ["roles"]}, "properties": {"value": "value"}, "sub_domains": [{"name": "Operations", "id": "id", "description": "description"}], "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Invoke method + response = _service.list_data_product_domains() + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_list_data_product_domains_required_params_with_retries(self): + # Enable retries and run test_list_data_product_domains_required_params. + _service.enable_retries() + self.test_list_data_product_domains_required_params() + + # Disable retries and run test_list_data_product_domains_required_params. + _service.disable_retries() + self.test_list_data_product_domains_required_params() + + +class TestCreateDataProductDomain: + """ + Test Class for create_data_product_domain + """ + + @responses.activate + def test_create_data_product_domain_all_params(self): + """ + create_data_product_domain() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "name": "Operations", "description": "This is a description of the data product domain.", "id": "id", "created_by": "created_by", "member_roles": {"user_iam_id": "user_iam_id", "roles": ["roles"]}, "properties": {"value": "value"}, "sub_domains": [{"name": "Operations", "id": "id", "description": "description"}], "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'ed580171-a6e4-4b93-973f-ae2f2f62991b' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a ErrorExtraResource model + error_extra_resource_model = {} + error_extra_resource_model['id'] = 'testString' + error_extra_resource_model['timestamp'] = '2019-01-01T12:00:00Z' + error_extra_resource_model['environment_name'] = 'testString' + error_extra_resource_model['http_status'] = 0 + error_extra_resource_model['source_cluster'] = 0 + error_extra_resource_model['source_component'] = 0 + error_extra_resource_model['transaction_id'] = 0 + + # Construct a dict representation of a ErrorModelResource model + error_model_resource_model = {} + error_model_resource_model['code'] = 'request_body_error' + error_model_resource_model['message'] = 'testString' + error_model_resource_model['extra'] = error_extra_resource_model + error_model_resource_model['more_info'] = 'testString' + + # Construct a dict representation of a MemberRolesSchema model + member_roles_schema_model = {} + member_roles_schema_model['user_iam_id'] = 'testString' + member_roles_schema_model['roles'] = ['testString'] + + # Construct a dict representation of a PropertiesSchema model + properties_schema_model = {} + properties_schema_model['value'] = 'testString' + + # Construct a dict representation of a InitializeSubDomain model + initialize_sub_domain_model = {} + initialize_sub_domain_model['name'] = 'Sub domain 1' + initialize_sub_domain_model['id'] = 'testString' + initialize_sub_domain_model['description'] = 'New sub domain 1' + + # Construct a dict representation of a ContainerIdentity model + container_identity_model = {} + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Set up parameter values + container = container_reference_model + trace = 'testString' + errors = [error_model_resource_model] + name = 'Test domain' + description = 'The sample description for new domain' + id = 'testString' + created_by = 'testString' + member_roles = member_roles_schema_model + properties = properties_schema_model + sub_domains = [initialize_sub_domain_model] + sub_container = container_identity_model + link_to_subcontainers = False + + # Invoke method + response = _service.create_data_product_domain( + container, + trace=trace, + errors=errors, + name=name, + description=description, + id=id, + created_by=created_by, + member_roles=member_roles, + properties=properties, + sub_domains=sub_domains, + sub_container=sub_container, + link_to_subcontainers=link_to_subcontainers, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'link_to_subcontainers={}'.format('true' if link_to_subcontainers else 'false') in query_string + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['container'] == container_reference_model + assert req_body['trace'] == 'testString' + assert req_body['errors'] == [error_model_resource_model] + assert req_body['name'] == 'Test domain' + assert req_body['description'] == 'The sample description for new domain' + assert req_body['id'] == 'testString' + assert req_body['created_by'] == 'testString' + assert req_body['member_roles'] == member_roles_schema_model + assert req_body['properties'] == properties_schema_model + assert req_body['sub_domains'] == [initialize_sub_domain_model] + assert req_body['sub_container'] == container_identity_model + + def test_create_data_product_domain_all_params_with_retries(self): + # Enable retries and run test_create_data_product_domain_all_params. + _service.enable_retries() + self.test_create_data_product_domain_all_params() + + # Disable retries and run test_create_data_product_domain_all_params. + _service.disable_retries() + self.test_create_data_product_domain_all_params() + + @responses.activate + def test_create_data_product_domain_required_params(self): + """ + test_create_data_product_domain_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "name": "Operations", "description": "This is a description of the data product domain.", "id": "id", "created_by": "created_by", "member_roles": {"user_iam_id": "user_iam_id", "roles": ["roles"]}, "properties": {"value": "value"}, "sub_domains": [{"name": "Operations", "id": "id", "description": "description"}], "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'ed580171-a6e4-4b93-973f-ae2f2f62991b' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a ErrorExtraResource model + error_extra_resource_model = {} + error_extra_resource_model['id'] = 'testString' + error_extra_resource_model['timestamp'] = '2019-01-01T12:00:00Z' + error_extra_resource_model['environment_name'] = 'testString' + error_extra_resource_model['http_status'] = 0 + error_extra_resource_model['source_cluster'] = 0 + error_extra_resource_model['source_component'] = 0 + error_extra_resource_model['transaction_id'] = 0 + + # Construct a dict representation of a ErrorModelResource model + error_model_resource_model = {} + error_model_resource_model['code'] = 'request_body_error' + error_model_resource_model['message'] = 'testString' + error_model_resource_model['extra'] = error_extra_resource_model + error_model_resource_model['more_info'] = 'testString' + + # Construct a dict representation of a MemberRolesSchema model + member_roles_schema_model = {} + member_roles_schema_model['user_iam_id'] = 'testString' + member_roles_schema_model['roles'] = ['testString'] + + # Construct a dict representation of a PropertiesSchema model + properties_schema_model = {} + properties_schema_model['value'] = 'testString' + + # Construct a dict representation of a InitializeSubDomain model + initialize_sub_domain_model = {} + initialize_sub_domain_model['name'] = 'Sub domain 1' + initialize_sub_domain_model['id'] = 'testString' + initialize_sub_domain_model['description'] = 'New sub domain 1' + + # Construct a dict representation of a ContainerIdentity model + container_identity_model = {} + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Set up parameter values + container = container_reference_model + trace = 'testString' + errors = [error_model_resource_model] + name = 'Test domain' + description = 'The sample description for new domain' + id = 'testString' + created_by = 'testString' + member_roles = member_roles_schema_model + properties = properties_schema_model + sub_domains = [initialize_sub_domain_model] + sub_container = container_identity_model + + # Invoke method + response = _service.create_data_product_domain( + container, + trace=trace, + errors=errors, + name=name, + description=description, + id=id, + created_by=created_by, + member_roles=member_roles, + properties=properties, + sub_domains=sub_domains, + sub_container=sub_container, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['container'] == container_reference_model + assert req_body['trace'] == 'testString' + assert req_body['errors'] == [error_model_resource_model] + assert req_body['name'] == 'Test domain' + assert req_body['description'] == 'The sample description for new domain' + assert req_body['id'] == 'testString' + assert req_body['created_by'] == 'testString' + assert req_body['member_roles'] == member_roles_schema_model + assert req_body['properties'] == properties_schema_model + assert req_body['sub_domains'] == [initialize_sub_domain_model] + assert req_body['sub_container'] == container_identity_model + + def test_create_data_product_domain_required_params_with_retries(self): + # Enable retries and run test_create_data_product_domain_required_params. + _service.enable_retries() + self.test_create_data_product_domain_required_params() + + # Disable retries and run test_create_data_product_domain_required_params. + _service.disable_retries() + self.test_create_data_product_domain_required_params() + + @responses.activate + def test_create_data_product_domain_value_error(self): + """ + test_create_data_product_domain_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "name": "Operations", "description": "This is a description of the data product domain.", "id": "id", "created_by": "created_by", "member_roles": {"user_iam_id": "user_iam_id", "roles": ["roles"]}, "properties": {"value": "value"}, "sub_domains": [{"name": "Operations", "id": "id", "description": "description"}], "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Construct a dict representation of a ContainerReference model + container_reference_model = {} + container_reference_model['id'] = 'ed580171-a6e4-4b93-973f-ae2f2f62991b' + container_reference_model['type'] = 'catalog' + + # Construct a dict representation of a ErrorExtraResource model + error_extra_resource_model = {} + error_extra_resource_model['id'] = 'testString' + error_extra_resource_model['timestamp'] = '2019-01-01T12:00:00Z' + error_extra_resource_model['environment_name'] = 'testString' + error_extra_resource_model['http_status'] = 0 + error_extra_resource_model['source_cluster'] = 0 + error_extra_resource_model['source_component'] = 0 + error_extra_resource_model['transaction_id'] = 0 + + # Construct a dict representation of a ErrorModelResource model + error_model_resource_model = {} + error_model_resource_model['code'] = 'request_body_error' + error_model_resource_model['message'] = 'testString' + error_model_resource_model['extra'] = error_extra_resource_model + error_model_resource_model['more_info'] = 'testString' + + # Construct a dict representation of a MemberRolesSchema model + member_roles_schema_model = {} + member_roles_schema_model['user_iam_id'] = 'testString' + member_roles_schema_model['roles'] = ['testString'] + + # Construct a dict representation of a PropertiesSchema model + properties_schema_model = {} + properties_schema_model['value'] = 'testString' + + # Construct a dict representation of a InitializeSubDomain model + initialize_sub_domain_model = {} + initialize_sub_domain_model['name'] = 'Sub domain 1' + initialize_sub_domain_model['id'] = 'testString' + initialize_sub_domain_model['description'] = 'New sub domain 1' + + # Construct a dict representation of a ContainerIdentity model + container_identity_model = {} + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Set up parameter values + container = container_reference_model + trace = 'testString' + errors = [error_model_resource_model] + name = 'Test domain' + description = 'The sample description for new domain' + id = 'testString' + created_by = 'testString' + member_roles = member_roles_schema_model + properties = properties_schema_model + sub_domains = [initialize_sub_domain_model] + sub_container = container_identity_model + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "container": container, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.create_data_product_domain(**req_copy) + + def test_create_data_product_domain_value_error_with_retries(self): + # Enable retries and run test_create_data_product_domain_value_error. + _service.enable_retries() + self.test_create_data_product_domain_value_error() + + # Disable retries and run test_create_data_product_domain_value_error. + _service.disable_retries() + self.test_create_data_product_domain_value_error() + + +class TestCreateDataProductSubdomain: + """ + Test Class for create_data_product_subdomain + """ + + @responses.activate + def test_create_data_product_subdomain_all_params(self): + """ + create_data_product_subdomain() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString/subdomains') + mock_response = '{"name": "Operations", "id": "id", "description": "description"}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Set up parameter values + domain_id = 'testString' + container_id = 'testString' + name = 'Sub domain 1' + id = 'testString' + description = 'New sub domain 1' + + # Invoke method + response = _service.create_data_product_subdomain( + domain_id, + container_id, + name=name, + id=id, + description=description, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body['name'] == 'Sub domain 1' + assert req_body['id'] == 'testString' + assert req_body['description'] == 'New sub domain 1' + + def test_create_data_product_subdomain_all_params_with_retries(self): + # Enable retries and run test_create_data_product_subdomain_all_params. + _service.enable_retries() + self.test_create_data_product_subdomain_all_params() + + # Disable retries and run test_create_data_product_subdomain_all_params. + _service.disable_retries() + self.test_create_data_product_subdomain_all_params() + + @responses.activate + def test_create_data_product_subdomain_value_error(self): + """ + test_create_data_product_subdomain_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString/subdomains') + mock_response = '{"name": "Operations", "id": "id", "description": "description"}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Set up parameter values + domain_id = 'testString' + container_id = 'testString' + name = 'Sub domain 1' + id = 'testString' + description = 'New sub domain 1' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "domain_id": domain_id, + "container_id": container_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.create_data_product_subdomain(**req_copy) + + def test_create_data_product_subdomain_value_error_with_retries(self): + # Enable retries and run test_create_data_product_subdomain_value_error. + _service.enable_retries() + self.test_create_data_product_subdomain_value_error() + + # Disable retries and run test_create_data_product_subdomain_value_error. + _service.disable_retries() + self.test_create_data_product_subdomain_value_error() + + +class TestGetDomain: + """ + Test Class for get_domain + """ + + @responses.activate + def test_get_domain_all_params(self): + """ + get_domain() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "name": "Operations", "description": "This is a description of the data product domain.", "id": "id", "created_by": "created_by", "member_roles": {"user_iam_id": "user_iam_id", "roles": ["roles"]}, "properties": {"value": "value"}, "sub_domains": [{"name": "Operations", "id": "id", "description": "description"}], "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + domain_id = 'testString' + + # Invoke method + response = _service.get_domain( + domain_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_domain_all_params_with_retries(self): + # Enable retries and run test_get_domain_all_params. + _service.enable_retries() + self.test_get_domain_all_params() + + # Disable retries and run test_get_domain_all_params. + _service.disable_retries() + self.test_get_domain_all_params() + + @responses.activate + def test_get_domain_value_error(self): + """ + test_get_domain_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "name": "Operations", "description": "This is a description of the data product domain.", "id": "id", "created_by": "created_by", "member_roles": {"user_iam_id": "user_iam_id", "roles": ["roles"]}, "properties": {"value": "value"}, "sub_domains": [{"name": "Operations", "id": "id", "description": "description"}], "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + domain_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "domain_id": domain_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_domain(**req_copy) + + def test_get_domain_value_error_with_retries(self): + # Enable retries and run test_get_domain_value_error. + _service.enable_retries() + self.test_get_domain_value_error() + + # Disable retries and run test_get_domain_value_error. + _service.disable_retries() + self.test_get_domain_value_error() + + +class TestDeleteDomain: + """ + Test Class for delete_domain + """ + + @responses.activate + def test_delete_domain_all_params(self): + """ + delete_domain() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString') + responses.add( + responses.DELETE, + url, + status=204, + ) + + # Set up parameter values + domain_id = 'testString' + + # Invoke method + response = _service.delete_domain( + domain_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 204 + + def test_delete_domain_all_params_with_retries(self): + # Enable retries and run test_delete_domain_all_params. + _service.enable_retries() + self.test_delete_domain_all_params() + + # Disable retries and run test_delete_domain_all_params. + _service.disable_retries() + self.test_delete_domain_all_params() + + @responses.activate + def test_delete_domain_value_error(self): + """ + test_delete_domain_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString') + responses.add( + responses.DELETE, + url, + status=204, + ) + + # Set up parameter values + domain_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "domain_id": domain_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.delete_domain(**req_copy) + + def test_delete_domain_value_error_with_retries(self): + # Enable retries and run test_delete_domain_value_error. + _service.enable_retries() + self.test_delete_domain_value_error() + + # Disable retries and run test_delete_domain_value_error. + _service.disable_retries() + self.test_delete_domain_value_error() + + +class TestUpdateDataProductDomain: + """ + Test Class for update_data_product_domain + """ + + @responses.activate + def test_update_data_product_domain_all_params(self): + """ + update_data_product_domain() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "name": "Operations", "description": "This is a description of the data product domain.", "id": "id", "created_by": "created_by", "member_roles": {"user_iam_id": "user_iam_id", "roles": ["roles"]}, "properties": {"value": "value"}, "sub_domains": [{"name": "Operations", "id": "id", "description": "description"}], "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + domain_id = 'testString' + container_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Invoke method + response = _service.update_data_product_domain( + domain_id, + container_id, + json_patch_instructions, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + # Validate body params + req_body = json.loads(str(responses.calls[0].request.body, 'utf-8')) + assert req_body == json_patch_instructions + + def test_update_data_product_domain_all_params_with_retries(self): + # Enable retries and run test_update_data_product_domain_all_params. + _service.enable_retries() + self.test_update_data_product_domain_all_params() + + # Disable retries and run test_update_data_product_domain_all_params. + _service.disable_retries() + self.test_update_data_product_domain_all_params() + + @responses.activate + def test_update_data_product_domain_value_error(self): + """ + test_update_data_product_domain_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString') + mock_response = '{"container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "trace": "trace", "errors": [{"code": "request_body_error", "message": "message", "extra": {"id": "id", "timestamp": "2019-01-01T12:00:00.000Z", "environment_name": "environment_name", "http_status": 0, "source_cluster": 0, "source_component": 0, "transaction_id": 0}, "more_info": "more_info"}], "name": "Operations", "description": "This is a description of the data product domain.", "id": "id", "created_by": "created_by", "member_roles": {"user_iam_id": "user_iam_id", "roles": ["roles"]}, "properties": {"value": "value"}, "sub_domains": [{"name": "Operations", "id": "id", "description": "description"}], "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}}' + responses.add( + responses.PATCH, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Construct a dict representation of a JsonPatchOperation model + json_patch_operation_model = {} + json_patch_operation_model['op'] = 'add' + json_patch_operation_model['path'] = 'testString' + json_patch_operation_model['from'] = 'testString' + json_patch_operation_model['value'] = 'testString' + + # Set up parameter values + domain_id = 'testString' + container_id = 'testString' + json_patch_instructions = [json_patch_operation_model] + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "domain_id": domain_id, + "container_id": container_id, + "json_patch_instructions": json_patch_instructions, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.update_data_product_domain(**req_copy) + + def test_update_data_product_domain_value_error_with_retries(self): + # Enable retries and run test_update_data_product_domain_value_error. + _service.enable_retries() + self.test_update_data_product_domain_value_error() + + # Disable retries and run test_update_data_product_domain_value_error. + _service.disable_retries() + self.test_update_data_product_domain_value_error() + + +class TestGetDataProductByDomain: + """ + Test Class for get_data_product_by_domain + """ + + @responses.activate + def test_get_data_product_by_domain_all_params(self): + """ + get_data_product_by_domain() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString/data_products') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "data_product_versions": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + domain_id = 'testString' + container_id = 'testString' + + # Invoke method + response = _service.get_data_product_by_domain( + domain_id, + container_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'container.id={}'.format(container_id) in query_string + + def test_get_data_product_by_domain_all_params_with_retries(self): + # Enable retries and run test_get_data_product_by_domain_all_params. + _service.enable_retries() + self.test_get_data_product_by_domain_all_params() + + # Disable retries and run test_get_data_product_by_domain_all_params. + _service.disable_retries() + self.test_get_data_product_by_domain_all_params() + + @responses.activate + def test_get_data_product_by_domain_value_error(self): + """ + test_get_data_product_by_domain_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/domains/testString/data_products') + mock_response = '{"limit": 200, "first": {"href": "https://api.example.com/collection"}, "next": {"href": "https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9", "start": "eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9"}, "total_results": 200, "data_product_versions": [{"version": "1.0.0", "state": "draft", "data_product": {"id": "b38df608-d34b-4d58-8136-ed25e6c6684e", "release": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}, "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "name": "My Data Product", "description": "This is a description of My Data Product.", "tags": ["tags"], "use_cases": [{"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}], "types": ["data"], "contract_terms": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "id": "id", "documents": [{"url": "url", "type": "terms_and_conditions", "name": "name", "id": "2b0bf220-079c-11ee-be56-0242ac120002", "attachment": {"id": "id"}, "upload_url": "upload_url"}], "error_msg": "error_msg", "overview": {"api_version": "v3.0.1", "kind": "DataContract", "name": "Sample Data Contract", "version": "0.0.0", "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "more_info": "List of links to sources that provide more details on the data contract."}, "description": {"purpose": "Used for customer behavior analysis.", "limitations": "Data cannot be used for marketing.", "usage": "Data should be used only for analytics.", "more_info": [{"type": "privacy-statement", "url": "https://moreinfo.example.com"}], "custom_properties": "{\\"property1\\":\\"value1\\"}"}, "organization": [{"user_id": "IBMid-691000IN4G", "role": "owner"}], "roles": [{"role": "owner"}], "price": {"amount": "100.0", "currency": "USD", "unit": "megabyte"}, "sla": [{"default_element": "Standard SLA Policy", "properties": [{"property": "Uptime Guarantee", "value": "99.9"}]}], "support_and_communication": [{"channel": "Email Support", "url": "https://support.example.com"}], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}], "contract_test": {"status": "pass", "last_tested_time": "last_tested_time", "message": "message"}, "servers": [{"server": "server", "asset": {"id": "id", "name": "name"}, "connection_id": "connection_id", "type": "type", "description": "description", "environment": "environment", "account": "account", "catalog": "catalog", "database": "database", "dataset": "dataset", "delimiter": "delimiter", "endpoint_url": "endpoint_url", "format": "format", "host": "host", "location": "location", "path": "path", "port": "port", "project": "project", "region": "region", "region_name": "region_name", "schema": "schema", "service_name": "service_name", "staging_dir": "staging_dir", "stream": "stream", "warehouse": "warehouse", "roles": ["roles"], "custom_properties": [{"key": "customPropertyKey", "value": "customPropertyValue"}]}], "schema": [{"asset_id": "2b0bf220-079c-11ee-be56-0242ac120002", "connection_id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "description": "description", "connection_path": "connection_path", "physical_type": "physical_type", "properties": [{"name": "name", "type": {"type": "type", "length": "length", "scale": "scale", "nullable": "nullable", "signed": "signed", "native_type": "native_type"}, "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}], "quality": [{"type": "sql", "description": "description", "rule": "rule", "implementation": "implementation", "engine": "engine", "must_be_less_than": "must_be_less_than", "must_be_less_or_equal_to": "must_be_less_or_equal_to", "must_be_greater_than": "must_be_greater_than", "must_be_greater_or_equal_to": "must_be_greater_or_equal_to", "must_be_between": ["must_be_between"], "must_not_be_between": ["must_not_be_between"], "must_be": "must_be", "must_not_be": "must_not_be", "name": "name", "unit": "unit", "query": "query"}]}]}], "domain": {"id": "id", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}, "parts_out": [{"asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "type": "data_asset"}, "delivery_methods": [{"id": "09cf5fcc-cb9d-4995-a8e4-16517b25229f", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}, "getproperties": {"producer_input": {"engine_details": {"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}, "engines": [{"display_name": "Iceberg Engine", "engine_id": "presto767", "engine_port": "34567", "engine_host": "a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud", "engine_type": "spark", "associated_catalogs": ["associated_catalogs"]}]}}}]}], "workflows": {"order_access_request": {"task_assignee_users": ["task_assignee_users"], "pre_approved_users": ["pre_approved_users"], "custom_workflow_definition": {"id": "18bdbde1-918e-4ecf-aa23-6727bf319e14"}}}, "dataview_enabled": true, "comments": "Comments by a producer that are provided either at the time of data product version creation or retiring", "access_control": {"owner": "IBMid-696000KYV9"}, "last_updated_at": "2019-01-01T12:00:00.000Z", "sub_container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd"}, "is_restricted": false, "id": "2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd", "asset": {"id": "2b0bf220-079c-11ee-be56-0242ac120002", "name": "name", "container": {"id": "d29c42eb-7100-4b7a-8257-c196dbcca1cd", "type": "catalog"}}}]}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + domain_id = 'testString' + container_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "domain_id": domain_id, + "container_id": container_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_data_product_by_domain(**req_copy) + + def test_get_data_product_by_domain_value_error_with_retries(self): + # Enable retries and run test_get_data_product_by_domain_value_error. + _service.enable_retries() + self.test_get_data_product_by_domain_value_error() + + # Disable retries and run test_get_data_product_by_domain_value_error. + _service.disable_retries() + self.test_get_data_product_by_domain_value_error() + + +# endregion +############################################################################## +# End of Service: DataProductDomains +############################################################################## + +############################################################################## +# Start of Service: BucketServices +############################################################################## +# region + + +class TestNewInstance: + """ + Test Class for new_instance + """ + + def test_new_instance(self): + """ + new_instance() + """ + os.environ['TEST_SERVICE_AUTH_TYPE'] = 'noAuth' + + service = DphV1.new_instance( + service_name='TEST_SERVICE', + ) + + assert service is not None + assert isinstance(service, DphV1) + + def test_new_instance_without_authenticator(self): + """ + new_instance_without_authenticator() + """ + with pytest.raises(ValueError, match='authenticator must be provided'): + service = DphV1.new_instance( + service_name='TEST_SERVICE_NOT_FOUND', + ) + + +class TestCreateS3Bucket: + """ + Test Class for create_s3_bucket + """ + + @responses.activate + def test_create_s3_bucket_all_params(self): + """ + create_s3_bucket() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/bucket') + mock_response = '{"bucket_name": "bucket_name", "bucket_location": "bucket_location", "role_arn": "role_arn", "bucket_type": "bucket_type", "shared": true}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Set up parameter values + is_shared = True + + # Invoke method + response = _service.create_s3_bucket( + is_shared, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 201 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'is_shared={}'.format('true' if is_shared else 'false') in query_string + + def test_create_s3_bucket_all_params_with_retries(self): + # Enable retries and run test_create_s3_bucket_all_params. + _service.enable_retries() + self.test_create_s3_bucket_all_params() + + # Disable retries and run test_create_s3_bucket_all_params. + _service.disable_retries() + self.test_create_s3_bucket_all_params() + + @responses.activate + def test_create_s3_bucket_value_error(self): + """ + test_create_s3_bucket_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/bucket') + mock_response = '{"bucket_name": "bucket_name", "bucket_location": "bucket_location", "role_arn": "role_arn", "bucket_type": "bucket_type", "shared": true}' + responses.add( + responses.POST, + url, + body=mock_response, + content_type='application/json', + status=201, + ) + + # Set up parameter values + is_shared = True + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "is_shared": is_shared, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.create_s3_bucket(**req_copy) + + def test_create_s3_bucket_value_error_with_retries(self): + # Enable retries and run test_create_s3_bucket_value_error. + _service.enable_retries() + self.test_create_s3_bucket_value_error() + + # Disable retries and run test_create_s3_bucket_value_error. + _service.disable_retries() + self.test_create_s3_bucket_value_error() + + +class TestGetS3BucketValidation: + """ + Test Class for get_s3_bucket_validation + """ + + @responses.activate + def test_get_s3_bucket_validation_all_params(self): + """ + get_s3_bucket_validation() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/bucket/validate/testString') + mock_response = '{"bucket_exists": false}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + bucket_name = 'testString' + + # Invoke method + response = _service.get_s3_bucket_validation( + bucket_name, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + + def test_get_s3_bucket_validation_all_params_with_retries(self): + # Enable retries and run test_get_s3_bucket_validation_all_params. + _service.enable_retries() + self.test_get_s3_bucket_validation_all_params() + + # Disable retries and run test_get_s3_bucket_validation_all_params. + _service.disable_retries() + self.test_get_s3_bucket_validation_all_params() + + @responses.activate + def test_get_s3_bucket_validation_value_error(self): + """ + test_get_s3_bucket_validation_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/bucket/validate/testString') + mock_response = '{"bucket_exists": false}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + bucket_name = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "bucket_name": bucket_name, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_s3_bucket_validation(**req_copy) + + def test_get_s3_bucket_validation_value_error_with_retries(self): + # Enable retries and run test_get_s3_bucket_validation_value_error. + _service.enable_retries() + self.test_get_s3_bucket_validation_value_error() + + # Disable retries and run test_get_s3_bucket_validation_value_error. + _service.disable_retries() + self.test_get_s3_bucket_validation_value_error() + + +# endregion +############################################################################## +# End of Service: BucketServices +############################################################################## + +############################################################################## +# Start of Service: DataProductRevokeAccessJobRuns +############################################################################## +# region + + +class TestNewInstance: + """ + Test Class for new_instance + """ + + def test_new_instance(self): + """ + new_instance() + """ + os.environ['TEST_SERVICE_AUTH_TYPE'] = 'noAuth' + + service = DphV1.new_instance( + service_name='TEST_SERVICE', + ) + + assert service is not None + assert isinstance(service, DphV1) + + def test_new_instance_without_authenticator(self): + """ + new_instance_without_authenticator() + """ + with pytest.raises(ValueError, match='authenticator must be provided'): + service = DphV1.new_instance( + service_name='TEST_SERVICE_NOT_FOUND', + ) + + +class TestGetRevokeAccessProcessState: + """ + Test Class for get_revoke_access_process_state + """ + + @responses.activate + def test_get_revoke_access_process_state_all_params(self): + """ + get_revoke_access_process_state() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_product_revoke_access/job_runs') + mock_response = '{"results": [{"metadata": {"anyKey": "anyValue"}, "entity": {"anyKey": "anyValue"}}], "total_count": 42, "next": {"query": "ibm_data_product_revoke_access.state:(SCHEDULED OR FAILED)", "limit": 1, "bookmark": "MQ==", "include": "entity", "skip": 0}}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + release_id = 'testString' + limit = 200 + start = 'testString' + + # Invoke method + response = _service.get_revoke_access_process_state( + release_id, + limit=limit, + start=start, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'release_id={}'.format(release_id) in query_string + assert 'limit={}'.format(limit) in query_string + assert 'start={}'.format(start) in query_string + + def test_get_revoke_access_process_state_all_params_with_retries(self): + # Enable retries and run test_get_revoke_access_process_state_all_params. + _service.enable_retries() + self.test_get_revoke_access_process_state_all_params() + + # Disable retries and run test_get_revoke_access_process_state_all_params. + _service.disable_retries() + self.test_get_revoke_access_process_state_all_params() + + @responses.activate + def test_get_revoke_access_process_state_required_params(self): + """ + test_get_revoke_access_process_state_required_params() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_product_revoke_access/job_runs') + mock_response = '{"results": [{"metadata": {"anyKey": "anyValue"}, "entity": {"anyKey": "anyValue"}}], "total_count": 42, "next": {"query": "ibm_data_product_revoke_access.state:(SCHEDULED OR FAILED)", "limit": 1, "bookmark": "MQ==", "include": "entity", "skip": 0}}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + release_id = 'testString' + + # Invoke method + response = _service.get_revoke_access_process_state( + release_id, + headers={}, + ) + + # Check for correct operation + assert len(responses.calls) == 1 + assert response.status_code == 200 + # Validate query params + query_string = responses.calls[0].request.url.split('?', 1)[1] + query_string = urllib.parse.unquote_plus(query_string) + assert 'release_id={}'.format(release_id) in query_string + + def test_get_revoke_access_process_state_required_params_with_retries(self): + # Enable retries and run test_get_revoke_access_process_state_required_params. + _service.enable_retries() + self.test_get_revoke_access_process_state_required_params() + + # Disable retries and run test_get_revoke_access_process_state_required_params. + _service.disable_retries() + self.test_get_revoke_access_process_state_required_params() + + @responses.activate + def test_get_revoke_access_process_state_value_error(self): + """ + test_get_revoke_access_process_state_value_error() + """ + # Set up mock + url = preprocess_url('/data_product_exchange/v1/data_product_revoke_access/job_runs') + mock_response = '{"results": [{"metadata": {"anyKey": "anyValue"}, "entity": {"anyKey": "anyValue"}}], "total_count": 42, "next": {"query": "ibm_data_product_revoke_access.state:(SCHEDULED OR FAILED)", "limit": 1, "bookmark": "MQ==", "include": "entity", "skip": 0}}' + responses.add( + responses.GET, + url, + body=mock_response, + content_type='application/json', + status=200, + ) + + # Set up parameter values + release_id = 'testString' + + # Pass in all but one required param and check for a ValueError + req_param_dict = { + "release_id": release_id, + } + for param in req_param_dict.keys(): + req_copy = {key: val if key is not param else None for (key, val) in req_param_dict.items()} + with pytest.raises(ValueError): + _service.get_revoke_access_process_state(**req_copy) + + def test_get_revoke_access_process_state_value_error_with_retries(self): + # Enable retries and run test_get_revoke_access_process_state_value_error. + _service.enable_retries() + self.test_get_revoke_access_process_state_value_error() + + # Disable retries and run test_get_revoke_access_process_state_value_error. + _service.disable_retries() + self.test_get_revoke_access_process_state_value_error() + + +# endregion +############################################################################## +# End of Service: DataProductRevokeAccessJobRuns +############################################################################## + + +############################################################################## +# Start of Model Tests +############################################################################## +# region + + +class TestModel_Asset: + """ + Test Class for Asset + """ + + def test_asset_serialization(self): + """ + Test serialization/deserialization for Asset + """ + + # Construct a json representation of a Asset model + asset_model_json = {} + asset_model_json['metadata'] = {'anyKey': 'anyValue'} + asset_model_json['entity'] = {'anyKey': 'anyValue'} + + # Construct a model instance of Asset by calling from_dict on the json representation + asset_model = Asset.from_dict(asset_model_json) + assert asset_model != False + + # Construct a model instance of Asset by calling from_dict on the json representation + asset_model_dict = Asset.from_dict(asset_model_json).__dict__ + asset_model2 = Asset(**asset_model_dict) + + # Verify the model instances are equivalent + assert asset_model == asset_model2 + + # Convert model instance back to dict and verify no loss of data + asset_model_json2 = asset_model.to_dict() + assert asset_model_json2 == asset_model_json + + +class TestModel_AssetListAccessControl: + """ + Test Class for AssetListAccessControl + """ + + def test_asset_list_access_control_serialization(self): + """ + Test serialization/deserialization for AssetListAccessControl + """ + + # Construct a json representation of a AssetListAccessControl model + asset_list_access_control_model_json = {} + asset_list_access_control_model_json['owner'] = 'IBMid-696000KYV9' + + # Construct a model instance of AssetListAccessControl by calling from_dict on the json representation + asset_list_access_control_model = AssetListAccessControl.from_dict(asset_list_access_control_model_json) + assert asset_list_access_control_model != False + + # Construct a model instance of AssetListAccessControl by calling from_dict on the json representation + asset_list_access_control_model_dict = AssetListAccessControl.from_dict(asset_list_access_control_model_json).__dict__ + asset_list_access_control_model2 = AssetListAccessControl(**asset_list_access_control_model_dict) + + # Verify the model instances are equivalent + assert asset_list_access_control_model == asset_list_access_control_model2 + + # Convert model instance back to dict and verify no loss of data + asset_list_access_control_model_json2 = asset_list_access_control_model.to_dict() + assert asset_list_access_control_model_json2 == asset_list_access_control_model_json + + +class TestModel_AssetPartReference: + """ + Test Class for AssetPartReference + """ + + def test_asset_part_reference_serialization(self): + """ + Test serialization/deserialization for AssetPartReference + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a AssetPartReference model + asset_part_reference_model_json = {} + asset_part_reference_model_json['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model_json['name'] = 'testString' + asset_part_reference_model_json['container'] = container_reference_model + asset_part_reference_model_json['type'] = 'data_asset' + + # Construct a model instance of AssetPartReference by calling from_dict on the json representation + asset_part_reference_model = AssetPartReference.from_dict(asset_part_reference_model_json) + assert asset_part_reference_model != False + + # Construct a model instance of AssetPartReference by calling from_dict on the json representation + asset_part_reference_model_dict = AssetPartReference.from_dict(asset_part_reference_model_json).__dict__ + asset_part_reference_model2 = AssetPartReference(**asset_part_reference_model_dict) + + # Verify the model instances are equivalent + assert asset_part_reference_model == asset_part_reference_model2 + + # Convert model instance back to dict and verify no loss of data + asset_part_reference_model_json2 = asset_part_reference_model.to_dict() + assert asset_part_reference_model_json2 == asset_part_reference_model_json + + +class TestModel_AssetPrototype: + """ + Test Class for AssetPrototype + """ + + def test_asset_prototype_serialization(self): + """ + Test serialization/deserialization for AssetPrototype + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a json representation of a AssetPrototype model + asset_prototype_model_json = {} + asset_prototype_model_json['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_prototype_model_json['container'] = container_identity_model + + # Construct a model instance of AssetPrototype by calling from_dict on the json representation + asset_prototype_model = AssetPrototype.from_dict(asset_prototype_model_json) + assert asset_prototype_model != False + + # Construct a model instance of AssetPrototype by calling from_dict on the json representation + asset_prototype_model_dict = AssetPrototype.from_dict(asset_prototype_model_json).__dict__ + asset_prototype_model2 = AssetPrototype(**asset_prototype_model_dict) + + # Verify the model instances are equivalent + assert asset_prototype_model == asset_prototype_model2 + + # Convert model instance back to dict and verify no loss of data + asset_prototype_model_json2 = asset_prototype_model.to_dict() + assert asset_prototype_model_json2 == asset_prototype_model_json + + +class TestModel_AssetReference: + """ + Test Class for AssetReference + """ + + def test_asset_reference_serialization(self): + """ + Test serialization/deserialization for AssetReference + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a AssetReference model + asset_reference_model_json = {} + asset_reference_model_json['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model_json['name'] = 'testString' + asset_reference_model_json['container'] = container_reference_model + + # Construct a model instance of AssetReference by calling from_dict on the json representation + asset_reference_model = AssetReference.from_dict(asset_reference_model_json) + assert asset_reference_model != False + + # Construct a model instance of AssetReference by calling from_dict on the json representation + asset_reference_model_dict = AssetReference.from_dict(asset_reference_model_json).__dict__ + asset_reference_model2 = AssetReference(**asset_reference_model_dict) + + # Verify the model instances are equivalent + assert asset_reference_model == asset_reference_model2 + + # Convert model instance back to dict and verify no loss of data + asset_reference_model_json2 = asset_reference_model.to_dict() + assert asset_reference_model_json2 == asset_reference_model_json + + +class TestModel_BucketResponse: + """ + Test Class for BucketResponse + """ + + def test_bucket_response_serialization(self): + """ + Test serialization/deserialization for BucketResponse + """ + + # Construct a json representation of a BucketResponse model + bucket_response_model_json = {} + bucket_response_model_json['bucket_name'] = 'testString' + bucket_response_model_json['bucket_location'] = 'testString' + bucket_response_model_json['role_arn'] = 'testString' + bucket_response_model_json['bucket_type'] = 'testString' + bucket_response_model_json['shared'] = True + + # Construct a model instance of BucketResponse by calling from_dict on the json representation + bucket_response_model = BucketResponse.from_dict(bucket_response_model_json) + assert bucket_response_model != False + + # Construct a model instance of BucketResponse by calling from_dict on the json representation + bucket_response_model_dict = BucketResponse.from_dict(bucket_response_model_json).__dict__ + bucket_response_model2 = BucketResponse(**bucket_response_model_dict) + + # Verify the model instances are equivalent + assert bucket_response_model == bucket_response_model2 + + # Convert model instance back to dict and verify no loss of data + bucket_response_model_json2 = bucket_response_model.to_dict() + assert bucket_response_model_json2 == bucket_response_model_json + + +class TestModel_BucketValidationResponse: + """ + Test Class for BucketValidationResponse + """ + + def test_bucket_validation_response_serialization(self): + """ + Test serialization/deserialization for BucketValidationResponse + """ + + # Construct a json representation of a BucketValidationResponse model + bucket_validation_response_model_json = {} + bucket_validation_response_model_json['bucket_exists'] = True + + # Construct a model instance of BucketValidationResponse by calling from_dict on the json representation + bucket_validation_response_model = BucketValidationResponse.from_dict(bucket_validation_response_model_json) + assert bucket_validation_response_model != False + + # Construct a model instance of BucketValidationResponse by calling from_dict on the json representation + bucket_validation_response_model_dict = BucketValidationResponse.from_dict(bucket_validation_response_model_json).__dict__ + bucket_validation_response_model2 = BucketValidationResponse(**bucket_validation_response_model_dict) + + # Verify the model instances are equivalent + assert bucket_validation_response_model == bucket_validation_response_model2 + + # Convert model instance back to dict and verify no loss of data + bucket_validation_response_model_json2 = bucket_validation_response_model.to_dict() + assert bucket_validation_response_model_json2 == bucket_validation_response_model_json + + +class TestModel_ContainerIdentity: + """ + Test Class for ContainerIdentity + """ + + def test_container_identity_serialization(self): + """ + Test serialization/deserialization for ContainerIdentity + """ + + # Construct a json representation of a ContainerIdentity model + container_identity_model_json = {} + container_identity_model_json['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a model instance of ContainerIdentity by calling from_dict on the json representation + container_identity_model = ContainerIdentity.from_dict(container_identity_model_json) + assert container_identity_model != False + + # Construct a model instance of ContainerIdentity by calling from_dict on the json representation + container_identity_model_dict = ContainerIdentity.from_dict(container_identity_model_json).__dict__ + container_identity_model2 = ContainerIdentity(**container_identity_model_dict) + + # Verify the model instances are equivalent + assert container_identity_model == container_identity_model2 + + # Convert model instance back to dict and verify no loss of data + container_identity_model_json2 = container_identity_model.to_dict() + assert container_identity_model_json2 == container_identity_model_json + + +class TestModel_ContainerReference: + """ + Test Class for ContainerReference + """ + + def test_container_reference_serialization(self): + """ + Test serialization/deserialization for ContainerReference + """ + + # Construct a json representation of a ContainerReference model + container_reference_model_json = {} + container_reference_model_json['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model_json['type'] = 'catalog' + + # Construct a model instance of ContainerReference by calling from_dict on the json representation + container_reference_model = ContainerReference.from_dict(container_reference_model_json) + assert container_reference_model != False + + # Construct a model instance of ContainerReference by calling from_dict on the json representation + container_reference_model_dict = ContainerReference.from_dict(container_reference_model_json).__dict__ + container_reference_model2 = ContainerReference(**container_reference_model_dict) + + # Verify the model instances are equivalent + assert container_reference_model == container_reference_model2 + + # Convert model instance back to dict and verify no loss of data + container_reference_model_json2 = container_reference_model.to_dict() + assert container_reference_model_json2 == container_reference_model_json + + +class TestModel_ContractAsset: + """ + Test Class for ContractAsset + """ + + def test_contract_asset_serialization(self): + """ + Test serialization/deserialization for ContractAsset + """ + + # Construct a json representation of a ContractAsset model + contract_asset_model_json = {} + contract_asset_model_json['id'] = 'testString' + contract_asset_model_json['name'] = 'testString' + + # Construct a model instance of ContractAsset by calling from_dict on the json representation + contract_asset_model = ContractAsset.from_dict(contract_asset_model_json) + assert contract_asset_model != False + + # Construct a model instance of ContractAsset by calling from_dict on the json representation + contract_asset_model_dict = ContractAsset.from_dict(contract_asset_model_json).__dict__ + contract_asset_model2 = ContractAsset(**contract_asset_model_dict) + + # Verify the model instances are equivalent + assert contract_asset_model == contract_asset_model2 + + # Convert model instance back to dict and verify no loss of data + contract_asset_model_json2 = contract_asset_model.to_dict() + assert contract_asset_model_json2 == contract_asset_model_json + + +class TestModel_ContractQualityRule: + """ + Test Class for ContractQualityRule + """ + + def test_contract_quality_rule_serialization(self): + """ + Test serialization/deserialization for ContractQualityRule + """ + + # Construct a json representation of a ContractQualityRule model + contract_quality_rule_model_json = {} + contract_quality_rule_model_json['type'] = 'sql' + contract_quality_rule_model_json['description'] = 'testString' + contract_quality_rule_model_json['rule'] = 'testString' + contract_quality_rule_model_json['implementation'] = 'testString' + contract_quality_rule_model_json['engine'] = 'testString' + contract_quality_rule_model_json['must_be_less_than'] = 'testString' + contract_quality_rule_model_json['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model_json['must_be_greater_than'] = 'testString' + contract_quality_rule_model_json['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model_json['must_be_between'] = ['testString'] + contract_quality_rule_model_json['must_not_be_between'] = ['testString'] + contract_quality_rule_model_json['must_be'] = 'testString' + contract_quality_rule_model_json['must_not_be'] = 'testString' + contract_quality_rule_model_json['name'] = 'testString' + contract_quality_rule_model_json['unit'] = 'testString' + contract_quality_rule_model_json['query'] = 'testString' + + # Construct a model instance of ContractQualityRule by calling from_dict on the json representation + contract_quality_rule_model = ContractQualityRule.from_dict(contract_quality_rule_model_json) + assert contract_quality_rule_model != False + + # Construct a model instance of ContractQualityRule by calling from_dict on the json representation + contract_quality_rule_model_dict = ContractQualityRule.from_dict(contract_quality_rule_model_json).__dict__ + contract_quality_rule_model2 = ContractQualityRule(**contract_quality_rule_model_dict) + + # Verify the model instances are equivalent + assert contract_quality_rule_model == contract_quality_rule_model2 + + # Convert model instance back to dict and verify no loss of data + contract_quality_rule_model_json2 = contract_quality_rule_model.to_dict() + assert contract_quality_rule_model_json2 == contract_quality_rule_model_json + + +class TestModel_ContractSchema: + """ + Test Class for ContractSchema + """ + + def test_contract_schema_serialization(self): + """ + Test serialization/deserialization for ContractSchema + """ + + # Construct dict forms of any model objects needed in order to build this model. + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + # Construct a json representation of a ContractSchema model + contract_schema_model_json = {} + contract_schema_model_json['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model_json['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model_json['name'] = 'testString' + contract_schema_model_json['description'] = 'testString' + contract_schema_model_json['connection_path'] = 'testString' + contract_schema_model_json['physical_type'] = 'testString' + contract_schema_model_json['properties'] = [contract_schema_property_model] + contract_schema_model_json['quality'] = [contract_quality_rule_model] + + # Construct a model instance of ContractSchema by calling from_dict on the json representation + contract_schema_model = ContractSchema.from_dict(contract_schema_model_json) + assert contract_schema_model != False + + # Construct a model instance of ContractSchema by calling from_dict on the json representation + contract_schema_model_dict = ContractSchema.from_dict(contract_schema_model_json).__dict__ + contract_schema_model2 = ContractSchema(**contract_schema_model_dict) + + # Verify the model instances are equivalent + assert contract_schema_model == contract_schema_model2 + + # Convert model instance back to dict and verify no loss of data + contract_schema_model_json2 = contract_schema_model.to_dict() + assert contract_schema_model_json2 == contract_schema_model_json + + +class TestModel_ContractSchemaProperty: + """ + Test Class for ContractSchemaProperty + """ + + def test_contract_schema_property_serialization(self): + """ + Test serialization/deserialization for ContractSchemaProperty + """ + + # Construct dict forms of any model objects needed in order to build this model. + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + # Construct a json representation of a ContractSchemaProperty model + contract_schema_property_model_json = {} + contract_schema_property_model_json['name'] = 'testString' + contract_schema_property_model_json['type'] = contract_schema_property_type_model + contract_schema_property_model_json['quality'] = [contract_quality_rule_model] + + # Construct a model instance of ContractSchemaProperty by calling from_dict on the json representation + contract_schema_property_model = ContractSchemaProperty.from_dict(contract_schema_property_model_json) + assert contract_schema_property_model != False + + # Construct a model instance of ContractSchemaProperty by calling from_dict on the json representation + contract_schema_property_model_dict = ContractSchemaProperty.from_dict(contract_schema_property_model_json).__dict__ + contract_schema_property_model2 = ContractSchemaProperty(**contract_schema_property_model_dict) + + # Verify the model instances are equivalent + assert contract_schema_property_model == contract_schema_property_model2 + + # Convert model instance back to dict and verify no loss of data + contract_schema_property_model_json2 = contract_schema_property_model.to_dict() + assert contract_schema_property_model_json2 == contract_schema_property_model_json + + +class TestModel_ContractSchemaPropertyType: + """ + Test Class for ContractSchemaPropertyType + """ + + def test_contract_schema_property_type_serialization(self): + """ + Test serialization/deserialization for ContractSchemaPropertyType + """ + + # Construct a json representation of a ContractSchemaPropertyType model + contract_schema_property_type_model_json = {} + contract_schema_property_type_model_json['type'] = 'testString' + contract_schema_property_type_model_json['length'] = 'testString' + contract_schema_property_type_model_json['scale'] = 'testString' + contract_schema_property_type_model_json['nullable'] = 'testString' + contract_schema_property_type_model_json['signed'] = 'testString' + contract_schema_property_type_model_json['native_type'] = 'testString' + + # Construct a model instance of ContractSchemaPropertyType by calling from_dict on the json representation + contract_schema_property_type_model = ContractSchemaPropertyType.from_dict(contract_schema_property_type_model_json) + assert contract_schema_property_type_model != False + + # Construct a model instance of ContractSchemaPropertyType by calling from_dict on the json representation + contract_schema_property_type_model_dict = ContractSchemaPropertyType.from_dict(contract_schema_property_type_model_json).__dict__ + contract_schema_property_type_model2 = ContractSchemaPropertyType(**contract_schema_property_type_model_dict) + + # Verify the model instances are equivalent + assert contract_schema_property_type_model == contract_schema_property_type_model2 + + # Convert model instance back to dict and verify no loss of data + contract_schema_property_type_model_json2 = contract_schema_property_type_model.to_dict() + assert contract_schema_property_type_model_json2 == contract_schema_property_type_model_json + + +class TestModel_ContractServer: + """ + Test Class for ContractServer + """ + + def test_contract_server_serialization(self): + """ + Test serialization/deserialization for ContractServer + """ + + # Construct dict forms of any model objects needed in order to build this model. + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + # Construct a json representation of a ContractServer model + contract_server_model_json = {} + contract_server_model_json['server'] = 'testString' + contract_server_model_json['asset'] = contract_asset_model + contract_server_model_json['connection_id'] = 'testString' + contract_server_model_json['type'] = 'testString' + contract_server_model_json['description'] = 'testString' + contract_server_model_json['environment'] = 'testString' + contract_server_model_json['account'] = 'testString' + contract_server_model_json['catalog'] = 'testString' + contract_server_model_json['database'] = 'testString' + contract_server_model_json['dataset'] = 'testString' + contract_server_model_json['delimiter'] = 'testString' + contract_server_model_json['endpoint_url'] = 'testString' + contract_server_model_json['format'] = 'testString' + contract_server_model_json['host'] = 'testString' + contract_server_model_json['location'] = 'testString' + contract_server_model_json['path'] = 'testString' + contract_server_model_json['port'] = 'testString' + contract_server_model_json['project'] = 'testString' + contract_server_model_json['region'] = 'testString' + contract_server_model_json['region_name'] = 'testString' + contract_server_model_json['schema'] = 'testString' + contract_server_model_json['service_name'] = 'testString' + contract_server_model_json['staging_dir'] = 'testString' + contract_server_model_json['stream'] = 'testString' + contract_server_model_json['warehouse'] = 'testString' + contract_server_model_json['roles'] = ['testString'] + contract_server_model_json['custom_properties'] = [contract_template_custom_property_model] + + # Construct a model instance of ContractServer by calling from_dict on the json representation + contract_server_model = ContractServer.from_dict(contract_server_model_json) + assert contract_server_model != False + + # Construct a model instance of ContractServer by calling from_dict on the json representation + contract_server_model_dict = ContractServer.from_dict(contract_server_model_json).__dict__ + contract_server_model2 = ContractServer(**contract_server_model_dict) + + # Verify the model instances are equivalent + assert contract_server_model == contract_server_model2 + + # Convert model instance back to dict and verify no loss of data + contract_server_model_json2 = contract_server_model.to_dict() + assert contract_server_model_json2 == contract_server_model_json + + +class TestModel_ContractTemplateCustomProperty: + """ + Test Class for ContractTemplateCustomProperty + """ + + def test_contract_template_custom_property_serialization(self): + """ + Test serialization/deserialization for ContractTemplateCustomProperty + """ + + # Construct a json representation of a ContractTemplateCustomProperty model + contract_template_custom_property_model_json = {} + contract_template_custom_property_model_json['key'] = 'customPropertyKey' + contract_template_custom_property_model_json['value'] = 'customPropertyValue' + + # Construct a model instance of ContractTemplateCustomProperty by calling from_dict on the json representation + contract_template_custom_property_model = ContractTemplateCustomProperty.from_dict(contract_template_custom_property_model_json) + assert contract_template_custom_property_model != False + + # Construct a model instance of ContractTemplateCustomProperty by calling from_dict on the json representation + contract_template_custom_property_model_dict = ContractTemplateCustomProperty.from_dict(contract_template_custom_property_model_json).__dict__ + contract_template_custom_property_model2 = ContractTemplateCustomProperty(**contract_template_custom_property_model_dict) + + # Verify the model instances are equivalent + assert contract_template_custom_property_model == contract_template_custom_property_model2 + + # Convert model instance back to dict and verify no loss of data + contract_template_custom_property_model_json2 = contract_template_custom_property_model.to_dict() + assert contract_template_custom_property_model_json2 == contract_template_custom_property_model_json + + +class TestModel_ContractTemplateOrganization: + """ + Test Class for ContractTemplateOrganization + """ + + def test_contract_template_organization_serialization(self): + """ + Test serialization/deserialization for ContractTemplateOrganization + """ + + # Construct a json representation of a ContractTemplateOrganization model + contract_template_organization_model_json = {} + contract_template_organization_model_json['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model_json['role'] = 'owner' + + # Construct a model instance of ContractTemplateOrganization by calling from_dict on the json representation + contract_template_organization_model = ContractTemplateOrganization.from_dict(contract_template_organization_model_json) + assert contract_template_organization_model != False + + # Construct a model instance of ContractTemplateOrganization by calling from_dict on the json representation + contract_template_organization_model_dict = ContractTemplateOrganization.from_dict(contract_template_organization_model_json).__dict__ + contract_template_organization_model2 = ContractTemplateOrganization(**contract_template_organization_model_dict) + + # Verify the model instances are equivalent + assert contract_template_organization_model == contract_template_organization_model2 + + # Convert model instance back to dict and verify no loss of data + contract_template_organization_model_json2 = contract_template_organization_model.to_dict() + assert contract_template_organization_model_json2 == contract_template_organization_model_json + + +class TestModel_ContractTemplateSLA: + """ + Test Class for ContractTemplateSLA + """ + + def test_contract_template_sla_serialization(self): + """ + Test serialization/deserialization for ContractTemplateSLA + """ + + # Construct dict forms of any model objects needed in order to build this model. + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + # Construct a json representation of a ContractTemplateSLA model + contract_template_sla_model_json = {} + contract_template_sla_model_json['default_element'] = 'Standard SLA Policy' + contract_template_sla_model_json['properties'] = [contract_template_sla_property_model] + + # Construct a model instance of ContractTemplateSLA by calling from_dict on the json representation + contract_template_sla_model = ContractTemplateSLA.from_dict(contract_template_sla_model_json) + assert contract_template_sla_model != False + + # Construct a model instance of ContractTemplateSLA by calling from_dict on the json representation + contract_template_sla_model_dict = ContractTemplateSLA.from_dict(contract_template_sla_model_json).__dict__ + contract_template_sla_model2 = ContractTemplateSLA(**contract_template_sla_model_dict) + + # Verify the model instances are equivalent + assert contract_template_sla_model == contract_template_sla_model2 + + # Convert model instance back to dict and verify no loss of data + contract_template_sla_model_json2 = contract_template_sla_model.to_dict() + assert contract_template_sla_model_json2 == contract_template_sla_model_json + + +class TestModel_ContractTemplateSLAProperty: + """ + Test Class for ContractTemplateSLAProperty + """ + + def test_contract_template_sla_property_serialization(self): + """ + Test serialization/deserialization for ContractTemplateSLAProperty + """ + + # Construct a json representation of a ContractTemplateSLAProperty model + contract_template_sla_property_model_json = {} + contract_template_sla_property_model_json['property'] = 'Uptime Guarantee' + contract_template_sla_property_model_json['value'] = '99.9' + + # Construct a model instance of ContractTemplateSLAProperty by calling from_dict on the json representation + contract_template_sla_property_model = ContractTemplateSLAProperty.from_dict(contract_template_sla_property_model_json) + assert contract_template_sla_property_model != False + + # Construct a model instance of ContractTemplateSLAProperty by calling from_dict on the json representation + contract_template_sla_property_model_dict = ContractTemplateSLAProperty.from_dict(contract_template_sla_property_model_json).__dict__ + contract_template_sla_property_model2 = ContractTemplateSLAProperty(**contract_template_sla_property_model_dict) + + # Verify the model instances are equivalent + assert contract_template_sla_property_model == contract_template_sla_property_model2 + + # Convert model instance back to dict and verify no loss of data + contract_template_sla_property_model_json2 = contract_template_sla_property_model.to_dict() + assert contract_template_sla_property_model_json2 == contract_template_sla_property_model_json + + +class TestModel_ContractTemplateSupportAndCommunication: + """ + Test Class for ContractTemplateSupportAndCommunication + """ + + def test_contract_template_support_and_communication_serialization(self): + """ + Test serialization/deserialization for ContractTemplateSupportAndCommunication + """ + + # Construct a json representation of a ContractTemplateSupportAndCommunication model + contract_template_support_and_communication_model_json = {} + contract_template_support_and_communication_model_json['channel'] = 'Email Support' + contract_template_support_and_communication_model_json['url'] = 'https://support.example.com' + + # Construct a model instance of ContractTemplateSupportAndCommunication by calling from_dict on the json representation + contract_template_support_and_communication_model = ContractTemplateSupportAndCommunication.from_dict(contract_template_support_and_communication_model_json) + assert contract_template_support_and_communication_model != False + + # Construct a model instance of ContractTemplateSupportAndCommunication by calling from_dict on the json representation + contract_template_support_and_communication_model_dict = ContractTemplateSupportAndCommunication.from_dict(contract_template_support_and_communication_model_json).__dict__ + contract_template_support_and_communication_model2 = ContractTemplateSupportAndCommunication(**contract_template_support_and_communication_model_dict) + + # Verify the model instances are equivalent + assert contract_template_support_and_communication_model == contract_template_support_and_communication_model2 + + # Convert model instance back to dict and verify no loss of data + contract_template_support_and_communication_model_json2 = contract_template_support_and_communication_model.to_dict() + assert contract_template_support_and_communication_model_json2 == contract_template_support_and_communication_model_json + + +class TestModel_ContractTerms: + """ + Test Class for ContractTerms + """ + + def test_contract_terms_serialization(self): + """ + Test serialization/deserialization for ContractTerms + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + # Construct a json representation of a ContractTerms model + contract_terms_model_json = {} + contract_terms_model_json['asset'] = asset_reference_model + contract_terms_model_json['id'] = 'testString' + contract_terms_model_json['documents'] = [contract_terms_document_model] + contract_terms_model_json['error_msg'] = 'testString' + contract_terms_model_json['overview'] = overview_model + contract_terms_model_json['description'] = description_model + contract_terms_model_json['organization'] = [contract_template_organization_model] + contract_terms_model_json['roles'] = [roles_model] + contract_terms_model_json['price'] = pricing_model + contract_terms_model_json['sla'] = [contract_template_sla_model] + contract_terms_model_json['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model_json['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model_json['contract_test'] = contract_test_model + contract_terms_model_json['servers'] = [contract_server_model] + contract_terms_model_json['schema'] = [contract_schema_model] + + # Construct a model instance of ContractTerms by calling from_dict on the json representation + contract_terms_model = ContractTerms.from_dict(contract_terms_model_json) + assert contract_terms_model != False + + # Construct a model instance of ContractTerms by calling from_dict on the json representation + contract_terms_model_dict = ContractTerms.from_dict(contract_terms_model_json).__dict__ + contract_terms_model2 = ContractTerms(**contract_terms_model_dict) + + # Verify the model instances are equivalent + assert contract_terms_model == contract_terms_model2 + + # Convert model instance back to dict and verify no loss of data + contract_terms_model_json2 = contract_terms_model.to_dict() + assert contract_terms_model_json2 == contract_terms_model_json + + +class TestModel_ContractTermsDocument: + """ + Test Class for ContractTermsDocument + """ + + def test_contract_terms_document_serialization(self): + """ + Test serialization/deserialization for ContractTermsDocument + """ + + # Construct dict forms of any model objects needed in order to build this model. + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + # Construct a json representation of a ContractTermsDocument model + contract_terms_document_model_json = {} + contract_terms_document_model_json['url'] = 'testString' + contract_terms_document_model_json['type'] = 'terms_and_conditions' + contract_terms_document_model_json['name'] = 'testString' + contract_terms_document_model_json['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model_json['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model_json['upload_url'] = 'testString' + + # Construct a model instance of ContractTermsDocument by calling from_dict on the json representation + contract_terms_document_model = ContractTermsDocument.from_dict(contract_terms_document_model_json) + assert contract_terms_document_model != False + + # Construct a model instance of ContractTermsDocument by calling from_dict on the json representation + contract_terms_document_model_dict = ContractTermsDocument.from_dict(contract_terms_document_model_json).__dict__ + contract_terms_document_model2 = ContractTermsDocument(**contract_terms_document_model_dict) + + # Verify the model instances are equivalent + assert contract_terms_document_model == contract_terms_document_model2 + + # Convert model instance back to dict and verify no loss of data + contract_terms_document_model_json2 = contract_terms_document_model.to_dict() + assert contract_terms_document_model_json2 == contract_terms_document_model_json + + +class TestModel_ContractTermsDocumentAttachment: + """ + Test Class for ContractTermsDocumentAttachment + """ + + def test_contract_terms_document_attachment_serialization(self): + """ + Test serialization/deserialization for ContractTermsDocumentAttachment + """ + + # Construct a json representation of a ContractTermsDocumentAttachment model + contract_terms_document_attachment_model_json = {} + contract_terms_document_attachment_model_json['id'] = 'testString' + + # Construct a model instance of ContractTermsDocumentAttachment by calling from_dict on the json representation + contract_terms_document_attachment_model = ContractTermsDocumentAttachment.from_dict(contract_terms_document_attachment_model_json) + assert contract_terms_document_attachment_model != False + + # Construct a model instance of ContractTermsDocumentAttachment by calling from_dict on the json representation + contract_terms_document_attachment_model_dict = ContractTermsDocumentAttachment.from_dict(contract_terms_document_attachment_model_json).__dict__ + contract_terms_document_attachment_model2 = ContractTermsDocumentAttachment(**contract_terms_document_attachment_model_dict) + + # Verify the model instances are equivalent + assert contract_terms_document_attachment_model == contract_terms_document_attachment_model2 + + # Convert model instance back to dict and verify no loss of data + contract_terms_document_attachment_model_json2 = contract_terms_document_attachment_model.to_dict() + assert contract_terms_document_attachment_model_json2 == contract_terms_document_attachment_model_json + + +class TestModel_ContractTermsMoreInfo: + """ + Test Class for ContractTermsMoreInfo + """ + + def test_contract_terms_more_info_serialization(self): + """ + Test serialization/deserialization for ContractTermsMoreInfo + """ + + # Construct a json representation of a ContractTermsMoreInfo model + contract_terms_more_info_model_json = {} + contract_terms_more_info_model_json['type'] = 'privacy-statement' + contract_terms_more_info_model_json['url'] = 'https://moreinfo.example.com' + + # Construct a model instance of ContractTermsMoreInfo by calling from_dict on the json representation + contract_terms_more_info_model = ContractTermsMoreInfo.from_dict(contract_terms_more_info_model_json) + assert contract_terms_more_info_model != False + + # Construct a model instance of ContractTermsMoreInfo by calling from_dict on the json representation + contract_terms_more_info_model_dict = ContractTermsMoreInfo.from_dict(contract_terms_more_info_model_json).__dict__ + contract_terms_more_info_model2 = ContractTermsMoreInfo(**contract_terms_more_info_model_dict) + + # Verify the model instances are equivalent + assert contract_terms_more_info_model == contract_terms_more_info_model2 + + # Convert model instance back to dict and verify no loss of data + contract_terms_more_info_model_json2 = contract_terms_more_info_model.to_dict() + assert contract_terms_more_info_model_json2 == contract_terms_more_info_model_json + + +class TestModel_ContractTest: + """ + Test Class for ContractTest + """ + + def test_contract_test_serialization(self): + """ + Test serialization/deserialization for ContractTest + """ + + # Construct a json representation of a ContractTest model + contract_test_model_json = {} + contract_test_model_json['status'] = 'pass' + contract_test_model_json['last_tested_time'] = 'testString' + contract_test_model_json['message'] = 'testString' + + # Construct a model instance of ContractTest by calling from_dict on the json representation + contract_test_model = ContractTest.from_dict(contract_test_model_json) + assert contract_test_model != False + + # Construct a model instance of ContractTest by calling from_dict on the json representation + contract_test_model_dict = ContractTest.from_dict(contract_test_model_json).__dict__ + contract_test_model2 = ContractTest(**contract_test_model_dict) + + # Verify the model instances are equivalent + assert contract_test_model == contract_test_model2 + + # Convert model instance back to dict and verify no loss of data + contract_test_model_json2 = contract_test_model.to_dict() + assert contract_test_model_json2 == contract_test_model_json + + +class TestModel_DataAssetRelationship: + """ + Test Class for DataAssetRelationship + """ + + def test_data_asset_relationship_serialization(self): + """ + Test serialization/deserialization for DataAssetRelationship + """ + + # Construct dict forms of any model objects needed in order to build this model. + + visualization_model = {} # Visualization + visualization_model['id'] = 'testString' + visualization_model['name'] = 'testString' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + error_message_model = {} # ErrorMessage + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + # Construct a json representation of a DataAssetRelationship model + data_asset_relationship_model_json = {} + data_asset_relationship_model_json['visualization'] = visualization_model + data_asset_relationship_model_json['asset'] = asset_reference_model + data_asset_relationship_model_json['related_asset'] = asset_reference_model + data_asset_relationship_model_json['error'] = error_message_model + + # Construct a model instance of DataAssetRelationship by calling from_dict on the json representation + data_asset_relationship_model = DataAssetRelationship.from_dict(data_asset_relationship_model_json) + assert data_asset_relationship_model != False + + # Construct a model instance of DataAssetRelationship by calling from_dict on the json representation + data_asset_relationship_model_dict = DataAssetRelationship.from_dict(data_asset_relationship_model_json).__dict__ + data_asset_relationship_model2 = DataAssetRelationship(**data_asset_relationship_model_dict) + + # Verify the model instances are equivalent + assert data_asset_relationship_model == data_asset_relationship_model2 + + # Convert model instance back to dict and verify no loss of data + data_asset_relationship_model_json2 = data_asset_relationship_model.to_dict() + assert data_asset_relationship_model_json2 == data_asset_relationship_model_json + + +class TestModel_DataAssetVisualizationRes: + """ + Test Class for DataAssetVisualizationRes + """ + + def test_data_asset_visualization_res_serialization(self): + """ + Test serialization/deserialization for DataAssetVisualizationRes + """ + + # Construct dict forms of any model objects needed in order to build this model. + + visualization_model = {} # Visualization + visualization_model['id'] = 'testString' + visualization_model['name'] = 'testString' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + error_message_model = {} # ErrorMessage + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + data_asset_relationship_model = {} # DataAssetRelationship + data_asset_relationship_model['visualization'] = visualization_model + data_asset_relationship_model['asset'] = asset_reference_model + data_asset_relationship_model['related_asset'] = asset_reference_model + data_asset_relationship_model['error'] = error_message_model + + # Construct a json representation of a DataAssetVisualizationRes model + data_asset_visualization_res_model_json = {} + data_asset_visualization_res_model_json['results'] = [data_asset_relationship_model] + + # Construct a model instance of DataAssetVisualizationRes by calling from_dict on the json representation + data_asset_visualization_res_model = DataAssetVisualizationRes.from_dict(data_asset_visualization_res_model_json) + assert data_asset_visualization_res_model != False + + # Construct a model instance of DataAssetVisualizationRes by calling from_dict on the json representation + data_asset_visualization_res_model_dict = DataAssetVisualizationRes.from_dict(data_asset_visualization_res_model_json).__dict__ + data_asset_visualization_res_model2 = DataAssetVisualizationRes(**data_asset_visualization_res_model_dict) + + # Verify the model instances are equivalent + assert data_asset_visualization_res_model == data_asset_visualization_res_model2 + + # Convert model instance back to dict and verify no loss of data + data_asset_visualization_res_model_json2 = data_asset_visualization_res_model.to_dict() + assert data_asset_visualization_res_model_json2 == data_asset_visualization_res_model_json + + +class TestModel_DataProduct: + """ + Test Class for DataProduct + """ + + def test_data_product_serialization(self): + """ + Test serialization/deserialization for DataProduct + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_version_summary_data_product_model = {} # DataProductVersionSummaryDataProduct + data_product_version_summary_data_product_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_version_summary_data_product_model['release'] = data_product_draft_version_release_model + data_product_version_summary_data_product_model['container'] = container_reference_model + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + data_product_version_summary_model = {} # DataProductVersionSummary + data_product_version_summary_model['version'] = '1.0.0' + data_product_version_summary_model['state'] = 'draft' + data_product_version_summary_model['data_product'] = data_product_version_summary_data_product_model + data_product_version_summary_model['name'] = 'My Data Product' + data_product_version_summary_model['description'] = 'This is a description of My Data Product.' + data_product_version_summary_model['tags'] = ['testString'] + data_product_version_summary_model['use_cases'] = [use_case_model] + data_product_version_summary_model['types'] = ['data'] + data_product_version_summary_model['contract_terms'] = [contract_terms_model] + data_product_version_summary_model['domain'] = domain_model + data_product_version_summary_model['parts_out'] = [data_product_part_model] + data_product_version_summary_model['workflows'] = data_product_workflows_model + data_product_version_summary_model['dataview_enabled'] = True + data_product_version_summary_model['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_version_summary_model['access_control'] = asset_list_access_control_model + data_product_version_summary_model['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_version_summary_model['sub_container'] = container_identity_model + data_product_version_summary_model['is_restricted'] = True + data_product_version_summary_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd' + data_product_version_summary_model['asset'] = asset_reference_model + + # Construct a json representation of a DataProduct model + data_product_model_json = {} + data_product_model_json['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_model_json['release'] = data_product_draft_version_release_model + data_product_model_json['container'] = container_reference_model + data_product_model_json['name'] = 'testString' + data_product_model_json['latest_release'] = data_product_version_summary_model + data_product_model_json['drafts'] = [data_product_version_summary_model] + + # Construct a model instance of DataProduct by calling from_dict on the json representation + data_product_model = DataProduct.from_dict(data_product_model_json) + assert data_product_model != False + + # Construct a model instance of DataProduct by calling from_dict on the json representation + data_product_model_dict = DataProduct.from_dict(data_product_model_json).__dict__ + data_product_model2 = DataProduct(**data_product_model_dict) + + # Verify the model instances are equivalent + assert data_product_model == data_product_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_model_json2 = data_product_model.to_dict() + assert data_product_model_json2 == data_product_model_json + + +class TestModel_DataProductCollection: + """ + Test Class for DataProductCollection + """ + + def test_data_product_collection_serialization(self): + """ + Test serialization/deserialization for DataProductCollection + """ + + # Construct dict forms of any model objects needed in order to build this model. + + first_page_model = {} # FirstPage + first_page_model['href'] = 'https://api.example.com/collection' + + next_page_model = {} # NextPage + next_page_model['href'] = 'https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + next_page_model['start'] = 'eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_summary_model = {} # DataProductSummary + data_product_summary_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_summary_model['release'] = data_product_draft_version_release_model + data_product_summary_model['container'] = container_reference_model + data_product_summary_model['name'] = 'testString' + + # Construct a json representation of a DataProductCollection model + data_product_collection_model_json = {} + data_product_collection_model_json['limit'] = 200 + data_product_collection_model_json['first'] = first_page_model + data_product_collection_model_json['next'] = next_page_model + data_product_collection_model_json['total_results'] = 200 + data_product_collection_model_json['data_products'] = [data_product_summary_model] + + # Construct a model instance of DataProductCollection by calling from_dict on the json representation + data_product_collection_model = DataProductCollection.from_dict(data_product_collection_model_json) + assert data_product_collection_model != False + + # Construct a model instance of DataProductCollection by calling from_dict on the json representation + data_product_collection_model_dict = DataProductCollection.from_dict(data_product_collection_model_json).__dict__ + data_product_collection_model2 = DataProductCollection(**data_product_collection_model_dict) + + # Verify the model instances are equivalent + assert data_product_collection_model == data_product_collection_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_collection_model_json2 = data_product_collection_model.to_dict() + assert data_product_collection_model_json2 == data_product_collection_model_json + + +class TestModel_DataProductContractTemplate: + """ + Test Class for DataProductContractTemplate + """ + + def test_data_product_contract_template_serialization(self): + """ + Test serialization/deserialization for DataProductContractTemplate + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + error_message_model = {} # ErrorMessage + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + # Construct a json representation of a DataProductContractTemplate model + data_product_contract_template_model_json = {} + data_product_contract_template_model_json['container'] = container_reference_model + data_product_contract_template_model_json['id'] = '20aa7c97-cfcc-4d16-ae76-2ca1847ce733' + data_product_contract_template_model_json['creator_id'] = 'IBMid-123456ABC' + data_product_contract_template_model_json['created_at'] = '2025-06-26T12:30:20.000Z' + data_product_contract_template_model_json['name'] = 'Sample Data Contract Template' + data_product_contract_template_model_json['error'] = error_message_model + data_product_contract_template_model_json['contract_terms'] = contract_terms_model + + # Construct a model instance of DataProductContractTemplate by calling from_dict on the json representation + data_product_contract_template_model = DataProductContractTemplate.from_dict(data_product_contract_template_model_json) + assert data_product_contract_template_model != False + + # Construct a model instance of DataProductContractTemplate by calling from_dict on the json representation + data_product_contract_template_model_dict = DataProductContractTemplate.from_dict(data_product_contract_template_model_json).__dict__ + data_product_contract_template_model2 = DataProductContractTemplate(**data_product_contract_template_model_dict) + + # Verify the model instances are equivalent + assert data_product_contract_template_model == data_product_contract_template_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_contract_template_model_json2 = data_product_contract_template_model.to_dict() + assert data_product_contract_template_model_json2 == data_product_contract_template_model_json + + +class TestModel_DataProductContractTemplateCollection: + """ + Test Class for DataProductContractTemplateCollection + """ + + def test_data_product_contract_template_collection_serialization(self): + """ + Test serialization/deserialization for DataProductContractTemplateCollection + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + error_message_model = {} # ErrorMessage + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + data_product_contract_template_model = {} # DataProductContractTemplate + data_product_contract_template_model['container'] = container_reference_model + data_product_contract_template_model['id'] = '20aa7c97-cfcc-4d16-ae76-2ca1847ce733' + data_product_contract_template_model['creator_id'] = 'IBMid-123456ABC' + data_product_contract_template_model['created_at'] = '2025-06-26T12:30:20.000Z' + data_product_contract_template_model['name'] = 'Sample Data Contract Template' + data_product_contract_template_model['error'] = error_message_model + data_product_contract_template_model['contract_terms'] = contract_terms_model + + # Construct a json representation of a DataProductContractTemplateCollection model + data_product_contract_template_collection_model_json = {} + data_product_contract_template_collection_model_json['contract_templates'] = [data_product_contract_template_model] + + # Construct a model instance of DataProductContractTemplateCollection by calling from_dict on the json representation + data_product_contract_template_collection_model = DataProductContractTemplateCollection.from_dict(data_product_contract_template_collection_model_json) + assert data_product_contract_template_collection_model != False + + # Construct a model instance of DataProductContractTemplateCollection by calling from_dict on the json representation + data_product_contract_template_collection_model_dict = DataProductContractTemplateCollection.from_dict(data_product_contract_template_collection_model_json).__dict__ + data_product_contract_template_collection_model2 = DataProductContractTemplateCollection(**data_product_contract_template_collection_model_dict) + + # Verify the model instances are equivalent + assert data_product_contract_template_collection_model == data_product_contract_template_collection_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_contract_template_collection_model_json2 = data_product_contract_template_collection_model.to_dict() + assert data_product_contract_template_collection_model_json2 == data_product_contract_template_collection_model_json + + +class TestModel_DataProductCustomWorkflowDefinition: + """ + Test Class for DataProductCustomWorkflowDefinition + """ + + def test_data_product_custom_workflow_definition_serialization(self): + """ + Test serialization/deserialization for DataProductCustomWorkflowDefinition + """ + + # Construct a json representation of a DataProductCustomWorkflowDefinition model + data_product_custom_workflow_definition_model_json = {} + data_product_custom_workflow_definition_model_json['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a model instance of DataProductCustomWorkflowDefinition by calling from_dict on the json representation + data_product_custom_workflow_definition_model = DataProductCustomWorkflowDefinition.from_dict(data_product_custom_workflow_definition_model_json) + assert data_product_custom_workflow_definition_model != False + + # Construct a model instance of DataProductCustomWorkflowDefinition by calling from_dict on the json representation + data_product_custom_workflow_definition_model_dict = DataProductCustomWorkflowDefinition.from_dict(data_product_custom_workflow_definition_model_json).__dict__ + data_product_custom_workflow_definition_model2 = DataProductCustomWorkflowDefinition(**data_product_custom_workflow_definition_model_dict) + + # Verify the model instances are equivalent + assert data_product_custom_workflow_definition_model == data_product_custom_workflow_definition_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_custom_workflow_definition_model_json2 = data_product_custom_workflow_definition_model.to_dict() + assert data_product_custom_workflow_definition_model_json2 == data_product_custom_workflow_definition_model_json + + +class TestModel_DataProductDomain: + """ + Test Class for DataProductDomain + """ + + def test_data_product_domain_serialization(self): + """ + Test serialization/deserialization for DataProductDomain + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + error_extra_resource_model = {} # ErrorExtraResource + error_extra_resource_model['id'] = 'testString' + error_extra_resource_model['timestamp'] = '2019-01-01T12:00:00Z' + error_extra_resource_model['environment_name'] = 'testString' + error_extra_resource_model['http_status'] = 0 + error_extra_resource_model['source_cluster'] = 0 + error_extra_resource_model['source_component'] = 0 + error_extra_resource_model['transaction_id'] = 0 + + error_model_resource_model = {} # ErrorModelResource + error_model_resource_model['code'] = 'request_body_error' + error_model_resource_model['message'] = 'testString' + error_model_resource_model['extra'] = error_extra_resource_model + error_model_resource_model['more_info'] = 'testString' + + member_roles_schema_model = {} # MemberRolesSchema + member_roles_schema_model['user_iam_id'] = 'testString' + member_roles_schema_model['roles'] = ['testString'] + + properties_schema_model = {} # PropertiesSchema + properties_schema_model['value'] = 'testString' + + initialize_sub_domain_model = {} # InitializeSubDomain + initialize_sub_domain_model['name'] = 'Operations' + initialize_sub_domain_model['id'] = 'testString' + initialize_sub_domain_model['description'] = 'testString' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a json representation of a DataProductDomain model + data_product_domain_model_json = {} + data_product_domain_model_json['container'] = container_reference_model + data_product_domain_model_json['trace'] = 'testString' + data_product_domain_model_json['errors'] = [error_model_resource_model] + data_product_domain_model_json['name'] = 'Operations' + data_product_domain_model_json['description'] = 'This is a description of the data product domain.' + data_product_domain_model_json['id'] = 'testString' + data_product_domain_model_json['created_by'] = 'testString' + data_product_domain_model_json['member_roles'] = member_roles_schema_model + data_product_domain_model_json['properties'] = properties_schema_model + data_product_domain_model_json['sub_domains'] = [initialize_sub_domain_model] + data_product_domain_model_json['sub_container'] = container_identity_model + + # Construct a model instance of DataProductDomain by calling from_dict on the json representation + data_product_domain_model = DataProductDomain.from_dict(data_product_domain_model_json) + assert data_product_domain_model != False + + # Construct a model instance of DataProductDomain by calling from_dict on the json representation + data_product_domain_model_dict = DataProductDomain.from_dict(data_product_domain_model_json).__dict__ + data_product_domain_model2 = DataProductDomain(**data_product_domain_model_dict) + + # Verify the model instances are equivalent + assert data_product_domain_model == data_product_domain_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_domain_model_json2 = data_product_domain_model.to_dict() + assert data_product_domain_model_json2 == data_product_domain_model_json + + +class TestModel_DataProductDomainCollection: + """ + Test Class for DataProductDomainCollection + """ + + def test_data_product_domain_collection_serialization(self): + """ + Test serialization/deserialization for DataProductDomainCollection + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + error_extra_resource_model = {} # ErrorExtraResource + error_extra_resource_model['id'] = 'testString' + error_extra_resource_model['timestamp'] = '2019-01-01T12:00:00Z' + error_extra_resource_model['environment_name'] = 'testString' + error_extra_resource_model['http_status'] = 0 + error_extra_resource_model['source_cluster'] = 0 + error_extra_resource_model['source_component'] = 0 + error_extra_resource_model['transaction_id'] = 0 + + error_model_resource_model = {} # ErrorModelResource + error_model_resource_model['code'] = 'request_body_error' + error_model_resource_model['message'] = 'testString' + error_model_resource_model['extra'] = error_extra_resource_model + error_model_resource_model['more_info'] = 'testString' + + member_roles_schema_model = {} # MemberRolesSchema + member_roles_schema_model['user_iam_id'] = 'testString' + member_roles_schema_model['roles'] = ['testString'] + + properties_schema_model = {} # PropertiesSchema + properties_schema_model['value'] = 'testString' + + initialize_sub_domain_model = {} # InitializeSubDomain + initialize_sub_domain_model['name'] = 'Operations' + initialize_sub_domain_model['id'] = 'testString' + initialize_sub_domain_model['description'] = 'testString' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + data_product_domain_model = {} # DataProductDomain + data_product_domain_model['container'] = container_reference_model + data_product_domain_model['trace'] = 'testString' + data_product_domain_model['errors'] = [error_model_resource_model] + data_product_domain_model['name'] = 'Operations' + data_product_domain_model['description'] = 'This is a description of the data product domain.' + data_product_domain_model['id'] = 'testString' + data_product_domain_model['created_by'] = 'testString' + data_product_domain_model['member_roles'] = member_roles_schema_model + data_product_domain_model['properties'] = properties_schema_model + data_product_domain_model['sub_domains'] = [initialize_sub_domain_model] + data_product_domain_model['sub_container'] = container_identity_model + + # Construct a json representation of a DataProductDomainCollection model + data_product_domain_collection_model_json = {} + data_product_domain_collection_model_json['domains'] = [data_product_domain_model] + + # Construct a model instance of DataProductDomainCollection by calling from_dict on the json representation + data_product_domain_collection_model = DataProductDomainCollection.from_dict(data_product_domain_collection_model_json) + assert data_product_domain_collection_model != False + + # Construct a model instance of DataProductDomainCollection by calling from_dict on the json representation + data_product_domain_collection_model_dict = DataProductDomainCollection.from_dict(data_product_domain_collection_model_json).__dict__ + data_product_domain_collection_model2 = DataProductDomainCollection(**data_product_domain_collection_model_dict) + + # Verify the model instances are equivalent + assert data_product_domain_collection_model == data_product_domain_collection_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_domain_collection_model_json2 = data_product_domain_collection_model.to_dict() + assert data_product_domain_collection_model_json2 == data_product_domain_collection_model_json + + +class TestModel_DataProductDraft: + """ + Test Class for DataProductDraft + """ + + def test_data_product_draft_serialization(self): + """ + Test serialization/deserialization for DataProductDraft + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_draft_data_product_model = {} # DataProductDraftDataProduct + data_product_draft_data_product_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_draft_data_product_model['release'] = data_product_draft_version_release_model + data_product_draft_data_product_model['container'] = container_reference_model + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + visualization_model = {} # Visualization + visualization_model['id'] = 'testString' + visualization_model['name'] = 'testString' + + error_message_model = {} # ErrorMessage + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + data_asset_relationship_model = {} # DataAssetRelationship + data_asset_relationship_model['visualization'] = visualization_model + data_asset_relationship_model['asset'] = asset_reference_model + data_asset_relationship_model['related_asset'] = asset_reference_model + data_asset_relationship_model['error'] = error_message_model + + # Construct a json representation of a DataProductDraft model + data_product_draft_model_json = {} + data_product_draft_model_json['version'] = '1.0.0' + data_product_draft_model_json['state'] = 'draft' + data_product_draft_model_json['data_product'] = data_product_draft_data_product_model + data_product_draft_model_json['name'] = 'My Data Product' + data_product_draft_model_json['description'] = 'This is a description of My Data Product.' + data_product_draft_model_json['tags'] = ['testString'] + data_product_draft_model_json['use_cases'] = [use_case_model] + data_product_draft_model_json['types'] = ['data'] + data_product_draft_model_json['contract_terms'] = [contract_terms_model] + data_product_draft_model_json['domain'] = domain_model + data_product_draft_model_json['parts_out'] = [data_product_part_model] + data_product_draft_model_json['workflows'] = data_product_workflows_model + data_product_draft_model_json['dataview_enabled'] = True + data_product_draft_model_json['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_draft_model_json['access_control'] = asset_list_access_control_model + data_product_draft_model_json['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_draft_model_json['sub_container'] = container_identity_model + data_product_draft_model_json['is_restricted'] = True + data_product_draft_model_json['id'] = '2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd' + data_product_draft_model_json['asset'] = asset_reference_model + data_product_draft_model_json['published_by'] = 'testString' + data_product_draft_model_json['published_at'] = '2019-01-01T12:00:00Z' + data_product_draft_model_json['created_by'] = 'testString' + data_product_draft_model_json['created_at'] = '2019-01-01T12:00:00Z' + data_product_draft_model_json['properties'] = {'anyKey': 'anyValue'} + data_product_draft_model_json['visualization_errors'] = [data_asset_relationship_model] + + # Construct a model instance of DataProductDraft by calling from_dict on the json representation + data_product_draft_model = DataProductDraft.from_dict(data_product_draft_model_json) + assert data_product_draft_model != False + + # Construct a model instance of DataProductDraft by calling from_dict on the json representation + data_product_draft_model_dict = DataProductDraft.from_dict(data_product_draft_model_json).__dict__ + data_product_draft_model2 = DataProductDraft(**data_product_draft_model_dict) + + # Verify the model instances are equivalent + assert data_product_draft_model == data_product_draft_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_draft_model_json2 = data_product_draft_model.to_dict() + assert data_product_draft_model_json2 == data_product_draft_model_json + + +class TestModel_DataProductDraftCollection: + """ + Test Class for DataProductDraftCollection + """ + + def test_data_product_draft_collection_serialization(self): + """ + Test serialization/deserialization for DataProductDraftCollection + """ + + # Construct dict forms of any model objects needed in order to build this model. + + first_page_model = {} # FirstPage + first_page_model['href'] = 'https://api.example.com/collection' + + next_page_model = {} # NextPage + next_page_model['href'] = 'https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + next_page_model['start'] = 'eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_draft_summary_data_product_model = {} # DataProductDraftSummaryDataProduct + data_product_draft_summary_data_product_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_draft_summary_data_product_model['release'] = data_product_draft_version_release_model + data_product_draft_summary_data_product_model['container'] = container_reference_model + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + data_product_draft_summary_model = {} # DataProductDraftSummary + data_product_draft_summary_model['version'] = '1.0.0' + data_product_draft_summary_model['state'] = 'draft' + data_product_draft_summary_model['data_product'] = data_product_draft_summary_data_product_model + data_product_draft_summary_model['name'] = 'My Data Product' + data_product_draft_summary_model['description'] = 'This is a description of My Data Product.' + data_product_draft_summary_model['tags'] = ['testString'] + data_product_draft_summary_model['use_cases'] = [use_case_model] + data_product_draft_summary_model['types'] = ['data'] + data_product_draft_summary_model['contract_terms'] = [contract_terms_model] + data_product_draft_summary_model['domain'] = domain_model + data_product_draft_summary_model['parts_out'] = [data_product_part_model] + data_product_draft_summary_model['workflows'] = data_product_workflows_model + data_product_draft_summary_model['dataview_enabled'] = True + data_product_draft_summary_model['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_draft_summary_model['access_control'] = asset_list_access_control_model + data_product_draft_summary_model['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_draft_summary_model['sub_container'] = container_identity_model + data_product_draft_summary_model['is_restricted'] = True + data_product_draft_summary_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd' + data_product_draft_summary_model['asset'] = asset_reference_model + + # Construct a json representation of a DataProductDraftCollection model + data_product_draft_collection_model_json = {} + data_product_draft_collection_model_json['limit'] = 200 + data_product_draft_collection_model_json['first'] = first_page_model + data_product_draft_collection_model_json['next'] = next_page_model + data_product_draft_collection_model_json['total_results'] = 200 + data_product_draft_collection_model_json['drafts'] = [data_product_draft_summary_model] + + # Construct a model instance of DataProductDraftCollection by calling from_dict on the json representation + data_product_draft_collection_model = DataProductDraftCollection.from_dict(data_product_draft_collection_model_json) + assert data_product_draft_collection_model != False + + # Construct a model instance of DataProductDraftCollection by calling from_dict on the json representation + data_product_draft_collection_model_dict = DataProductDraftCollection.from_dict(data_product_draft_collection_model_json).__dict__ + data_product_draft_collection_model2 = DataProductDraftCollection(**data_product_draft_collection_model_dict) + + # Verify the model instances are equivalent + assert data_product_draft_collection_model == data_product_draft_collection_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_draft_collection_model_json2 = data_product_draft_collection_model.to_dict() + assert data_product_draft_collection_model_json2 == data_product_draft_collection_model_json + + +class TestModel_DataProductDraftDataProduct: + """ + Test Class for DataProductDraftDataProduct + """ + + def test_data_product_draft_data_product_serialization(self): + """ + Test serialization/deserialization for DataProductDraftDataProduct + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a DataProductDraftDataProduct model + data_product_draft_data_product_model_json = {} + data_product_draft_data_product_model_json['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_draft_data_product_model_json['release'] = data_product_draft_version_release_model + data_product_draft_data_product_model_json['container'] = container_reference_model + + # Construct a model instance of DataProductDraftDataProduct by calling from_dict on the json representation + data_product_draft_data_product_model = DataProductDraftDataProduct.from_dict(data_product_draft_data_product_model_json) + assert data_product_draft_data_product_model != False + + # Construct a model instance of DataProductDraftDataProduct by calling from_dict on the json representation + data_product_draft_data_product_model_dict = DataProductDraftDataProduct.from_dict(data_product_draft_data_product_model_json).__dict__ + data_product_draft_data_product_model2 = DataProductDraftDataProduct(**data_product_draft_data_product_model_dict) + + # Verify the model instances are equivalent + assert data_product_draft_data_product_model == data_product_draft_data_product_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_draft_data_product_model_json2 = data_product_draft_data_product_model.to_dict() + assert data_product_draft_data_product_model_json2 == data_product_draft_data_product_model_json + + +class TestModel_DataProductDraftPrototype: + """ + Test Class for DataProductDraftPrototype + """ + + def test_data_product_draft_prototype_serialization(self): + """ + Test serialization/deserialization for DataProductDraftPrototype + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_identity_model = {} # DataProductIdentity + data_product_identity_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_identity_model['release'] = data_product_draft_version_release_model + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + asset_prototype_model = {} # AssetPrototype + asset_prototype_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_prototype_model['container'] = container_identity_model + + # Construct a json representation of a DataProductDraftPrototype model + data_product_draft_prototype_model_json = {} + data_product_draft_prototype_model_json['version'] = '1.0.0' + data_product_draft_prototype_model_json['state'] = 'draft' + data_product_draft_prototype_model_json['data_product'] = data_product_identity_model + data_product_draft_prototype_model_json['name'] = 'My Data Product' + data_product_draft_prototype_model_json['description'] = 'This is a description of My Data Product.' + data_product_draft_prototype_model_json['tags'] = ['testString'] + data_product_draft_prototype_model_json['use_cases'] = [use_case_model] + data_product_draft_prototype_model_json['types'] = ['data'] + data_product_draft_prototype_model_json['contract_terms'] = [contract_terms_model] + data_product_draft_prototype_model_json['domain'] = domain_model + data_product_draft_prototype_model_json['parts_out'] = [data_product_part_model] + data_product_draft_prototype_model_json['workflows'] = data_product_workflows_model + data_product_draft_prototype_model_json['dataview_enabled'] = True + data_product_draft_prototype_model_json['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_draft_prototype_model_json['access_control'] = asset_list_access_control_model + data_product_draft_prototype_model_json['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_draft_prototype_model_json['sub_container'] = container_identity_model + data_product_draft_prototype_model_json['is_restricted'] = True + data_product_draft_prototype_model_json['asset'] = asset_prototype_model + + # Construct a model instance of DataProductDraftPrototype by calling from_dict on the json representation + data_product_draft_prototype_model = DataProductDraftPrototype.from_dict(data_product_draft_prototype_model_json) + assert data_product_draft_prototype_model != False + + # Construct a model instance of DataProductDraftPrototype by calling from_dict on the json representation + data_product_draft_prototype_model_dict = DataProductDraftPrototype.from_dict(data_product_draft_prototype_model_json).__dict__ + data_product_draft_prototype_model2 = DataProductDraftPrototype(**data_product_draft_prototype_model_dict) + + # Verify the model instances are equivalent + assert data_product_draft_prototype_model == data_product_draft_prototype_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_draft_prototype_model_json2 = data_product_draft_prototype_model.to_dict() + assert data_product_draft_prototype_model_json2 == data_product_draft_prototype_model_json + + +class TestModel_DataProductDraftSummary: + """ + Test Class for DataProductDraftSummary + """ + + def test_data_product_draft_summary_serialization(self): + """ + Test serialization/deserialization for DataProductDraftSummary + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_draft_summary_data_product_model = {} # DataProductDraftSummaryDataProduct + data_product_draft_summary_data_product_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_draft_summary_data_product_model['release'] = data_product_draft_version_release_model + data_product_draft_summary_data_product_model['container'] = container_reference_model + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a json representation of a DataProductDraftSummary model + data_product_draft_summary_model_json = {} + data_product_draft_summary_model_json['version'] = '1.0.0' + data_product_draft_summary_model_json['state'] = 'draft' + data_product_draft_summary_model_json['data_product'] = data_product_draft_summary_data_product_model + data_product_draft_summary_model_json['name'] = 'My Data Product' + data_product_draft_summary_model_json['description'] = 'This is a description of My Data Product.' + data_product_draft_summary_model_json['tags'] = ['testString'] + data_product_draft_summary_model_json['use_cases'] = [use_case_model] + data_product_draft_summary_model_json['types'] = ['data'] + data_product_draft_summary_model_json['contract_terms'] = [contract_terms_model] + data_product_draft_summary_model_json['domain'] = domain_model + data_product_draft_summary_model_json['parts_out'] = [data_product_part_model] + data_product_draft_summary_model_json['workflows'] = data_product_workflows_model + data_product_draft_summary_model_json['dataview_enabled'] = True + data_product_draft_summary_model_json['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_draft_summary_model_json['access_control'] = asset_list_access_control_model + data_product_draft_summary_model_json['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_draft_summary_model_json['sub_container'] = container_identity_model + data_product_draft_summary_model_json['is_restricted'] = True + data_product_draft_summary_model_json['id'] = '2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd' + data_product_draft_summary_model_json['asset'] = asset_reference_model + + # Construct a model instance of DataProductDraftSummary by calling from_dict on the json representation + data_product_draft_summary_model = DataProductDraftSummary.from_dict(data_product_draft_summary_model_json) + assert data_product_draft_summary_model != False + + # Construct a model instance of DataProductDraftSummary by calling from_dict on the json representation + data_product_draft_summary_model_dict = DataProductDraftSummary.from_dict(data_product_draft_summary_model_json).__dict__ + data_product_draft_summary_model2 = DataProductDraftSummary(**data_product_draft_summary_model_dict) + + # Verify the model instances are equivalent + assert data_product_draft_summary_model == data_product_draft_summary_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_draft_summary_model_json2 = data_product_draft_summary_model.to_dict() + assert data_product_draft_summary_model_json2 == data_product_draft_summary_model_json + + +class TestModel_DataProductDraftSummaryDataProduct: + """ + Test Class for DataProductDraftSummaryDataProduct + """ + + def test_data_product_draft_summary_data_product_serialization(self): + """ + Test serialization/deserialization for DataProductDraftSummaryDataProduct + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a DataProductDraftSummaryDataProduct model + data_product_draft_summary_data_product_model_json = {} + data_product_draft_summary_data_product_model_json['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_draft_summary_data_product_model_json['release'] = data_product_draft_version_release_model + data_product_draft_summary_data_product_model_json['container'] = container_reference_model + + # Construct a model instance of DataProductDraftSummaryDataProduct by calling from_dict on the json representation + data_product_draft_summary_data_product_model = DataProductDraftSummaryDataProduct.from_dict(data_product_draft_summary_data_product_model_json) + assert data_product_draft_summary_data_product_model != False + + # Construct a model instance of DataProductDraftSummaryDataProduct by calling from_dict on the json representation + data_product_draft_summary_data_product_model_dict = DataProductDraftSummaryDataProduct.from_dict(data_product_draft_summary_data_product_model_json).__dict__ + data_product_draft_summary_data_product_model2 = DataProductDraftSummaryDataProduct(**data_product_draft_summary_data_product_model_dict) + + # Verify the model instances are equivalent + assert data_product_draft_summary_data_product_model == data_product_draft_summary_data_product_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_draft_summary_data_product_model_json2 = data_product_draft_summary_data_product_model.to_dict() + assert data_product_draft_summary_data_product_model_json2 == data_product_draft_summary_data_product_model_json + + +class TestModel_DataProductDraftVersionRelease: + """ + Test Class for DataProductDraftVersionRelease + """ + + def test_data_product_draft_version_release_serialization(self): + """ + Test serialization/deserialization for DataProductDraftVersionRelease + """ + + # Construct a json representation of a DataProductDraftVersionRelease model + data_product_draft_version_release_model_json = {} + data_product_draft_version_release_model_json['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a model instance of DataProductDraftVersionRelease by calling from_dict on the json representation + data_product_draft_version_release_model = DataProductDraftVersionRelease.from_dict(data_product_draft_version_release_model_json) + assert data_product_draft_version_release_model != False + + # Construct a model instance of DataProductDraftVersionRelease by calling from_dict on the json representation + data_product_draft_version_release_model_dict = DataProductDraftVersionRelease.from_dict(data_product_draft_version_release_model_json).__dict__ + data_product_draft_version_release_model2 = DataProductDraftVersionRelease(**data_product_draft_version_release_model_dict) + + # Verify the model instances are equivalent + assert data_product_draft_version_release_model == data_product_draft_version_release_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_draft_version_release_model_json2 = data_product_draft_version_release_model.to_dict() + assert data_product_draft_version_release_model_json2 == data_product_draft_version_release_model_json + + +class TestModel_DataProductIdentity: + """ + Test Class for DataProductIdentity + """ + + def test_data_product_identity_serialization(self): + """ + Test serialization/deserialization for DataProductIdentity + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a json representation of a DataProductIdentity model + data_product_identity_model_json = {} + data_product_identity_model_json['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_identity_model_json['release'] = data_product_draft_version_release_model + + # Construct a model instance of DataProductIdentity by calling from_dict on the json representation + data_product_identity_model = DataProductIdentity.from_dict(data_product_identity_model_json) + assert data_product_identity_model != False + + # Construct a model instance of DataProductIdentity by calling from_dict on the json representation + data_product_identity_model_dict = DataProductIdentity.from_dict(data_product_identity_model_json).__dict__ + data_product_identity_model2 = DataProductIdentity(**data_product_identity_model_dict) + + # Verify the model instances are equivalent + assert data_product_identity_model == data_product_identity_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_identity_model_json2 = data_product_identity_model.to_dict() + assert data_product_identity_model_json2 == data_product_identity_model_json + + +class TestModel_DataProductOrderAccessRequest: + """ + Test Class for DataProductOrderAccessRequest + """ + + def test_data_product_order_access_request_serialization(self): + """ + Test serialization/deserialization for DataProductOrderAccessRequest + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a json representation of a DataProductOrderAccessRequest model + data_product_order_access_request_model_json = {} + data_product_order_access_request_model_json['task_assignee_users'] = ['testString'] + data_product_order_access_request_model_json['pre_approved_users'] = ['testString'] + data_product_order_access_request_model_json['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + # Construct a model instance of DataProductOrderAccessRequest by calling from_dict on the json representation + data_product_order_access_request_model = DataProductOrderAccessRequest.from_dict(data_product_order_access_request_model_json) + assert data_product_order_access_request_model != False + + # Construct a model instance of DataProductOrderAccessRequest by calling from_dict on the json representation + data_product_order_access_request_model_dict = DataProductOrderAccessRequest.from_dict(data_product_order_access_request_model_json).__dict__ + data_product_order_access_request_model2 = DataProductOrderAccessRequest(**data_product_order_access_request_model_dict) + + # Verify the model instances are equivalent + assert data_product_order_access_request_model == data_product_order_access_request_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_order_access_request_model_json2 = data_product_order_access_request_model.to_dict() + assert data_product_order_access_request_model_json2 == data_product_order_access_request_model_json + + +class TestModel_DataProductPart: + """ + Test Class for DataProductPart + """ + + def test_data_product_part_serialization(self): + """ + Test serialization/deserialization for DataProductPart + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + # Construct a json representation of a DataProductPart model + data_product_part_model_json = {} + data_product_part_model_json['asset'] = asset_part_reference_model + data_product_part_model_json['delivery_methods'] = [delivery_method_model] + + # Construct a model instance of DataProductPart by calling from_dict on the json representation + data_product_part_model = DataProductPart.from_dict(data_product_part_model_json) + assert data_product_part_model != False + + # Construct a model instance of DataProductPart by calling from_dict on the json representation + data_product_part_model_dict = DataProductPart.from_dict(data_product_part_model_json).__dict__ + data_product_part_model2 = DataProductPart(**data_product_part_model_dict) + + # Verify the model instances are equivalent + assert data_product_part_model == data_product_part_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_part_model_json2 = data_product_part_model.to_dict() + assert data_product_part_model_json2 == data_product_part_model_json + + +class TestModel_DataProductRelease: + """ + Test Class for DataProductRelease + """ + + def test_data_product_release_serialization(self): + """ + Test serialization/deserialization for DataProductRelease + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_release_data_product_model = {} # DataProductReleaseDataProduct + data_product_release_data_product_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_release_data_product_model['release'] = data_product_draft_version_release_model + data_product_release_data_product_model['container'] = container_reference_model + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + visualization_model = {} # Visualization + visualization_model['id'] = 'testString' + visualization_model['name'] = 'testString' + + error_message_model = {} # ErrorMessage + error_message_model['code'] = 'testString' + error_message_model['message'] = 'testString' + + data_asset_relationship_model = {} # DataAssetRelationship + data_asset_relationship_model['visualization'] = visualization_model + data_asset_relationship_model['asset'] = asset_reference_model + data_asset_relationship_model['related_asset'] = asset_reference_model + data_asset_relationship_model['error'] = error_message_model + + # Construct a json representation of a DataProductRelease model + data_product_release_model_json = {} + data_product_release_model_json['version'] = '1.0.0' + data_product_release_model_json['state'] = 'draft' + data_product_release_model_json['data_product'] = data_product_release_data_product_model + data_product_release_model_json['name'] = 'My Data Product' + data_product_release_model_json['description'] = 'This is a description of My Data Product.' + data_product_release_model_json['tags'] = ['testString'] + data_product_release_model_json['use_cases'] = [use_case_model] + data_product_release_model_json['types'] = ['data'] + data_product_release_model_json['contract_terms'] = [contract_terms_model] + data_product_release_model_json['domain'] = domain_model + data_product_release_model_json['parts_out'] = [data_product_part_model] + data_product_release_model_json['workflows'] = data_product_workflows_model + data_product_release_model_json['dataview_enabled'] = True + data_product_release_model_json['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_release_model_json['access_control'] = asset_list_access_control_model + data_product_release_model_json['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_release_model_json['sub_container'] = container_identity_model + data_product_release_model_json['is_restricted'] = True + data_product_release_model_json['id'] = '2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd' + data_product_release_model_json['asset'] = asset_reference_model + data_product_release_model_json['published_by'] = 'testString' + data_product_release_model_json['published_at'] = '2019-01-01T12:00:00Z' + data_product_release_model_json['created_by'] = 'testString' + data_product_release_model_json['created_at'] = '2019-01-01T12:00:00Z' + data_product_release_model_json['properties'] = {'anyKey': 'anyValue'} + data_product_release_model_json['visualization_errors'] = [data_asset_relationship_model] + + # Construct a model instance of DataProductRelease by calling from_dict on the json representation + data_product_release_model = DataProductRelease.from_dict(data_product_release_model_json) + assert data_product_release_model != False + + # Construct a model instance of DataProductRelease by calling from_dict on the json representation + data_product_release_model_dict = DataProductRelease.from_dict(data_product_release_model_json).__dict__ + data_product_release_model2 = DataProductRelease(**data_product_release_model_dict) + + # Verify the model instances are equivalent + assert data_product_release_model == data_product_release_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_release_model_json2 = data_product_release_model.to_dict() + assert data_product_release_model_json2 == data_product_release_model_json + + +class TestModel_DataProductReleaseCollection: + """ + Test Class for DataProductReleaseCollection + """ + + def test_data_product_release_collection_serialization(self): + """ + Test serialization/deserialization for DataProductReleaseCollection + """ + + # Construct dict forms of any model objects needed in order to build this model. + + first_page_model = {} # FirstPage + first_page_model['href'] = 'https://api.example.com/collection' + + next_page_model = {} # NextPage + next_page_model['href'] = 'https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + next_page_model['start'] = 'eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_release_summary_data_product_model = {} # DataProductReleaseSummaryDataProduct + data_product_release_summary_data_product_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_release_summary_data_product_model['release'] = data_product_draft_version_release_model + data_product_release_summary_data_product_model['container'] = container_reference_model + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + data_product_release_summary_model = {} # DataProductReleaseSummary + data_product_release_summary_model['version'] = '1.0.0' + data_product_release_summary_model['state'] = 'draft' + data_product_release_summary_model['data_product'] = data_product_release_summary_data_product_model + data_product_release_summary_model['name'] = 'My Data Product' + data_product_release_summary_model['description'] = 'This is a description of My Data Product.' + data_product_release_summary_model['tags'] = ['testString'] + data_product_release_summary_model['use_cases'] = [use_case_model] + data_product_release_summary_model['types'] = ['data'] + data_product_release_summary_model['contract_terms'] = [contract_terms_model] + data_product_release_summary_model['domain'] = domain_model + data_product_release_summary_model['parts_out'] = [data_product_part_model] + data_product_release_summary_model['workflows'] = data_product_workflows_model + data_product_release_summary_model['dataview_enabled'] = True + data_product_release_summary_model['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_release_summary_model['access_control'] = asset_list_access_control_model + data_product_release_summary_model['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_release_summary_model['sub_container'] = container_identity_model + data_product_release_summary_model['is_restricted'] = True + data_product_release_summary_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd' + data_product_release_summary_model['asset'] = asset_reference_model + + # Construct a json representation of a DataProductReleaseCollection model + data_product_release_collection_model_json = {} + data_product_release_collection_model_json['limit'] = 200 + data_product_release_collection_model_json['first'] = first_page_model + data_product_release_collection_model_json['next'] = next_page_model + data_product_release_collection_model_json['total_results'] = 200 + data_product_release_collection_model_json['releases'] = [data_product_release_summary_model] + + # Construct a model instance of DataProductReleaseCollection by calling from_dict on the json representation + data_product_release_collection_model = DataProductReleaseCollection.from_dict(data_product_release_collection_model_json) + assert data_product_release_collection_model != False + + # Construct a model instance of DataProductReleaseCollection by calling from_dict on the json representation + data_product_release_collection_model_dict = DataProductReleaseCollection.from_dict(data_product_release_collection_model_json).__dict__ + data_product_release_collection_model2 = DataProductReleaseCollection(**data_product_release_collection_model_dict) + + # Verify the model instances are equivalent + assert data_product_release_collection_model == data_product_release_collection_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_release_collection_model_json2 = data_product_release_collection_model.to_dict() + assert data_product_release_collection_model_json2 == data_product_release_collection_model_json + + +class TestModel_DataProductReleaseDataProduct: + """ + Test Class for DataProductReleaseDataProduct + """ + + def test_data_product_release_data_product_serialization(self): + """ + Test serialization/deserialization for DataProductReleaseDataProduct + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a DataProductReleaseDataProduct model + data_product_release_data_product_model_json = {} + data_product_release_data_product_model_json['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_release_data_product_model_json['release'] = data_product_draft_version_release_model + data_product_release_data_product_model_json['container'] = container_reference_model + + # Construct a model instance of DataProductReleaseDataProduct by calling from_dict on the json representation + data_product_release_data_product_model = DataProductReleaseDataProduct.from_dict(data_product_release_data_product_model_json) + assert data_product_release_data_product_model != False + + # Construct a model instance of DataProductReleaseDataProduct by calling from_dict on the json representation + data_product_release_data_product_model_dict = DataProductReleaseDataProduct.from_dict(data_product_release_data_product_model_json).__dict__ + data_product_release_data_product_model2 = DataProductReleaseDataProduct(**data_product_release_data_product_model_dict) + + # Verify the model instances are equivalent + assert data_product_release_data_product_model == data_product_release_data_product_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_release_data_product_model_json2 = data_product_release_data_product_model.to_dict() + assert data_product_release_data_product_model_json2 == data_product_release_data_product_model_json + + +class TestModel_DataProductReleaseSummary: + """ + Test Class for DataProductReleaseSummary + """ + + def test_data_product_release_summary_serialization(self): + """ + Test serialization/deserialization for DataProductReleaseSummary + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_release_summary_data_product_model = {} # DataProductReleaseSummaryDataProduct + data_product_release_summary_data_product_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_release_summary_data_product_model['release'] = data_product_draft_version_release_model + data_product_release_summary_data_product_model['container'] = container_reference_model + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a json representation of a DataProductReleaseSummary model + data_product_release_summary_model_json = {} + data_product_release_summary_model_json['version'] = '1.0.0' + data_product_release_summary_model_json['state'] = 'draft' + data_product_release_summary_model_json['data_product'] = data_product_release_summary_data_product_model + data_product_release_summary_model_json['name'] = 'My Data Product' + data_product_release_summary_model_json['description'] = 'This is a description of My Data Product.' + data_product_release_summary_model_json['tags'] = ['testString'] + data_product_release_summary_model_json['use_cases'] = [use_case_model] + data_product_release_summary_model_json['types'] = ['data'] + data_product_release_summary_model_json['contract_terms'] = [contract_terms_model] + data_product_release_summary_model_json['domain'] = domain_model + data_product_release_summary_model_json['parts_out'] = [data_product_part_model] + data_product_release_summary_model_json['workflows'] = data_product_workflows_model + data_product_release_summary_model_json['dataview_enabled'] = True + data_product_release_summary_model_json['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_release_summary_model_json['access_control'] = asset_list_access_control_model + data_product_release_summary_model_json['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_release_summary_model_json['sub_container'] = container_identity_model + data_product_release_summary_model_json['is_restricted'] = True + data_product_release_summary_model_json['id'] = '2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd' + data_product_release_summary_model_json['asset'] = asset_reference_model + + # Construct a model instance of DataProductReleaseSummary by calling from_dict on the json representation + data_product_release_summary_model = DataProductReleaseSummary.from_dict(data_product_release_summary_model_json) + assert data_product_release_summary_model != False + + # Construct a model instance of DataProductReleaseSummary by calling from_dict on the json representation + data_product_release_summary_model_dict = DataProductReleaseSummary.from_dict(data_product_release_summary_model_json).__dict__ + data_product_release_summary_model2 = DataProductReleaseSummary(**data_product_release_summary_model_dict) + + # Verify the model instances are equivalent + assert data_product_release_summary_model == data_product_release_summary_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_release_summary_model_json2 = data_product_release_summary_model.to_dict() + assert data_product_release_summary_model_json2 == data_product_release_summary_model_json + + +class TestModel_DataProductReleaseSummaryDataProduct: + """ + Test Class for DataProductReleaseSummaryDataProduct + """ + + def test_data_product_release_summary_data_product_serialization(self): + """ + Test serialization/deserialization for DataProductReleaseSummaryDataProduct + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a DataProductReleaseSummaryDataProduct model + data_product_release_summary_data_product_model_json = {} + data_product_release_summary_data_product_model_json['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_release_summary_data_product_model_json['release'] = data_product_draft_version_release_model + data_product_release_summary_data_product_model_json['container'] = container_reference_model + + # Construct a model instance of DataProductReleaseSummaryDataProduct by calling from_dict on the json representation + data_product_release_summary_data_product_model = DataProductReleaseSummaryDataProduct.from_dict(data_product_release_summary_data_product_model_json) + assert data_product_release_summary_data_product_model != False + + # Construct a model instance of DataProductReleaseSummaryDataProduct by calling from_dict on the json representation + data_product_release_summary_data_product_model_dict = DataProductReleaseSummaryDataProduct.from_dict(data_product_release_summary_data_product_model_json).__dict__ + data_product_release_summary_data_product_model2 = DataProductReleaseSummaryDataProduct(**data_product_release_summary_data_product_model_dict) + + # Verify the model instances are equivalent + assert data_product_release_summary_data_product_model == data_product_release_summary_data_product_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_release_summary_data_product_model_json2 = data_product_release_summary_data_product_model.to_dict() + assert data_product_release_summary_data_product_model_json2 == data_product_release_summary_data_product_model_json + + +class TestModel_DataProductSummary: + """ + Test Class for DataProductSummary + """ + + def test_data_product_summary_serialization(self): + """ + Test serialization/deserialization for DataProductSummary + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a DataProductSummary model + data_product_summary_model_json = {} + data_product_summary_model_json['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_summary_model_json['release'] = data_product_draft_version_release_model + data_product_summary_model_json['container'] = container_reference_model + data_product_summary_model_json['name'] = 'testString' + + # Construct a model instance of DataProductSummary by calling from_dict on the json representation + data_product_summary_model = DataProductSummary.from_dict(data_product_summary_model_json) + assert data_product_summary_model != False + + # Construct a model instance of DataProductSummary by calling from_dict on the json representation + data_product_summary_model_dict = DataProductSummary.from_dict(data_product_summary_model_json).__dict__ + data_product_summary_model2 = DataProductSummary(**data_product_summary_model_dict) + + # Verify the model instances are equivalent + assert data_product_summary_model == data_product_summary_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_summary_model_json2 = data_product_summary_model.to_dict() + assert data_product_summary_model_json2 == data_product_summary_model_json + + +class TestModel_DataProductVersionCollection: + """ + Test Class for DataProductVersionCollection + """ + + def test_data_product_version_collection_serialization(self): + """ + Test serialization/deserialization for DataProductVersionCollection + """ + + # Construct dict forms of any model objects needed in order to build this model. + + first_page_model = {} # FirstPage + first_page_model['href'] = 'https://api.example.com/collection' + + next_page_model = {} # NextPage + next_page_model['href'] = 'https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + next_page_model['start'] = 'eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_version_summary_data_product_model = {} # DataProductVersionSummaryDataProduct + data_product_version_summary_data_product_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_version_summary_data_product_model['release'] = data_product_draft_version_release_model + data_product_version_summary_data_product_model['container'] = container_reference_model + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + data_product_version_summary_model = {} # DataProductVersionSummary + data_product_version_summary_model['version'] = '1.0.0' + data_product_version_summary_model['state'] = 'draft' + data_product_version_summary_model['data_product'] = data_product_version_summary_data_product_model + data_product_version_summary_model['name'] = 'My Data Product' + data_product_version_summary_model['description'] = 'This is a description of My Data Product.' + data_product_version_summary_model['tags'] = ['testString'] + data_product_version_summary_model['use_cases'] = [use_case_model] + data_product_version_summary_model['types'] = ['data'] + data_product_version_summary_model['contract_terms'] = [contract_terms_model] + data_product_version_summary_model['domain'] = domain_model + data_product_version_summary_model['parts_out'] = [data_product_part_model] + data_product_version_summary_model['workflows'] = data_product_workflows_model + data_product_version_summary_model['dataview_enabled'] = True + data_product_version_summary_model['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_version_summary_model['access_control'] = asset_list_access_control_model + data_product_version_summary_model['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_version_summary_model['sub_container'] = container_identity_model + data_product_version_summary_model['is_restricted'] = True + data_product_version_summary_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd' + data_product_version_summary_model['asset'] = asset_reference_model + + # Construct a json representation of a DataProductVersionCollection model + data_product_version_collection_model_json = {} + data_product_version_collection_model_json['limit'] = 200 + data_product_version_collection_model_json['first'] = first_page_model + data_product_version_collection_model_json['next'] = next_page_model + data_product_version_collection_model_json['total_results'] = 200 + data_product_version_collection_model_json['data_product_versions'] = [data_product_version_summary_model] + + # Construct a model instance of DataProductVersionCollection by calling from_dict on the json representation + data_product_version_collection_model = DataProductVersionCollection.from_dict(data_product_version_collection_model_json) + assert data_product_version_collection_model != False + + # Construct a model instance of DataProductVersionCollection by calling from_dict on the json representation + data_product_version_collection_model_dict = DataProductVersionCollection.from_dict(data_product_version_collection_model_json).__dict__ + data_product_version_collection_model2 = DataProductVersionCollection(**data_product_version_collection_model_dict) + + # Verify the model instances are equivalent + assert data_product_version_collection_model == data_product_version_collection_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_version_collection_model_json2 = data_product_version_collection_model.to_dict() + assert data_product_version_collection_model_json2 == data_product_version_collection_model_json + + +class TestModel_DataProductVersionSummary: + """ + Test Class for DataProductVersionSummary + """ + + def test_data_product_version_summary_serialization(self): + """ + Test serialization/deserialization for DataProductVersionSummary + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + data_product_version_summary_data_product_model = {} # DataProductVersionSummaryDataProduct + data_product_version_summary_data_product_model['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_version_summary_data_product_model['release'] = data_product_draft_version_release_model + data_product_version_summary_data_product_model['container'] = container_reference_model + + use_case_model = {} # UseCase + use_case_model['id'] = 'testString' + use_case_model['name'] = 'testString' + use_case_model['container'] = container_reference_model + + asset_reference_model = {} # AssetReference + asset_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_reference_model['name'] = 'testString' + asset_reference_model['container'] = container_reference_model + + contract_terms_document_attachment_model = {} # ContractTermsDocumentAttachment + contract_terms_document_attachment_model['id'] = 'testString' + + contract_terms_document_model = {} # ContractTermsDocument + contract_terms_document_model['url'] = 'testString' + contract_terms_document_model['type'] = 'terms_and_conditions' + contract_terms_document_model['name'] = 'testString' + contract_terms_document_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_terms_document_model['attachment'] = contract_terms_document_attachment_model + contract_terms_document_model['upload_url'] = 'testString' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + overview_model = {} # Overview + overview_model['api_version'] = 'v3.0.1' + overview_model['kind'] = 'DataContract' + overview_model['name'] = 'Sample Data Contract' + overview_model['version'] = '0.0.0' + overview_model['domain'] = domain_model + overview_model['more_info'] = 'List of links to sources that provide more details on the data contract.' + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + description_model = {} # Description + description_model['purpose'] = 'Used for customer behavior analysis.' + description_model['limitations'] = 'Data cannot be used for marketing.' + description_model['usage'] = 'Data should be used only for analytics.' + description_model['more_info'] = [contract_terms_more_info_model] + description_model['custom_properties'] = '{"property1":"value1"}' + + contract_template_organization_model = {} # ContractTemplateOrganization + contract_template_organization_model['user_id'] = 'IBMid-691000IN4G' + contract_template_organization_model['role'] = 'owner' + + roles_model = {} # Roles + roles_model['role'] = 'owner' + + pricing_model = {} # Pricing + pricing_model['amount'] = '100.0' + pricing_model['currency'] = 'USD' + pricing_model['unit'] = 'megabyte' + + contract_template_sla_property_model = {} # ContractTemplateSLAProperty + contract_template_sla_property_model['property'] = 'Uptime Guarantee' + contract_template_sla_property_model['value'] = '99.9' + + contract_template_sla_model = {} # ContractTemplateSLA + contract_template_sla_model['default_element'] = 'Standard SLA Policy' + contract_template_sla_model['properties'] = [contract_template_sla_property_model] + + contract_template_support_and_communication_model = {} # ContractTemplateSupportAndCommunication + contract_template_support_and_communication_model['channel'] = 'Email Support' + contract_template_support_and_communication_model['url'] = 'https://support.example.com' + + contract_template_custom_property_model = {} # ContractTemplateCustomProperty + contract_template_custom_property_model['key'] = 'customPropertyKey' + contract_template_custom_property_model['value'] = 'customPropertyValue' + + contract_test_model = {} # ContractTest + contract_test_model['status'] = 'pass' + contract_test_model['last_tested_time'] = 'testString' + contract_test_model['message'] = 'testString' + + contract_asset_model = {} # ContractAsset + contract_asset_model['id'] = 'testString' + contract_asset_model['name'] = 'testString' + + contract_server_model = {} # ContractServer + contract_server_model['server'] = 'testString' + contract_server_model['asset'] = contract_asset_model + contract_server_model['connection_id'] = 'testString' + contract_server_model['type'] = 'testString' + contract_server_model['description'] = 'testString' + contract_server_model['environment'] = 'testString' + contract_server_model['account'] = 'testString' + contract_server_model['catalog'] = 'testString' + contract_server_model['database'] = 'testString' + contract_server_model['dataset'] = 'testString' + contract_server_model['delimiter'] = 'testString' + contract_server_model['endpoint_url'] = 'testString' + contract_server_model['format'] = 'testString' + contract_server_model['host'] = 'testString' + contract_server_model['location'] = 'testString' + contract_server_model['path'] = 'testString' + contract_server_model['port'] = 'testString' + contract_server_model['project'] = 'testString' + contract_server_model['region'] = 'testString' + contract_server_model['region_name'] = 'testString' + contract_server_model['schema'] = 'testString' + contract_server_model['service_name'] = 'testString' + contract_server_model['staging_dir'] = 'testString' + contract_server_model['stream'] = 'testString' + contract_server_model['warehouse'] = 'testString' + contract_server_model['roles'] = ['testString'] + contract_server_model['custom_properties'] = [contract_template_custom_property_model] + + contract_schema_property_type_model = {} # ContractSchemaPropertyType + contract_schema_property_type_model['type'] = 'testString' + contract_schema_property_type_model['length'] = 'testString' + contract_schema_property_type_model['scale'] = 'testString' + contract_schema_property_type_model['nullable'] = 'testString' + contract_schema_property_type_model['signed'] = 'testString' + contract_schema_property_type_model['native_type'] = 'testString' + + contract_quality_rule_model = {} # ContractQualityRule + contract_quality_rule_model['type'] = 'sql' + contract_quality_rule_model['description'] = 'testString' + contract_quality_rule_model['rule'] = 'testString' + contract_quality_rule_model['implementation'] = 'testString' + contract_quality_rule_model['engine'] = 'testString' + contract_quality_rule_model['must_be_less_than'] = 'testString' + contract_quality_rule_model['must_be_less_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_greater_than'] = 'testString' + contract_quality_rule_model['must_be_greater_or_equal_to'] = 'testString' + contract_quality_rule_model['must_be_between'] = ['testString'] + contract_quality_rule_model['must_not_be_between'] = ['testString'] + contract_quality_rule_model['must_be'] = 'testString' + contract_quality_rule_model['must_not_be'] = 'testString' + contract_quality_rule_model['name'] = 'testString' + contract_quality_rule_model['unit'] = 'testString' + contract_quality_rule_model['query'] = 'testString' + + contract_schema_property_model = {} # ContractSchemaProperty + contract_schema_property_model['name'] = 'testString' + contract_schema_property_model['type'] = contract_schema_property_type_model + contract_schema_property_model['quality'] = [contract_quality_rule_model] + + contract_schema_model = {} # ContractSchema + contract_schema_model['asset_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['connection_id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + contract_schema_model['name'] = 'testString' + contract_schema_model['description'] = 'testString' + contract_schema_model['connection_path'] = 'testString' + contract_schema_model['physical_type'] = 'testString' + contract_schema_model['properties'] = [contract_schema_property_model] + contract_schema_model['quality'] = [contract_quality_rule_model] + + contract_terms_model = {} # ContractTerms + contract_terms_model['asset'] = asset_reference_model + contract_terms_model['id'] = 'testString' + contract_terms_model['documents'] = [contract_terms_document_model] + contract_terms_model['error_msg'] = 'testString' + contract_terms_model['overview'] = overview_model + contract_terms_model['description'] = description_model + contract_terms_model['organization'] = [contract_template_organization_model] + contract_terms_model['roles'] = [roles_model] + contract_terms_model['price'] = pricing_model + contract_terms_model['sla'] = [contract_template_sla_model] + contract_terms_model['support_and_communication'] = [contract_template_support_and_communication_model] + contract_terms_model['custom_properties'] = [contract_template_custom_property_model] + contract_terms_model['contract_test'] = contract_test_model + contract_terms_model['servers'] = [contract_server_model] + contract_terms_model['schema'] = [contract_schema_model] + + asset_part_reference_model = {} # AssetPartReference + asset_part_reference_model['id'] = '2b0bf220-079c-11ee-be56-0242ac120002' + asset_part_reference_model['name'] = 'testString' + asset_part_reference_model['container'] = container_reference_model + asset_part_reference_model['type'] = 'data_asset' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + delivery_method_model = {} # DeliveryMethod + delivery_method_model['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model['container'] = container_reference_model + delivery_method_model['getproperties'] = delivery_method_properties_model_model + + data_product_part_model = {} # DataProductPart + data_product_part_model['asset'] = asset_part_reference_model + data_product_part_model['delivery_methods'] = [delivery_method_model] + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + data_product_workflows_model = {} # DataProductWorkflows + data_product_workflows_model['order_access_request'] = data_product_order_access_request_model + + asset_list_access_control_model = {} # AssetListAccessControl + asset_list_access_control_model['owner'] = 'IBMid-696000KYV9' + + container_identity_model = {} # ContainerIdentity + container_identity_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + + # Construct a json representation of a DataProductVersionSummary model + data_product_version_summary_model_json = {} + data_product_version_summary_model_json['version'] = '1.0.0' + data_product_version_summary_model_json['state'] = 'draft' + data_product_version_summary_model_json['data_product'] = data_product_version_summary_data_product_model + data_product_version_summary_model_json['name'] = 'My Data Product' + data_product_version_summary_model_json['description'] = 'This is a description of My Data Product.' + data_product_version_summary_model_json['tags'] = ['testString'] + data_product_version_summary_model_json['use_cases'] = [use_case_model] + data_product_version_summary_model_json['types'] = ['data'] + data_product_version_summary_model_json['contract_terms'] = [contract_terms_model] + data_product_version_summary_model_json['domain'] = domain_model + data_product_version_summary_model_json['parts_out'] = [data_product_part_model] + data_product_version_summary_model_json['workflows'] = data_product_workflows_model + data_product_version_summary_model_json['dataview_enabled'] = True + data_product_version_summary_model_json['comments'] = 'Comments by a producer that are provided either at the time of data product version creation or retiring' + data_product_version_summary_model_json['access_control'] = asset_list_access_control_model + data_product_version_summary_model_json['last_updated_at'] = '2019-01-01T12:00:00Z' + data_product_version_summary_model_json['sub_container'] = container_identity_model + data_product_version_summary_model_json['is_restricted'] = True + data_product_version_summary_model_json['id'] = '2b0bf220-079c-11ee-be56-0242ac120002@d29c42eb-7100-4b7a-8257-c196dbcca1cd' + data_product_version_summary_model_json['asset'] = asset_reference_model + + # Construct a model instance of DataProductVersionSummary by calling from_dict on the json representation + data_product_version_summary_model = DataProductVersionSummary.from_dict(data_product_version_summary_model_json) + assert data_product_version_summary_model != False + + # Construct a model instance of DataProductVersionSummary by calling from_dict on the json representation + data_product_version_summary_model_dict = DataProductVersionSummary.from_dict(data_product_version_summary_model_json).__dict__ + data_product_version_summary_model2 = DataProductVersionSummary(**data_product_version_summary_model_dict) + + # Verify the model instances are equivalent + assert data_product_version_summary_model == data_product_version_summary_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_version_summary_model_json2 = data_product_version_summary_model.to_dict() + assert data_product_version_summary_model_json2 == data_product_version_summary_model_json + + +class TestModel_DataProductVersionSummaryDataProduct: + """ + Test Class for DataProductVersionSummaryDataProduct + """ + + def test_data_product_version_summary_data_product_serialization(self): + """ + Test serialization/deserialization for DataProductVersionSummaryDataProduct + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_draft_version_release_model = {} # DataProductDraftVersionRelease + data_product_draft_version_release_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a DataProductVersionSummaryDataProduct model + data_product_version_summary_data_product_model_json = {} + data_product_version_summary_data_product_model_json['id'] = 'b38df608-d34b-4d58-8136-ed25e6c6684e' + data_product_version_summary_data_product_model_json['release'] = data_product_draft_version_release_model + data_product_version_summary_data_product_model_json['container'] = container_reference_model + + # Construct a model instance of DataProductVersionSummaryDataProduct by calling from_dict on the json representation + data_product_version_summary_data_product_model = DataProductVersionSummaryDataProduct.from_dict(data_product_version_summary_data_product_model_json) + assert data_product_version_summary_data_product_model != False + + # Construct a model instance of DataProductVersionSummaryDataProduct by calling from_dict on the json representation + data_product_version_summary_data_product_model_dict = DataProductVersionSummaryDataProduct.from_dict(data_product_version_summary_data_product_model_json).__dict__ + data_product_version_summary_data_product_model2 = DataProductVersionSummaryDataProduct(**data_product_version_summary_data_product_model_dict) + + # Verify the model instances are equivalent + assert data_product_version_summary_data_product_model == data_product_version_summary_data_product_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_version_summary_data_product_model_json2 = data_product_version_summary_data_product_model.to_dict() + assert data_product_version_summary_data_product_model_json2 == data_product_version_summary_data_product_model_json + + +class TestModel_DataProductWorkflows: + """ + Test Class for DataProductWorkflows + """ + + def test_data_product_workflows_serialization(self): + """ + Test serialization/deserialization for DataProductWorkflows + """ + + # Construct dict forms of any model objects needed in order to build this model. + + data_product_custom_workflow_definition_model = {} # DataProductCustomWorkflowDefinition + data_product_custom_workflow_definition_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + data_product_order_access_request_model = {} # DataProductOrderAccessRequest + data_product_order_access_request_model['task_assignee_users'] = ['testString'] + data_product_order_access_request_model['pre_approved_users'] = ['testString'] + data_product_order_access_request_model['custom_workflow_definition'] = data_product_custom_workflow_definition_model + + # Construct a json representation of a DataProductWorkflows model + data_product_workflows_model_json = {} + data_product_workflows_model_json['order_access_request'] = data_product_order_access_request_model + + # Construct a model instance of DataProductWorkflows by calling from_dict on the json representation + data_product_workflows_model = DataProductWorkflows.from_dict(data_product_workflows_model_json) + assert data_product_workflows_model != False + + # Construct a model instance of DataProductWorkflows by calling from_dict on the json representation + data_product_workflows_model_dict = DataProductWorkflows.from_dict(data_product_workflows_model_json).__dict__ + data_product_workflows_model2 = DataProductWorkflows(**data_product_workflows_model_dict) + + # Verify the model instances are equivalent + assert data_product_workflows_model == data_product_workflows_model2 + + # Convert model instance back to dict and verify no loss of data + data_product_workflows_model_json2 = data_product_workflows_model.to_dict() + assert data_product_workflows_model_json2 == data_product_workflows_model_json + + +class TestModel_DeliveryMethod: + """ + Test Class for DeliveryMethod + """ + + def test_delivery_method_serialization(self): + """ + Test serialization/deserialization for DeliveryMethod + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + delivery_method_properties_model_model = {} # DeliveryMethodPropertiesModel + delivery_method_properties_model_model['producer_input'] = producer_input_model_model + + # Construct a json representation of a DeliveryMethod model + delivery_method_model_json = {} + delivery_method_model_json['id'] = '09cf5fcc-cb9d-4995-a8e4-16517b25229f' + delivery_method_model_json['container'] = container_reference_model + delivery_method_model_json['getproperties'] = delivery_method_properties_model_model + + # Construct a model instance of DeliveryMethod by calling from_dict on the json representation + delivery_method_model = DeliveryMethod.from_dict(delivery_method_model_json) + assert delivery_method_model != False + + # Construct a model instance of DeliveryMethod by calling from_dict on the json representation + delivery_method_model_dict = DeliveryMethod.from_dict(delivery_method_model_json).__dict__ + delivery_method_model2 = DeliveryMethod(**delivery_method_model_dict) + + # Verify the model instances are equivalent + assert delivery_method_model == delivery_method_model2 + + # Convert model instance back to dict and verify no loss of data + delivery_method_model_json2 = delivery_method_model.to_dict() + assert delivery_method_model_json2 == delivery_method_model_json + + +class TestModel_DeliveryMethodPropertiesModel: + """ + Test Class for DeliveryMethodPropertiesModel + """ + + def test_delivery_method_properties_model_serialization(self): + """ + Test serialization/deserialization for DeliveryMethodPropertiesModel + """ + + # Construct dict forms of any model objects needed in order to build this model. + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + producer_input_model_model = {} # ProducerInputModel + producer_input_model_model['engine_details'] = engine_details_model_model + producer_input_model_model['engines'] = [engine_details_model_model] + + # Construct a json representation of a DeliveryMethodPropertiesModel model + delivery_method_properties_model_model_json = {} + delivery_method_properties_model_model_json['producer_input'] = producer_input_model_model + + # Construct a model instance of DeliveryMethodPropertiesModel by calling from_dict on the json representation + delivery_method_properties_model_model = DeliveryMethodPropertiesModel.from_dict(delivery_method_properties_model_model_json) + assert delivery_method_properties_model_model != False + + # Construct a model instance of DeliveryMethodPropertiesModel by calling from_dict on the json representation + delivery_method_properties_model_model_dict = DeliveryMethodPropertiesModel.from_dict(delivery_method_properties_model_model_json).__dict__ + delivery_method_properties_model_model2 = DeliveryMethodPropertiesModel(**delivery_method_properties_model_model_dict) + + # Verify the model instances are equivalent + assert delivery_method_properties_model_model == delivery_method_properties_model_model2 + + # Convert model instance back to dict and verify no loss of data + delivery_method_properties_model_model_json2 = delivery_method_properties_model_model.to_dict() + assert delivery_method_properties_model_model_json2 == delivery_method_properties_model_model_json + + +class TestModel_Description: + """ + Test Class for Description + """ + + def test_description_serialization(self): + """ + Test serialization/deserialization for Description + """ + + # Construct dict forms of any model objects needed in order to build this model. + + contract_terms_more_info_model = {} # ContractTermsMoreInfo + contract_terms_more_info_model['type'] = 'privacy-statement' + contract_terms_more_info_model['url'] = 'https://moreinfo.example.com' + + # Construct a json representation of a Description model + description_model_json = {} + description_model_json['purpose'] = 'Used for customer behavior analysis.' + description_model_json['limitations'] = 'Data cannot be used for marketing.' + description_model_json['usage'] = 'Data should be used only for analytics.' + description_model_json['more_info'] = [contract_terms_more_info_model] + description_model_json['custom_properties'] = '{"property1":"value1"}' + + # Construct a model instance of Description by calling from_dict on the json representation + description_model = Description.from_dict(description_model_json) + assert description_model != False + + # Construct a model instance of Description by calling from_dict on the json representation + description_model_dict = Description.from_dict(description_model_json).__dict__ + description_model2 = Description(**description_model_dict) + + # Verify the model instances are equivalent + assert description_model == description_model2 + + # Convert model instance back to dict and verify no loss of data + description_model_json2 = description_model.to_dict() + assert description_model_json2 == description_model_json + + +class TestModel_Domain: + """ + Test Class for Domain + """ + + def test_domain_serialization(self): + """ + Test serialization/deserialization for Domain + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a Domain model + domain_model_json = {} + domain_model_json['id'] = 'testString' + domain_model_json['name'] = 'testString' + domain_model_json['container'] = container_reference_model + + # Construct a model instance of Domain by calling from_dict on the json representation + domain_model = Domain.from_dict(domain_model_json) + assert domain_model != False + + # Construct a model instance of Domain by calling from_dict on the json representation + domain_model_dict = Domain.from_dict(domain_model_json).__dict__ + domain_model2 = Domain(**domain_model_dict) + + # Verify the model instances are equivalent + assert domain_model == domain_model2 + + # Convert model instance back to dict and verify no loss of data + domain_model_json2 = domain_model.to_dict() + assert domain_model_json2 == domain_model_json + + +class TestModel_EngineDetailsModel: + """ + Test Class for EngineDetailsModel + """ + + def test_engine_details_model_serialization(self): + """ + Test serialization/deserialization for EngineDetailsModel + """ + + # Construct a json representation of a EngineDetailsModel model + engine_details_model_model_json = {} + engine_details_model_model_json['display_name'] = 'Iceberg Engine' + engine_details_model_model_json['engine_id'] = 'presto767' + engine_details_model_model_json['engine_port'] = '34567' + engine_details_model_model_json['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model_json['engine_type'] = 'spark' + engine_details_model_model_json['associated_catalogs'] = ['testString'] + + # Construct a model instance of EngineDetailsModel by calling from_dict on the json representation + engine_details_model_model = EngineDetailsModel.from_dict(engine_details_model_model_json) + assert engine_details_model_model != False + + # Construct a model instance of EngineDetailsModel by calling from_dict on the json representation + engine_details_model_model_dict = EngineDetailsModel.from_dict(engine_details_model_model_json).__dict__ + engine_details_model_model2 = EngineDetailsModel(**engine_details_model_model_dict) + + # Verify the model instances are equivalent + assert engine_details_model_model == engine_details_model_model2 + + # Convert model instance back to dict and verify no loss of data + engine_details_model_model_json2 = engine_details_model_model.to_dict() + assert engine_details_model_model_json2 == engine_details_model_model_json + + +class TestModel_ErrorExtraResource: + """ + Test Class for ErrorExtraResource + """ + + def test_error_extra_resource_serialization(self): + """ + Test serialization/deserialization for ErrorExtraResource + """ + + # Construct a json representation of a ErrorExtraResource model + error_extra_resource_model_json = {} + error_extra_resource_model_json['id'] = 'testString' + error_extra_resource_model_json['timestamp'] = '2019-01-01T12:00:00Z' + error_extra_resource_model_json['environment_name'] = 'testString' + error_extra_resource_model_json['http_status'] = 0 + error_extra_resource_model_json['source_cluster'] = 0 + error_extra_resource_model_json['source_component'] = 0 + error_extra_resource_model_json['transaction_id'] = 0 + + # Construct a model instance of ErrorExtraResource by calling from_dict on the json representation + error_extra_resource_model = ErrorExtraResource.from_dict(error_extra_resource_model_json) + assert error_extra_resource_model != False + + # Construct a model instance of ErrorExtraResource by calling from_dict on the json representation + error_extra_resource_model_dict = ErrorExtraResource.from_dict(error_extra_resource_model_json).__dict__ + error_extra_resource_model2 = ErrorExtraResource(**error_extra_resource_model_dict) + + # Verify the model instances are equivalent + assert error_extra_resource_model == error_extra_resource_model2 + + # Convert model instance back to dict and verify no loss of data + error_extra_resource_model_json2 = error_extra_resource_model.to_dict() + assert error_extra_resource_model_json2 == error_extra_resource_model_json + + +class TestModel_ErrorMessage: + """ + Test Class for ErrorMessage + """ + + def test_error_message_serialization(self): + """ + Test serialization/deserialization for ErrorMessage + """ + + # Construct a json representation of a ErrorMessage model + error_message_model_json = {} + error_message_model_json['code'] = 'testString' + error_message_model_json['message'] = 'testString' + + # Construct a model instance of ErrorMessage by calling from_dict on the json representation + error_message_model = ErrorMessage.from_dict(error_message_model_json) + assert error_message_model != False + + # Construct a model instance of ErrorMessage by calling from_dict on the json representation + error_message_model_dict = ErrorMessage.from_dict(error_message_model_json).__dict__ + error_message_model2 = ErrorMessage(**error_message_model_dict) + + # Verify the model instances are equivalent + assert error_message_model == error_message_model2 + + # Convert model instance back to dict and verify no loss of data + error_message_model_json2 = error_message_model.to_dict() + assert error_message_model_json2 == error_message_model_json + + +class TestModel_ErrorModelResource: + """ + Test Class for ErrorModelResource + """ + + def test_error_model_resource_serialization(self): + """ + Test serialization/deserialization for ErrorModelResource + """ + + # Construct dict forms of any model objects needed in order to build this model. + + error_extra_resource_model = {} # ErrorExtraResource + error_extra_resource_model['id'] = 'testString' + error_extra_resource_model['timestamp'] = '2019-01-01T12:00:00Z' + error_extra_resource_model['environment_name'] = 'testString' + error_extra_resource_model['http_status'] = 0 + error_extra_resource_model['source_cluster'] = 0 + error_extra_resource_model['source_component'] = 0 + error_extra_resource_model['transaction_id'] = 0 + + # Construct a json representation of a ErrorModelResource model + error_model_resource_model_json = {} + error_model_resource_model_json['code'] = 'request_body_error' + error_model_resource_model_json['message'] = 'testString' + error_model_resource_model_json['extra'] = error_extra_resource_model + error_model_resource_model_json['more_info'] = 'testString' + + # Construct a model instance of ErrorModelResource by calling from_dict on the json representation + error_model_resource_model = ErrorModelResource.from_dict(error_model_resource_model_json) + assert error_model_resource_model != False + + # Construct a model instance of ErrorModelResource by calling from_dict on the json representation + error_model_resource_model_dict = ErrorModelResource.from_dict(error_model_resource_model_json).__dict__ + error_model_resource_model2 = ErrorModelResource(**error_model_resource_model_dict) + + # Verify the model instances are equivalent + assert error_model_resource_model == error_model_resource_model2 + + # Convert model instance back to dict and verify no loss of data + error_model_resource_model_json2 = error_model_resource_model.to_dict() + assert error_model_resource_model_json2 == error_model_resource_model_json + + +class TestModel_FirstPage: + """ + Test Class for FirstPage + """ + + def test_first_page_serialization(self): + """ + Test serialization/deserialization for FirstPage + """ + + # Construct a json representation of a FirstPage model + first_page_model_json = {} + first_page_model_json['href'] = 'https://api.example.com/collection' + + # Construct a model instance of FirstPage by calling from_dict on the json representation + first_page_model = FirstPage.from_dict(first_page_model_json) + assert first_page_model != False + + # Construct a model instance of FirstPage by calling from_dict on the json representation + first_page_model_dict = FirstPage.from_dict(first_page_model_json).__dict__ + first_page_model2 = FirstPage(**first_page_model_dict) + + # Verify the model instances are equivalent + assert first_page_model == first_page_model2 + + # Convert model instance back to dict and verify no loss of data + first_page_model_json2 = first_page_model.to_dict() + assert first_page_model_json2 == first_page_model_json + + +class TestModel_InitializeResource: + """ + Test Class for InitializeResource + """ + + def test_initialize_resource_serialization(self): + """ + Test serialization/deserialization for InitializeResource + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + error_extra_resource_model = {} # ErrorExtraResource + error_extra_resource_model['id'] = 'testString' + error_extra_resource_model['timestamp'] = '2019-01-01T12:00:00Z' + error_extra_resource_model['environment_name'] = 'testString' + error_extra_resource_model['http_status'] = 0 + error_extra_resource_model['source_cluster'] = 0 + error_extra_resource_model['source_component'] = 0 + error_extra_resource_model['transaction_id'] = 0 + + error_model_resource_model = {} # ErrorModelResource + error_model_resource_model['code'] = 'request_body_error' + error_model_resource_model['message'] = 'testString' + error_model_resource_model['extra'] = error_extra_resource_model + error_model_resource_model['more_info'] = 'testString' + + initialized_option_model = {} # InitializedOption + initialized_option_model['name'] = 'testString' + initialized_option_model['version'] = 1 + + workflow_definition_reference_model = {} # WorkflowDefinitionReference + workflow_definition_reference_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + provided_workflow_resource_model = {} # ProvidedWorkflowResource + provided_workflow_resource_model['definition'] = workflow_definition_reference_model + + provided_catalog_workflows_model = {} # ProvidedCatalogWorkflows + provided_catalog_workflows_model['data_access'] = provided_workflow_resource_model + provided_catalog_workflows_model['request_new_product'] = provided_workflow_resource_model + + # Construct a json representation of a InitializeResource model + initialize_resource_model_json = {} + initialize_resource_model_json['container'] = container_reference_model + initialize_resource_model_json['href'] = 'https://api.example.com/configuration/initialize/status?catalog_id=d29c42eb-7100-4b7a-8257-c196dbcca1cd' + initialize_resource_model_json['status'] = 'not_started' + initialize_resource_model_json['trace'] = 'testString' + initialize_resource_model_json['errors'] = [error_model_resource_model] + initialize_resource_model_json['last_started_at'] = '2023-08-21T15:24:06.021000Z' + initialize_resource_model_json['last_finished_at'] = '2023-08-21T20:24:34.450000Z' + initialize_resource_model_json['initialized_options'] = [initialized_option_model] + initialize_resource_model_json['workflows'] = provided_catalog_workflows_model + + # Construct a model instance of InitializeResource by calling from_dict on the json representation + initialize_resource_model = InitializeResource.from_dict(initialize_resource_model_json) + assert initialize_resource_model != False + + # Construct a model instance of InitializeResource by calling from_dict on the json representation + initialize_resource_model_dict = InitializeResource.from_dict(initialize_resource_model_json).__dict__ + initialize_resource_model2 = InitializeResource(**initialize_resource_model_dict) + + # Verify the model instances are equivalent + assert initialize_resource_model == initialize_resource_model2 + + # Convert model instance back to dict and verify no loss of data + initialize_resource_model_json2 = initialize_resource_model.to_dict() + assert initialize_resource_model_json2 == initialize_resource_model_json + + +class TestModel_InitializeSubDomain: + """ + Test Class for InitializeSubDomain + """ + + def test_initialize_sub_domain_serialization(self): + """ + Test serialization/deserialization for InitializeSubDomain + """ + + # Construct a json representation of a InitializeSubDomain model + initialize_sub_domain_model_json = {} + initialize_sub_domain_model_json['name'] = 'Operations' + initialize_sub_domain_model_json['id'] = 'testString' + initialize_sub_domain_model_json['description'] = 'testString' + + # Construct a model instance of InitializeSubDomain by calling from_dict on the json representation + initialize_sub_domain_model = InitializeSubDomain.from_dict(initialize_sub_domain_model_json) + assert initialize_sub_domain_model != False + + # Construct a model instance of InitializeSubDomain by calling from_dict on the json representation + initialize_sub_domain_model_dict = InitializeSubDomain.from_dict(initialize_sub_domain_model_json).__dict__ + initialize_sub_domain_model2 = InitializeSubDomain(**initialize_sub_domain_model_dict) + + # Verify the model instances are equivalent + assert initialize_sub_domain_model == initialize_sub_domain_model2 + + # Convert model instance back to dict and verify no loss of data + initialize_sub_domain_model_json2 = initialize_sub_domain_model.to_dict() + assert initialize_sub_domain_model_json2 == initialize_sub_domain_model_json + + +class TestModel_InitializedOption: + """ + Test Class for InitializedOption + """ + + def test_initialized_option_serialization(self): + """ + Test serialization/deserialization for InitializedOption + """ + + # Construct a json representation of a InitializedOption model + initialized_option_model_json = {} + initialized_option_model_json['name'] = 'testString' + initialized_option_model_json['version'] = 1 + + # Construct a model instance of InitializedOption by calling from_dict on the json representation + initialized_option_model = InitializedOption.from_dict(initialized_option_model_json) + assert initialized_option_model != False + + # Construct a model instance of InitializedOption by calling from_dict on the json representation + initialized_option_model_dict = InitializedOption.from_dict(initialized_option_model_json).__dict__ + initialized_option_model2 = InitializedOption(**initialized_option_model_dict) + + # Verify the model instances are equivalent + assert initialized_option_model == initialized_option_model2 + + # Convert model instance back to dict and verify no loss of data + initialized_option_model_json2 = initialized_option_model.to_dict() + assert initialized_option_model_json2 == initialized_option_model_json + + +class TestModel_JsonPatchOperation: + """ + Test Class for JsonPatchOperation + """ + + def test_json_patch_operation_serialization(self): + """ + Test serialization/deserialization for JsonPatchOperation + """ + + # Construct a json representation of a JsonPatchOperation model + json_patch_operation_model_json = {} + json_patch_operation_model_json['op'] = 'add' + json_patch_operation_model_json['path'] = 'testString' + json_patch_operation_model_json['from'] = 'testString' + json_patch_operation_model_json['value'] = 'testString' + + # Construct a model instance of JsonPatchOperation by calling from_dict on the json representation + json_patch_operation_model = JsonPatchOperation.from_dict(json_patch_operation_model_json) + assert json_patch_operation_model != False + + # Construct a model instance of JsonPatchOperation by calling from_dict on the json representation + json_patch_operation_model_dict = JsonPatchOperation.from_dict(json_patch_operation_model_json).__dict__ + json_patch_operation_model2 = JsonPatchOperation(**json_patch_operation_model_dict) + + # Verify the model instances are equivalent + assert json_patch_operation_model == json_patch_operation_model2 + + # Convert model instance back to dict and verify no loss of data + json_patch_operation_model_json2 = json_patch_operation_model.to_dict() + assert json_patch_operation_model_json2 == json_patch_operation_model_json + + +class TestModel_MemberRolesSchema: + """ + Test Class for MemberRolesSchema + """ + + def test_member_roles_schema_serialization(self): + """ + Test serialization/deserialization for MemberRolesSchema + """ + + # Construct a json representation of a MemberRolesSchema model + member_roles_schema_model_json = {} + member_roles_schema_model_json['user_iam_id'] = 'testString' + member_roles_schema_model_json['roles'] = ['testString'] + + # Construct a model instance of MemberRolesSchema by calling from_dict on the json representation + member_roles_schema_model = MemberRolesSchema.from_dict(member_roles_schema_model_json) + assert member_roles_schema_model != False + + # Construct a model instance of MemberRolesSchema by calling from_dict on the json representation + member_roles_schema_model_dict = MemberRolesSchema.from_dict(member_roles_schema_model_json).__dict__ + member_roles_schema_model2 = MemberRolesSchema(**member_roles_schema_model_dict) + + # Verify the model instances are equivalent + assert member_roles_schema_model == member_roles_schema_model2 + + # Convert model instance back to dict and verify no loss of data + member_roles_schema_model_json2 = member_roles_schema_model.to_dict() + assert member_roles_schema_model_json2 == member_roles_schema_model_json + + +class TestModel_NextPage: + """ + Test Class for NextPage + """ + + def test_next_page_serialization(self): + """ + Test serialization/deserialization for NextPage + """ + + # Construct a json representation of a NextPage model + next_page_model_json = {} + next_page_model_json['href'] = 'https://api.example.com/collection?start=eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + next_page_model_json['start'] = 'eyJvZmZzZXQiOjAsImRvbmUiOnRydWV9' + + # Construct a model instance of NextPage by calling from_dict on the json representation + next_page_model = NextPage.from_dict(next_page_model_json) + assert next_page_model != False + + # Construct a model instance of NextPage by calling from_dict on the json representation + next_page_model_dict = NextPage.from_dict(next_page_model_json).__dict__ + next_page_model2 = NextPage(**next_page_model_dict) + + # Verify the model instances are equivalent + assert next_page_model == next_page_model2 + + # Convert model instance back to dict and verify no loss of data + next_page_model_json2 = next_page_model.to_dict() + assert next_page_model_json2 == next_page_model_json + + +class TestModel_Overview: + """ + Test Class for Overview + """ + + def test_overview_serialization(self): + """ + Test serialization/deserialization for Overview + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + domain_model = {} # Domain + domain_model['id'] = 'testString' + domain_model['name'] = 'testString' + domain_model['container'] = container_reference_model + + # Construct a json representation of a Overview model + overview_model_json = {} + overview_model_json['api_version'] = 'v3.0.1' + overview_model_json['kind'] = 'DataContract' + overview_model_json['name'] = 'Sample Data Contract' + overview_model_json['version'] = '0.0.0' + overview_model_json['domain'] = domain_model + overview_model_json['more_info'] = 'List of links to sources that provide more details on the data contract.' + + # Construct a model instance of Overview by calling from_dict on the json representation + overview_model = Overview.from_dict(overview_model_json) + assert overview_model != False + + # Construct a model instance of Overview by calling from_dict on the json representation + overview_model_dict = Overview.from_dict(overview_model_json).__dict__ + overview_model2 = Overview(**overview_model_dict) + + # Verify the model instances are equivalent + assert overview_model == overview_model2 + + # Convert model instance back to dict and verify no loss of data + overview_model_json2 = overview_model.to_dict() + assert overview_model_json2 == overview_model_json + + +class TestModel_Pricing: + """ + Test Class for Pricing + """ + + def test_pricing_serialization(self): + """ + Test serialization/deserialization for Pricing + """ + + # Construct a json representation of a Pricing model + pricing_model_json = {} + pricing_model_json['amount'] = '100.0' + pricing_model_json['currency'] = 'USD' + pricing_model_json['unit'] = 'megabyte' + + # Construct a model instance of Pricing by calling from_dict on the json representation + pricing_model = Pricing.from_dict(pricing_model_json) + assert pricing_model != False + + # Construct a model instance of Pricing by calling from_dict on the json representation + pricing_model_dict = Pricing.from_dict(pricing_model_json).__dict__ + pricing_model2 = Pricing(**pricing_model_dict) + + # Verify the model instances are equivalent + assert pricing_model == pricing_model2 + + # Convert model instance back to dict and verify no loss of data + pricing_model_json2 = pricing_model.to_dict() + assert pricing_model_json2 == pricing_model_json + + +class TestModel_ProducerInputModel: + """ + Test Class for ProducerInputModel + """ + + def test_producer_input_model_serialization(self): + """ + Test serialization/deserialization for ProducerInputModel + """ + + # Construct dict forms of any model objects needed in order to build this model. + + engine_details_model_model = {} # EngineDetailsModel + engine_details_model_model['display_name'] = 'Iceberg Engine' + engine_details_model_model['engine_id'] = 'presto767' + engine_details_model_model['engine_port'] = '34567' + engine_details_model_model['engine_host'] = 'a109e0f6-2dfc-4954-a0ff-343d70f7da7b.someId.lakehouse.appdomain.cloud' + engine_details_model_model['engine_type'] = 'spark' + engine_details_model_model['associated_catalogs'] = ['testString'] + + # Construct a json representation of a ProducerInputModel model + producer_input_model_model_json = {} + producer_input_model_model_json['engine_details'] = engine_details_model_model + producer_input_model_model_json['engines'] = [engine_details_model_model] + + # Construct a model instance of ProducerInputModel by calling from_dict on the json representation + producer_input_model_model = ProducerInputModel.from_dict(producer_input_model_model_json) + assert producer_input_model_model != False + + # Construct a model instance of ProducerInputModel by calling from_dict on the json representation + producer_input_model_model_dict = ProducerInputModel.from_dict(producer_input_model_model_json).__dict__ + producer_input_model_model2 = ProducerInputModel(**producer_input_model_model_dict) + + # Verify the model instances are equivalent + assert producer_input_model_model == producer_input_model_model2 + + # Convert model instance back to dict and verify no loss of data + producer_input_model_model_json2 = producer_input_model_model.to_dict() + assert producer_input_model_model_json2 == producer_input_model_model_json + + +class TestModel_PropertiesSchema: + """ + Test Class for PropertiesSchema + """ + + def test_properties_schema_serialization(self): + """ + Test serialization/deserialization for PropertiesSchema + """ + + # Construct a json representation of a PropertiesSchema model + properties_schema_model_json = {} + properties_schema_model_json['value'] = 'testString' + + # Construct a model instance of PropertiesSchema by calling from_dict on the json representation + properties_schema_model = PropertiesSchema.from_dict(properties_schema_model_json) + assert properties_schema_model != False + + # Construct a model instance of PropertiesSchema by calling from_dict on the json representation + properties_schema_model_dict = PropertiesSchema.from_dict(properties_schema_model_json).__dict__ + properties_schema_model2 = PropertiesSchema(**properties_schema_model_dict) + + # Verify the model instances are equivalent + assert properties_schema_model == properties_schema_model2 + + # Convert model instance back to dict and verify no loss of data + properties_schema_model_json2 = properties_schema_model.to_dict() + assert properties_schema_model_json2 == properties_schema_model_json + + +class TestModel_ProvidedCatalogWorkflows: + """ + Test Class for ProvidedCatalogWorkflows + """ + + def test_provided_catalog_workflows_serialization(self): + """ + Test serialization/deserialization for ProvidedCatalogWorkflows + """ + + # Construct dict forms of any model objects needed in order to build this model. + + workflow_definition_reference_model = {} # WorkflowDefinitionReference + workflow_definition_reference_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + provided_workflow_resource_model = {} # ProvidedWorkflowResource + provided_workflow_resource_model['definition'] = workflow_definition_reference_model + + # Construct a json representation of a ProvidedCatalogWorkflows model + provided_catalog_workflows_model_json = {} + provided_catalog_workflows_model_json['data_access'] = provided_workflow_resource_model + provided_catalog_workflows_model_json['request_new_product'] = provided_workflow_resource_model + + # Construct a model instance of ProvidedCatalogWorkflows by calling from_dict on the json representation + provided_catalog_workflows_model = ProvidedCatalogWorkflows.from_dict(provided_catalog_workflows_model_json) + assert provided_catalog_workflows_model != False + + # Construct a model instance of ProvidedCatalogWorkflows by calling from_dict on the json representation + provided_catalog_workflows_model_dict = ProvidedCatalogWorkflows.from_dict(provided_catalog_workflows_model_json).__dict__ + provided_catalog_workflows_model2 = ProvidedCatalogWorkflows(**provided_catalog_workflows_model_dict) + + # Verify the model instances are equivalent + assert provided_catalog_workflows_model == provided_catalog_workflows_model2 + + # Convert model instance back to dict and verify no loss of data + provided_catalog_workflows_model_json2 = provided_catalog_workflows_model.to_dict() + assert provided_catalog_workflows_model_json2 == provided_catalog_workflows_model_json + + +class TestModel_ProvidedWorkflowResource: + """ + Test Class for ProvidedWorkflowResource + """ + + def test_provided_workflow_resource_serialization(self): + """ + Test serialization/deserialization for ProvidedWorkflowResource + """ + + # Construct dict forms of any model objects needed in order to build this model. + + workflow_definition_reference_model = {} # WorkflowDefinitionReference + workflow_definition_reference_model['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a json representation of a ProvidedWorkflowResource model + provided_workflow_resource_model_json = {} + provided_workflow_resource_model_json['definition'] = workflow_definition_reference_model + + # Construct a model instance of ProvidedWorkflowResource by calling from_dict on the json representation + provided_workflow_resource_model = ProvidedWorkflowResource.from_dict(provided_workflow_resource_model_json) + assert provided_workflow_resource_model != False + + # Construct a model instance of ProvidedWorkflowResource by calling from_dict on the json representation + provided_workflow_resource_model_dict = ProvidedWorkflowResource.from_dict(provided_workflow_resource_model_json).__dict__ + provided_workflow_resource_model2 = ProvidedWorkflowResource(**provided_workflow_resource_model_dict) + + # Verify the model instances are equivalent + assert provided_workflow_resource_model == provided_workflow_resource_model2 + + # Convert model instance back to dict and verify no loss of data + provided_workflow_resource_model_json2 = provided_workflow_resource_model.to_dict() + assert provided_workflow_resource_model_json2 == provided_workflow_resource_model_json + + +class TestModel_RevokeAccessResponse: + """ + Test Class for RevokeAccessResponse + """ + + def test_revoke_access_response_serialization(self): + """ + Test serialization/deserialization for RevokeAccessResponse + """ + + # Construct a json representation of a RevokeAccessResponse model + revoke_access_response_model_json = {} + revoke_access_response_model_json['message'] = 'testString' + + # Construct a model instance of RevokeAccessResponse by calling from_dict on the json representation + revoke_access_response_model = RevokeAccessResponse.from_dict(revoke_access_response_model_json) + assert revoke_access_response_model != False + + # Construct a model instance of RevokeAccessResponse by calling from_dict on the json representation + revoke_access_response_model_dict = RevokeAccessResponse.from_dict(revoke_access_response_model_json).__dict__ + revoke_access_response_model2 = RevokeAccessResponse(**revoke_access_response_model_dict) + + # Verify the model instances are equivalent + assert revoke_access_response_model == revoke_access_response_model2 + + # Convert model instance back to dict and verify no loss of data + revoke_access_response_model_json2 = revoke_access_response_model.to_dict() + assert revoke_access_response_model_json2 == revoke_access_response_model_json + + +class TestModel_RevokeAccessStateResponse: + """ + Test Class for RevokeAccessStateResponse + """ + + def test_revoke_access_state_response_serialization(self): + """ + Test serialization/deserialization for RevokeAccessStateResponse + """ + + # Construct dict forms of any model objects needed in order to build this model. + + asset_model = {} # Asset + asset_model['metadata'] = {'anyKey': 'anyValue'} + asset_model['entity'] = {'anyKey': 'anyValue'} + + search_asset_pagination_info_model = {} # SearchAssetPaginationInfo + search_asset_pagination_info_model['query'] = 'ibm_data_product_revoke_access.state:(SCHEDULED OR FAILED)' + search_asset_pagination_info_model['limit'] = 1 + search_asset_pagination_info_model['bookmark'] = 'MQ==' + search_asset_pagination_info_model['include'] = 'entity' + search_asset_pagination_info_model['skip'] = 0 + + # Construct a json representation of a RevokeAccessStateResponse model + revoke_access_state_response_model_json = {} + revoke_access_state_response_model_json['results'] = [asset_model] + revoke_access_state_response_model_json['total_count'] = 42 + revoke_access_state_response_model_json['next'] = search_asset_pagination_info_model + + # Construct a model instance of RevokeAccessStateResponse by calling from_dict on the json representation + revoke_access_state_response_model = RevokeAccessStateResponse.from_dict(revoke_access_state_response_model_json) + assert revoke_access_state_response_model != False + + # Construct a model instance of RevokeAccessStateResponse by calling from_dict on the json representation + revoke_access_state_response_model_dict = RevokeAccessStateResponse.from_dict(revoke_access_state_response_model_json).__dict__ + revoke_access_state_response_model2 = RevokeAccessStateResponse(**revoke_access_state_response_model_dict) + + # Verify the model instances are equivalent + assert revoke_access_state_response_model == revoke_access_state_response_model2 + + # Convert model instance back to dict and verify no loss of data + revoke_access_state_response_model_json2 = revoke_access_state_response_model.to_dict() + assert revoke_access_state_response_model_json2 == revoke_access_state_response_model_json + + +class TestModel_Roles: + """ + Test Class for Roles + """ + + def test_roles_serialization(self): + """ + Test serialization/deserialization for Roles + """ + + # Construct a json representation of a Roles model + roles_model_json = {} + roles_model_json['role'] = 'owner' + + # Construct a model instance of Roles by calling from_dict on the json representation + roles_model = Roles.from_dict(roles_model_json) + assert roles_model != False + + # Construct a model instance of Roles by calling from_dict on the json representation + roles_model_dict = Roles.from_dict(roles_model_json).__dict__ + roles_model2 = Roles(**roles_model_dict) + + # Verify the model instances are equivalent + assert roles_model == roles_model2 + + # Convert model instance back to dict and verify no loss of data + roles_model_json2 = roles_model.to_dict() + assert roles_model_json2 == roles_model_json + + +class TestModel_SearchAssetPaginationInfo: + """ + Test Class for SearchAssetPaginationInfo + """ + + def test_search_asset_pagination_info_serialization(self): + """ + Test serialization/deserialization for SearchAssetPaginationInfo + """ + + # Construct a json representation of a SearchAssetPaginationInfo model + search_asset_pagination_info_model_json = {} + search_asset_pagination_info_model_json['query'] = 'ibm_data_product_revoke_access.state:(SCHEDULED OR FAILED)' + search_asset_pagination_info_model_json['limit'] = 1 + search_asset_pagination_info_model_json['bookmark'] = 'MQ==' + search_asset_pagination_info_model_json['include'] = 'entity' + search_asset_pagination_info_model_json['skip'] = 0 + + # Construct a model instance of SearchAssetPaginationInfo by calling from_dict on the json representation + search_asset_pagination_info_model = SearchAssetPaginationInfo.from_dict(search_asset_pagination_info_model_json) + assert search_asset_pagination_info_model != False + + # Construct a model instance of SearchAssetPaginationInfo by calling from_dict on the json representation + search_asset_pagination_info_model_dict = SearchAssetPaginationInfo.from_dict(search_asset_pagination_info_model_json).__dict__ + search_asset_pagination_info_model2 = SearchAssetPaginationInfo(**search_asset_pagination_info_model_dict) + + # Verify the model instances are equivalent + assert search_asset_pagination_info_model == search_asset_pagination_info_model2 + + # Convert model instance back to dict and verify no loss of data + search_asset_pagination_info_model_json2 = search_asset_pagination_info_model.to_dict() + assert search_asset_pagination_info_model_json2 == search_asset_pagination_info_model_json + + +class TestModel_ServiceIdCredentials: + """ + Test Class for ServiceIdCredentials + """ + + def test_service_id_credentials_serialization(self): + """ + Test serialization/deserialization for ServiceIdCredentials + """ + + # Construct a json representation of a ServiceIdCredentials model + service_id_credentials_model_json = {} + service_id_credentials_model_json['name'] = 'data-product-admin-service-id-API-key' + service_id_credentials_model_json['created_at'] = '2019-01-01T12:00:00Z' + + # Construct a model instance of ServiceIdCredentials by calling from_dict on the json representation + service_id_credentials_model = ServiceIdCredentials.from_dict(service_id_credentials_model_json) + assert service_id_credentials_model != False + + # Construct a model instance of ServiceIdCredentials by calling from_dict on the json representation + service_id_credentials_model_dict = ServiceIdCredentials.from_dict(service_id_credentials_model_json).__dict__ + service_id_credentials_model2 = ServiceIdCredentials(**service_id_credentials_model_dict) + + # Verify the model instances are equivalent + assert service_id_credentials_model == service_id_credentials_model2 + + # Convert model instance back to dict and verify no loss of data + service_id_credentials_model_json2 = service_id_credentials_model.to_dict() + assert service_id_credentials_model_json2 == service_id_credentials_model_json + + +class TestModel_UseCase: + """ + Test Class for UseCase + """ + + def test_use_case_serialization(self): + """ + Test serialization/deserialization for UseCase + """ + + # Construct dict forms of any model objects needed in order to build this model. + + container_reference_model = {} # ContainerReference + container_reference_model['id'] = 'd29c42eb-7100-4b7a-8257-c196dbcca1cd' + container_reference_model['type'] = 'catalog' + + # Construct a json representation of a UseCase model + use_case_model_json = {} + use_case_model_json['id'] = 'testString' + use_case_model_json['name'] = 'testString' + use_case_model_json['container'] = container_reference_model + + # Construct a model instance of UseCase by calling from_dict on the json representation + use_case_model = UseCase.from_dict(use_case_model_json) + assert use_case_model != False + + # Construct a model instance of UseCase by calling from_dict on the json representation + use_case_model_dict = UseCase.from_dict(use_case_model_json).__dict__ + use_case_model2 = UseCase(**use_case_model_dict) + + # Verify the model instances are equivalent + assert use_case_model == use_case_model2 + + # Convert model instance back to dict and verify no loss of data + use_case_model_json2 = use_case_model.to_dict() + assert use_case_model_json2 == use_case_model_json + + +class TestModel_Visualization: + """ + Test Class for Visualization + """ + + def test_visualization_serialization(self): + """ + Test serialization/deserialization for Visualization + """ + + # Construct a json representation of a Visualization model + visualization_model_json = {} + visualization_model_json['id'] = 'testString' + visualization_model_json['name'] = 'testString' + + # Construct a model instance of Visualization by calling from_dict on the json representation + visualization_model = Visualization.from_dict(visualization_model_json) + assert visualization_model != False + + # Construct a model instance of Visualization by calling from_dict on the json representation + visualization_model_dict = Visualization.from_dict(visualization_model_json).__dict__ + visualization_model2 = Visualization(**visualization_model_dict) + + # Verify the model instances are equivalent + assert visualization_model == visualization_model2 + + # Convert model instance back to dict and verify no loss of data + visualization_model_json2 = visualization_model.to_dict() + assert visualization_model_json2 == visualization_model_json + + +class TestModel_WorkflowDefinitionReference: + """ + Test Class for WorkflowDefinitionReference + """ + + def test_workflow_definition_reference_serialization(self): + """ + Test serialization/deserialization for WorkflowDefinitionReference + """ + + # Construct a json representation of a WorkflowDefinitionReference model + workflow_definition_reference_model_json = {} + workflow_definition_reference_model_json['id'] = '18bdbde1-918e-4ecf-aa23-6727bf319e14' + + # Construct a model instance of WorkflowDefinitionReference by calling from_dict on the json representation + workflow_definition_reference_model = WorkflowDefinitionReference.from_dict(workflow_definition_reference_model_json) + assert workflow_definition_reference_model != False + + # Construct a model instance of WorkflowDefinitionReference by calling from_dict on the json representation + workflow_definition_reference_model_dict = WorkflowDefinitionReference.from_dict(workflow_definition_reference_model_json).__dict__ + workflow_definition_reference_model2 = WorkflowDefinitionReference(**workflow_definition_reference_model_dict) + + # Verify the model instances are equivalent + assert workflow_definition_reference_model == workflow_definition_reference_model2 + + # Convert model instance back to dict and verify no loss of data + workflow_definition_reference_model_json2 = workflow_definition_reference_model.to_dict() + assert workflow_definition_reference_model_json2 == workflow_definition_reference_model_json + + +# endregion +############################################################################## +# End of Model Tests +############################################################################## diff --git a/tests/src/dq_validator/__init__.py b/tests/src/dq_validator/__init__.py new file mode 100644 index 0000000..b646327 --- /dev/null +++ b/tests/src/dq_validator/__init__.py @@ -0,0 +1,20 @@ +# Copyright 2026 IBM Corporation +# Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# See the LICENSE file in the project root for license information. + +""" +Data quality validator tests +""" + +# This file is only here to get pylint to check the files in this directory diff --git a/tests/src/dq_validator/provider/test_assets.py b/tests/src/dq_validator/provider/test_assets.py new file mode 100644 index 0000000..15070a0 --- /dev/null +++ b/tests/src/dq_validator/provider/test_assets.py @@ -0,0 +1,445 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" + +import pytest +from unittest.mock import Mock, patch +import json + +from wxdi.dq_validator.provider import ProviderConfig, DQAssetsProvider + + +class TestDQAssetsProvider: + """Test suite for DQAssetsProvider class.""" + + @pytest.fixture + def config(self): + """Create a test configuration.""" + return ProviderConfig( + url="https://test-instance.com", + auth_token="Bearer test-token" + ) + + @pytest.fixture + def provider(self, config): + """Create a test DQAssetsProvider instance.""" + with patch('wxdi.dq_validator.provider.base_provider.Session') as mock_session_class: + mock_session = Mock() + mock_session_class.return_value = mock_session + provider = DQAssetsProvider(config) + yield provider + + # ==================== get_assets Tests ==================== + + def test_get_assets_with_project_id(self, provider): + """Test getting assets with project_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [ + { + "id": "asset-1", + "name": "Table1", + "type": "table" + }, + { + "id": "asset-2", + "name": "Column1", + "type": "column" + } + ], + "total_count": 2 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets(project_id="project-123") + + # Verify + assert "assets" in result + assert len(result["assets"]) == 2 + assert result["assets"][0]["id"] == "asset-1" + assert result["total_count"] == 2 + + # Verify the API call + provider.session.get.assert_called_once() + call_args = provider.session.get.call_args + + # Check URL + assert "https://test-instance.com/data_quality/v4/assets" in call_args[0][0] + assert "project_id=project-123" in call_args[0][0] + + # Check headers + headers = call_args[1]["headers"] + assert headers["Authorization"] == "Bearer test-token" + assert headers["Content-Type"] == "application/json" + + def test_get_assets_with_catalog_id(self, provider): + """Test getting assets with catalog_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [ + { + "id": "asset-3", + "name": "Table2", + "type": "table" + } + ], + "total_count": 1 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets(catalog_id="catalog-456") + + # Verify + assert "assets" in result + assert len(result["assets"]) == 1 + assert result["assets"][0]["id"] == "asset-3" + + # Verify the API call + call_args = provider.session.get.call_args + + # Check URL contains catalog_id + assert "catalog_id=catalog-456" in call_args[0][0] + assert "project_id" not in call_args[0][0] + + def test_get_assets_with_all_optional_params(self, provider): + """Test getting assets with all optional parameters.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [], + "total_count": 0, + "next": { + "start": "next-token" + } + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets( + project_id="project-123", + start="start-token", + limit=50, + include_children=True, + asset_type="column" + ) + + # Verify + assert "assets" in result + assert result["total_count"] == 0 + + # Verify the API call + call_args = provider.session.get.call_args + + # Check URL contains all parameters + assert "project_id=project-123" in call_args[0][0] + assert "start=start-token" in call_args[0][0] + assert "limit=50" in call_args[0][0] + assert "include_children=true" in call_args[0][0] + assert "type=column" in call_args[0][0] + + def test_get_assets_with_include_children_false(self, provider): + """Test getting assets with include_children=False.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [], + "total_count": 0 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets( + project_id="project-123", + include_children=False + ) + + # Verify + assert "assets" in result + + # Verify the API call + call_args = provider.session.get.call_args + + # Check URL contains include_children=false + assert "include_children=false" in call_args[0][0] + + def test_get_assets_with_limit(self, provider): + """Test getting assets with limit parameter.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [{"id": f"asset-{i}"} for i in range(10)], + "total_count": 10 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets( + project_id="project-123", + limit=10 + ) + + # Verify + assert len(result["assets"]) == 10 + + # Verify the API call + call_args = provider.session.get.call_args + + # Check URL contains limit + assert "limit=10" in call_args[0][0] + + def test_get_assets_with_start_token(self, provider): + """Test getting assets with start token for pagination.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [{"id": "asset-next"}], + "total_count": 1 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets( + project_id="project-123", + start="pagination-token-123" + ) + + # Verify + assert len(result["assets"]) == 1 + + # Verify the API call + call_args = provider.session.get.call_args + + # Check URL contains start token + assert "start=pagination-token-123" in call_args[0][0] + + def test_get_assets_with_asset_type_table(self, provider): + """Test getting assets filtered by type=table.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [ + {"id": "asset-1", "type": "table"}, + {"id": "asset-2", "type": "table"} + ], + "total_count": 2 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets( + project_id="project-123", + asset_type="table" + ) + + # Verify + assert len(result["assets"]) == 2 + assert all(asset["type"] == "table" for asset in result["assets"]) + + # Verify the API call + call_args = provider.session.get.call_args + + # Check URL contains type parameter + assert "type=table" in call_args[0][0] + + def test_get_assets_with_asset_type_column(self, provider): + """Test getting assets filtered by type=column.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [ + {"id": "asset-col-1", "type": "column"}, + {"id": "asset-col-2", "type": "column"} + ], + "total_count": 2 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets( + catalog_id="catalog-456", + asset_type="column" + ) + + # Verify + assert len(result["assets"]) == 2 + assert all(asset["type"] == "column" for asset in result["assets"]) + + # Verify the API call + call_args = provider.session.get.call_args + + # Check URL contains type parameter + assert "type=column" in call_args[0][0] + + def test_get_assets_missing_both_ids(self, provider): + """Test getting assets without project_id or catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.get_assets() + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_get_assets_both_ids_provided(self, provider): + """Test getting assets with both project_id and catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.get_assets( + project_id="project-123", + catalog_id="catalog-456" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + def test_get_assets_api_failure(self, provider): + """Test failed get assets request.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 404 + mock_response.text = "Project not found" + provider.session.get.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.get_assets(project_id="invalid-project") + + assert "Failed to get assets" in str(exc_info.value) + assert "404" in str(exc_info.value) + assert "Project not found" in str(exc_info.value) + + def test_get_assets_empty_result(self, provider): + """Test getting assets with no results.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [], + "total_count": 0 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets(project_id="project-empty") + + # Verify + assert result["assets"] == [] + assert result["total_count"] == 0 + + def test_get_assets_with_pagination_info(self, provider): + """Test getting assets with pagination information.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [{"id": f"asset-{i}"} for i in range(100)], + "total_count": 250, + "next": { + "start": "next-page-token" + } + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets( + project_id="project-123", + limit=100 + ) + + # Verify + assert len(result["assets"]) == 100 + assert result["total_count"] == 250 + assert "next" in result + assert result["next"]["start"] == "next-page-token" + + def test_get_assets_unauthorized(self, provider): + """Test getting assets with unauthorized access.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 401 + mock_response.text = "Unauthorized" + provider.session.get.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.get_assets(project_id="project-123") + + assert "Failed to get assets" in str(exc_info.value) + assert "401" in str(exc_info.value) + + def test_get_assets_server_error(self, provider): + """Test getting assets with server error.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 500 + mock_response.text = "Internal server error" + provider.session.get.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.get_assets(catalog_id="catalog-123") + + assert "Failed to get assets" in str(exc_info.value) + assert "500" in str(exc_info.value) + assert "Internal server error" in str(exc_info.value) + + def test_get_assets_with_complex_response(self, provider): + """Test getting assets with complex nested response.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "assets": [ + { + "id": "asset-complex-1", + "name": "ComplexTable", + "type": "table", + "metadata": { + "columns": ["col1", "col2"], + "row_count": 1000 + }, + "children": [ + {"id": "child-1", "type": "column"}, + {"id": "child-2", "type": "column"} + ] + } + ], + "total_count": 1 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_assets( + project_id="project-123", + include_children=True + ) + + # Verify + assert len(result["assets"]) == 1 + asset = result["assets"][0] + assert asset["id"] == "asset-complex-1" + assert "metadata" in asset + assert "children" in asset + assert len(asset["children"]) == 2 \ No newline at end of file diff --git a/tests/src/dq_validator/provider/test_cams.py b/tests/src/dq_validator/provider/test_cams.py new file mode 100644 index 0000000..05e2856 --- /dev/null +++ b/tests/src/dq_validator/provider/test_cams.py @@ -0,0 +1,513 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Test suite for CamsProvider module +""" + +import pytest +import json +from pathlib import Path +from wxdi.dq_validator.provider.cams import CamsProvider +from wxdi.dq_validator.provider.config import ProviderConfig +from wxdi.dq_validator.provider.data_asset_model import DataAsset + + +class TestCamsProvider: + """Test cases for CamsProvider class""" + + @pytest.fixture + def base_url(self): + """Base URL for API""" + return "https://api.example.com" + + @pytest.fixture + def auth_token(self): + """Authentication token""" + return "Bearer test-token-12345" + + @pytest.fixture + def project_id(self): + """Project ID""" + return "72d21c1d-499b-4784-a3c7-6f84507f9a20" + + @pytest.fixture + def catalog_id(self): + """Catalog ID""" + return "a1b2c3d4-e5f6-7890-abcd-ef1234567890" + + @pytest.fixture + def config_with_project(self, base_url, auth_token, project_id): + """Create ProviderConfig with project_id""" + return ProviderConfig(base_url, auth_token, project_id=project_id) + + @pytest.fixture + def config_with_catalog(self, base_url, auth_token, catalog_id): + """Create ProviderConfig with catalog_id""" + return ProviderConfig(base_url, auth_token, catalog_id=catalog_id) + + @pytest.fixture + def config_without_project_or_catalog(self, base_url, auth_token): + """Create ProviderConfig without project_id or catalog_id - should fail""" + return ProviderConfig(base_url, auth_token) + + @pytest.fixture + def data_asset_json(self): + """Load the data asset response JSON""" + json_path = Path(__file__).parent.parent.parent.parent / "data" / "data_asset_response.json" + with open(json_path, "r") as f: + return json.load(f) + + @pytest.fixture + def mock_data_asset(self, data_asset_json): + """Create DataAsset from JSON""" + return DataAsset.from_dict(data_asset_json) + + def test_provider_initialization(self, config_with_project): + """Test CamsProvider initialization""" + provider = CamsProvider(config_with_project) + assert provider.config == config_with_project + assert provider.config.url == "https://api.example.com" + assert provider.config.auth_token == "Bearer test-token-12345" + assert provider.config.project_id == "72d21c1d-499b-4784-a3c7-6f84507f9a20" + + def test_get_asset_by_id_success( + self, config_with_project, data_asset_json, mocker + ): + """Test successful asset retrieval""" + provider = CamsProvider(config_with_project) + + # Mock the session.get call + mock_response = mocker.Mock() + mock_response.text = json.dumps(data_asset_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + asset = provider.get_asset_by_id("6862f3ba-81f5-4122-8286-62bb4c5d6543") + + assert isinstance(asset, DataAsset) + assert asset.metadata.asset_id == "6862f3ba-81f5-4122-8286-62bb4c5d6543" + assert asset.metadata.name == "DEPARTMENT" + + def test_get_asset_by_id_with_project_id( + self, config_with_project, data_asset_json, mocker + ): + """Test that project_id is included in query parameters""" + provider = CamsProvider(config_with_project) + + mock_response = mocker.Mock() + mock_response.text = json.dumps(data_asset_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session_instance = mock_session.return_value + mock_session_instance.get.return_value = mock_response + + # Mock get_url_with_query_params to capture the call + mock_get_url = mocker.patch( + "wxdi.dq_validator.provider.cams.get_url_with_query_params" + ) + mock_get_url.return_value = ( + "https://api.example.com/v2/assets/asset-123?project_id=project-456" + ) + + provider.get_asset_by_id("asset-123") + + # Verify get_url_with_query_params was called with project_id + mock_get_url.assert_called_once() + call_args = mock_get_url.call_args + assert call_args[0][0] == "https://api.example.com/v2/assets/asset-123" + assert call_args[0][1]["project_id"] == "72d21c1d-499b-4784-a3c7-6f84507f9a20" + + def test_get_asset_by_id_without_project_id( + self, config_without_project_or_catalog, data_asset_json, mocker + ): + """Test asset retrieval without project_id""" + provider = CamsProvider(config_without_project_or_catalog) + + mock_response = mocker.Mock() + mock_response.text = json.dumps(data_asset_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + mock_get_url = mocker.patch( + "wxdi.dq_validator.provider.cams.get_url_with_query_params" + ) + mock_get_url.return_value = "https://api.example.com/v2/assets/asset-123" + + provider.get_asset_by_id("asset-123") + + # Verify get_url_with_query_params was called without project_id + mock_get_url.assert_called_once() + call_args = mock_get_url.call_args + assert "project_id" not in call_args[0][1] + + def test_get_asset_by_id_with_options( + self, config_with_project, data_asset_json, mocker + ): + """Test asset retrieval with additional options""" + provider = CamsProvider(config_with_project) + + mock_response = mocker.Mock() + mock_response.text = json.dumps(data_asset_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + mock_get_url = mocker.patch( + "wxdi.dq_validator.provider.cams.get_url_with_query_params" + ) + mock_get_url.return_value = "https://api.example.com/v2/assets/asset-123?project_id=project-456&include=metadata" + + options = {"include": "metadata"} + provider.get_asset_by_id("asset-123", options) + + # Verify both project_id and custom options are included + mock_get_url.assert_called_once() + call_args = mock_get_url.call_args + query_params = call_args[0][1] + assert query_params["project_id"] == "72d21c1d-499b-4784-a3c7-6f84507f9a20" + assert query_params["include"] == "metadata" + + def test_get_asset_by_id_url_construction( + self, config_with_project, data_asset_json, mocker + ): + """Test correct URL construction""" + provider = CamsProvider(config_with_project) + + mock_response = mocker.Mock() + mock_response.text = json.dumps(data_asset_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session_instance = mock_session.return_value + mock_session_instance.get.return_value = mock_response + + asset_id = "6862f3ba-81f5-4122-8286-62bb4c5d6543" + provider.get_asset_by_id(asset_id) + + # Verify the session.get was called + mock_session_instance.get.assert_called_once() + call_args = mock_session_instance.get.call_args + + # Check that headers were passed + assert "headers" in call_args[1] + assert call_args[1]["verify"] is False + + def test_get_asset_by_id_headers( + self, config_with_project, data_asset_json, mocker + ): + """Test that correct headers are sent""" + provider = CamsProvider(config_with_project) + + mock_response = mocker.Mock() + mock_response.text = json.dumps(data_asset_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session_instance = mock_session.return_value + mock_session_instance.get.return_value = mock_response + + # Mock get_request_headers to verify it's called + mock_get_headers = mocker.patch( + "wxdi.dq_validator.provider.cams.get_request_headers" + ) + mock_get_headers.return_value = { + "Authorization": "Bearer test-token-12345", + "Content-Type": "application/json", + } + + provider.get_asset_by_id("asset-123") + + # Verify get_request_headers was called with auth_token + mock_get_headers.assert_called_once_with("Bearer test-token-12345") + + # Verify session.get was called with the headers + mock_session_instance.get.assert_called_once() + call_args = mock_session_instance.get.call_args + assert call_args[1]["headers"] == { + "Authorization": "Bearer test-token-12345", + "Content-Type": "application/json", + } + + def test_get_asset_by_id_returns_data_asset( + self, config_with_project, data_asset_json, mocker + ): + """Test that DataAsset object is correctly constructed""" + provider = CamsProvider(config_with_project) + + mock_response = mocker.Mock() + mock_response.text = json.dumps(data_asset_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + asset = provider.get_asset_by_id("6862f3ba-81f5-4122-8286-62bb4c5d6543") + + # Verify DataAsset structure + assert isinstance(asset, DataAsset) + assert hasattr(asset, "metadata") + assert hasattr(asset, "entity") + assert hasattr(asset.entity, "data_asset") + assert hasattr(asset.entity, "column_info") + + # Verify specific data + assert len(asset.entity.data_asset.columns) == 5 + assert "DEPTNO" in asset.entity.column_info + assert "MGRNO" in asset.entity.column_info + + def test_get_asset_by_id_json_parsing(self, config_with_project, mocker): + """Test JSON parsing of response""" + provider = CamsProvider(config_with_project) + + # Create a minimal valid JSON response + minimal_json = { + "metadata": { + "project_id": "test-project", + "name": "TEST_TABLE", + "tags": [], + "asset_type": "data_asset", + "catalog_id": "test-catalog", + "created": 1234567890, + "created_at": "2024-01-01T00:00:00Z", + "owner_id": "owner-123", + "size": 0, + "version": 1, + "asset_state": "available", + "asset_attributes": [], + "asset_id": "asset-123", + "asset_category": "USER", + "creator_id": "creator-123", + }, + "entity": { + "data_asset": { + "columns": [], + "dataset": True, + "mime_type": "application/x-ibm-rel-table", + "properties": [], + }, + "column_info": {}, + "data_profile": {"attribute_classes": []}, + "key_analyses": { + "fk_defined": 0, + "pk_defined": 0, + "fk_assigned": 0, + "pk_assigned": 0, + "fk_suggested": 0, + "pk_suggested": 0, + "primary_keys": [], + "fk_defined_as_pk": 0, + "overlap_assigned": 0, + "fk_assigned_as_pk": 0, + "overlap_suggested": 0, + "fk_suggested_as_pk": 0, + "key_analysis_area_id": "area-123", + }, + "discovered_asset": {"extended_metadata": []}, + "dataview_visualization": {"jobs": {}}, + "metadata_enrichment_info": {"MDE_instrumented": False}, + "asset_data_quality_constraint": { + "asset_checks": [], + "rejected_checks": [], + "suggested_checks": [], + }, + "metadata_enrichment_area_info": { + "job_id": "job-123", + "area_id": "area-123", + "added_date": 1234567890, + "job_run_id": "run-123", + "last_enrichment_status": "finished", + "last_enrichment_timestamp": 1234567890, + }, + }, + "href": "/v2/assets/asset-123", + } + + mock_response = mocker.Mock() + mock_response.text = json.dumps(minimal_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + asset = provider.get_asset_by_id("asset-123") + + assert isinstance(asset, DataAsset) + assert asset.metadata.name == "TEST_TABLE" + assert asset.metadata.asset_id == "asset-123" + + def test_get_asset_by_id_http_error(self, config_with_project, mocker): + """Test handling of HTTP errors (404, 500, etc.)""" + provider = CamsProvider(config_with_project) + + # Mock a 404 response + mock_response = mocker.Mock() + mock_response.status_code = 404 + mock_response.ok = False + mock_response.text = json.dumps({"error": "Asset not found"}) + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + # The implementation checks response.ok and raises ValueError + with pytest.raises(ValueError) as exc_info: + provider.get_asset_by_id("non-existent-asset") + + # Verify the error message + assert "Could not find data asset" in str(exc_info.value) + assert "non-existent-asset" in str(exc_info.value) + assert config_with_project.project_id in str(exc_info.value) + + def test_get_asset_by_id_invalid_json(self, config_with_project, mocker): + """Test handling of invalid JSON response""" + provider = CamsProvider(config_with_project) + + # Mock a response with invalid JSON + mock_response = mocker.Mock() + mock_response.text = "This is not valid JSON" + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + # Should raise JSONDecodeError + with pytest.raises(json.JSONDecodeError): + provider.get_asset_by_id("asset-123") + + def test_get_asset_by_id_malformed_response(self, config_with_project, mocker): + """Test handling of malformed response (valid JSON but invalid structure)""" + provider = CamsProvider(config_with_project) + + # Mock a response with valid JSON but missing required fields + malformed_json = {"some_field": "some_value"} + mock_response = mocker.Mock() + mock_response.text = json.dumps(malformed_json) + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + # Should raise ValidationError from Pydantic + with pytest.raises(Exception): # Pydantic ValidationError + provider.get_asset_by_id("asset-123") + + def test_get_asset_by_id_network_error(self, config_with_project, mocker): + """Test handling of network errors""" + provider = CamsProvider(config_with_project) + + # Mock a network error + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.side_effect = ConnectionError("Network error") + + # Should raise ConnectionError + with pytest.raises(ConnectionError): + provider.get_asset_by_id("asset-123") + + def test_get_asset_by_id_timeout(self, config_with_project, mocker): + """Test handling of timeout errors""" + provider = CamsProvider(config_with_project) + + # Mock a timeout error + from requests.exceptions import Timeout + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.side_effect = Timeout("Request timeout") + + # Should raise Timeout + with pytest.raises(Timeout): + provider.get_asset_by_id("asset-123") + + def test_config_with_catalog_id(self, config_with_catalog, catalog_id): + """Test ProviderConfig initialization with catalog_id""" + provider = CamsProvider(config_with_catalog) + assert provider.config.catalog_id == catalog_id + assert provider.config.project_id is None + + def test_get_asset_by_id_with_catalog_id( + self, config_with_catalog, data_asset_json, mocker + ): + """Test that catalog_id is included in query parameters""" + provider = CamsProvider(config_with_catalog) + + mock_response = mocker.Mock() + mock_response.text = json.dumps(data_asset_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session_instance = mock_session.return_value + mock_session_instance.get.return_value = mock_response + + # Mock get_url_with_query_params to capture the call + mock_get_url = mocker.patch( + "wxdi.dq_validator.provider.cams.get_url_with_query_params" + ) + mock_get_url.return_value = ( + "https://api.example.com/v2/assets/asset-123?catalog_id=catalog-456" + ) + + provider.get_asset_by_id("asset-123") + + # Verify get_url_with_query_params was called with catalog_id + mock_get_url.assert_called_once() + call_args = mock_get_url.call_args + assert call_args[0][0] == "https://api.example.com/v2/assets/asset-123" + assert call_args[0][1]["catalog_id"] == "a1b2c3d4-e5f6-7890-abcd-ef1234567890" + assert "project_id" not in call_args[0][1] + + def test_get_asset_by_id_catalog_error_message(self, config_with_catalog, mocker): + """Test error message includes catalog_id when using catalog""" + provider = CamsProvider(config_with_catalog) + + # Mock a 404 response + mock_response = mocker.Mock() + mock_response.status_code = 404 + mock_response.ok = False + mock_response.text = json.dumps({"error": "Asset not found"}) + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + with pytest.raises(ValueError) as exc_info: + provider.get_asset_by_id("non-existent-asset") + + # Verify the error message includes catalog context + assert "Could not find data asset" in str(exc_info.value) + assert "non-existent-asset" in str(exc_info.value) + assert "catalog" in str(exc_info.value) + assert config_with_catalog.catalog_id in str(exc_info.value) + + def test_get_asset_by_id_with_catalog_and_options( + self, config_with_catalog, data_asset_json, mocker + ): + """Test asset retrieval with catalog_id and additional options""" + # Mock a timeout error + from requests.exceptions import Timeout + + provider = CamsProvider(config_with_catalog) + + mock_response = mocker.Mock() + mock_response.text = json.dumps(data_asset_json) + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + mock_session.return_value.get.side_effect = [mocker.DEFAULT, Timeout("Request timeout")] + + mock_get_url = mocker.patch( + "wxdi.dq_validator.provider.cams.get_url_with_query_params" + ) + mock_get_url.return_value = "https://api.example.com/v2/assets/asset-123?catalog_id=catalog-456&include=metadata" + + options = {"include": "metadata"} + provider.get_asset_by_id("asset-123", options) + + # Verify both catalog_id and custom options are included + mock_get_url.assert_called_once() + call_args = mock_get_url.call_args + query_params = call_args[0][1] + assert query_params["catalog_id"] == "a1b2c3d4-e5f6-7890-abcd-ef1234567890" + assert query_params["include"] == "metadata" + assert "project_id" not in query_params + + # Should raise Timeout with side effect set above + with pytest.raises(Timeout): + provider.get_asset_by_id("asset-123") + + +# Made with Bob diff --git a/tests/src/dq_validator/provider/test_checks.py b/tests/src/dq_validator/provider/test_checks.py new file mode 100644 index 0000000..8bd3462 --- /dev/null +++ b/tests/src/dq_validator/provider/test_checks.py @@ -0,0 +1,796 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" + +import pytest +from unittest.mock import Mock, patch +import json + +from wxdi.dq_validator.provider import ProviderConfig, ChecksProvider + + +class TestChecksProvider: + """Test suite for ChecksProvider class.""" + + @pytest.fixture + def config(self): + """Create a test configuration.""" + return ProviderConfig( + url="https://test-instance.com", + auth_token="Bearer test-token" + ) + + @pytest.fixture + def provider(self, config): + """Create a test ChecksProvider instance.""" + with patch('wxdi.dq_validator.provider.base_provider.Session') as mock_session_class: + mock_session = Mock() + mock_session_class.return_value = mock_session + provider = ChecksProvider(config) + yield provider + + # ==================== create_check Tests ==================== + + def test_create_check_with_project_id(self, provider): + """Test creating a check with project_id - returns check ID only.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + expected_body = { + "id": "check-123", + "name": "check_uniqueness_of_id", + "type": "uniqueness", + "dimension": { + "id": "dimension-456" + }, + "native_id": "asset-789/check-123" + } + mock_response.text = json.dumps(expected_body) + provider.session.post.return_value = mock_response + + # Execute + result = provider.create_check( + name="check_uniqueness_of_id", + dimension_id="dimension-456", + native_id="asset-789/check-123", + check_type="uniqueness", + project_id="project-123" + ) + + # Verify - should return only check ID (string) + assert isinstance(result, str) + assert result == "check-123" + + # Verify the API call + provider.session.post.assert_called_once() + call_args = provider.session.post.call_args + + # Check URL + assert "https://test-instance.com/data_quality/v4/checks" in call_args[0][0] + assert "project_id=project-123" in call_args[0][0] + + # Check headers + headers = call_args[1]["headers"] + assert headers["Authorization"] == "Bearer test-token" + assert headers["Content-Type"] == "application/json" + + # Check payload + payload = json.loads(call_args[1]["data"]) + assert payload["name"] == "check_uniqueness_of_id" + assert payload["type"] == "uniqueness" + assert payload["dimension"]["id"] == "dimension-456" + assert payload["native_id"] == "asset-789/check-123" + assert json.loads(payload["details"])["origin"] == "SDK" + + def test_create_check_with_catalog_id(self, provider): + """Test creating a check with catalog_id""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + expected_body = { + "id": "check-456", + "name": "check_completeness", + "type": "completeness" + } + mock_response.text = json.dumps(expected_body) + provider.session.post.return_value = mock_response + + # Execute + result = provider.create_check( + name="check_completeness", + dimension_id="dimension-789", + native_id="asset-111/check-456", + catalog_id="catalog-999" + ) + + # Verify - should return only check ID (string) + assert isinstance(result, str) + assert result == "check-456" + + # Verify the API call + call_args = provider.session.post.call_args + + # Check URL contains catalog_id + assert "catalog_id=catalog-999" in call_args[0][0] + assert "project_id" not in call_args[0][0] + + def test_create_check_without_check_type(self, provider): + """Test creating a check without check_type (should default to name) - returns check ID only.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + expected_body = { + "id": "check-789", + "name": "format_check", + "type": "format_check" + } + mock_response.text = json.dumps(expected_body) + provider.session.post.return_value = mock_response + + # Execute + result = provider.create_check( + name="format_check", + dimension_id="dimension-111", + native_id="asset-222/check-789", + project_id="project-123" + ) + + # Verify - should return only check ID (string) + assert isinstance(result, str) + assert result == "check-789" + + # Verify the API call + call_args = provider.session.post.call_args + payload = json.loads(call_args[1]["data"]) + + # Check that type defaults to name + assert payload["type"] == "format_check" + assert payload["name"] == "format_check" + + def test_create_check_missing_both_ids(self, provider): + """Test creating a check without project_id or catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.create_check( + name="check_test", + dimension_id="dimension-123", + native_id="asset-456/check-789" + ) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_create_check_both_ids_provided(self, provider): + """Test creating a check with both project_id and catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.create_check( + name="check_test", + dimension_id="dimension-123", + native_id="asset-456/check-789", + project_id="project-123", + catalog_id="catalog-456" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + def test_create_check_api_failure(self, provider): + """Test failed check creation.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 400 + mock_response.text = "Bad request" + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.create_check( + name="check_test", + dimension_id="dimension-123", + native_id="asset-456/check-789", + project_id="project-123" + ) + + assert "Failed to create check" in str(exc_info.value) + assert "400" in str(exc_info.value) + assert "Bad request" in str(exc_info.value) + + def test_create_check_missing_id_in_response(self, provider): + """Test check creation with missing id in response.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "name": "check_test", + "type": "test" + # Missing "id" field + }) + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.create_check( + name="check_test", + dimension_id="dimension-123", + native_id="asset-456/check-789", + project_id="project-123" + ) + + assert "Check ID not found in response" in str(exc_info.value) + + def test_create_check_with_special_characters_in_native_id(self, provider): + """Test creating a check with special characters in native_id - returns check ID only.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + expected_body = { + "id": "check-special-123", + "name": "check_format", + "native_id": "asset-abc-123/check-xyz-456" + } + mock_response.text = json.dumps(expected_body) + provider.session.post.return_value = mock_response + + # Execute + result = provider.create_check( + name="check_format", + dimension_id="dimension-999", + native_id="asset-abc-123/check-xyz-456", + project_id="project-123" + ) + + # Verify - should return only check ID (string) + assert isinstance(result, str) + assert result == "check-special-123" + + # Verify the payload contains the special native_id + call_args = provider.session.post.call_args + payload = json.loads(call_args[1]["data"]) + assert payload["native_id"] == "asset-abc-123/check-xyz-456" + + def test_create_check_with_parent_check_id(self, provider): + """Test creating a check with parent_check_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + expected_body = { + "id": "child-check-123", + "name": "check_format_email", + "type": "format", + "parent": { + "id": "parent-check-456" + } + } + mock_response.text = json.dumps(expected_body) + provider.session.post.return_value = mock_response + + # Execute + result = provider.create_check( + name="check_format_email", + dimension_id="dimension-789", + native_id="asset-111/email/format", + check_type="format", + project_id="project-123", + parent_check_id="parent-check-456" + ) + + # Verify - should return only check ID (string) + assert isinstance(result, str) + assert result == "child-check-123" + + # Verify the API call includes parent in payload + provider.session.post.assert_called_once() + call_args = provider.session.post.call_args + + # Check payload includes parent + payload = json.loads(call_args[1]["data"]) + assert "parent" in payload + assert payload["parent"]["id"] == "parent-check-456" + assert payload["native_id"] == "asset-111/email/format" + + # ==================== _create_check_full Tests ==================== + + def test_create_check_full_with_project_id(self, provider): + """Test _create_check_full returns full check body with project_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + expected_body = { + "id": "check-full-123", + "name": "check_uniqueness_of_id", + "type": "uniqueness", + "native_id": "asset-456/check-789", + "dimension": { + "id": "dimension-123" + }, + "details": '{"origin": "SDK"}' + } + mock_response.text = json.dumps(expected_body) + provider.session.post.return_value = mock_response + + # Execute + result = provider._create_check_full( + name="check_uniqueness_of_id", + dimension_id="dimension-123", + native_id="asset-456/check-789", + check_type="uniqueness", + project_id="project-123" + ) + + # Verify - should return full check body (dict) + assert isinstance(result, dict) + assert result["id"] == "check-full-123" + assert result["name"] == "check_uniqueness_of_id" + assert result["type"] == "uniqueness" + assert result["native_id"] == "asset-456/check-789" + assert result["dimension"]["id"] == "dimension-123" + + # Verify the API call + provider.session.post.assert_called_once() + call_args = provider.session.post.call_args + + # Check URL + assert "https://test-instance.com/data_quality/v4/checks" in call_args[0][0] + assert "project_id=project-123" in call_args[0][0] + + # Check payload + payload = json.loads(call_args[1]["data"]) + assert payload["name"] == "check_uniqueness_of_id" + assert payload["type"] == "uniqueness" + assert payload["dimension"]["id"] == "dimension-123" + assert json.loads(payload["details"])["origin"] == "SDK" + + def test_create_check_full_with_catalog_id(self, provider): + """Test _create_check_full returns full check body with catalog_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + expected_body = { + "id": "check-full-456", + "name": "check_completeness", + "type": "completeness", + "native_id": "asset-111/check-456" + } + mock_response.text = json.dumps(expected_body) + provider.session.post.return_value = mock_response + + # Execute + result = provider._create_check_full( + name="check_completeness", + dimension_id="dimension-789", + native_id="asset-111/check-456", + catalog_id="catalog-999" + ) + + # Verify - should return full check body (dict) + assert isinstance(result, dict) + assert result["id"] == "check-full-456" + assert result["name"] == "check_completeness" + assert result["type"] == "completeness" + + # Verify the API call + call_args = provider.session.post.call_args + + # Check URL contains catalog_id + assert "catalog_id=catalog-999" in call_args[0][0] + assert "project_id" not in call_args[0][0] + + def test_create_check_full_with_parent_check_id(self, provider): + """Test _create_check_full returns full check body with parent.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + expected_body = { + "id": "child-check-full-123", + "name": "check_format_email", + "type": "format", + "parent": { + "id": "parent-check-456", + "name": "parent_check" + } + } + mock_response.text = json.dumps(expected_body) + provider.session.post.return_value = mock_response + + # Execute + result = provider._create_check_full( + name="check_format_email", + dimension_id="dimension-789", + native_id="asset-111/email/format", + check_type="format", + project_id="project-123", + parent_check_id="parent-check-456" + ) + + # Verify - should return full check body with parent + assert isinstance(result, dict) + assert result["id"] == "child-check-full-123" + assert result["parent"]["id"] == "parent-check-456" + assert result["parent"]["name"] == "parent_check" + + # Verify the API call includes parent in payload + provider.session.post.assert_called_once() + call_args = provider.session.post.call_args + + # Check payload includes parent + payload = json.loads(call_args[1]["data"]) + assert "parent" in payload + assert payload["parent"]["id"] == "parent-check-456" + + def test_create_check_full_without_check_type(self, provider): + """Test _create_check_full defaults check_type to name.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + expected_body = { + "id": "check-full-789", + "name": "format_check", + "type": "format_check" + } + mock_response.text = json.dumps(expected_body) + provider.session.post.return_value = mock_response + + # Execute + result = provider._create_check_full( + name="format_check", + dimension_id="dimension-111", + native_id="asset-222/check-789", + project_id="project-123" + ) + + # Verify - should return full check body + assert isinstance(result, dict) + assert result["id"] == "check-full-789" + assert result["name"] == "format_check" + assert result["type"] == "format_check" + + # Verify the API call + call_args = provider.session.post.call_args + payload = json.loads(call_args[1]["data"]) + + # Check that type defaults to name + assert payload["type"] == "format_check" + assert payload["name"] == "format_check" + + def test_create_check_full_missing_both_ids(self, provider): + """Test _create_check_full without project_id or catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider._create_check_full( + name="check_test", + dimension_id="dimension-123", + native_id="asset-456/check-789" + ) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_create_check_full_both_ids_provided(self, provider): + """Test _create_check_full with both project_id and catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider._create_check_full( + name="check_test", + dimension_id="dimension-123", + native_id="asset-456/check-789", + project_id="project-123", + catalog_id="catalog-456" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + def test_create_check_full_api_failure(self, provider): + """Test _create_check_full with failed API request.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 500 + mock_response.text = "Internal server error" + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider._create_check_full( + name="check_test", + dimension_id="dimension-123", + native_id="asset-456/check-789", + project_id="project-123" + ) + + assert "Failed to create check" in str(exc_info.value) + assert "500" in str(exc_info.value) + assert "Internal server error" in str(exc_info.value) + + def test_create_check_full_missing_id_in_response(self, provider): + """Test _create_check_full with missing id in response.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "name": "check_test", + "type": "test" + # Missing "id" field + }) + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider._create_check_full( + name="check_test", + dimension_id="dimension-123", + native_id="asset-456/check-789", + project_id="project-123" + ) + + assert "Check ID not found in response" in str(exc_info.value) + + # ==================== get_checks Tests ==================== + + def test_get_checks_with_project_id(self, provider): + """Test getting checks with project_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "checks": [ + { + "id": "check-1", + "name": "case_check", + "type": "case", + "asset": { + "id": "asset-123" + } + }, + { + "id": "check-2", + "name": "case_check_2", + "type": "case", + "asset": { + "id": "asset-123" + } + } + ] + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_checks( + dq_asset_id="asset-123", + check_type="case", + project_id="project-456" + ) + + # Verify + assert len(result) == 2 + assert result[0]["id"] == "check-1" + assert result[1]["id"] == "check-2" + assert result[0]["type"] == "case" + + # Verify the API call + provider.session.get.assert_called_once() + call_args = provider.session.get.call_args + + # Check URL + assert "https://test-instance.com/data_quality/v4/checks" in call_args[0][0] + assert "asset.id=asset-123" in call_args[0][0] + assert "type=case" in call_args[0][0] + assert "project_id=project-456" in call_args[0][0] + assert "include_children=true" in call_args[0][0] + + # Check headers + headers = call_args[1]["headers"] + assert headers["Authorization"] == "Bearer test-token" + + def test_get_checks_with_catalog_id(self, provider): + """Test getting checks with catalog_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "checks": [ + { + "id": "check-3", + "type": "completeness" + } + ] + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_checks( + dq_asset_id="asset-789", + check_type="completeness", + catalog_id="catalog-999" + ) + + # Verify + assert len(result) == 1 + assert result[0]["id"] == "check-3" + + # Verify the API call + call_args = provider.session.get.call_args + + # Check URL contains catalog_id + assert "catalog_id=catalog-999" in call_args[0][0] + assert "project_id" not in call_args[0][0] + + def test_get_checks_with_include_children_false(self, provider): + """Test getting checks with include_children=False.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "checks": [] + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_checks( + dq_asset_id="asset-111", + check_type="format", + project_id="project-222", + include_children=False + ) + + # Verify + assert len(result) == 0 + + # Verify the API call + call_args = provider.session.get.call_args + + # Check URL contains include_children=false + assert "include_children=false" in call_args[0][0] + + def test_get_checks_empty_result(self, provider): + """Test getting checks with no results.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "checks": [] + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_checks( + dq_asset_id="asset-nonexistent", + check_type="unknown", + project_id="project-123" + ) + + # Verify + assert result == [] + assert len(result) == 0 + + def test_get_checks_missing_both_ids(self, provider): + """Test getting checks without project_id or catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.get_checks( + dq_asset_id="asset-123", + check_type="case" + ) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_get_checks_both_ids_provided(self, provider): + """Test getting checks with both project_id and catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.get_checks( + dq_asset_id="asset-123", + check_type="case", + project_id="project-123", + catalog_id="catalog-456" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + def test_get_checks_api_failure(self, provider): + """Test failed get checks request.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 404 + mock_response.text = "Asset not found" + provider.session.get.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.get_checks( + dq_asset_id="asset-invalid", + check_type="case", + project_id="project-123" + ) + + assert "Failed to get checks" in str(exc_info.value) + assert "404" in str(exc_info.value) + assert "Asset not found" in str(exc_info.value) + + def test_get_checks_missing_checks_in_response(self, provider): + """Test get checks with missing 'checks' field in response.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "data": [] + # Missing "checks" field + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_checks( + dq_asset_id="asset-123", + check_type="case", + project_id="project-123" + ) + + # Verify - should return empty list when checks field is missing + assert result == [] + + def test_get_checks_with_multiple_check_types(self, provider): + """Test getting checks filters by specific check_type.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "checks": [ + { + "id": "check-format-1", + "type": "format" + }, + { + "id": "check-format-2", + "type": "format" + } + ] + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_checks( + dq_asset_id="asset-multi", + check_type="format", + project_id="project-123" + ) + + # Verify + assert len(result) == 2 + assert all(check["type"] == "format" for check in result) + + # Verify the API call includes the check_type filter + call_args = provider.session.get.call_args + assert "type=format" in call_args[0][0] + + def test_get_checks_with_comparison_check_type(self, provider): + """Test getting checks with comparison check type.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "checks": [ + { + "id": "check-comparison-1", + "type": "comparison", + "name": "compare_columns" + } + ] + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_checks( + dq_asset_id="asset-comparison", + check_type="comparison", + project_id="project-123" + ) + + # Verify + assert len(result) == 1 + assert result[0]["type"] == "comparison" + assert result[0]["name"] == "compare_columns" \ No newline at end of file diff --git a/tests/src/dq_validator/provider/test_config.py b/tests/src/dq_validator/provider/test_config.py new file mode 100644 index 0000000..3158db3 --- /dev/null +++ b/tests/src/dq_validator/provider/test_config.py @@ -0,0 +1,284 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Test suite for ProviderConfig module +""" + +import pytest +from unittest.mock import Mock, patch +from wxdi.dq_validator.provider.config import ProviderConfig +from wxdi.common.auth.auth_config import AuthConfig, EnvironmentType + + +class TestProviderConfig: + """Test cases for ProviderConfig class""" + + @pytest.fixture + def base_url(self): + """Base URL for API""" + return "https://api.example.com" + + @pytest.fixture + def auth_token(self): + """Authentication token""" + return "Bearer test-token-12345" + + @pytest.fixture + def project_id(self): + """Project ID""" + return "72d21c1d-499b-4784-a3c7-6f84507f9a20" + + @pytest.fixture + def catalog_id(self): + """Catalog ID""" + return "a1b2c3d4-e5f6-7890-abcd-ef1234567890" + + @pytest.fixture + def auth_config_ibm_cloud(self): + """AuthConfig for IBM Cloud""" + return AuthConfig( + environment_type=EnvironmentType.IBM_CLOUD, + api_key="test-api-key-12345" + ) + + @pytest.fixture + def auth_config_aws_cloud(self): + """AuthConfig for AWS Cloud""" + return AuthConfig( + environment_type=EnvironmentType.AWS_CLOUD, + api_key="test-api-key-12345", + account_id="test-account-id" + ) + + @pytest.fixture + def auth_config_gov_cloud(self): + """AuthConfig for Government Cloud""" + return AuthConfig( + environment_type=EnvironmentType.GOV_CLOUD, + api_key="test-api-key-12345" + ) + + @pytest.fixture + def auth_config_on_prem(self): + """AuthConfig for On-Premises""" + return AuthConfig( + environment_type=EnvironmentType.ON_PREM, + url="https://on-prem.example.com", + username="test-user", + api_key="test-api-key" + ) + + def test_config_with_auth_token_only(self, base_url, auth_token): + """Test ProviderConfig initialization with auth_token only""" + config = ProviderConfig(base_url, auth_token=auth_token) + + assert config.url == base_url + assert config._auth_token == auth_token + assert config.project_id is None + assert config.catalog_id is None + assert config.auth_provider is None + assert config.auth_token == auth_token + + def test_config_with_project_id(self, base_url, auth_token, project_id): + """Test ProviderConfig initialization with project_id""" + config = ProviderConfig(base_url, auth_token=auth_token, project_id=project_id) + + assert config.url == base_url + assert config.project_id == project_id + assert config.catalog_id is None + + def test_config_with_catalog_id(self, base_url, auth_token, catalog_id): + """Test ProviderConfig initialization with catalog_id""" + config = ProviderConfig(base_url, auth_token=auth_token, catalog_id=catalog_id) + + assert config.url == base_url + assert config.catalog_id == catalog_id + assert config.project_id is None + + def test_config_with_auth_config_ibm_cloud(self, base_url, auth_config_ibm_cloud): + """Test ProviderConfig initialization with AuthConfig for IBM Cloud""" + with patch('wxdi.dq_validator.provider.config.AuthProvider') as mock_auth_provider: + mock_provider_instance = Mock() + mock_provider_instance.get_token.return_value = "mocked-token-12345" + mock_auth_provider.return_value = mock_provider_instance + + config = ProviderConfig(base_url, auth_config=auth_config_ibm_cloud) + + assert config.url == base_url + assert config.auth_provider is not None + assert config._auth_token is None + mock_auth_provider.assert_called_once_with(auth_config_ibm_cloud) + + # Test that auth_token property calls get_token + token = config.auth_token + assert token == "mocked-token-12345" + mock_provider_instance.get_token.assert_called_once() + + def test_config_with_auth_config_aws_cloud(self, base_url, auth_config_aws_cloud): + """Test ProviderConfig initialization with AuthConfig for AWS Cloud""" + with patch('wxdi.dq_validator.provider.config.AuthProvider') as mock_auth_provider: + mock_provider_instance = Mock() + mock_provider_instance.get_token.return_value = "aws-token-12345" + mock_auth_provider.return_value = mock_provider_instance + + config = ProviderConfig(base_url, auth_config=auth_config_aws_cloud) + + assert config.auth_provider is not None + mock_auth_provider.assert_called_once_with(auth_config_aws_cloud) + + token = config.auth_token + assert token == "aws-token-12345" + + def test_config_with_auth_config_gov_cloud(self, base_url, auth_config_gov_cloud): + """Test ProviderConfig initialization with AuthConfig for Government Cloud""" + with patch('wxdi.dq_validator.provider.config.AuthProvider') as mock_auth_provider: + mock_provider_instance = Mock() + mock_provider_instance.get_token.return_value = "gov-token-12345" + mock_auth_provider.return_value = mock_provider_instance + + config = ProviderConfig(base_url, auth_config=auth_config_gov_cloud) + + assert config.auth_provider is not None + token = config.auth_token + assert token == "gov-token-12345" + + def test_config_with_auth_config_on_prem(self, base_url, auth_config_on_prem): + """Test ProviderConfig initialization with AuthConfig for On-Premises""" + with patch('wxdi.dq_validator.provider.config.AuthProvider') as mock_auth_provider: + mock_provider_instance = Mock() + mock_provider_instance.get_token.return_value = "on-prem-token-12345" + mock_auth_provider.return_value = mock_provider_instance + + config = ProviderConfig(base_url, auth_config=auth_config_on_prem) + + assert config.auth_provider is not None + token = config.auth_token + assert token == "on-prem-token-12345" + + def test_config_with_both_auth_token_and_auth_config( + self, base_url, auth_token, auth_config_ibm_cloud + ): + """Test ProviderConfig with both auth_token and auth_config (auth_config takes precedence)""" + with patch('wxdi.dq_validator.provider.config.AuthProvider') as mock_auth_provider: + mock_provider_instance = Mock() + mock_provider_instance.get_token.return_value = "config-token-12345" + mock_auth_provider.return_value = mock_provider_instance + + config = ProviderConfig( + base_url, + auth_token=auth_token, + auth_config=auth_config_ibm_cloud + ) + + # auth_config should take precedence + assert config.auth_provider is not None + assert config._auth_token == auth_token + + # When getting token, auth_provider should be used first + token = config.auth_token + assert token == "config-token-12345" + mock_provider_instance.get_token.assert_called_once() + + def test_config_auth_token_property_with_auth_provider( + self, base_url, auth_config_ibm_cloud + ): + """Test auth_token property when auth_provider is set""" + with patch('wxdi.dq_validator.provider.config.AuthProvider') as mock_auth_provider: + mock_provider_instance = Mock() + mock_provider_instance.get_token.return_value = "provider-token" + mock_auth_provider.return_value = mock_provider_instance + + config = ProviderConfig(base_url, auth_config=auth_config_ibm_cloud) + + # First call + token1 = config.auth_token + assert token1 == "provider-token" + + # Second call - should call get_token again (no caching) + token2 = config.auth_token + assert token2 == "provider-token" + assert mock_provider_instance.get_token.call_count == 2 + + def test_config_auth_token_property_with_static_token(self, base_url, auth_token): + """Test auth_token property when only static token is set""" + config = ProviderConfig(base_url, auth_token=auth_token) + + token = config.auth_token + assert token == auth_token + + def test_config_auth_token_property_no_auth(self, base_url): + """Test auth_token property raises error when no authentication is provided""" + config = ProviderConfig(base_url) + + assert config.auth_token == '' + + def test_config_get_auth_token_no_auth(self, base_url): + """Test auth_token property raises error when no authentication is provided""" + config = ProviderConfig(base_url) + + with pytest.raises(ValueError) as exc_info: + _ = config.get_auth_token() + + assert "No authentication token provided" in str(exc_info.value) + + def test_config_with_all_parameters( + self, base_url, auth_token, project_id, catalog_id, auth_config_ibm_cloud + ): + """Test ProviderConfig with all parameters""" + with patch('wxdi.dq_validator.provider.config.AuthProvider') as mock_auth_provider: + mock_provider_instance = Mock() + returned_token = "full-config-token" + mock_provider_instance.get_token.return_value = returned_token + mock_auth_provider.return_value = mock_provider_instance + + config = ProviderConfig( + base_url, + auth_token=auth_token, + project_id=project_id, + catalog_id=catalog_id, + auth_config=auth_config_ibm_cloud + ) + + assert config.url == base_url + assert config._auth_token == auth_token + assert config.project_id == project_id + assert config.catalog_id == catalog_id + assert config.auth_provider is not None + assert config.auth_token == returned_token + + def test_config_auth_provider_none_when_no_auth_config(self, base_url, auth_token): + """Test that auth_provider is None when auth_config is not provided""" + config = ProviderConfig(base_url, auth_token=auth_token) + + assert config.auth_provider is None + + def test_config_with_project_and_catalog( + self, base_url, auth_token, project_id, catalog_id + ): + """Test ProviderConfig with both project_id and catalog_id""" + config = ProviderConfig( + base_url, + auth_token=auth_token, + project_id=project_id, + catalog_id=catalog_id + ) + + assert config.project_id == project_id + assert config.catalog_id == catalog_id + assert config.auth_token == auth_token + + +# Made with Bob \ No newline at end of file diff --git a/tests/src/dq_validator/provider/test_dimensions.py b/tests/src/dq_validator/provider/test_dimensions.py new file mode 100644 index 0000000..396fc04 --- /dev/null +++ b/tests/src/dq_validator/provider/test_dimensions.py @@ -0,0 +1,227 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" + +import pytest +from unittest.mock import Mock, patch +import json + +from wxdi.dq_validator.provider import ProviderConfig, DimensionsProvider + + +class TestDimensionsProvider: + """Test suite for DimensionsProvider class.""" + + @pytest.fixture + def config(self): + """Create a test configuration.""" + return ProviderConfig( + url="https://test-instance.com", + auth_token="Bearer test-token" + ) + + @pytest.fixture + def provider(self, config): + """Create a test DimensionsProvider instance.""" + with patch('wxdi.dq_validator.provider.base_provider.Session') as mock_session_class: + mock_session = Mock() + mock_session_class.return_value = mock_session + provider = DimensionsProvider(config) + yield provider + + # ==================== search_dimension Tests ==================== + + def test_search_dimension_success(self, provider): + """Test successful dimension search.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "dimensions": [ + { + "id": "371114cd-5516-4691-8b2e-1e66edf66486", + "name": "Completeness", + "description": "Data is complete if it contains all required values.", + "is_default": True + } + ] + }) + provider.session.post.return_value = mock_response + + # Execute + result = provider.search_dimension("Completeness") + + # Verify + assert result == "371114cd-5516-4691-8b2e-1e66edf66486" + + # Verify the API call + provider.session.post.assert_called_once() + call_args = provider.session.post.call_args + + # Check URL + assert "https://test-instance.com/data_quality/v4/search_dq_dimension" in call_args[0][0] + assert "name=Completeness" in call_args[0][0] + + # Check headers + headers = call_args[1]["headers"] + assert headers["Authorization"] == "Bearer test-token" + assert headers["Content-Type"] == "application/json" + + def test_search_dimension_case_insensitive(self, provider): + """Test case-insensitive dimension search.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "dimensions": [ + { + "id": "accuracy-id-123", + "name": "Accuracy", + "description": "Data accuracy dimension" + } + ] + }) + provider.session.post.return_value = mock_response + + # Execute with lowercase + result = provider.search_dimension("accuracy") + + # Verify - should match case-insensitively + assert result == "accuracy-id-123" + + def test_search_dimension_multiple_results(self, provider): + """Test searching when API returns multiple dimensions.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "dimensions": [ + { + "id": "completeness-id", + "name": "Completeness", + "description": "Completeness dimension" + }, + { + "id": "accuracy-id", + "name": "Accuracy", + "description": "Accuracy dimension" + }, + { + "id": "validity-id", + "name": "Validity", + "description": "Validity dimension" + } + ] + }) + provider.session.post.return_value = mock_response + + # Execute - search for Accuracy + result = provider.search_dimension("Accuracy") + + # Verify - should find the correct one + assert result == "accuracy-id" + + def test_search_dimension_api_failure(self, provider): + """Test failed dimension search request.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 500 + mock_response.text = "Internal server error" + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.search_dimension("Completeness") + + assert "Failed to get dimension" in str(exc_info.value) + assert "500" in str(exc_info.value) + + def test_search_dimension_not_found_empty_list(self, provider): + """Test searching when API returns empty list.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "dimensions": [] + }) + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.search_dimension("NonExistent") + + assert "Dimension with name 'NonExistent' not found" in str(exc_info.value) + + def test_search_dimension_not_found_no_match(self, provider): + """Test searching when name doesn't match any result.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "dimensions": [ + { + "id": "completeness-id", + "name": "Completeness", + "description": "Completeness dimension" + } + ] + }) + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.search_dimension("InvalidDimension") + + assert "Dimension with name 'InvalidDimension' not found in results" in str(exc_info.value) + + def test_search_dimension_missing_id_in_response(self, provider): + """Test searching when ID is missing in response.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "dimensions": [ + { + "name": "Completeness", + "description": "Completeness dimension" + # Missing "id" field + } + ] + }) + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.search_dimension("Completeness") + + assert "Dimension ID not found in response" in str(exc_info.value) + + def test_search_dimension_missing_dimensions_key(self, provider): + """Test searching when response doesn't have 'dimensions' key.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "data": [] + # Missing "dimensions" key + }) + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.search_dimension("Completeness") + + assert "Dimension with name 'Completeness' not found" in str(exc_info.value) \ No newline at end of file diff --git a/tests/src/dq_validator/provider/test_dq_search.py b/tests/src/dq_validator/provider/test_dq_search.py new file mode 100644 index 0000000..dbaa81a --- /dev/null +++ b/tests/src/dq_validator/provider/test_dq_search.py @@ -0,0 +1,279 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Tests for DQSearchProvider +""" + +import pytest +from unittest.mock import Mock, patch, MagicMock +from wxdi.dq_validator.provider import ProviderConfig, DQSearchProvider + + +@pytest.fixture +def config(): + """Create a test configuration""" + return ProviderConfig( + url="https://test.example.com", + auth_token="Bearer test-token", + project_id="test-project-id" + ) + + +@pytest.fixture +def provider(config): + """Create a DQSearchProvider instance""" + with patch('wxdi.dq_validator.provider.base_provider.Session') as mock_session_class: + mock_session = Mock() + mock_session_class.return_value = mock_session + provider = DQSearchProvider(config) + yield provider + + +class TestSearchDQCheck: + """Tests for search_dq_check method""" + + def test_search_dq_check_success_with_project_id(self, provider): + """Test successful DQ check search with project_id""" + mock_response = Mock() + mock_response.ok = True + mock_response.text = '''{ + "id": "ad277842-dea7-44ef-8e4b-d940df0f79aa", + "account_id": "999", + "created_at": "2025-12-19T06:37:23.519Z", + "dimension": { + "id": "ec453723-669c-48bb-82c1-11b69b3b8c93", + "name": "Validity" + }, + "name": "Format check", + "native_id": "b2debda2-6ab9-4a39-8c23-17954e004dcf/7377e2cd-ac0e-4833-8760-fd0e8cb682aa", + "wkc_container_id": "24419069-d649-45cb-a2c1-64d6eed650d5", + "type": "format" + }''' + provider.session.post.return_value = mock_response + + result = provider.search_dq_check( + native_id="b2debda2-6ab9-4a39-8c23-17954e004dcf/7377e2cd-ac0e-4833-8760-fd0e8cb682aa", + check_type="format", + project_id="24419069-d649-45cb-a2c1-64d6eed650d5" + ) + + assert result['id'] == "ad277842-dea7-44ef-8e4b-d940df0f79aa" + assert result['name'] == "Format check" + assert result['type'] == "format" + assert result['native_id'] == "b2debda2-6ab9-4a39-8c23-17954e004dcf/7377e2cd-ac0e-4833-8760-fd0e8cb682aa" + + # Verify project_id is in the URL + call_args = provider.session.post.call_args + assert "project_id=24419069-d649-45cb-a2c1-64d6eed650d5" in call_args[0][0] + + def test_search_dq_check_success_with_catalog_id(self, provider): + """Test successful DQ check search with catalog_id""" + mock_response = Mock() + mock_response.ok = True + mock_response.text = '''{ + "id": "ad277842-dea7-44ef-8e4b-d940df0f79aa", + "name": "Format check", + "type": "format" + }''' + provider.session.post.return_value = mock_response + + result = provider.search_dq_check( + native_id="test-native-id", + check_type="format", + catalog_id="catalog-123" + ) + + assert result['id'] == "ad277842-dea7-44ef-8e4b-d940df0f79aa" + + # Verify catalog_id is in the URL + call_args = provider.session.post.call_args + assert "catalog_id=catalog-123" in call_args[0][0] + assert "project_id" not in call_args[0][0] + + def test_search_dq_check_missing_both_ids(self, provider): + """Test DQ check search without project_id or catalog_id""" + with pytest.raises(ValueError) as exc_info: + provider.search_dq_check( + native_id="test-id", + check_type="format" + ) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_search_dq_check_both_ids_provided(self, provider): + """Test DQ check search with both project_id and catalog_id""" + with pytest.raises(ValueError) as exc_info: + provider.search_dq_check( + native_id="test-id", + check_type="format", + project_id="project-123", + catalog_id="catalog-123" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + def test_search_dq_check_failure(self, provider): + """Test DQ check search failure""" + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 404 + mock_response.text = "Not found" + provider.session.post.return_value = mock_response + + with pytest.raises(ValueError) as exc_info: + provider.search_dq_check( + native_id="invalid-id", + check_type="format", + project_id="test-project" + ) + + assert "Failed to search DQ check" in str(exc_info.value) + assert "404" in str(exc_info.value) + + def test_search_dq_check_with_include_children(self, provider): + """Test DQ check search with include_children parameter""" + mock_response = Mock() + mock_response.ok = True + mock_response.text = '{"id": "test-id", "type": "format"}' + provider.session.post.return_value = mock_response + + provider.search_dq_check( + native_id="test-native-id", + check_type="format", + project_id="test-project", + include_children=False + ) + + # Verify the URL contains include_children=false + call_args = provider.session.post.call_args + assert "include_children=false" in call_args[0][0] + + +class TestSearchDQAsset: + """Tests for search_dq_asset method""" + + def test_search_dq_asset_success_with_project_id(self, provider): + """Test successful DQ asset search with project_id""" + mock_response = Mock() + mock_response.ok = True + mock_response.text = '''{ + "id": "1488a413-99f9-4bed-906d-c33b505d5728", + "account_id": "999", + "created_at": "2026-01-28T14:08:08.380Z", + "name": "RTN", + "native_id": "b2debda2-6ab9-4a39-8c23-17954e004dcf/RTN", + "wkc_container_id": "24419069-d649-45cb-a2c1-64d6eed650d5", + "type": "column" + }''' + provider.session.post.return_value = mock_response + + result = provider.search_dq_asset( + native_id="b2debda2-6ab9-4a39-8c23-17954e004dcf/RTN", + project_id="24419069-d649-45cb-a2c1-64d6eed650d5", + asset_type="column" + ) + + assert result['id'] == "1488a413-99f9-4bed-906d-c33b505d5728" + assert result['name'] == "RTN" + assert result['type'] == "column" + assert result['native_id'] == "b2debda2-6ab9-4a39-8c23-17954e004dcf/RTN" + + # Verify project_id is in the URL + call_args = provider.session.post.call_args + assert "project_id=24419069-d649-45cb-a2c1-64d6eed650d5" in call_args[0][0] + + def test_search_dq_asset_success_with_catalog_id(self, provider): + """Test successful DQ asset search with catalog_id""" + mock_response = Mock() + mock_response.ok = True + mock_response.text = '''{ + "id": "test-asset-id", + "name": "Test Asset", + "type": "column" + }''' + provider.session.post.return_value = mock_response + + result = provider.search_dq_asset( + native_id="test-native-id", + catalog_id="catalog-123", + asset_type="column" + ) + + assert result['id'] == "test-asset-id" + + # Verify catalog_id is in the URL + call_args = provider.session.post.call_args + assert "catalog_id=catalog-123" in call_args[0][0] + assert "project_id" not in call_args[0][0] + + def test_search_dq_asset_missing_both_ids(self, provider): + """Test DQ asset search without project_id or catalog_id""" + with pytest.raises(ValueError) as exc_info: + provider.search_dq_asset( + native_id="test-id" + ) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_search_dq_asset_both_ids_provided(self, provider): + """Test DQ asset search with both project_id and catalog_id""" + with pytest.raises(ValueError) as exc_info: + provider.search_dq_asset( + native_id="test-id", + project_id="project-123", + catalog_id="catalog-123" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + def test_search_dq_asset_failure(self, provider): + """Test DQ asset search failure""" + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 404 + mock_response.text = "Not found" + provider.session.post.return_value = mock_response + + with pytest.raises(ValueError) as exc_info: + provider.search_dq_asset( + native_id="invalid-id", + project_id="test-project" + ) + + assert "Failed to search DQ asset" in str(exc_info.value) + assert "404" in str(exc_info.value) + + def test_search_dq_asset_with_optional_params(self, provider): + """Test DQ asset search with optional parameters""" + mock_response = Mock() + mock_response.ok = True + mock_response.text = '{"id": "test-id", "type": "column"}' + provider.session.post.return_value = mock_response + + provider.search_dq_asset( + native_id="test-native-id", + project_id="test-project", + asset_type="table", + include_children=False, + get_actual_asset=True + ) + + # Verify the URL contains the parameters + call_args = provider.session.post.call_args + url = call_args[0][0] + assert "type=table" in url + assert "include_children=false" in url + assert "get_actual_asset=true" in url \ No newline at end of file diff --git a/tests/src/dq_validator/provider/test_glossary.py b/tests/src/dq_validator/provider/test_glossary.py new file mode 100644 index 0000000..464c6c3 --- /dev/null +++ b/tests/src/dq_validator/provider/test_glossary.py @@ -0,0 +1,697 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Comprehensive unit tests for the GlossaryProvider module. + +This test suite validates all aspects of the GlossaryProvider including: + - Configuration validation + - Fetching published artifacts by ID + - Fetching terms by version ID + - Response parsing and model creation + - Error handling (HTTP errors, network errors, invalid responses) + - Query parameter handling + +Test Coverage: + - TestProviderConfig: Configuration class validation + - TestGlossaryProvider: Provider functionality + - TestGlossaryTermModel: Response model parsing + - TestUtilityFunctions: Helper function validation + +Running Tests: + Run all glossary tests: + $ pytest tests/src/provider/test_glossary.py -v + + Run specific test: + $ pytest tests/src/provider/test_glossary.py::TestGlossaryProvider::test_get_published_artifact_success -v + + Run with coverage: + $ pytest tests/src/provider/test_glossary.py --cov=dq_validator.provider --cov-report=html + +Note: + Tests use pytest-mock to avoid actual network calls. + Test data is loaded from tests/data/ directory. + +See Also: + - src/dq_validator/provider/glossary.py: GlossaryProvider implementation + - src/dq_validator/provider/response_model.py: Response models +""" + +import json +import pytest +from pathlib import Path +from wxdi.dq_validator.provider.glossary import GlossaryProvider +from wxdi.dq_validator.provider.config import ProviderConfig +from wxdi.dq_validator.provider.response_model import GlossaryTerm +from wxdi.dq_validator.utils import get_request_headers, get_url_with_query_params + + +# Test constants +MOCK_URL = "https://test.example.com" +MOCK_AUTH_TOKEN = "Bearer test-token-123" +MOCK_PROJECT_ID = "test-project-id" +MOCK_TERM_ID = "30d1b847-0aa9-4840-a182-dd157fe977a0" +MOCK_VERSION_ID = "bdeef8cc-d9ab-4822-b3df-cef82b4de538_0" + +# Get test data directory +TEST_DATA_DIR = Path(__file__).parent.parent.parent.parent / "data" + + +def load_test_data(filename: str) -> str: + """Load test data from JSON file""" + file_path = TEST_DATA_DIR / filename + with open(file_path, "r") as f: + return f.read() + + +# ============================================================================ +# ProviderConfig Tests +# ============================================================================ + + +class TestProviderConfig: + """Test ProviderConfig class""" + + def test_config_with_all_parameters(self): + """Test creating config with all parameters""" + config = ProviderConfig( + url=MOCK_URL, auth_token=MOCK_AUTH_TOKEN, project_id=MOCK_PROJECT_ID + ) + assert config.url == MOCK_URL + assert config.auth_token == MOCK_AUTH_TOKEN + assert config.project_id == MOCK_PROJECT_ID + + def test_config_without_project_id(self): + """Test creating config without project_id""" + config = ProviderConfig(url=MOCK_URL, auth_token=MOCK_AUTH_TOKEN) + assert config.url == MOCK_URL + assert config.auth_token == MOCK_AUTH_TOKEN + assert config.project_id is None + + def test_config_attributes_accessible(self): + """Test that config attributes are accessible""" + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN, MOCK_PROJECT_ID) + assert hasattr(config, "url") + assert hasattr(config, "auth_token") + assert hasattr(config, "project_id") + + +# ============================================================================ +# Utility Functions Tests +# ============================================================================ + + +class TestUtilityFunctions: + """Test utility functions""" + + def test_get_request_headers_with_auth(self): + """Test getting request headers with auth token""" + headers = get_request_headers(MOCK_AUTH_TOKEN) + assert headers["Authorization"] == MOCK_AUTH_TOKEN + assert headers["Content-Type"] == "application/json" + + def test_get_request_headers_custom_content_type(self): + """Test getting request headers with custom content type""" + headers = get_request_headers(MOCK_AUTH_TOKEN, "text/plain") + assert headers["Authorization"] == MOCK_AUTH_TOKEN + assert headers["Content-Type"] == "text/plain" + + def test_get_request_headers_no_auth(self): + """Test getting request headers without auth token""" + headers = get_request_headers("") + assert "Authorization" not in headers + assert headers["Content-Type"] == "application/json" + + def test_get_url_with_query_params(self): + """Test adding query parameters to URL""" + url = "https://example.com/api" + params = {"key1": "value1", "key2": "value2"} + result = get_url_with_query_params(url, params) + assert "key1=value1" in result + assert "key2=value2" in result + assert result.startswith(url) + + def test_get_url_without_query_params(self): + """Test URL without query parameters""" + url = "https://example.com/api" + result = get_url_with_query_params(url, None) + assert result == url + + +# ============================================================================ +# GlossaryProvider Tests +# ============================================================================ + + +class TestGlossaryProvider: + """Test GlossaryProvider class""" + + def test_provider_initialization(self): + """Test provider initialization with config""" + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + assert provider.config == config + assert provider.config.url == MOCK_URL + assert provider.config.auth_token == MOCK_AUTH_TOKEN + + def test_get_published_artifact_success(self, mocker): + """Test successful retrieval of published artifact""" + # Load test data + test_data = load_test_data("term_latest_version.json") + + # Mock response + mock_response = mocker.MagicMock() + mock_response.text = test_data + mock_response.status_code = 200 + + # Mock Session.get + mock_session = mocker.MagicMock() + mock_session.get.return_value = mock_response + mocker.patch( + "wxdi.dq_validator.provider.base_provider.Session", return_value=mock_session + ) + + # Create provider and call method + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + result = provider.get_published_artifact_by_id(MOCK_TERM_ID) + + # Verify + assert isinstance(result, GlossaryTerm) + assert result.metadata.artifact_id == MOCK_TERM_ID + assert result.metadata.name == "mango" + assert result.metadata.state == "PUBLISHED" + + # Verify URL was constructed correctly + expected_url = ( + f"{MOCK_URL}/v3/governance_artifact_types/glossary_term/{MOCK_TERM_ID}" + ) + mock_session.get.assert_called_once() + call_args = mock_session.get.call_args + assert call_args[0][0] == expected_url + + def test_get_published_artifact_with_options(self, mocker): + """Test retrieval with query options""" + test_data = load_test_data("term_latest_version.json") + + mock_response = mocker.MagicMock() + mock_response.text = test_data + mock_response.status_code = 200 + + mock_session = mocker.MagicMock() + mock_session.get.return_value = mock_response + mocker.patch( + "wxdi.dq_validator.provider.base_provider.Session", return_value=mock_session + ) + + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + options = {"include": "metadata", "limit": "10"} + result = provider.get_published_artifact_by_id(MOCK_TERM_ID, options) + + # Verify result + assert isinstance(result, GlossaryTerm) + + # Verify URL includes query parameters + call_args = mock_session.get.call_args + called_url = call_args[0][0] + assert "include=metadata" in called_url + assert "limit=10" in called_url + + def test_get_term_by_version_id_success(self, mocker): + """Test successful retrieval of term by version ID""" + # Load test data with full details + test_data = load_test_data("term_response.json") + + mock_response = mocker.MagicMock() + mock_response.text = test_data + mock_response.status_code = 200 + + mock_session = mocker.MagicMock() + mock_session.get.return_value = mock_response + mocker.patch( + "wxdi.dq_validator.provider.base_provider.Session", return_value=mock_session + ) + + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + result = provider.get_term_by_version_id(MOCK_TERM_ID, MOCK_VERSION_ID) + + # Verify + assert isinstance(result, GlossaryTerm) + assert result.metadata.artifact_id == MOCK_TERM_ID + assert result.metadata.version_id == MOCK_VERSION_ID + assert result.metadata.name == "mango" + assert result.metadata.effective_start_date is not None + + # Verify extended attributes + assert result.entity is not None + assert len(result.entity.extended_attribute_groups.dq_constraints) == 2 + + # Verify URL was constructed correctly + expected_url = ( + f"{MOCK_URL}/v3/glossary_terms/{MOCK_TERM_ID}/versions/{MOCK_VERSION_ID}" + ) + mock_session.get.assert_called_once() + call_args = mock_session.get.call_args + assert call_args[0][0] == expected_url + + def test_get_term_by_version_id_with_options(self, mocker): + """Test retrieval by version ID with query options""" + test_data = load_test_data("term_response.json") + + mock_response = mocker.MagicMock() + mock_response.text = test_data + mock_response.status_code = 200 + + mock_session = mocker.MagicMock() + mock_session.get.return_value = mock_response + mocker.patch( + "wxdi.dq_validator.provider.base_provider.Session", return_value=mock_session + ) + + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + options = {"included_extended_attribute_groups": "dq_constraints"} + result = provider.get_term_by_version_id(MOCK_TERM_ID, MOCK_VERSION_ID, options) + + # Verify result + assert isinstance(result, GlossaryTerm) + + # Verify URL includes query parameters + call_args = mock_session.get.call_args + called_url = call_args[0][0] + assert "included_extended_attribute_groups=dq_constraints" in called_url + + def test_get_term_by_version_id_draft(self, mocker): + """Test successful retrieval of term by version ID""" + # Load test data with full details + test_data = load_test_data("term_draft.json") + + mock_response = mocker.MagicMock() + mock_response.text = test_data + mock_response.status_code = 200 + + mock_session = mocker.MagicMock() + mock_session.get.return_value = mock_response + mocker.patch( + "wxdi.dq_validator.provider.base_provider.Session", return_value=mock_session + ) + + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + result = provider.get_term_by_version_id(MOCK_TERM_ID, MOCK_VERSION_ID) + + # Verify + assert isinstance(result, GlossaryTerm) + assert result.metadata.artifact_id == MOCK_TERM_ID + assert result.metadata.version_id == MOCK_VERSION_ID + assert result.metadata.name == "mango" + assert result.metadata.effective_start_date is None + assert result.metadata.draft_mode is not None + assert result.metadata.workflow_id is not None + assert result.metadata.workflow_state is not None + assert result.metadata.state == "DRAFT_HISTORY" + + # Verify extended attributes + assert result.entity.extended_attribute_groups is None + + # Verify URL was constructed correctly + expected_url = ( + f"{MOCK_URL}/v3/glossary_terms/{MOCK_TERM_ID}/versions/{MOCK_VERSION_ID}" + ) + mock_session.get.assert_called_once() + call_args = mock_session.get.call_args + assert call_args[0][0] == expected_url + + def test_request_headers_included(self, mocker): + """Test that authorization headers are included in requests""" + test_data = load_test_data("term_latest_version.json") + + mock_response = mocker.MagicMock() + mock_response.text = test_data + + mock_session = mocker.MagicMock() + mock_session.get.return_value = mock_response + mocker.patch( + "wxdi.dq_validator.provider.base_provider.Session", return_value=mock_session + ) + + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + provider.get_published_artifact_by_id(MOCK_TERM_ID) + + # Verify headers were passed as keyword argument + call_args = mock_session.get.call_args + assert "headers" in call_args.kwargs + headers = call_args.kwargs["headers"] + assert "Authorization" in headers + assert headers["Authorization"] == MOCK_AUTH_TOKEN + + +# ============================================================================ +# GlossaryTerm Model Tests +# ============================================================================ + + +class TestGlossaryTermModel: + """Test GlossaryTerm response model""" + + def test_parse_latest_version_response(self): + """Test parsing latest version response""" + test_data = load_test_data("term_latest_version.json") + term = GlossaryTerm.from_json(test_data) + + assert term.metadata.artifact_id == MOCK_TERM_ID + assert term.metadata.version_id == MOCK_VERSION_ID + assert term.metadata.name == "mango" + assert term.metadata.state == "PUBLISHED" + + def test_parse_full_term_response(self): + """Test parsing full term response with DQ constraints""" + test_data = load_test_data("term_response.json") + term = GlossaryTerm.from_json(test_data) + + # Verify metadata + assert term.metadata.artifact_id == MOCK_TERM_ID + assert term.metadata.name == "mango" + + # Verify DQ constraints + dq_constraints = term.entity.extended_attribute_groups.dq_constraints + assert len(dq_constraints) == 2 + + # First constraint: data_type + first_constraint = dq_constraints[0] + assert first_constraint.metadata.type == "data_type" + assert len(first_constraint.check) == 2 + + # Second constraint: length + second_constraint = dq_constraints[1] + assert second_constraint.metadata.type == "length" + assert len(second_constraint.check) == 2 + + def test_model_to_dict(self): + """Test converting model to dictionary""" + test_data = load_test_data("term_latest_version.json") + term = GlossaryTerm.from_json(test_data) + + result_dict = term.to_dict() + assert isinstance(result_dict, dict) + assert "metadata" in result_dict + assert "entity" in result_dict + assert result_dict["metadata"]["artifact_id"] == MOCK_TERM_ID + + def test_model_to_json(self): + """Test converting model to JSON string""" + test_data = load_test_data("term_latest_version.json") + term = GlossaryTerm.from_json(test_data) + + result_json = term.to_json() + assert isinstance(result_json, str) + + # Verify it's valid JSON + parsed = json.loads(result_json) + assert parsed["metadata"]["artifact_id"] == MOCK_TERM_ID + + def test_model_from_dict(self): + """Test creating model from dictionary""" + test_data = load_test_data("term_latest_version.json") + data_dict = json.loads(test_data) + + term = GlossaryTerm.from_dict(data_dict) + assert isinstance(term, GlossaryTerm) + assert term.metadata.artifact_id == MOCK_TERM_ID + + def test_dq_constraint_check_values(self): + """Test extracting check constraint values""" + test_data = load_test_data("term_response.json") + term = GlossaryTerm.from_json(test_data) + + # Get first constraint (data_type) + first_constraint = term.entity.extended_attribute_groups.dq_constraints[0] + + # Verify check values + checks = {check.name: check for check in first_constraint.check} + assert "data_type" in checks + assert checks["data_type"].value == "STRING" + assert "length" in checks + assert checks["length"].numeric_value == 80 + + # Get second constraint (length) + second_constraint = term.entity.extended_attribute_groups.dq_constraints[1] + checks = {check.name: check for check in second_constraint.check} + assert "min" in checks + assert checks["min"].numeric_value == 3 + assert "max" in checks + assert checks["max"].numeric_value == 80 + + +# ============================================================================ +# Integration Tests +# ============================================================================ + + +class TestGlossaryProviderIntegration: + """Integration tests for GlossaryProvider workflow""" + + def test_full_workflow_get_latest_then_version(self, mocker): + """Test complete workflow: get latest version, then get full details""" + # Load test data + latest_data = load_test_data("term_latest_version.json") + full_data = load_test_data("term_response.json") + + # Mock responses + mock_response_latest = mocker.MagicMock() + mock_response_latest.text = latest_data + + mock_response_full = mocker.MagicMock() + mock_response_full.text = full_data + + # Mock Session to return different responses + mock_session = mocker.MagicMock() + mock_session.get.side_effect = [mock_response_latest, mock_response_full] + mocker.patch( + "wxdi.dq_validator.provider.base_provider.Session", return_value=mock_session + ) + + # Create provider + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Step 1: Get latest version + latest_term = provider.get_published_artifact_by_id(MOCK_TERM_ID) + assert latest_term.metadata.version_id == MOCK_VERSION_ID + + # Step 2: Get full details using version_id + full_term = provider.get_term_by_version_id( + MOCK_TERM_ID, latest_term.metadata.version_id + ) + + # Verify full details + assert full_term.metadata.artifact_id == MOCK_TERM_ID + assert len(full_term.entity.extended_attribute_groups.dq_constraints) == 2 + + +# ============================================================================ +# Error Handling Tests +# ============================================================================ + + +class TestGlossaryProviderErrorHandling: + """Test error handling in GlossaryProvider""" + + def test_get_published_artifact_http_error(self, mocker): + """Test handling of HTTP errors (404, 500, etc.) in get_published_artifact_by_id""" + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a 404 response + mock_response = mocker.Mock() + mock_response.status_code = 404 + mock_response.ok = False + mock_response.text = json.dumps({"error": "Artifact not found"}) + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + # The implementation checks response.ok and raises ValueError + with pytest.raises(ValueError) as exc_info: + provider.get_published_artifact_by_id("non-existent-artifact") + + # Verify the error message contains artifact_id + assert "Cannot get artifact" in str(exc_info.value) + assert "non-existent-artifact" in str(exc_info.value) + + def test_get_term_by_version_id_http_error(self, mocker): + """Test handling of HTTP errors in get_term_by_version_id""" + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a 404 response + mock_response = mocker.Mock() + mock_response.status_code = 404 + mock_response.ok = False + mock_response.text = json.dumps({"error": "Version not found"}) + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + # The implementation checks response.ok and raises ValueError + with pytest.raises(ValueError) as exc_info: + provider.get_term_by_version_id("artifact-123", "version-456") + + # Verify the error message contains both artifact_id and version_id + assert "Cannot get artifact" in str(exc_info.value) + assert "artifact-123" in str(exc_info.value) + assert "version-456" in str(exc_info.value) + + def test_get_published_artifact_invalid_json(self, mocker): + """Test handling of invalid JSON response in get_published_artifact_by_id""" + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a response with invalid JSON + mock_response = mocker.Mock() + mock_response.ok = True + mock_response.text = "This is not valid JSON" + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + # Should raise JSONDecodeError + with pytest.raises(json.JSONDecodeError): + provider.get_published_artifact_by_id(MOCK_TERM_ID) + + def test_get_term_by_version_id_invalid_json(self, mocker): + """Test handling of invalid JSON response in get_term_by_version_id""" + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a response with invalid JSON + mock_response = mocker.Mock() + mock_response.ok = True + mock_response.text = "This is not valid JSON" + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + # Should raise JSONDecodeError + with pytest.raises(json.JSONDecodeError): + provider.get_term_by_version_id(MOCK_TERM_ID, MOCK_VERSION_ID) + + def test_get_published_artifact_malformed_response(self, mocker): + """Test handling of malformed response in get_published_artifact_by_id""" + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a response with valid JSON but missing required fields + malformed_json = {"some_field": "some_value"} + mock_response = mocker.Mock() + mock_response.ok = True + mock_response.text = json.dumps(malformed_json) + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + # Should raise ValidationError from Pydantic + with pytest.raises(Exception): # Pydantic ValidationError + provider.get_published_artifact_by_id(MOCK_TERM_ID) + + def test_get_term_by_version_id_malformed_response(self, mocker): + """Test handling of malformed response in get_term_by_version_id""" + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a response with valid JSON but missing required fields + malformed_json = {"some_field": "some_value"} + mock_response = mocker.Mock() + mock_response.ok = True + mock_response.text = json.dumps(malformed_json) + + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.return_value = mock_response + + # Should raise ValidationError from Pydantic + with pytest.raises(Exception): # Pydantic ValidationError + provider.get_term_by_version_id(MOCK_TERM_ID, MOCK_VERSION_ID) + + def test_get_published_artifact_network_error(self, mocker): + """Test handling of network errors in get_published_artifact_by_id""" + from requests.exceptions import ConnectionError + + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a network error + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.side_effect = ConnectionError( + "Network unreachable" + ) + + # Should raise ConnectionError + with pytest.raises(ConnectionError): + provider.get_published_artifact_by_id(MOCK_TERM_ID) + + def test_get_term_by_version_id_network_error(self, mocker): + """Test handling of network errors in get_term_by_version_id""" + from requests.exceptions import ConnectionError + + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a network error + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.side_effect = ConnectionError( + "Network unreachable" + ) + + # Should raise ConnectionError + with pytest.raises(ConnectionError): + provider.get_term_by_version_id(MOCK_TERM_ID, MOCK_VERSION_ID) + + def test_get_published_artifact_timeout(self, mocker): + """Test handling of timeout errors in get_published_artifact_by_id""" + from requests.exceptions import Timeout + + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a timeout error + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.side_effect = Timeout("Request timed out") + + # Should raise Timeout + with pytest.raises(Timeout): + provider.get_published_artifact_by_id(MOCK_TERM_ID) + + def test_get_term_by_version_id_timeout(self, mocker): + """Test handling of timeout errors in get_term_by_version_id""" + from requests.exceptions import Timeout + + config = ProviderConfig(MOCK_URL, MOCK_AUTH_TOKEN) + provider = GlossaryProvider(config) + + # Mock a timeout error + mock_session = mocker.patch("wxdi.dq_validator.provider.base_provider.Session") + mock_session.return_value.get.side_effect = Timeout("Request timed out") + + # Should raise Timeout + with pytest.raises(Timeout): + provider.get_term_by_version_id(MOCK_TERM_ID, MOCK_VERSION_ID) + + +# Made with Bob diff --git a/tests/src/dq_validator/provider/test_issues.py b/tests/src/dq_validator/provider/test_issues.py new file mode 100644 index 0000000..80ee668 --- /dev/null +++ b/tests/src/dq_validator/provider/test_issues.py @@ -0,0 +1,1129 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +import pytest +from unittest.mock import Mock, patch, MagicMock +import json + +from wxdi.dq_validator.provider import ProviderConfig, IssuesProvider + + +class TestIssuesProvider: + """Test suite for IssuesProvider class.""" + + @pytest.fixture + def config(self): + """Create a test configuration.""" + return ProviderConfig( + url="https://test-instance.com", + auth_token="Bearer test-token" + ) + + @pytest.fixture + def provider(self, config): + """Create a test IssuesProvider instance.""" + with patch('wxdi.dq_validator.provider.base_provider.Session') as mock_session_class: + mock_session = Mock() + mock_session_class.return_value = mock_session + provider = IssuesProvider(config) + yield provider + + def test_update_issue_values_both_fields_with_project_id(self, provider): + """Test updating both occurrences and tested records with project_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issue_id": "issue-123", + "number_of_occurrences": 777, + "number_of_tested_records": 1100, + "status": "updated" + }) + provider.session.patch.return_value = mock_response + + # Execute + result = provider.update_issue_values("issue-123", occurrences=10, tested_records=100, project_id="project-123") + + # Verify + assert result["issue_id"] == "issue-123" + assert result["number_of_occurrences"] == 777 + assert result["number_of_tested_records"] == 1100 + assert result["status"] == "updated" + + # Verify the API call + provider.session.patch.assert_called_once() + call_args = provider.session.patch.call_args + + # Check URL + assert "https://test-instance.com/data_quality/v4/issues/issue-123" in call_args[0][0] + assert "project_id=project-123" in call_args[0][0] + + # Check headers + headers = call_args[1]["headers"] + assert headers["Authorization"] == "Bearer test-token" + assert headers["Content-Type"] == "application/json-patch+json" + + # Check payload - should have both operations + payload = json.loads(call_args[1]["data"]) + assert len(payload) == 2 + assert payload[0]["op"] == "add" + assert payload[0]["path"] == "/number_of_occurrences" + assert payload[0]["value"] == 10 + assert payload[1]["op"] == "add" + assert payload[1]["path"] == "/number_of_tested_records" + assert payload[1]["value"] == 100 + + def test_update_issue_values_both_fields_with_catalog_id(self, provider): + """Test updating both occurrences and tested records with catalog_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issue_id": "issue-123", + "number_of_occurrences": 777, + "number_of_tested_records": 1100, + "status": "updated" + }) + provider.session.patch.return_value = mock_response + + # Execute + result = provider.update_issue_values("issue-123", occurrences=10, tested_records=100, catalog_id="catalog-456") + + # Verify + assert result["issue_id"] == "issue-123" + + # Verify the API call + call_args = provider.session.patch.call_args + + # Check URL contains catalog_id + assert "catalog_id=catalog-456" in call_args[0][0] + assert "project_id" not in call_args[0][0] + + def test_update_issue_values_missing_both_ids(self, provider): + """Test updating without project_id or catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.update_issue_values("issue-123", occurrences=10, tested_records=100) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_update_issue_values_both_ids_provided(self, provider): + """Test updating with both project_id and catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.update_issue_values( + "issue-123", + occurrences=10, + tested_records=100, + project_id="project-123", + catalog_id="catalog-456" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + + def test_update_issue_values_with_replace_operation(self, provider): + """Test updating with replace operation.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issue_id": "issue-999", + "number_of_occurrences": 100, + "number_of_tested_records": 1000, + "status": "updated" + }) + provider.session.patch.return_value = mock_response + + # Execute + result = provider.update_issue_values( + "issue-999", + occurrences=100, + tested_records=1000, + project_id="project-123", + operation="replace" + ) + + # Verify + assert result["issue_id"] == "issue-999" + assert result["number_of_occurrences"] == 100 + assert result["number_of_tested_records"] == 1000 + + # Verify the API call + call_args = provider.session.patch.call_args + payload = json.loads(call_args[1]["data"]) + assert len(payload) == 2 + assert payload[0]["op"] == "replace" + assert payload[0]["path"] == "/number_of_occurrences" + assert payload[0]["value"] == 100 + assert payload[1]["op"] == "replace" + assert payload[1]["path"] == "/number_of_tested_records" + assert payload[1]["value"] == 1000 + + def test_update_issue_values_failure(self, provider): + """Test failed update of issue values.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 404 + mock_response.text = "Issue not found" + provider.session.patch.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.update_issue_values("invalid-issue", occurrences=10, tested_records=100, project_id="project-123") + + assert "Failed to update issue metrics for issue invalid-issue" in str(exc_info.value) + assert "404" in str(exc_info.value) + + def test_update_issue_values_with_zero_values(self, provider): + """Test updating with zero values (should be included).""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issue_id": "issue-000", + "number_of_occurrences": 0, + "number_of_tested_records": 0, + "status": "updated" + }) + provider.session.patch.return_value = mock_response + + # Execute + result = provider.update_issue_values("issue-000", occurrences=0, tested_records=0, project_id="project-123") + + # Verify + assert result["number_of_occurrences"] == 0 + assert result["number_of_tested_records"] == 0 + + # Verify the API call includes both zero values + call_args = provider.session.patch.call_args + payload = json.loads(call_args[1]["data"]) + assert len(payload) == 2 + assert payload[0]["value"] == 0 + assert payload[1]["value"] == 0 + + def test_get_issue_id_success_with_project_id(self, provider): + """Test successful retrieval of issue ID with project_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "id": "b8f4252b-cd35-4668-9b35-4635bfc6e2e0", + "account_id": "999", + "created_at": "2025-12-19T06:37:24.955Z", + "check": { + "id": "ad277842-dea7-44ef-8e4b-d940df0f79aa", + "account_id": "999", + "created_at": "2025-12-19T06:37:23.519Z", + "dimension": { + "id": "ec453723-669c-48bb-82c1-11b69b3b8c93", + "name": "Validity", + "description": "Data is valid if it conforms to the syntax (format, type, range) of its definition.", + "is_default": True + }, + "name": "Format check", + "native_id": "b2debda2-6ab9-4a39-8c23-17954e004dcf/7377e2cd-ac0e-4833-8760-fd0e8cb682aa", + "wkc_container_id": "24419069-d649-45cb-a2c1-64d6eed650d5", + "parent": { + "id": "f3fca1af-f00f-42b7-af42-f0965673237e" + }, + "type": "format" + }, + "reported_for": { + "id": "1488a413-99f9-4bed-906d-c33b505d5728", + "native_id": "b2debda2-6ab9-4a39-8c23-17954e004dcf/RTN" + }, + "number_of_occurrences": 0, + "number_of_tested_records": 1000, + "percent_occurrences": 0, + "status": "actual", + "ignored": False, + "details": "{\"sampling\":{\"size\":1000,\"min_records\":0,\"type\":\"SEQUENTIAL\"}}", + "archived_issues": [] + }) + provider.session.post.return_value = mock_response + + # Execute + result = provider.get_issue_id( + reported_for_id="1488a413-99f9-4bed-906d-c33b505d5728", + dq_check_id="ad277842-dea7-44ef-8e4b-d940df0f79aa", + project_id="24419069-d649-45cb-a2c1-64d6eed650d5" + ) + + # Verify + assert result == "b8f4252b-cd35-4668-9b35-4635bfc6e2e0" + + # Verify the API call + provider.session.post.assert_called_once() + call_args = provider.session.post.call_args + url = call_args[0][0] + + # Check URL contains required parameters + assert "project_id=24419069-d649-45cb-a2c1-64d6eed650d5" in url + assert "reported_for.id=1488a413-99f9-4bed-906d-c33b505d5728" in url + assert "check.id=ad277842-dea7-44ef-8e4b-d940df0f79aa" in url + + def test_get_issue_id_success_with_catalog_id(self, provider): + """Test successful retrieval of issue ID with catalog_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "id": "issue-id-789", + "catalog_id": "catalog-123" + }) + provider.session.post.return_value = mock_response + + # Execute + result = provider.get_issue_id( + reported_for_id="asset-id", + dq_check_id="check-id", + catalog_id="catalog-123" + ) + + # Verify + assert result == "issue-id-789" + + # Verify the API call + call_args = provider.session.post.call_args + url = call_args[0][0] + + # Check URL contains catalog_id + assert "catalog_id=catalog-123" in url + assert "project_id" not in url + + def test_get_issue_id_missing_both_ids(self, provider): + """Test get_issue_id without project_id or catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.get_issue_id( + reported_for_id="asset-id", + dq_check_id="check-id" + ) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_get_issue_id_both_ids_provided(self, provider): + """Test get_issue_id with both project_id and catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.get_issue_id( + reported_for_id="asset-id", + dq_check_id="check-id", + project_id="project-123", + catalog_id="catalog-456" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + def test_get_issue_id_failure(self, provider): + """Test failed retrieval of issue ID.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 404 + mock_response.text = "Not found" + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.get_issue_id( + reported_for_id="invalid-asset", + dq_check_id="invalid-check", + project_id="invalid-project" + ) + + assert "Failed to search for issue" in str(exc_info.value) + assert "404" in str(exc_info.value) + + def test_update_issue_metrics(self, provider): + """Test updating issue metrics using CAMS asset and check IDs.""" + with patch('wxdi.dq_validator.provider.dq_search.DQSearchProvider') as mock_search_provider_class: + # Setup mock search provider + mock_search_provider = Mock() + mock_search_provider_class.return_value = mock_search_provider + + # Mock search_dq_asset response + mock_search_provider.search_dq_asset.return_value = { + "id": "dq-asset-123", + "name": "Test Asset", + "type": "column" + } + + # Mock get_issues response (replaces search_dq_check and get_issue_id) + mock_get_issues_response = Mock() + mock_get_issues_response.ok = True + mock_get_issues_response.text = json.dumps({ + "issues": [ + { + "id": "issue-789", + "check": { + "id": "dq-check-456", + "native_id": "cams-asset-abc/cams-check-def" + }, + "project_id": "project-123" + } + ], + "total_count": 1 + }) + provider.session.get.return_value = mock_get_issues_response + + # Mock update_issue_values response + mock_update_response = Mock() + mock_update_response.ok = True + mock_update_response.text = json.dumps({ + "issue_id": "issue-789", + "number_of_occurrences": 10, + "number_of_tested_records": 100 + }) + provider.session.patch.return_value = mock_update_response + + # Execute + result = provider.update_issue_metrics( + asset_id="cams-asset-abc", + check_id="cams-check-def", + occurrences=10, + tested_records=100, + column_name="test_column", + check_type="format", + project_id="project-123", + asset_type="column" + ) + + # Verify + assert result["issue_id"] == "issue-789" + assert result["number_of_occurrences"] == 10 + assert result["number_of_tested_records"] == 100 + + # Verify search_dq_asset was called correctly + mock_search_provider.search_dq_asset.assert_called_once_with( + native_id="cams-asset-abc/test_column", + project_id="project-123", + catalog_id=None, + asset_type="column" + ) + + # Verify get_issues (GET request) was called instead of search_dq_check + provider.session.get.assert_called_once() + call_args = provider.session.get.call_args + url = call_args[0][0] + assert "reported_for.id=dq-asset-123" in url + assert "type=format" in url + assert "include_children=false" in url + + def test_update_issue_metrics_with_check_native_id(self, provider): + """Test updating issue metrics using check_native_id instead of asset_id and check_id.""" + with patch('wxdi.dq_validator.provider.dq_search.DQSearchProvider') as mock_search_provider_class: + # Setup mock search provider + mock_search_provider = Mock() + mock_search_provider_class.return_value = mock_search_provider + + # Mock search_dq_asset response + mock_search_provider.search_dq_asset.return_value = { + "id": "dq-asset-456", + "name": "Test Asset", + "type": "column" + } + + # Mock get_issues response + mock_get_issues_response = Mock() + mock_get_issues_response.ok = True + mock_get_issues_response.text = json.dumps({ + "issues": [ + { + "id": "issue-999", + "check": { + "id": "dq-check-789", + "native_id": "asset-abc/column_name/case" + }, + "project_id": "project-456" + } + ], + "total_count": 1 + }) + provider.session.get.return_value = mock_get_issues_response + + # Mock update_issue_values response + mock_update_response = Mock() + mock_update_response.ok = True + mock_update_response.text = json.dumps({ + "issue_id": "issue-999", + "number_of_occurrences": 25, + "number_of_tested_records": 250 + }) + provider.session.patch.return_value = mock_update_response + + # Execute - using check_native_id instead of asset_id and check_id + result = provider.update_issue_metrics( + check_native_id="asset-abc/column_name/case", + occurrences=25, + tested_records=250, + column_name="column_name", + check_type="case", + project_id="project-456", + asset_type="column" + ) + + # Verify + assert result["issue_id"] == "issue-999" + assert result["number_of_occurrences"] == 25 + assert result["number_of_tested_records"] == 250 + + # Verify search_dq_asset was called with extracted asset_id + mock_search_provider.search_dq_asset.assert_called_once_with( + native_id="asset-abc/column_name", + project_id="project-456", + catalog_id=None, + asset_type="column" + ) + + def test_validate_and_resolve_ids_with_asset_and_check_ids(self, provider): + """Test _validate_and_resolve_ids with asset_id and check_id provided.""" + asset_id, check_id, check_native_id = provider._validate_and_resolve_ids( + asset_id="asset-123", + check_id="check-456", + check_native_id=None + ) + + assert asset_id == "asset-123" + assert check_id == "check-456" + assert check_native_id == "asset-123/check-456" + + def test_validate_and_resolve_ids_with_check_native_id(self, provider): + """Test _validate_and_resolve_ids with check_native_id provided.""" + asset_id, check_id, check_native_id = provider._validate_and_resolve_ids( + asset_id=None, + check_id=None, + check_native_id="asset-abc/check-def" + ) + + assert asset_id == "asset-abc" + assert check_id == "check-def" + assert check_native_id == "asset-abc/check-def" + + def test_validate_and_resolve_ids_with_check_native_id_containing_slashes(self, provider): + """Test _validate_and_resolve_ids with check_native_id containing multiple slashes.""" + asset_id, check_id, check_native_id = provider._validate_and_resolve_ids( + asset_id=None, + check_id=None, + check_native_id="asset-123/column/check-type" + ) + + assert asset_id == "asset-123" + assert check_id == "column/check-type" + assert check_native_id == "asset-123/column/check-type" + + def test_validate_and_resolve_ids_with_both_provided(self, provider): + """Test _validate_and_resolve_ids when both asset_id/check_id and check_native_id are provided.""" + # When both are provided, the original values are kept (no parsing or construction happens) + asset_id, check_id, check_native_id = provider._validate_and_resolve_ids( + asset_id="asset-111", + check_id="check-222", + check_native_id="asset-999/check-888" + ) + + # Original values should be returned as-is + assert asset_id == "asset-111" + assert check_id == "check-222" + assert check_native_id == "asset-999/check-888" + + def test_validate_and_resolve_ids_with_neither_provided(self, provider): + """Test _validate_and_resolve_ids raises error when neither IDs are provided.""" + with pytest.raises(ValueError) as exc_info: + provider._validate_and_resolve_ids( + asset_id=None, + check_id=None, + check_native_id=None + ) + + assert "Either (asset_id and check_id) or check_native_id must be provided" in str(exc_info.value) + + def test_validate_and_resolve_ids_with_only_asset_id(self, provider): + """Test _validate_and_resolve_ids raises error when only asset_id is provided.""" + with pytest.raises(ValueError) as exc_info: + provider._validate_and_resolve_ids( + asset_id="asset-123", + check_id=None, + check_native_id=None + ) + + assert "Either (asset_id and check_id) or check_native_id must be provided" in str(exc_info.value) + + def test_validate_and_resolve_ids_with_only_check_id(self, provider): + """Test _validate_and_resolve_ids raises error when only check_id is provided.""" + with pytest.raises(ValueError) as exc_info: + provider._validate_and_resolve_ids( + asset_id=None, + check_id="check-456", + check_native_id=None + ) + + assert "Either (asset_id and check_id) or check_native_id must be provided" in str(exc_info.value) + + def test_validate_and_resolve_ids_with_invalid_check_native_id_format(self, provider): + """Test _validate_and_resolve_ids raises error for invalid check_native_id format.""" + with pytest.raises(ValueError) as exc_info: + provider._validate_and_resolve_ids( + asset_id=None, + check_id=None, + check_native_id="invalid-format-no-slash" + ) + + assert "Invalid check_native_id format (missing /)" in str(exc_info.value) + assert "invalid-format-no-slash" in str(exc_info.value) + + def test_get_issues_with_catalog_id(self, provider): + """Test getting issues with catalog_id and check_id filter.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issues": [ + { + "id": "issue-1", + "check": { + "id": "check-1", + "native_id": "asset-id/065c2b72-4600-4d15-8c48-298a2abf66cd" + }, + "check_name": "Completeness Check", + "number_of_occurrences": 5, + "number_of_tested_records": 100 + }, + { + "id": "issue-2", + "check": { + "id": "check-2", + "native_id": "asset-id/other-check-id" + }, + "check_name": "Format Check", + "number_of_occurrences": 3, + "number_of_tested_records": 100 + } + ], + "total_count": 2 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_issues( + dq_asset_id="08b139ca-35a6-4b61-b87b-aa832870d89c", + check_type="completeness", + check_id="065c2b72-4600-4d15-8c48-298a2abf66cd", + catalog_id="07708fd8-8d77-4a07-a01b-0132130bce0e", + limit=20, + latest_only=True, + include_children=False, + sort_by="check_name", + sort_direction="asc" + ) + + # Verify - should return only the matching issue + assert result is not None + assert result["id"] == "issue-1" + assert result["check"]["id"] == "check-1" + + # Verify the API call + provider.session.get.assert_called_once() + call_args = provider.session.get.call_args + url = call_args[0][0] + + # Check URL contains required parameters + assert "catalog_id=07708fd8-8d77-4a07-a01b-0132130bce0e" in url + assert "reported_for.id=08b139ca-35a6-4b61-b87b-aa832870d89c" in url + assert "type=completeness" in url + assert "limit=20" in url + assert "latest_only=true" in url + assert "include_children=false" in url + assert "sort_by=check_name" in url + assert "sort_direction=asc" in url + + def test_get_issues_with_project_id(self, provider): + """Test getting issues with project_id and check_id filter.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issues": [ + { + "id": "issue-3", + "check": { + "id": "check-3", + "native_id": "asset-id/test-check-id" + }, + "check_name": "Range Check", + "number_of_occurrences": 10, + "number_of_tested_records": 200 + } + ], + "total_count": 1 + }) + provider.session.get.return_value = mock_response + + # Execute + result = provider.get_issues( + dq_asset_id="asset-id-123", + check_type="range", + check_id="test-check-id", + project_id="project-456" + ) + + # Verify - should return the matching issue + assert result is not None + assert result["id"] == "issue-3" + + # Verify the API call + call_args = provider.session.get.call_args + url = call_args[0][0] + + # Check URL contains project_id + assert "project_id=project-456" in url + assert "catalog_id" not in url + assert "include_children=false" in url + + def test_get_issues_missing_both_ids(self, provider): + """Test get_issues without project_id or catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.get_issues( + dq_asset_id="asset-id", + check_type="completeness", + check_id="test-check-id" + ) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_get_issues_both_ids_provided(self, provider): + """Test get_issues with both project_id and catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.get_issues( + dq_asset_id="asset-id", + check_type="completeness", + check_id="test-check-id", + project_id="project-123", + catalog_id="catalog-456" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + def test_get_issues_failure(self, provider): + """Test failed retrieval of issues.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 404 + mock_response.text = "Not found" + provider.session.get.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.get_issues( + dq_asset_id="invalid-asset", + check_type="completeness", + check_id="test-check-id", + project_id="invalid-project" + ) + + assert "Failed to get issues" in str(exc_info.value) + assert "404" in str(exc_info.value) + + def test_get_issues_with_minimal_params(self, provider): + """Test getting issues with minimal parameters (using defaults).""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issues": [ + { + "id": "issue-4", + "check": { + "id": "check-4", + "native_id": "asset-id/minimal-check-id" + } + } + ], + "total_count": 1 + }) + provider.session.get.return_value = mock_response + + # Execute with minimal params + result = provider.get_issues( + dq_asset_id="asset-id", + check_type="datatype", + check_id="minimal-check-id", + catalog_id="catalog-123" + ) + + # Verify - should return the matching issue + assert result is not None + assert result["id"] == "issue-4" + + # Verify the API call includes default values + call_args = provider.session.get.call_args + url = call_args[0][0] + + assert "limit=20" in url + assert "latest_only=true" in url + assert "include_children=false" in url + assert "sort_by=check_name" in url + assert "sort_direction=asc" in url + + def test_get_issues_no_match_found(self, provider): + """Test get_issues when no matching check_id is found.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issues": [ + { + "id": "issue-5", + "check": { + "id": "check-5", + "native_id": "asset-id/different-check-id" + } + } + ], + "total_count": 1 + }) + provider.session.get.return_value = mock_response + + # Execute with a check_id that doesn't match + result = provider.get_issues( + dq_asset_id="asset-id", + check_type="format", + check_id="non-existent-check-id", + catalog_id="catalog-123" + ) + + # Verify - should return None when no match found + assert result is None + + def test_create_issue_with_project_id(self, provider): + """Test creating an issue with project_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "id": "046605b5-48d9-489e-b846-8ef96a7a1aba", + "check": { + "id": "6be18374-573a-4cf8-8ab7-e428506e428b" + }, + "reported_for": { + "id": "894d01fd-bdfc-4a4f-b68b-62751e06e06a" + }, + "number_of_occurrences": 123, + "number_of_tested_records": 456789, + "status": "actual", + "ignored": False + }) + provider.session.post.return_value = mock_response + + # Execute + result = provider.create_issue( + dq_check_id="6be18374-573a-4cf8-8ab7-e428506e428b", + reported_for_id="894d01fd-bdfc-4a4f-b68b-62751e06e06a", + number_of_occurrences=123, + number_of_tested_records=456789, + project_id="project-123" + ) + + # Verify + assert result == "046605b5-48d9-489e-b846-8ef96a7a1aba" + + # Verify the API call + provider.session.post.assert_called_once() + call_args = provider.session.post.call_args + + # Check URL + assert "https://test-instance.com/data_quality/v4/issues" in call_args[0][0] + assert "project_id=project-123" in call_args[0][0] + + # Check payload + payload = json.loads(call_args[1]["data"]) + assert payload["check"]["id"] == "6be18374-573a-4cf8-8ab7-e428506e428b" + assert payload["reported_for"]["id"] == "894d01fd-bdfc-4a4f-b68b-62751e06e06a" + assert payload["number_of_occurrences"] == 123 + assert payload["number_of_tested_records"] == 456789 + + def test_create_issue_missing_both_ids(self, provider): + """Test creating an issue without project_id or catalog_id.""" + with pytest.raises(ValueError) as exc_info: + provider.create_issue( + dq_check_id="check-123", + reported_for_id="asset-456", + number_of_occurrences=10, + number_of_tested_records=100 + ) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_create_issue_failure(self, provider): + """Test failed creation of issue.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 400 + mock_response.text = "Bad request" + provider.session.post.return_value = mock_response + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.create_issue( + dq_check_id="invalid-check", + reported_for_id="invalid-asset", + number_of_occurrences=10, + number_of_tested_records=100, + project_id="project-123" + ) + + assert "Failed to create issue" in str(exc_info.value) + assert "400" in str(exc_info.value) + + def test_create_issues_bulk_with_project_id(self, provider): + """Test creating multiple issues in bulk with project_id.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issues": [ + { + "id": "issue-bulk-1", + "check": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/Validity", + "type": "format" + }, + "status": "aggregation" + }, + { + "id": "issue-bulk-2", + "check": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/sample3", + "type": "format" + }, + "status": "actual" + } + ], + "assets": [ + { + "id": "asset-bulk-1", + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f", + "type": "data_asset" + }, + { + "id": "asset-bulk-2", + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/NAME", + "type": "column" + } + ] + }) + provider.session.post.return_value = mock_response + + # Prepare bulk payload + bulk_payload = { + "issues": [ + { + "check": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/Validity", + "type": "format" + }, + "reported_for": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f", + "type": "data_asset" + }, + "number_of_occurrences": 200, + "number_of_tested_records": 1000, + "status": "aggregation", + "ignored": False + }, + { + "check": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/sample3", + "type": "format" + }, + "reported_for": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/NAME", + "type": "column" + }, + "number_of_occurrences": 200, + "number_of_tested_records": 1000, + "status": "actual", + "ignored": False + } + ], + "assets": [ + { + "name": "ACCOUNT_HOLDERS.csv", + "type": "data_asset", + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f", + "weight": 1 + }, + { + "name": "NAME", + "type": "column", + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/NAME", + "parent": { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f", + "type": "data_asset" + }, + "weight": 1 + } + ], + "existing_checks": [ + { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/Validity", + "type": "format" + }, + { + "native_id": "ba23145a-6d0a-46db-b314-41526b1e465f/format/sample3", + "type": "format" + } + ] + } + + # Execute + result = provider.create_issues_bulk( + payload=bulk_payload, + project_id="project-123", + incremental_reporting=False, + refresh_assets=False + ) + + # Verify + assert "issues" in result + assert len(result["issues"]) == 2 + assert result["issues"][0]["id"] == "issue-bulk-1" + assert result["issues"][1]["id"] == "issue-bulk-2" + + # Verify the API call + provider.session.post.assert_called_once() + call_args = provider.session.post.call_args + + # Check URL + assert "https://test-instance.com/data_quality/v4/create_issues" in call_args[0][0] + assert "project_id=project-123" in call_args[0][0] + assert "incremental_reporting=false" in call_args[0][0] + assert "refresh_assets=false" in call_args[0][0] + + # Check payload + payload = json.loads(call_args[1]["data"]) + assert len(payload["issues"]) == 2 + assert len(payload["assets"]) == 2 + assert len(payload["existing_checks"]) == 2 + + def test_create_issues_bulk_with_catalog_id_and_incremental_reporting(self, provider): + """Test creating bulk issues with catalog_id and incremental_reporting=True.""" + # Setup mock + mock_response = Mock() + mock_response.ok = True + mock_response.text = json.dumps({ + "issues": [ + {"id": "issue-1", "status": "aggregation"}, + {"id": "issue-2", "status": "actual"} + ] + }) + provider.session.post.return_value = mock_response + + # Minimal bulk payload + bulk_payload = { + "issues": [ + { + "check": {"native_id": "asset/check1", "type": "format"}, + "reported_for": {"native_id": "asset", "type": "data_asset"}, + "number_of_occurrences": 10, + "number_of_tested_records": 100, + "status": "aggregation", + "ignored": False + } + ], + "assets": [ + {"name": "Test Asset", "type": "data_asset", "native_id": "asset", "weight": 1} + ], + "existing_checks": [ + {"native_id": "asset/check1", "type": "format"} + ] + } + + # Execute + result = provider.create_issues_bulk( + payload=bulk_payload, + catalog_id="catalog-456", + incremental_reporting=True, + refresh_assets=True + ) + + # Verify + assert "issues" in result + assert len(result["issues"]) == 2 + + # Verify the API call + call_args = provider.session.post.call_args + url = call_args[0][0] + + # Check URL contains catalog_id and boolean params + assert "catalog_id=catalog-456" in url + assert "project_id" not in url + assert "incremental_reporting=true" in url + assert "refresh_assets=true" in url + + def test_create_issues_bulk_missing_both_ids(self, provider): + """Test creating bulk issues without project_id or catalog_id.""" + bulk_payload = { + "issues": [], + "assets": [], + "existing_checks": [] + } + + with pytest.raises(ValueError) as exc_info: + provider.create_issues_bulk(payload=bulk_payload) + + assert "Either project_id or catalog_id must be provided" in str(exc_info.value) + + def test_create_issues_bulk_both_ids_provided(self, provider): + """Test creating bulk issues with both project_id and catalog_id.""" + bulk_payload = { + "issues": [], + "assets": [], + "existing_checks": [] + } + + with pytest.raises(ValueError) as exc_info: + provider.create_issues_bulk( + payload=bulk_payload, + project_id="project-123", + catalog_id="catalog-456" + ) + + assert "Only one of project_id or catalog_id should be provided" in str(exc_info.value) + + def test_create_issues_bulk_failure(self, provider): + """Test failed bulk issue creation.""" + # Setup mock + mock_response = Mock() + mock_response.ok = False + mock_response.status_code = 400 + mock_response.text = "Invalid payload" + provider.session.post.return_value = mock_response + + bulk_payload = { + "issues": [], + "assets": [], + "existing_checks": [] + } + + # Execute and verify exception + with pytest.raises(ValueError) as exc_info: + provider.create_issues_bulk( + payload=bulk_payload, + project_id="project-123" + ) + + assert "Failed to create issues in bulk" in str(exc_info.value) + assert "400" in str(exc_info.value) + assert "Invalid payload" in str(exc_info.value) \ No newline at end of file diff --git a/tests/src/dq_validator/provider/test_thread_safety.py b/tests/src/dq_validator/provider/test_thread_safety.py new file mode 100644 index 0000000..1f9ae19 --- /dev/null +++ b/tests/src/dq_validator/provider/test_thread_safety.py @@ -0,0 +1,129 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Tests for thread safety of provider classes +""" + +import pytest +import threading +from unittest.mock import Mock, patch +from wxdi.dq_validator.provider import ProviderConfig, DQSearchProvider, IssuesProvider + + +class TestThreadSafety: + """Test suite for thread safety of provider classes.""" + + @pytest.fixture + def config(self): + """Create a test configuration.""" + return ProviderConfig( + url="https://test-instance.com", + auth_token="Bearer test-token" + ) + + def test_dq_search_provider_thread_local_sessions(self, config): + """Test that each thread gets its own session in DQSearchProvider.""" + with patch('wxdi.dq_validator.provider.base_provider.Session') as mock_session_class: + provider = DQSearchProvider(config) + + # Track session instances created + sessions_created = [] + + def mock_session_factory(): + session = Mock() + sessions_created.append(session) + return session + + mock_session_class.side_effect = mock_session_factory + + # Access session from main thread + main_session = provider.session + assert len(sessions_created) == 1 + + # Access session from another thread + thread_sessions = [] + + def access_session(): + thread_sessions.append(provider.session) + + thread = threading.Thread(target=access_session) + thread.start() + thread.join() + + # Should have created a second session for the new thread + assert len(sessions_created) == 2 + assert main_session is sessions_created[0] + assert thread_sessions[0] is sessions_created[1] + assert main_session is not thread_sessions[0] + + def test_issues_provider_thread_local_sessions(self, config): + """Test that each thread gets its own session in IssuesProvider.""" + with patch('wxdi.dq_validator.provider.base_provider.Session') as mock_session_class: + provider = IssuesProvider(config) + + # Track session instances created + sessions_created = [] + + def mock_session_factory(): + session = Mock() + sessions_created.append(session) + return session + + mock_session_class.side_effect = mock_session_factory + + # Access session from main thread + main_session = provider.session + assert len(sessions_created) == 1 + + # Access session from another thread + thread_sessions = [] + + def access_session(): + thread_sessions.append(provider.session) + + thread = threading.Thread(target=access_session) + thread.start() + thread.join() + + # Should have created a second session for the new thread + assert len(sessions_created) == 2 + assert main_session is sessions_created[0] + assert thread_sessions[0] is sessions_created[1] + assert main_session is not thread_sessions[0] + + def test_dq_search_provider_session_reuse_within_thread(self, config): + """Test that session is reused within the same thread.""" + with patch('wxdi.dq_validator.provider.base_provider.Session') as mock_session_class: + provider = DQSearchProvider(config) + + sessions_created = [] + + def mock_session_factory(): + session = Mock() + sessions_created.append(session) + return session + + mock_session_class.side_effect = mock_session_factory + + # Access session multiple times in the same thread + session1 = provider.session + session2 = provider.session + session3 = provider.session + + # Should only create one session + assert len(sessions_created) == 1 + assert session1 is session2 + assert session2 is session3 \ No newline at end of file diff --git a/tests/src/dq_validator/test_case_check.py b/tests/src/dq_validator/test_case_check.py new file mode 100644 index 0000000..94cf648 --- /dev/null +++ b/tests/src/dq_validator/test_case_check.py @@ -0,0 +1,258 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for CaseCheck +""" + +import pytest +from datetime import date +from wxdi.dq_validator.checks.case_check import CaseCheck, ColumnCaseEnum +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + + +class TestCaseCheckInitialization: + """Test CaseCheck initialization and parameter validation""" + + def test_init_with_no_parameter(self): + """Test initialization without a paramater defaults to CaseCheckEnum : ANY_CASE """ + check = CaseCheck() + assert check.case_type == ColumnCaseEnum.ANY_CASE + + def test_init_with_parameter(self): + """Test initialization with a specific case type""" + check = CaseCheck(case_type=ColumnCaseEnum.UPPER_CASE) + assert check.case_type == ColumnCaseEnum.UPPER_CASE + + def test_init_with_different_datatype_raises_error(self): + """Test initialization with a different datatype raises ValueError""" + with pytest.raises(ValueError) as exc_info: + CaseCheck("AnyCase") + assert "case_type must be a ColumnCaseEnum" in str(exc_info.value) + + def test_get_check_name(self): + """Test get_check_name returns correct name""" + check = CaseCheck() + assert check.get_check_name() == "case_check" + + def test_get_dimension(self): + """Test get_dimension returns correct dimension""" + check = CaseCheck() + assert check.get_dimension() == DataQualityDimension.CONSISTENCY + + def test_set_dimension(self): + """Test set_dimension changes the dimension""" + check = CaseCheck() + assert check.get_dimension() == DataQualityDimension.CONSISTENCY + + check.set_dimension(DataQualityDimension.VALIDITY) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + +class TestCaseCheckHelperMethods: + """Test helper methods for word and sentence delimiters""" + + def test_is_word_delimiter(self): + """Test detection of word delimiters (non-alphabetic characters)""" + check = CaseCheck() + assert check._is_word_delimiter(" ") is True + assert check._is_word_delimiter("-") is True + assert check._is_word_delimiter("_") is True + assert check._is_word_delimiter("1") is True + assert check._is_word_delimiter("'") is True + assert check._is_word_delimiter(",") is True + + def test_not_word_delimiter(self): + """Test detection of word delimiters fails for alphabetic characters""" + check = CaseCheck() + assert check._is_word_delimiter("a") is False + assert check._is_word_delimiter("Z") is False + + def test_is_sentence_delimiter(self): + """Test detection of sentence boundaries (. ! ?) """ + check = CaseCheck() + assert check._is_sentence_delimiter(".") is True + assert check._is_sentence_delimiter("!") is True + assert check._is_sentence_delimiter("?") is True + + def test_not_sentence_delimiter(self): + """Test detection of sentence boundaries (. ! ?) fails """ + check = CaseCheck() + assert check._is_sentence_delimiter(" ") is False + assert check._is_sentence_delimiter(",") is False + assert check._is_sentence_delimiter("a") is False + assert check._is_sentence_delimiter("\n") is False + + +class TestCaseCheckValidation: + """Test validation logic for various case scenarios""" + + def test_any_case_passes(self): + """Test AnyCase accepts any string format""" + check = CaseCheck(ColumnCaseEnum.ANY_CASE) + context = {'column_name': 'description'} + result = check.validate("anything GOES here 123!", context) + assert result is None + + def test_upper_case_passes(self): + """Test UpperCase validation success""" + check = CaseCheck(ColumnCaseEnum.UPPER_CASE) + context = {'column_name': 'country_code'} + result = check.validate("USA IS IN WEST", context) + assert result is None + + def test_upper_case_fails(self): + """Test UpperCase validation failure""" + check = CaseCheck(ColumnCaseEnum.UPPER_CASE) + context = {'column_name': 'country_code'} + result = check.validate("USA is in west", context) + assert result is not None + assert "country_code does not follow UpperCase" in result.message + + def test_lower_case_passes(self): + """Test LowerCase validation success""" + check = CaseCheck(ColumnCaseEnum.LOWER_CASE) + context = {'column_name': 'email'} + result = check.validate("test@example.com", context) + assert result is None + + def test_lower_case_fails(self): + """Test LowerCase validation failure""" + check = CaseCheck(ColumnCaseEnum.LOWER_CASE) + context = {'column_name': 'email'} + result = check.validate("Test@example.com", context) + assert result is not None + assert "email does not follow LowerCase" in result.message + + def test_name_case_passes(self): + """Test NameCase validation success""" + check = CaseCheck(ColumnCaseEnum.NAME_CASE) + context = {'column_name': 'full_name'} + result = check.validate("Jean-Luc O'Niel", context) + assert result is None + + def test_name_case_fails(self): + """Test NameCase validation failure""" + check = CaseCheck(ColumnCaseEnum.NAME_CASE) + context = {'column_name': 'full_name'} + result = check.validate("john Doe", context) + assert result is not None + result = check.validate("JoHn Doe", context) + assert result is not None + assert "full_name does not follow NameCase" in result.message + + def test_sentence_case_passes(self): + """Test SentenceCase validation success""" + check = CaseCheck(ColumnCaseEnum.SENTENCE_CASE) + context = {'column_name': 'comment'} + result = check.validate("This is a sentence.", context) + assert result is None + result = check.validate("First sentence. Second sentence!", context) + assert result is None + + def test_sentence_case_fails(self): + """Test SentenceCase validation failure""" + check = CaseCheck(ColumnCaseEnum.SENTENCE_CASE) + context = {'column_name': 'comment'} + result = check.validate("First sentence. second sentence.", context) + assert result is not None + assert "comment does not follow SentenceCase" in result.message + + def test_none_value_fails(self): + """Test None input returns error""" + check = CaseCheck(ColumnCaseEnum.UPPER_CASE) + context = {'column_name': 'username'} + result = check.validate(None, context) + assert result is not None + assert "username is None, cannot check the case" in result.message + + +class TestCaseCheckNonStringTypes: + """Test validation of different non-string datatypes via auto-conversion""" + + def test_int_input_passes_upper_case(self): + """Test Integer input to string passes UpperCase.""" + check = CaseCheck(ColumnCaseEnum.UPPER_CASE) + context = {'column_name': 'id_number'} + result = check.validate(100, context) + assert result is None + + def test_float_input_passes_lower_case(self): + """Test Float input to string passes LowerCase.""" + check = CaseCheck(ColumnCaseEnum.LOWER_CASE) + context = {'column_name': 'score'} + result = check.validate(99.9, context) + assert result is None + + def test_boolean_true_fails_upper_case(self): + """Test Bool input to string fails UpperCase.""" + check = CaseCheck(ColumnCaseEnum.UPPER_CASE) + context = {'column_name': 'is_active'} + result = check.validate(True, context) + assert result is not None + # Verify result.value maintains the original boolean type + assert isinstance(result.value, bool) + assert "is_active does not follow UpperCase" in result.message + + def test_boolean_false_passes_name_case(self): + """Test Bool input to string passes NameCase """ + check = CaseCheck(ColumnCaseEnum.NAME_CASE) + context = {'column_name': 'flag'} + result = check.validate(False, context) + assert result is None + + def test_date_input_passes_sentence_case(self): + """Test Date input passes SentenceCase.""" + check = CaseCheck(ColumnCaseEnum.SENTENCE_CASE) + context = {'column_name': 'created_at'} + result = check.validate(date(2024, 1, 1), context) + assert result is None + + def test_list_input_fails_lower_case(self): + """Test List input to string fails LowerCase.""" + check = CaseCheck(ColumnCaseEnum.LOWER_CASE) + context = {'column_name': 'tags'} + result = check.validate([1, "A"], context) + assert result is not None + assert "tags" in result.message + assert "does not follow LowerCase" in result.message + + def test_custom_object_conversion(self): + """Test a custom object whose str() representation fails UpperCase""" + class MockObj: + def __str__(self): + return "lower" + + check = CaseCheck(ColumnCaseEnum.UPPER_CASE) + result = check.validate(MockObj(), {'column_name': 'obj'}) + assert result is not None + assert "obj does not follow UpperCase" in result.message + + +class TestCaseCheckEdgeCases: + """Test edge cases for CaseCheck""" + + def test_empty_string_passes(self): + """Test that empty strings pass case validation """ + check = CaseCheck(ColumnCaseEnum.UPPER_CASE) + result = check.validate("", {'column_name': 'test'}) + result is None + + def test_repr(self): + """Test __repr__ output""" + check = CaseCheck(ColumnCaseEnum.NAME_CASE) + repr_str = repr(check) + assert "CaseCheck" in repr_str + assert "case_type=NameCase" in repr_str \ No newline at end of file diff --git a/tests/src/dq_validator/test_comparison_check.py b/tests/src/dq_validator/test_comparison_check.py new file mode 100644 index 0000000..526d24c --- /dev/null +++ b/tests/src/dq_validator/test_comparison_check.py @@ -0,0 +1,491 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for ComparisonCheck +""" + +import pytest +from datetime import datetime, date +from wxdi.dq_validator.checks.comparison_check import ComparisonCheck, ComparisonOperator +from wxdi.dq_validator.metadata import AssetMetadata, ColumnMetadata, DataType +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + + +class TestComparisonCheckInitialization: + """Test ComparisonCheck initialization and parameter validation""" + + def test_init_with_enum_operator_and_value(self): + """Test initialization with enum operator and target value""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=18) + assert check.operator == ComparisonOperator.GREATER_THAN + assert check.target_value == 18 + assert check.target_column is None + assert check.is_column_comparison == False + + def test_init_with_string_operator_and_value(self): + """Test initialization with string operator and target value""" + check = ComparisonCheck(operator='>', target_value=18) + assert check.operator == ComparisonOperator.GREATER_THAN + assert check.target_value == 18 + + def test_init_with_enum_operator_and_column(self): + """Test initialization with enum operator and target column""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_column='min_value') + assert check.operator == ComparisonOperator.GREATER_THAN + assert check.target_column == 'min_value' + assert check.target_value is None + assert check.is_column_comparison == True + + def test_init_all_operators_as_enum(self): + """Test initialization with all operator enums""" + operators = [ + ComparisonOperator.GREATER_THAN, + ComparisonOperator.LESS_THAN, + ComparisonOperator.GREATER_THAN_OR_EQUAL, + ComparisonOperator.LESS_THAN_OR_EQUAL, + ComparisonOperator.EQUAL, + ComparisonOperator.NOT_EQUAL + ] + for op in operators: + check = ComparisonCheck(operator=op, target_value=10) + assert check.operator == op + + def test_init_all_operators_as_string(self): + """Test initialization with all operator strings""" + operators = ['>', '<', '>=', '<=', '==', '!='] + for op in operators: + check = ComparisonCheck(operator=op, target_value=10) + assert check.operator.value == op + + def test_init_invalid_string_operator_raises_error(self): + """Test that invalid string operator raises ValueError""" + with pytest.raises(ValueError) as exc_info: + ComparisonCheck(operator='<>', target_value=10) + assert "Invalid operator '<>'" in str(exc_info.value) + + def test_init_invalid_operator_type_raises_error(self): + """Test that invalid operator type raises TypeError""" + with pytest.raises(TypeError) as exc_info: + ComparisonCheck(operator=123, target_value=10) # type: ignore[arg-type] + assert "operator must be ComparisonOperator or str" in str(exc_info.value) + + def test_init_no_target_raises_error(self): + """Test that no target raises ValueError""" + with pytest.raises(ValueError) as exc_info: + ComparisonCheck(operator=ComparisonOperator.GREATER_THAN) + assert "Either target_column or target_value must be specified" in str(exc_info.value) + + def test_init_both_targets_raises_error(self): + """Test that both targets raises ValueError""" + with pytest.raises(ValueError) as exc_info: + ComparisonCheck( + operator=ComparisonOperator.GREATER_THAN, + target_column='col1', + target_value=10 + ) + assert "Cannot specify both target_column and target_value" in str(exc_info.value) + + def test_get_check_name(self): + """Test get_check_name returns correct name""" + check = ComparisonCheck(operator='>', target_value=10) + assert check.get_check_name() == "comparison_check" + + def test_get_dimension(self): + """Test get_dimension returns correct dimension""" + check = ComparisonCheck(operator='>', target_value=10) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + def test_set_dimension(self): + """Test set_dimension changes the dimension""" + check = ComparisonCheck(operator='>', target_value=10) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + check.set_dimension(DataQualityDimension.CONSISTENCY) + assert check.get_dimension() == DataQualityDimension.CONSISTENCY + + +class TestComparisonCheckColumnToValue: + """Test ComparisonCheck with column-to-value comparisons""" + + def test_greater_than_passes(self): + """Test greater than comparison passes""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=18) + context = {'column_name': 'age'} + result = check.validate(25, context) + assert result is None + + def test_greater_than_fails(self): + """Test greater than comparison fails""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=18) + context = {'column_name': 'age'} + result = check.validate(15, context) + assert result is not None + assert "age (15) > 18 failed" in result.message + + def test_less_than_passes(self): + """Test less than comparison passes""" + check = ComparisonCheck(operator=ComparisonOperator.LESS_THAN, target_value=100) + context = {'column_name': 'score'} + result = check.validate(75, context) + assert result is None + + def test_less_than_fails(self): + """Test less than comparison fails""" + check = ComparisonCheck(operator=ComparisonOperator.LESS_THAN, target_value=100) + context = {'column_name': 'score'} + result = check.validate(150, context) + assert result is not None + assert "score (150) < 100 failed" in result.message + + def test_greater_than_or_equal_passes_equal(self): + """Test >= passes when equal""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN_OR_EQUAL, target_value=18) + context = {'column_name': 'age'} + result = check.validate(18, context) + assert result is None + + def test_greater_than_or_equal_passes_greater(self): + """Test >= passes when greater""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN_OR_EQUAL, target_value=18) + context = {'column_name': 'age'} + result = check.validate(25, context) + assert result is None + + def test_greater_than_or_equal_fails(self): + """Test >= fails when less""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN_OR_EQUAL, target_value=18) + context = {'column_name': 'age'} + result = check.validate(17, context) + assert result is not None + assert "age (17) >= 18 failed" in result.message + + def test_less_than_or_equal_passes_equal(self): + """Test <= passes when equal""" + check = ComparisonCheck(operator=ComparisonOperator.LESS_THAN_OR_EQUAL, target_value=100) + context = {'column_name': 'discount'} + result = check.validate(100, context) + assert result is None + + def test_less_than_or_equal_passes_less(self): + """Test <= passes when less""" + check = ComparisonCheck(operator=ComparisonOperator.LESS_THAN_OR_EQUAL, target_value=100) + context = {'column_name': 'discount'} + result = check.validate(75, context) + assert result is None + + def test_less_than_or_equal_fails(self): + """Test <= fails when greater""" + check = ComparisonCheck(operator=ComparisonOperator.LESS_THAN_OR_EQUAL, target_value=100) + context = {'column_name': 'discount'} + result = check.validate(150, context) + assert result is not None + assert "discount (150) <= 100 failed" in result.message + + def test_equal_passes(self): + """Test == passes when equal""" + check = ComparisonCheck(operator=ComparisonOperator.EQUAL, target_value=200) + context = {'column_name': 'status_code'} + result = check.validate(200, context) + assert result is None + + def test_equal_fails(self): + """Test == fails when not equal""" + check = ComparisonCheck(operator=ComparisonOperator.EQUAL, target_value=200) + context = {'column_name': 'status_code'} + result = check.validate(404, context) + assert result is not None + assert "status_code (404) == 200 failed" in result.message + + def test_not_equal_passes(self): + """Test != passes when not equal""" + check = ComparisonCheck(operator=ComparisonOperator.NOT_EQUAL, target_value=0) + context = {'column_name': 'amount'} + result = check.validate(100, context) + assert result is None + + def test_not_equal_fails(self): + """Test != fails when equal""" + check = ComparisonCheck(operator=ComparisonOperator.NOT_EQUAL, target_value=0) + context = {'column_name': 'amount'} + result = check.validate(0, context) + assert result is not None + assert "amount (0) != 0 failed" in result.message + + +class TestComparisonCheckColumnToColumn: + """Test ComparisonCheck with column-to-column comparisons""" + + def setup_method(self): + """Set up metadata for column-to-column tests""" + self.metadata = AssetMetadata( + table_name='test', + columns=[ + ColumnMetadata('salary', DataType.DECIMAL), + ColumnMetadata('min_salary', DataType.DECIMAL) + ] + ) + + def test_column_to_column_greater_than_passes(self): + """Test column-to-column > passes""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_column='min_salary') + record = [60000.00, 50000.00] + context = {'column_name': 'salary', 'record': record, 'metadata': self.metadata} + result = check.validate(60000.00, context) + assert result is None + + def test_column_to_column_greater_than_fails(self): + """Test column-to-column > fails""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_column='min_salary') + record = [45000.00, 50000.00] + context = {'column_name': 'salary', 'record': record, 'metadata': self.metadata} + result = check.validate(45000.00, context) + assert result is not None + assert "salary (45000.0) > min_salary (50000.0) failed" in result.message + + def test_column_to_column_target_not_found(self): + """Test error when target column not found""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_column='nonexistent') + record = [60000.00, 50000.00] + context = {'column_name': 'salary', 'record': record, 'metadata': self.metadata} + result = check.validate(60000.00, context) + assert result is not None + assert "Target column 'nonexistent' not found" in result.message + + def test_column_to_column_target_is_none(self): + """Test error when target column value is None""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_column='min_salary') + record = [60000.00, None] + context = {'column_name': 'salary', 'record': record, 'metadata': self.metadata} + result = check.validate(60000.00, context) + assert result is not None + assert "min_salary=None" in result.message + + +class TestComparisonCheckStringComparison: + """Test ComparisonCheck with string values""" + + def test_string_greater_than_passes(self): + """Test string lexicographic > passes""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value='apple') + context = {'column_name': 'fruit'} + result = check.validate('banana', context) + assert result is None # 'banana' > 'apple' + + def test_string_less_than_passes(self): + """Test string lexicographic < passes""" + check = ComparisonCheck(operator=ComparisonOperator.LESS_THAN, target_value='zebra') + context = {'column_name': 'word'} + result = check.validate('apple', context) + assert result is None # 'apple' < 'zebra' + + def test_string_equal_passes(self): + """Test string == passes""" + check = ComparisonCheck(operator=ComparisonOperator.EQUAL, target_value='test') + context = {'column_name': 'value'} + result = check.validate('test', context) + assert result is None + + +class TestComparisonCheckDateComparison: + """Test ComparisonCheck with date and datetime values""" + + def test_date_greater_than_passes(self): + """Test date > passes""" + check = ComparisonCheck( + operator=ComparisonOperator.GREATER_THAN, + target_value=date(2024, 1, 1) + ) + context = {'column_name': 'end_date'} + result = check.validate(date(2024, 12, 31), context) + assert result is None + + def test_date_less_than_passes(self): + """Test date < passes""" + check = ComparisonCheck( + operator=ComparisonOperator.LESS_THAN, + target_value=date(2024, 12, 31) + ) + context = {'column_name': 'start_date'} + result = check.validate(date(2024, 1, 1), context) + assert result is None + + def test_datetime_greater_than_passes(self): + """Test datetime > passes""" + check = ComparisonCheck( + operator=ComparisonOperator.GREATER_THAN, + target_value=datetime(2024, 1, 1, 0, 0, 0) + ) + context = {'column_name': 'timestamp'} + result = check.validate(datetime(2024, 6, 15, 12, 0, 0), context) + assert result is None + + def test_datetime_less_than_or_equal_passes(self): + """Test datetime <= passes""" + check = ComparisonCheck( + operator=ComparisonOperator.LESS_THAN_OR_EQUAL, + target_value=datetime(2024, 12, 31, 23, 59, 59) + ) + context = {'column_name': 'timestamp'} + result = check.validate(datetime(2024, 6, 15, 12, 0, 0), context) + assert result is None + + +class TestComparisonCheckBooleanComparison: + """Test ComparisonCheck with boolean values""" + + def test_boolean_equal_true_passes(self): + """Test boolean == True passes""" + check = ComparisonCheck(operator=ComparisonOperator.EQUAL, target_value=True) + context = {'column_name': 'is_active'} + result = check.validate(True, context) + assert result is None + + def test_boolean_equal_false_passes(self): + """Test boolean == False passes""" + check = ComparisonCheck(operator=ComparisonOperator.EQUAL, target_value=False) + context = {'column_name': 'is_active'} + result = check.validate(False, context) + assert result is None + + def test_boolean_not_equal_passes(self): + """Test boolean != passes""" + check = ComparisonCheck(operator=ComparisonOperator.NOT_EQUAL, target_value=False) + context = {'column_name': 'is_active'} + result = check.validate(True, context) + assert result is None + + def test_boolean_greater_than_passes(self): + """Test boolean > passes (True > False in Python)""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=False) + context = {'column_name': 'flag'} + result = check.validate(True, context) + assert result is None + + +class TestComparisonCheckNoneHandling: + """Test ComparisonCheck with None values""" + + def test_none_value_fails(self): + """Test None value returns error""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=50000) + context = {'column_name': 'salary'} + result = check.validate(None, context) + assert result is not None + assert "is None, cannot perform comparison" in result.message + + +class TestComparisonCheckTypeMismatch: + """Test ComparisonCheck with type mismatches""" + + def test_type_mismatch_string_vs_int_fails(self): + """Test type mismatch between string and int fails""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=50000) + context = {'column_name': 'salary'} + result = check.validate('60000', context) + assert result is not None + assert "Type mismatch" in result.message + assert "str" in result.message + assert "int" in result.message + + def test_type_mismatch_int_vs_float_fails(self): + """Test type mismatch between int and float fails""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=50000) + context = {'column_name': 'value'} + result = check.validate(60000.0, context) + assert result is not None + assert "Type mismatch" in result.message + + def test_type_mismatch_column_to_column(self): + """Test type mismatch in column-to-column comparison""" + metadata = AssetMetadata( + table_name='test', + columns=[ + ColumnMetadata('value1', DataType.STRING), + ColumnMetadata('value2', DataType.INTEGER) + ] + ) + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_column='value2') + record = ['100', 50] + context = {'column_name': 'value1', 'record': record, 'metadata': metadata} + result = check.validate('100', context) + assert result is not None + assert "Type mismatch" in result.message + + +class TestComparisonCheckFloatComparison: + """Test ComparisonCheck with float values""" + + def test_float_greater_than_passes(self): + """Test float > passes""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=50.5) + context = {'column_name': 'amount'} + result = check.validate(75.3, context) + assert result is None + + def test_float_less_than_passes(self): + """Test float < passes""" + check = ComparisonCheck(operator=ComparisonOperator.LESS_THAN, target_value=100.0) + context = {'column_name': 'amount'} + result = check.validate(99.99, context) + assert result is None + + def test_float_equal_passes(self): + """Test float == passes""" + check = ComparisonCheck(operator=ComparisonOperator.EQUAL, target_value=3.14) + context = {'column_name': 'pi'} + result = check.validate(3.14, context) + assert result is None + + +class TestComparisonCheckEdgeCases: + """Test ComparisonCheck edge cases""" + + def test_zero_comparison(self): + """Test comparison with zero""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=0) + context = {'column_name': 'value'} + result = check.validate(1, context) + assert result is None + + def test_negative_numbers(self): + """Test comparison with negative numbers""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=-10) + context = {'column_name': 'temperature'} + result = check.validate(-5, context) + assert result is None + + def test_very_large_numbers(self): + """Test comparison with very large numbers""" + check = ComparisonCheck(operator=ComparisonOperator.LESS_THAN, target_value=10**100) + context = {'column_name': 'big_number'} + result = check.validate(10**50, context) + assert result is None + + def test_repr(self): + """Test __repr__ method""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_value=10) + repr_str = repr(check) + assert "ComparisonCheck" in repr_str + assert ">" in repr_str + assert "10" in repr_str + + def test_repr_with_column(self): + """Test __repr__ method with target column""" + check = ComparisonCheck(operator=ComparisonOperator.GREATER_THAN, target_column='min_value') + repr_str = repr(check) + assert "ComparisonCheck" in repr_str + assert "min_value" in repr_str + diff --git a/tests/src/dq_validator/test_completeness_check.py b/tests/src/dq_validator/test_completeness_check.py new file mode 100644 index 0000000..83dfe19 --- /dev/null +++ b/tests/src/dq_validator/test_completeness_check.py @@ -0,0 +1,131 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +import pytest +from wxdi.dq_validator.checks.completeness_check import CompletenessCheck +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + +class TestCompletenessCheckInitialization: + """Comprehensive tests for the CompletenessCheck class""" + + def test_init_without_missing_values_allowed(self): + """Test default initialization (missing values should NOT be allowed)""" + check = CompletenessCheck() + assert check.missing_values_allowed is False + + def test_init_with_missing_values_allowed(self): + """Test initialization with missing_values_allowed explicitly""" + check = CompletenessCheck(missing_values_allowed=True) + assert check.missing_values_allowed is True + + def test_get_check_name(self): + """Ensure check name is correct""" + check = CompletenessCheck() + assert check.get_check_name() == "completeness_check" + + def test_get_dimension(self): + """Test get_dimension returns correct dimension""" + check = CompletenessCheck() + assert check.get_dimension() == DataQualityDimension.COMPLETENESS + + def test_set_dimension(self): + """Test set_dimension changes the dimension""" + check = CompletenessCheck() + assert check.get_dimension() == DataQualityDimension.COMPLETENESS + + check.set_dimension(DataQualityDimension.VALIDITY) + assert check.get_dimension() == DataQualityDimension.VALIDITY + +class TestValidationMissingValuesNotAllowed: + """ Test completeness check for various value inputs""" + + def test_validate_string_passes(self): + """Test non-empty string should pass""" + check = CompletenessCheck(missing_values_allowed=False) + result = check.validate("Hello", {"column_name": "name"}) + assert result is None + + def test_validate_numerical_passes(self): + """Test numerical non-empty values should pass""" + check = CompletenessCheck(missing_values_allowed=False) + result = check.validate(0, {"column_name": "score"}) + assert result is None + + def test_validate_boolean_passes(self): + """Booleans should pass""" + check = CompletenessCheck(missing_values_allowed=False) + result = check.validate(False, {"column_name": "is_valid"}) + assert result is None + + def test_validate_none_fails(self): + """None should fail when not allowed""" + check = CompletenessCheck(missing_values_allowed=False) + result = check.validate(None, {"column_name": "user_id"}) + assert result is not None + assert "user_id is missing" in result.message + assert result.value is None + + def test_validate_empty_string_fails(self): + """Purely empty string should fail""" + check = CompletenessCheck(missing_values_allowed=False) + result = check.validate("", {"column_name": "email"}) + assert result is not None + + def test_validate_white_space_string_passes(self): + """Strings with only spaces should fail""" + check = CompletenessCheck(missing_values_allowed=False) + result = check.validate(" ", {"column_name": "comments"}) + assert result is None + + +class TestValidationMissingValuesAllowed: + + def test_missing_allowed_string_passes(self): + """Test non-empty string should pass""" + check = CompletenessCheck(missing_values_allowed=True) + result = check.validate("Hello", {"column_name": "name"}) + assert result is None + + def test_missing_allowed_numerical_passes(self): + """Test numerical non-empty values should pass""" + check = CompletenessCheck(missing_values_allowed=True) + result = check.validate(0, {"column_name": "score"}) + assert result is None + + def test_missing_allowed_boolean_passes(self): + """Booleans should pass""" + check = CompletenessCheck(missing_values_allowed=True) + result = check.validate(False, {"column_name": "is_valid"}) + assert result is None + + def test_missing_allowed_none_passes(self): + """None should pass when missing_values_allowed is True""" + check = CompletenessCheck(missing_values_allowed=True) + result = check.validate(None, {"column_name": "optional_field"}) + assert result is None + + def test_missing_allowed_whitespace_passes(self): + """Whitespace strings should pass when missing_values_allowed""" + check = CompletenessCheck(missing_values_allowed=True) + result = check.validate(" ", {"column_name": "notes"}) + assert result is None + + +class TestValidateEdgeCases: + + def test_repr(self): + """Test the string representation""" + check = CompletenessCheck(missing_values_allowed=True) + assert "missing_values_allowed=True" in repr(check) \ No newline at end of file diff --git a/tests/src/dq_validator/test_data_asset_model.py b/tests/src/dq_validator/test_data_asset_model.py new file mode 100644 index 0000000..fd247a7 --- /dev/null +++ b/tests/src/dq_validator/test_data_asset_model.py @@ -0,0 +1,187 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Test suite for DataAsset model +""" + +import json +import pytest +from pathlib import Path +from wxdi.dq_validator.provider.data_asset_model import DataAsset + + +class TestDataAssetModel: + """Test cases for DataAsset Pydantic model""" + + @pytest.fixture + def data_asset_json(self): + """Load the data asset response JSON""" + json_path = Path(__file__).parent.parent.parent / "data" / "data_asset_response.json" + with open(json_path, "r") as f: + return json.load(f) + + def test_parse_data_asset_response(self, data_asset_json): + """Test parsing the complete data asset response""" + data_asset = DataAsset.from_dict(data_asset_json) + + assert data_asset is not None + assert isinstance(data_asset, DataAsset) + + def test_metadata_fields(self, data_asset_json): + """Test metadata fields are correctly parsed""" + data_asset = DataAsset.from_dict(data_asset_json) + + assert data_asset.metadata.asset_id == "6862f3ba-81f5-4122-8286-62bb4c5d6543" + assert data_asset.metadata.name == "DEPARTMENT" + assert data_asset.metadata.asset_type == "data_asset" + assert data_asset.metadata.project_id == "72d21c1d-499b-4784-a3c7-6f84507f9a20" + + def test_columns_parsed(self, data_asset_json): + """Test that columns are correctly parsed""" + data_asset = DataAsset.from_dict(data_asset_json) + + columns = data_asset.entity.data_asset.columns + assert len(columns) == 5 + + # Check first column + assert columns[0].name == "DEPTNO" + assert columns[0].type.type == "char" + assert columns[0].type.length == 3 + assert columns[0].type.nullable is False + + def test_column_info_parsed(self, data_asset_json): + """Test that column_info is correctly parsed""" + data_asset = DataAsset.from_dict(data_asset_json) + + column_info = data_asset.entity.column_info + assert "DEPTNO" in column_info + assert "MGRNO" in column_info + assert "LOCATION" in column_info + + def test_column_checks_parsed(self, data_asset_json): + """Test that column checks are correctly parsed""" + data_asset = DataAsset.from_dict(data_asset_json) + + # DEPTNO has 4 checks + deptno_info = data_asset.entity.column_info["DEPTNO"] + assert len(deptno_info.column_checks) == 4 + + # Check types + check_types = [check.metadata.type for check in deptno_info.column_checks] + assert "format" in check_types + assert "uniqueness" in check_types + assert "completeness" in check_types + assert "data_class" in check_types + + def test_data_class_info(self, data_asset_json): + """Test data class information""" + data_asset = DataAsset.from_dict(data_asset_json) + + deptno_info = data_asset.entity.column_info["DEPTNO"] + assert deptno_info.data_class.selected_data_class.name == "ICD-10" + assert len(deptno_info.data_class.suggested_classes) == 2 + + def test_inferred_type(self, data_asset_json): + """Test inferred type information""" + data_asset = DataAsset.from_dict(data_asset_json) + + mgrno_info = data_asset.entity.column_info["MGRNO"] + assert mgrno_info.inferred_type.type == "INT8" + assert mgrno_info.inferred_type.display_name == "tinyint" + assert mgrno_info.inferred_type.precision == 3 + + def test_column_without_checks(self, data_asset_json): + """Test column without any checks (LOCATION)""" + data_asset = DataAsset.from_dict(data_asset_json) + + location_info = data_asset.entity.column_info["LOCATION"] + assert len(location_info.column_checks) == 0 + assert location_info.data_class.selected_data_class.name == "NoClassDetected" + + def test_asset_properties(self, data_asset_json): + """Test asset properties""" + data_asset = DataAsset.from_dict(data_asset_json) + + properties = data_asset.entity.data_asset.properties + assert len(properties) == 2 + + schema_prop = next(p for p in properties if p.name == "schema_name") + assert schema_prop.value == "DB2INST1" + + table_prop = next(p for p in properties if p.name == "table_name") + assert table_prop.value == "DEPARTMENT" + + def test_check_constraint_details(self, data_asset_json): + """Test detailed check constraint information""" + data_asset = DataAsset.from_dict(data_asset_json) + + # Get format check from MGRNO + mgrno_info = data_asset.entity.column_info["MGRNO"] + format_check = next( + c for c in mgrno_info.column_checks + if c.metadata.type == "format" + ) + + assert len(format_check.check) == 1 + assert format_check.check[0].name == "formats" + assert format_check.check[0].list_value == ["999999"] + + def test_completeness_check(self, data_asset_json): + """Test completeness check parsing""" + data_asset = DataAsset.from_dict(data_asset_json) + + deptno_info = data_asset.entity.column_info["DEPTNO"] + completeness_check = next( + c for c in deptno_info.column_checks + if c.metadata.type == "completeness" + ) + + assert len(completeness_check.check) == 1 + assert completeness_check.check[0].name == "missing_values_allowed" + assert completeness_check.check[0].boolean_value is False + + def test_range_check(self, data_asset_json): + """Test range check parsing""" + data_asset = DataAsset.from_dict(data_asset_json) + + mgrno_info = data_asset.entity.column_info["MGRNO"] + range_check = next( + c for c in mgrno_info.column_checks + if c.metadata.type == "range" + ) + + assert len(range_check.check) == 2 + range_type = next(c for c in range_check.check if c.name == "range_type") + assert range_type.value == "number" + + min_value = next(c for c in range_check.check if c.name == "min") + assert min_value.numeric_value == 0 + + def test_case_check(self, data_asset_json): + """Test case check parsing""" + data_asset = DataAsset.from_dict(data_asset_json) + + deptname_info = data_asset.entity.column_info["DEPTNAME"] + case_check = next( + c for c in deptname_info.column_checks + if c.metadata.type == "case" + ) + + assert len(case_check.check) == 1 + assert case_check.check[0].name == "case_type" + assert case_check.check[0].value == "UpperCase" + +# Made with Bob diff --git a/tests/src/dq_validator/test_datatype_check.py b/tests/src/dq_validator/test_datatype_check.py new file mode 100644 index 0000000..eb90200 --- /dev/null +++ b/tests/src/dq_validator/test_datatype_check.py @@ -0,0 +1,424 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for DataType Check +""" + +from wxdi.dq_validator.checks.datatype_check import DataTypeCheck +from wxdi.dq_validator.datatypes import DataType, DataTypeEnum +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + + +class TestDataTypeCheckInitialization: + """Tests for the datatype_check class initialization""" + + def test_init_with_datatype(self): + """Test initialization with a completely intitialized datatype object""" + expected = DataType(DataTypeEnum.INT32) + check = DataTypeCheck(expected) + assert check.expected_type == expected + + def test_init_without_datatype(self): + """Test initialization without a datatype object""" + check = DataTypeCheck() + assert check.expected_type == None + + def test_init_with_empty_datatype(self): + """Test initialization with an empty datatype object""" + expected = DataType() + check = DataTypeCheck(expected) + assert check.expected_type.dtype == DataTypeEnum.UNKNOWN + + def test_get_check_name(self): + """Test get_check_name returns correct name""" + check = DataTypeCheck(DataType(DataTypeEnum.STRING, length=10)) + assert check.get_check_name() == "datatype_check" + + def test_get_dimension(self): + """Test get_dimension returns correct dimension""" + check = DataTypeCheck(DataType(DataTypeEnum.STRING, length=10)) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + def test_set_dimension(self): + """Test set_dimension changes the dimension""" + check = DataTypeCheck(DataType(DataTypeEnum.STRING, length=10)) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + check.set_dimension(DataQualityDimension.CONSISTENCY) + assert check.get_dimension() == DataQualityDimension.CONSISTENCY + + +class TestDataTypeCheckValidation: + """Test validation results for empty and non-empty values""" + + def test_empty_datatype_with_value_passes(self): + """UNKNOWN of empty datatype doesnt match integer""" + expected = DataType() + check = DataTypeCheck(expected) + result = check.validate(123, {"column_name": "age"}) + assert result is not None + assert "not compatible" in result.message + + def test_matching_datatype_passes(self): + """Value matches expected type""" + expected = DataType(DataTypeEnum.INT32) + check = DataTypeCheck(expected) + result = check.validate(100, {"column_name": "count"}) + assert result is None + + def test_mismatching_datatype_fails(self): + """Value inferred as STRING but expected INT""" + expected = DataType(DataTypeEnum.INT32) + check = DataTypeCheck(expected) + result = check.validate("abc", {"column_name": "count"}) + assert result is not None + assert "not compatible" in result.message + + def test_empty_datatype_with_none_passes(self): + """None value allowed for an empty expected type""" + expected = DataType() + check = DataTypeCheck(expected) + result = check.validate(None, {"column_name": "optional"}) + assert result is None + + def test_no_datatype_with_none_passes(self): + """With no expected type any value is allowed""" + check = DataTypeCheck(None) + result = check.validate(None, {"column_name": "optional"}) + assert result is None + + def test_datatype_with_none_passes(self): + """Null value allowed regardless of datatype""" + expected = DataType(DataTypeEnum.DATE) + check = DataTypeCheck(expected) + result = check.validate(None, {"column_name": "created_at"}) + assert result is None + + +class TestIntegerCompatibility: + """Test validation results for various values against expected Integer types""" + + def test_int8_pass(self): + """Test int8 value passes""" + check = DataTypeCheck(DataType(DataTypeEnum.INT8)) + result = check.validate(127, {"column_name": "age"}) + assert result is None + + def test_int8_fail_overflow(self): + """Test int8 value fails due to overflow""" + check = DataTypeCheck(DataType(DataTypeEnum.INT8)) + result = check.validate(200, {"column_name": "age"}) + assert result is not None + assert "not compatible" in result.message + + def test_int16_accepts_int8(self): + """Test expected int16 accepts int8""" + check = DataTypeCheck(DataType(DataTypeEnum.INT16)) + result = check.validate(100, {"column_name": "score"}) + assert result is None + + def test_int32_accepts_int16(self): + """Test expected int32 accepts int16""" + check = DataTypeCheck(DataType(DataTypeEnum.INT32)) + result = check.validate(30000, {"column_name": "population"}) + assert result is None + + def test_int64_accepts_int32(self): + """Test expected int64 accepts int32""" + check = DataTypeCheck(DataType(DataTypeEnum.INT64)) + result = check.validate(2000000000, {"column_name": "id"}) + assert result is None + + def test_int32_fail_decimal(self): + """Test int32 should reject decimal values""" + check = DataTypeCheck(DataType(DataTypeEnum.INT32)) + result = check.validate(12.34, {"column_name": "price"}) + assert result is not None + assert "not compatible" in result.message + + def test_int32_fail_non_numeric_string(self): + """Test int32 should reject non-numeric strings""" + check = DataTypeCheck(DataType(DataTypeEnum.INT32)) + result = check.validate("abc", {"column_name": "age"}) + assert result is not None + assert "not compatible" in result.message + + def test_int_accepts_numeric_string(self): + """Numeric string value is parsed to compatible numerical type""" + check = DataTypeCheck(DataType(DataTypeEnum.INT32)) + result = check.validate("12345", {"column_name": "age"}) + assert result is None + + + +class TestDecimalCompatibility: + """Test validation results for various values against expected Decimal types""" + + def test_decimal_exact(self): + """Test decimal value passes""" + expected = DataType(DataTypeEnum.DECIMAL, precision=5, scale=2) + check = DataTypeCheck(expected) + result = check.validate("123.45", {"column_name": "price"}) + assert result is None + + def test_decimal_scale_too_large(self): + """Test decimal rejects scale overflow""" + expected = DataType(DataTypeEnum.DECIMAL, precision=5, scale=2) + check = DataTypeCheck(expected) + result = check.validate("12.345", {"column_name": "price"}) + assert result is not None + + def test_decimal_integer_allowed(self): + """Test decimal accepts integer""" + expected = DataType(DataTypeEnum.DECIMAL, precision=10, scale=2) + check = DataTypeCheck(expected) + result = check.validate(100, {"column_name": "amount"}) + assert result is None + + def test_decimal_precision_overflow(self): + """Test decimal rejects precision overflow""" + expected = DataType(DataTypeEnum.DECIMAL, precision=4, scale=2) + check = DataTypeCheck(expected) + result = check.validate("123.45", {"column_name": "amount"}) + assert result is not None + + def test_decimal_rejects_date(self): + """Testcdecimal should reject date values""" + expected = DataType(DataTypeEnum.DECIMAL, precision=10, scale=2) + check = DataTypeCheck(expected) + result = check.validate("2024-01-15", {"column_name": "price"}) + assert result is not None + assert "not compatible" in result.message + + +class TestStringCompatibility: + """Test validation results for various values against expected String types""" + + def test_string_within_length(self): + check = DataTypeCheck(DataType(DataTypeEnum.STRING, length=10)) + assert check.validate("hello", {"column_name": "name"}) is None + + def test_string_exact_length(self): + check = DataTypeCheck(DataType(DataTypeEnum.STRING, length=5)) + assert check.validate("hello", {"column_name": "name"}) is None + + def test_string_too_long(self): + check = DataTypeCheck(DataType(DataTypeEnum.STRING, length=5)) + result = check.validate("toolong", {"column_name": "name"}) + assert result is not None + assert "not compatible" in result.message + + def test_string_numeric_passes(self): + """Input string value can be parsed to Numerical value""" + check = DataTypeCheck(DataType(DataTypeEnum.INT64, length=10)) + result = check.validate("12345", {"column_name": "code"}) + assert result is None + + def test_string_rejects_date(self): + """String constraint should accept appropriate date values""" + check = DataTypeCheck(DataType(DataTypeEnum.STRING, length=10)) + result = check.validate("2024-01-15", {"column_name": "dob"}) + assert result is not None + assert "not compatible" in result.message + + +class TestDateTimeCompatibility: + """Test validation results for various values against expected DateTime types""" + + def test_date_string(self): + """Test date value passes""" + check = DataTypeCheck(DataType(DataTypeEnum.DATE)) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is None + + def test_invalid_date_string(self): + """Test date value fails""" + check = DataTypeCheck(DataType(DataTypeEnum.DATE)) + result = check.validate("hello", {"column_name": "dob"}) + assert result is not None + assert "not compatible" in result.message + + def test_time_string(self): + """Test time value passes""" + check = DataTypeCheck(DataType(DataTypeEnum.TIME)) + result = check.validate("14:23:01", {"column_name": "login_time"}) + assert result is None + + def test_timestamp_string(self): + """Test timestamp value passes""" + check = DataTypeCheck(DataType(DataTypeEnum.TIMESTAMP)) + result = check.validate("2024-01-10 14:23:01", {"column_name": "created_at"}) + assert result is None + + def test_timestamp_against_date(self): + """Test timestamp value fails against expected date""" + check = DataTypeCheck(DataType(DataTypeEnum.DATE)) + result = check.validate("2024-01-01 10:11:12", {"column_name": "created"}) + assert result is not None + + +class TestDateFormats: + """Test validation results for various date formats""" + + def test_date_dash_format(self): + """YYYY-MM-DD format should be detected as DATE""" + check = DataTypeCheck(DataType(DataTypeEnum.DATE)) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is None + + def test_date_slash_format(self): + """MM/YYYY format should be detected as DATE""" + check = DataTypeCheck(DataType(DataTypeEnum.DATE)) + result = check.validate("01/2024", {"column_name": "dob"}) + assert result is None + + def test_date_dot_format(self): + """DD.MM.YYYY format should be detected as DATE""" + check = DataTypeCheck(DataType(DataTypeEnum.DATE)) + result = check.validate("10.01.2024", {"column_name": "dob"}) + assert result is None + + def test_date_month_name(self): + """Mon-DD-YY format should be detected as DATE""" + check = DataTypeCheck(DataType(DataTypeEnum.DATE)) + result = check.validate("Jan-10-24", {"column_name": "dob"}) + assert result is None + + def test_date_textual(self): + """Month DD, YYYY format should be detected as DATE""" + check = DataTypeCheck(DataType(DataTypeEnum.DATE)) + result = check.validate("Jan 10, 2024", {"column_name": "dob"}) + assert result is None + + +class TestTimeFormats: + """Test validation results for various time formats""" + + def test_time_24h_seconds(self): + """24-hour HH:MM:SS format should be detected as TIME""" + check = DataTypeCheck(DataType(DataTypeEnum.TIME)) + result = check.validate("14:23:01", {"column_name": "login_time"}) + assert result is None + + def test_time_24h_no_seconds(self): + """24-hour HH:MM format should be detected as TIME""" + check = DataTypeCheck(DataType(DataTypeEnum.TIME)) + result = check.validate("09:45", {"column_name": "login_time"}) + assert result is None + + def test_time_milliseconds(self): + """Time with milliseconds should be detected as TIME""" + check = DataTypeCheck(DataType(DataTypeEnum.TIME)) + result = check.validate("14:23:01.123", {"column_name": "login_time"}) + assert result is None + + def test_time_12h_with_am_pm(self): + """12-hour time with AM/PM should be detected as TIME""" + check = DataTypeCheck(DataType(DataTypeEnum.TIME)) + result = check.validate("02:23 PM", {"column_name": "login_time"}) + assert result is None + + def test_time_12h_with_seconds(self): + """12-hour time with seconds and AM/PM should be detected as TIME""" + check = DataTypeCheck(DataType(DataTypeEnum.TIME)) + result = check.validate("02:23:01PM", {"column_name": "login_time"}) + assert result is None + + def test_time_with_timezone(self): + """Time with timezone offset should be detected as TIME""" + check = DataTypeCheck(DataType(DataTypeEnum.TIME)) + result = check.validate("14:23:01+0530", {"column_name": "login_time"}) + assert result is None + + +class TestTimeStampFormats: + """Test validation results for various timestamp formats""" + + def test_timestamp_standard(self): + """Standard YYYY-MM-DD HH:MM:SS format should be detected as TIMESTAMP""" + check = DataTypeCheck(DataType(DataTypeEnum.TIMESTAMP)) + result = check.validate("2024-01-10 14:23:01", {"column_name": "created_at"}) + assert result is None + + def test_timestamp_iso_t(self): + """ISO T-separated timestamp should be detected as TIMESTAMP""" + check = DataTypeCheck(DataType(DataTypeEnum.TIMESTAMP)) + result = check.validate("2024-01-10T14:23:01", {"column_name": "created_at"}) + assert result is None + + def test_timestamp_with_millis(self): + """Timestamp with milliseconds should be detected as TIMESTAMP""" + check = DataTypeCheck(DataType(DataTypeEnum.TIMESTAMP)) + result = check.validate("2024-01-10 14:23:01.456", {"column_name": "created_at"}) + assert result is None + + def test_timestamp_with_timezone(self): + """Timestamp with timezone offset should be detected as TIMESTAMP""" + check = DataTypeCheck(DataType(DataTypeEnum.TIMESTAMP)) + result = check.validate("2024-01-10 14:23:01+0530", {"column_name": "created_at"}) + assert result is None + + def test_timestamp_12h_clock(self): + """12-hour timestamp with AM/PM should be detected as TIMESTAMP""" + check = DataTypeCheck(DataType(DataTypeEnum.TIMESTAMP)) + result = check.validate("2024-01-10 02:23:01PM", {"column_name": "created_at"}) + assert result is None + + def test_timestamp_european(self): + """European DD-MM-YYYY HH:MM timestamp should be detected as TIMESTAMP""" + check = DataTypeCheck(DataType(DataTypeEnum.TIMESTAMP)) + result = check.validate("10-01-2024 14:23", {"column_name": "created_at"}) + assert result is None + + def test_timestamp_us_format(self): + """US MM-DD-YYYY HH:MM:SS timestamp should be detected as TIMESTAMP""" + check = DataTypeCheck(DataType(DataTypeEnum.TIMESTAMP)) + result = check.validate("01-10-2024 14:23:01", {"column_name": "created_at"}) + assert result is None + + +class TestCrossTypeFailures: + """Test failure of values against cross-types""" + + def test_string_vs_int(self): + """Test string fails against int32""" + check = DataTypeCheck(DataType(DataTypeEnum.INT32)) + result = check.validate("abc", {"column_name": "age"}) + assert result is not None + + def test_date_vs_decimal(self): + """Test date fails against decimal""" + check = DataTypeCheck(DataType(DataTypeEnum.DECIMAL, precision=5, scale=2)) + result = check.validate("2024-01-01", {"column_name": "amount"}) + assert result is not None + + +class TestFormatAndTypeIntegration: + """Test proper initialization of parameters in datatypes""" + + def test_inferred_format_is_set(self): + """Test initialization of inferred_type and inferred_format""" + check = DataTypeCheck(DataType(DataTypeEnum.DATE)) + engine = check.engine + check.validate("2024-01-10", {"column_name": "dob"}) + assert engine.inferred_type.dtype == DataTypeEnum.DATE + assert engine.inferred_format is not None + + def test_repr(self): + """Test __repr__ method""" + check = DataTypeCheck(DataType(DataTypeEnum.INT32)) + assert "datatype_check" in repr(check) + assert "int32" in repr(check) \ No newline at end of file diff --git a/tests/src/dq_validator/test_format_check.py b/tests/src/dq_validator/test_format_check.py new file mode 100644 index 0000000..76f530a --- /dev/null +++ b/tests/src/dq_validator/test_format_check.py @@ -0,0 +1,357 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for Format Check +""" + +from wxdi.dq_validator.checks.format_check import FormatCheck, FormatConstraintType +from wxdi.dq_validator.datetime_formats import DateTimeFormats +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + +class TestFormatCheckInitialization: + """Test FormatCheck initialization""" + + def test_init_valid_formats(self): + """Test initialization with ValidFormats and format set passes""" + check = FormatCheck(FormatConstraintType.ValidFormats, {DateTimeFormats.ISO_DATE}) + assert check.constraint_type == FormatConstraintType.ValidFormats + assert DateTimeFormats.ISO_DATE in check.formats + + def test_init_invalid_formats(self): + """Test initialization with InvalidFormats and format set""" + check = FormatCheck(FormatConstraintType.InvalidFormats, {"ABx"}) + assert check.constraint_type == FormatConstraintType.InvalidFormats + + def test_init_multiple_formats(self): + """Test formats correctly stores multiple formats in the set""" + formats = {DateTimeFormats.ISO_DATE, DateTimeFormats.UK_DATE, "AAA"} + check = FormatCheck(FormatConstraintType.ValidFormats, formats) + assert check.constraint_type == FormatConstraintType.ValidFormats + assert check.formats == formats + assert len(check.formats) == 3 + + def test_init_without_formats(self): + """Test initialization without format set""" + check = FormatCheck(constraint_type=FormatConstraintType.InvalidFormats) + assert check.constraint_type == FormatConstraintType.InvalidFormats + assert len(check.formats) == 0 + + def test_init_without_constraint_type(self): + """Test initialization without FormatConstraintType""" + check = FormatCheck(formats={DateTimeFormats.ISO_DATE}) + assert check.constraint_type == FormatConstraintType.ValidFormats + assert DateTimeFormats.ISO_DATE in check.formats + + def test_init_without_parameters(self): + """Test initialization without any parameters""" + check = FormatCheck() + assert check.constraint_type == FormatConstraintType.ValidFormats + assert len(check.formats) == 0 + + def test_get_check_name(self): + """Test get_check_name returns correct name""" + check = FormatCheck(FormatConstraintType.ValidFormats, set()) + assert check.get_check_name() == "format_check" + + def test_get_dimension(self): + """Test get_dimension returns correct dimension""" + check = FormatCheck(FormatConstraintType.ValidFormats, set()) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + def test_set_dimension(self): + """Test set_dimension changes the dimension""" + check = FormatCheck(FormatConstraintType.ValidFormats, set()) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + check.set_dimension(DataQualityDimension.CONSISTENCY) + assert check.get_dimension() == DataQualityDimension.CONSISTENCY + + +class TestValidFormats: + """Test ValidFormats validation where value must match one of the allowed formats""" + + def test_valid_date_format_passes(self): + """Date format in allowed list should pass""" + check = FormatCheck(FormatConstraintType.ValidFormats, {DateTimeFormats.ISO_DATE}) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is None + + def test_invalid_date_format_fails(self): + """Date not in allowed list should fail""" + check = FormatCheck(FormatConstraintType.ValidFormats, {DateTimeFormats.UK_DATE}) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is not None + assert "violates" in result.message + + def test_valid_time_passes(self): + """Time in allowed list should pass""" + check = FormatCheck(FormatConstraintType.ValidFormats, {DateTimeFormats.TIME_24H}) + result = check.validate("14:23:01", {"column_name": "login_time"}) + assert result is None + + def test_invalid_time_fails(self): + """Time not in allowed list should fail""" + check = FormatCheck(FormatConstraintType.ValidFormats, {"%hh:%nn %a"}) + result = check.validate("14:23:01", {"column_name": "login_time"}) + assert result is not None + assert "violates" in result.message + + def test_valid_timestamp_passes(self): + """Timestamp in allowed list should pass""" + check = FormatCheck(FormatConstraintType.ValidFormats, {"%yyyy-%mm-%dd %HH:%nn:%ss"}) + result = check.validate("2024-01-10 14:23:01", {"column_name": "created_at"}) + assert result is None + + +class TestComplexFormatScenarios: + """High-complexity format validation scenarios""" + + def test_complex_alphanumeric_with_symbols_valid(self): + """Mixed case, digits and symbols should match expected format""" + check = FormatCheck( + FormatConstraintType.ValidFormats, + {"Aa@99#Aa"} + ) + result = check.validate("Ab@12#Xy", {"column_name": "password"}) + assert result is None + + def test_email_like_string_valid_format(self): + """Email-like structure should match detected format""" + check = FormatCheck( + FormatConstraintType.ValidFormats, + {"aaaa@aaaa.aaa"} + ) + result = check.validate("user@mail.com", {"column_name": "email"}) + assert result is None + + def test_ideographic_common_sandwich_valid(self): + """Common script between ideographs should be treated as ideographic""" + check = FormatCheck( + FormatConstraintType.ValidFormats, + {"III"} + ) + result = check.validate("中·国", {"column_name": "country"}) + assert result is None + + def test_emoji_surrogate_valid_format(self): + """Emoji (surrogate pair) should be preserved as-is""" + check = FormatCheck( + FormatConstraintType.ValidFormats, + {"😀"} + ) + result = check.validate("😀", {"column_name": "icon"}) + assert result is None + + def test_string_longer_than_255_maps_to_na(self): + """Strings longer than 255 UTF-16 units should map to """ + long_value = "a" * 256 + check = FormatCheck( + FormatConstraintType.ValidFormats, + {""} + ) + result = check.validate(long_value, {"column_name": "payload"}) + assert result is None + + def test_numeric_value_vs_numeric_string_difference(self): + """Numeric value should not be treated the same as numeric string""" + check = FormatCheck( + FormatConstraintType.ValidFormats, + {"999"} + ) + result = check.validate(123, {"column_name": "code"}) + assert result is None + + +class TestInvalidFormats: + """Test InvalidFormats validation where value must not match any forbidden format""" + + def test_forbidden_date_fails(self): + """Forbidden date format should fail""" + check = FormatCheck(FormatConstraintType.InvalidFormats, {DateTimeFormats.ISO_DATE}) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is not None + assert "violates" in result.message + + def test_allowed_date_passes(self): + """Date not in forbidden list should pass""" + check = FormatCheck(FormatConstraintType.InvalidFormats, {DateTimeFormats.UK_DATE}) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is None + + def test_forbidden_time_fails(self): + """Forbidden time format should fail""" + check = FormatCheck(FormatConstraintType.InvalidFormats, {DateTimeFormats.TIME_24H}) + result = check.validate("14:23:01", {"column_name": "time"}) + assert result is not None + assert "violates" in result.message + + def test_allowed_time_passes(self): + """Time not in forbidden list should pass""" + check = FormatCheck(FormatConstraintType.InvalidFormats, {"%hh:%nn %a"}) + result = check.validate("14:23:01", {"column_name": "time"}) + assert result is None + + +class TestNoDetectedFormat: + """Test behavior when InferredTypeEngine cannot detect any format""" + + def test_string_with_no_format_valid_formats(self): + """Non-date string should fail ValidFormats""" + check = FormatCheck(FormatConstraintType.ValidFormats, {"%yyyy-%nn-%dd"}) + result = check.validate("hello", {"column_name": "name"}) + assert result is not None + assert "violates" in result.message + + def test_string_with_no_format_invalid_formats(self): + """Non-date string should pass InvalidFormats""" + check = FormatCheck(FormatConstraintType.InvalidFormats, {"%yyyy-%nn-%dd"}) + result = check.validate("hello", {"column_name": "name"}) + assert result is None + + +class TestEmptyAndNAFormats: + """Tests for and format handling""" + + def test_empty_string_allowed_when_empty_format_present(self): + """Empty string should map to and pass when allowed""" + check = FormatCheck(FormatConstraintType.ValidFormats, {""}) + result = check.validate("", {"column_name": "comment"}) + assert result is None + + def test_empty_string_rejected_when_empty_format_not_allowed(self): + """Empty string should fail ValidFormats when not allowed""" + check = FormatCheck(FormatConstraintType.ValidFormats, {"AAA"}) + result = check.validate("", {"column_name": "comment"}) + assert result is not None + assert "violates" in result.message + + def test_empty_string_blocked_by_invalid_formats(self): + """Empty string should fail InvalidFormats when is forbidden""" + check = FormatCheck(FormatConstraintType.InvalidFormats, {""}) + result = check.validate("", {"column_name": "comment"}) + assert result is not None + assert "violates" in result.message + + def test_long_string_maps_to_na_and_passes_when_allowed(self): + """String longer than 255 UTF-16 units should map to and pass when allowed""" + long_value = "A" * 300 + check = FormatCheck(FormatConstraintType.ValidFormats, {""}) + result = check.validate(long_value, {"column_name": "payload"}) + assert result is None + + def test_long_string_rejected_when_na_not_allowed(self): + """String mapping to should fail ValidFormats when not allowed""" + long_value = "A" * 300 + check = FormatCheck(FormatConstraintType.ValidFormats, {"AAA"}) + result = check.validate(long_value, {"column_name": "payload"}) + assert result is not None + assert "violates" in result.message + + def test_na_blocked_by_invalid_formats(self): + """ should fail InvalidFormats when explicitly forbidden""" + long_value = "A" * 300 + check = FormatCheck(FormatConstraintType.InvalidFormats, {""}) + result = check.validate(long_value, {"column_name": "payload"}) + assert result is not None + assert "violates" in result.message + + +class TestIdeographicCommonSandwich: + """Tests ideograph + COMMON + ideograph → III behavior""" + + def test_han_common_han(self): + """Han ideographs with COMMON in between should map to III""" + check = FormatCheck(FormatConstraintType.ValidFormats, {"III"}) + result = check.validate("中·国", {"column_name": "value"}) + assert result is None + + def test_hiragana_common_hiragana(self): + """Hiragana with COMMON in between should map to III""" + check = FormatCheck(FormatConstraintType.ValidFormats, {"III"}) + result = check.validate("あ・い", {"column_name": "value"}) + assert result is None + + def test_katakana_common_katakana(self): + """Katakana with COMMON in between should map to III""" + check = FormatCheck(FormatConstraintType.ValidFormats, {"III"}) + result = check.validate("カ・タ", {"column_name": "value"}) + assert result is None + + def test_hangul_common_hangul(self): + """Hangul with COMMON in between should map to III""" + check = FormatCheck(FormatConstraintType.ValidFormats, {"III"}) + result = check.validate("한·국", {"column_name": "value"}) + assert result is None + + + +class TestFormatCheckConstraintEdgeCases: + """Test FormatCheck behavior when constraint type or format sets are missing or empty""" + + def test_valid_formats_empty_rejects_date(self): + """With ValidFormats and empty set, any detected format should fail""" + check = FormatCheck(FormatConstraintType.ValidFormats, set()) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is not None + assert "violates" in result.message + + def test_invalid_formats_empty_allows_date(self): + """With InvalidFormats and empty set, any detected format should pass""" + check = FormatCheck(FormatConstraintType.InvalidFormats, set()) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is None + + def test_valid_formats_none_rejects(self): + """formats=None behaves like empty set for ValidFormats""" + check = FormatCheck(FormatConstraintType.ValidFormats, None) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is not None + assert "violates" in result.message + + def test_invalid_formats_none_allows(self): + """formats=None behaves like empty set for InvalidFormats""" + check = FormatCheck(FormatConstraintType.InvalidFormats, None) + result = check.validate("2024-01-10", {"column_name": "dob"}) + assert result is None + + +class TestNumericValues: + """Test behavior when numeric values are passed""" + + def test_numeric_value_valid_formats(self): + """Numbers should fail ValidFormats for the date format""" + check = FormatCheck(FormatConstraintType.ValidFormats, {DateTimeFormats.ISO_DATE}) + result = check.validate(123, {"column_name": "value"}) + assert result is not None + assert "violates" in result.message + + def test_numeric_value_invalid_formats(self): + """Numbers should pass in case of invalid date formats""" + check = FormatCheck(FormatConstraintType.InvalidFormats, {DateTimeFormats.ISO_DATE}) + result = check.validate(123, {"column_name": "value"}) + assert result is None + +class TestTimeZones: + """Test handling of timezone and Zulu (Z) timestamps""" + + def test_zulu_timestamp(self): + """Zulu timestamp should match %z-based formats""" + check = FormatCheck( + FormatConstraintType.ValidFormats, + {"%yyyy-%mm-%dd %HH:%nn:%ssZ"} + ) + result = check.validate("2024-01-10 14:23:01Z", {"column_name": "ts"}) + assert result is None diff --git a/tests/src/dq_validator/test_integration.py b/tests/src/dq_validator/test_integration.py new file mode 100644 index 0000000..44cdba1 --- /dev/null +++ b/tests/src/dq_validator/test_integration.py @@ -0,0 +1,466 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Integration tests for Data Intelligence SDK +""" + +import math +import pytest +from wxdi.dq_validator import ( + AssetMetadata, + ColumnMetadata, + DataType, + Validator, + ValidationRule, + ComparisonCheck, + ComparisonOperator, + ValidValuesCheck, + LengthCheck, +) + + +class TestBasicIntegration: + """Test basic integration scenarios""" + + def setup_method(self): + """Set up test metadata""" + self.metadata = AssetMetadata( + table_name="test_table", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("name", DataType.STRING, length=100), + ColumnMetadata("age", DataType.INTEGER), + ColumnMetadata("status", DataType.STRING, length=20), + ], + ) + + def test_single_rule_single_check_passes(self): + """Test single rule with single check passes""" + validator = Validator(self.metadata) + validator.add_rule( + ValidationRule("name").add_check(LengthCheck(min_length=2, max_length=50)) + ) + + record = [1, "John Doe", 25, "active"] + result = validator.validate(record) + + assert result.is_valid + assert result.score == "1/1" + assert math.isclose(result.pass_rate, 100.0) + assert len(result.errors) == 0 + + def test_single_rule_single_check_fails(self): + """Test single rule with single check fails""" + validator = Validator(self.metadata) + validator.add_rule( + ValidationRule("name").add_check(LengthCheck(min_length=10, max_length=50)) + ) + + record = [1, "John", 25, "active"] + result = validator.validate(record) + + assert not result.is_valid + assert result.score == "0/1" + assert math.isclose(result.pass_rate, 0.0, abs_tol=1e-9) + assert len(result.errors) == 1 + assert result.errors[0].column_name == "name" + + def test_multiple_rules_all_pass(self): + """Test multiple rules all pass""" + validator = Validator(self.metadata) + validator.add_rule( + ValidationRule("name").add_check(LengthCheck(min_length=2, max_length=50)) + ) + validator.add_rule( + ValidationRule("age").add_check( + ComparisonCheck(operator=">=", target_value=18) + ) + ) + validator.add_rule( + ValidationRule("status").add_check(ValidValuesCheck(["active", "inactive"])) + ) + + record = [1, "John Doe", 25, "active"] + result = validator.validate(record) + + assert result.is_valid + assert result.score == "3/3" + assert math.isclose(result.pass_rate, 100.0) + + def test_multiple_rules_some_fail(self): + """Test multiple rules with some failures""" + validator = Validator(self.metadata) + validator.add_rule( + ValidationRule("name").add_check(LengthCheck(min_length=2, max_length=50)) + ) + validator.add_rule( + ValidationRule("age").add_check( + ComparisonCheck(operator=">=", target_value=18) + ) + ) + validator.add_rule( + ValidationRule("status").add_check(ValidValuesCheck(["active", "inactive"])) + ) + + record = [1, "John Doe", 15, "pending"] # age < 18, status invalid + result = validator.validate(record) + + assert not result.is_valid + assert result.score == "1/3" + assert result.pass_rate == pytest.approx(33.33, rel=0.1) + assert len(result.errors) == 2 + + def test_single_rule_multiple_checks(self): + """Test single rule with multiple checks""" + validator = Validator(self.metadata) + validator.add_rule( + ValidationRule("age") + .add_check(ComparisonCheck(operator=">=", target_value=18)) + .add_check(ComparisonCheck(operator="<=", target_value=65)) + ) + + record = [1, "John Doe", 30, "active"] + result = validator.validate(record) + + assert result.is_valid + assert result.score == "2/2" + assert math.isclose(result.pass_rate, 100.0) + + +class TestBatchValidation: + """Test batch validation scenarios""" + + def setup_method(self): + """Set up test metadata and validator""" + self.metadata = AssetMetadata( + table_name="users", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("username", DataType.STRING), + ColumnMetadata("age", DataType.INTEGER), + ], + ) + + self.validator = Validator(self.metadata) + self.validator.add_rule( + ValidationRule("username").add_check( + LengthCheck(min_length=3, max_length=20) + ) + ) + self.validator.add_rule( + ValidationRule("age").add_check( + ComparisonCheck(operator=">=", target_value=18) + ) + ) + + def test_batch_all_pass(self): + """Test batch validation with all records passing""" + records = [[1, "alice", 25], [2, "bob", 30], [3, "charlie", 22]] + + results = self.validator.validate_batch(records) + + assert len(results) == 3 + assert all(r.is_valid for r in results) + assert all(r.score == "2/2" for r in results) + + def test_batch_some_fail(self): + """Test batch validation with some failures""" + records = [ + [1, "alice", 25], # Pass + [2, "ab", 30], # Fail: username too short + [3, "charlie", 15], # Fail: age < 18 + ] + + results = self.validator.validate_batch(records) + + assert len(results) == 3 + assert results[0].is_valid + assert not results[1].is_valid + assert not results[2].is_valid + + # Check specific errors + assert len(results[1].errors) == 1 + assert results[1].errors[0].column_name == "username" + + assert len(results[2].errors) == 1 + assert results[2].errors[0].column_name == "age" + + def test_batch_record_indices(self): + """Test batch validation tracks record indices""" + records = [ + [1, "alice", 25], + [2, "bob", 30], + ] + + results = self.validator.validate_batch(records) + + assert results[0].record_index == 0 + assert results[1].record_index == 1 + + +class TestComplexScenarios: + """Test complex validation scenarios""" + + def test_column_to_column_comparison(self): + """Test column-to-column comparison""" + metadata = AssetMetadata( + table_name="transactions", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("amount", DataType.DECIMAL), + ColumnMetadata("min_amount", DataType.DECIMAL), + ], + ) + + validator = Validator(metadata) + validator.add_rule( + ValidationRule("amount").add_check( + ComparisonCheck(operator=">", target_column="min_amount") + ) + ) + + # Pass case + record_pass = [1, 100.00, 50.00] + result_pass = validator.validate(record_pass) + assert result_pass.is_valid + + # Fail case + record_fail = [2, 30.00, 50.00] + result_fail = validator.validate(record_fail) + assert not result_fail.is_valid + assert ( + "amount (30.0) > min_amount (50.0) failed" in result_fail.errors[0].message + ) + + def test_case_insensitive_validation(self): + """Test case-insensitive validation""" + metadata = AssetMetadata( + table_name="data", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("status", DataType.STRING), + ], + ) + + validator = Validator(metadata) + validator.add_rule( + ValidationRule("status").add_check( + ValidValuesCheck(["active", "inactive"], case_sensitive=False) + ) + ) + + # Test various cases + test_cases = [ + [1, "active"], + [2, "ACTIVE"], + [3, "Active"], + [4, "AcTiVe"], + ] + + results = validator.validate_batch(test_cases) + assert all(r.is_valid for r in results) + + def test_multiple_checks_per_field(self): + """Test multiple checks on same field""" + metadata = AssetMetadata( + table_name="data", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("age", DataType.INTEGER), + ], + ) + + validator = Validator(metadata) + validator.add_rule( + ValidationRule("age") + .add_check(ComparisonCheck(operator=">=", target_value=18)) + .add_check(ComparisonCheck(operator="<=", target_value=65)) + .add_check(ComparisonCheck(operator="!=", target_value=0)) + ) + + # All checks pass + record_pass = [1, 30] + result_pass = validator.validate(record_pass) + assert result_pass.is_valid + assert result_pass.score == "3/3" + + # Some checks fail + record_fail = [2, 70] # > 65 + result_fail = validator.validate(record_fail) + assert not result_fail.is_valid + assert result_fail.score == "2/3" + + def test_length_check_with_integers(self): + """Test length check converts integers to strings""" + metadata = AssetMetadata( + table_name="data", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ], + ) + + validator = Validator(metadata) + validator.add_rule( + ValidationRule("id").add_check(LengthCheck(min_length=4, max_length=6)) + ) + + # Pass: 12345 -> "12345" (length 5) + result_pass = validator.validate([12345]) + assert result_pass.is_valid + + # Fail: 12 -> "12" (length 2) + result_fail = validator.validate([12]) + assert not result_fail.is_valid + + +class TestValidationResultDetails: + """Test ValidationResult details""" + + def test_result_to_dict(self): + """Test ValidationResult.to_dict()""" + metadata = AssetMetadata( + table_name="test", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("name", DataType.STRING), + ], + ) + + validator = Validator(metadata) + validator.add_rule(ValidationRule("name").add_check(LengthCheck(min_length=5))) + + record = [1, "abc"] + result = validator.validate(record) + + result_dict = result.to_dict() + + assert "record_index" in result_dict + assert "is_valid" in result_dict + assert "score" in result_dict + assert "pass_rate" in result_dict + assert "total_checks" in result_dict + assert "passed_checks" in result_dict + assert "failed_checks" in result_dict + assert "errors" in result_dict + + assert result_dict["is_valid"] == False + assert result_dict["total_checks"] == 1 + assert result_dict["failed_checks"] == 1 + assert len(result_dict["errors"]) == 1 + + def test_error_to_dict(self): + """Test ValidationError.to_dict()""" + metadata = AssetMetadata( + table_name="test", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("age", DataType.INTEGER), + ], + ) + + validator = Validator(metadata) + validator.add_rule( + ValidationRule("age").add_check( + ComparisonCheck(operator=">=", target_value=18) + ) + ) + + record = [1, 15] + result = validator.validate(record) + + error_dict = result.errors[0].to_dict() + + assert "column" in error_dict + assert "check" in error_dict + assert "message" in error_dict + assert "value" in error_dict + assert "expected" in error_dict + + assert error_dict["column"] == "age" + assert error_dict["check"] == "comparison_check" + + +class TestEdgeCases: + """Test edge cases and error conditions""" + + def test_empty_record_array(self): + """Test validation with empty record array""" + metadata = AssetMetadata( + table_name="test", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ], + ) + + validator = Validator(metadata) + validator.add_rule( + ValidationRule("id").add_check( + ComparisonCheck(operator=">", target_value=0) + ) + ) + + # Empty record should cause error + record = [] + result = validator.validate(record) + + assert not result.is_valid + assert len(result.errors) > 0 + + def test_validator_with_no_rules(self): + """Test validator with no rules""" + metadata = AssetMetadata( + table_name="test", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ], + ) + + validator = Validator(metadata) + + record = [1] + result = validator.validate(record) + + assert result.is_valid + assert result.total_checks == 0 + assert result.score == "0/0" + + def test_fluent_api_chaining(self): + """Test fluent API method chaining""" + metadata = AssetMetadata( + table_name="test", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("name", DataType.STRING), + ColumnMetadata("age", DataType.INTEGER), + ], + ) + + # Chain multiple add_rule calls + validator = ( + Validator(metadata) + .add_rule(ValidationRule("name").add_check(LengthCheck(min_length=2))) + .add_rule( + ValidationRule("age").add_check( + ComparisonCheck(operator=">=", target_value=0) + ) + ) + ) + + record = [1, "John", 25] + result = validator.validate(record) + + assert result.is_valid + assert result.total_checks == 2 diff --git a/tests/src/dq_validator/test_issue_reporting.py b/tests/src/dq_validator/test_issue_reporting.py new file mode 100644 index 0000000..760d819 --- /dev/null +++ b/tests/src/dq_validator/test_issue_reporting.py @@ -0,0 +1,1609 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" + +""" +Tests for IssueReporter utility. +""" + +import pytest +from unittest.mock import Mock, patch, MagicMock, call +from wxdi.dq_validator.issue_reporting import IssueReporter +from wxdi.dq_validator.provider import ProviderConfig +from wxdi.dq_validator import Validator, ValidationRule, AssetMetadata, ColumnMetadata +from wxdi.dq_validator.checks import ( + FormatCheck, CompletenessCheck, ComparisonCheck, + LengthCheck, RangeCheck, RegexCheck, CaseCheck, + DataTypeCheck, ValidValuesCheck +) +from wxdi.dq_validator.metadata import DataType +from wxdi.dq_validator.checks.comparison_check import ComparisonOperator +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + + +@pytest.fixture +def config(): + """Create a test configuration.""" + return ProviderConfig( + url="https://test-instance.com", + auth_token="Bearer test-token", + project_id="project-123" + ) + + +@pytest.fixture +def reporter(config): + """Create a test IssueReporter instance with mocked providers.""" + with patch('wxdi.dq_validator.issue_reporting.ChecksProvider'), \ + patch('wxdi.dq_validator.issue_reporting.IssuesProvider'), \ + patch('wxdi.dq_validator.issue_reporting.DimensionsProvider'), \ + patch('wxdi.dq_validator.issue_reporting.DQAssetsProvider'), \ + patch('wxdi.dq_validator.issue_reporting.CamsProvider'), \ + patch('wxdi.dq_validator.issue_reporting.DQSearchProvider'): + reporter = IssueReporter(config) + yield reporter + + +@pytest.fixture +def sample_validator(): + """Create a sample validator for testing""" + metadata = AssetMetadata('test_table', [ + ColumnMetadata('email', DataType.STRING), + ColumnMetadata('name', DataType.STRING), + ColumnMetadata('age', DataType.INTEGER), + ColumnMetadata('salary', DataType.FLOAT) + ]) + validator = Validator(metadata) + validator.add_rule(ValidationRule('email').add_check(FormatCheck(formats={'email'}))) + validator.add_rule(ValidationRule('name').add_check(CompletenessCheck()).add_check(LengthCheck(min_length=2))) + validator.add_rule(ValidationRule('age').add_check(RangeCheck(min_value=0, max_value=120))) + validator.add_rule(ValidationRule('salary').add_check(ComparisonCheck( + operator=ComparisonOperator.GREATER_THAN, + target_column='min_salary' + ))) + return validator + + +class TestIssueReporterInitialization: + """Test IssueReporter initialization""" + + def test_init_creates_all_providers(self, config): + """Test that initialization creates all required providers.""" + with patch('wxdi.dq_validator.issue_reporting.ChecksProvider') as mock_checks, \ + patch('wxdi.dq_validator.issue_reporting.IssuesProvider') as mock_issues, \ + patch('wxdi.dq_validator.issue_reporting.DimensionsProvider') as mock_dimensions, \ + patch('wxdi.dq_validator.issue_reporting.DQAssetsProvider') as mock_assets, \ + patch('wxdi.dq_validator.issue_reporting.DQSearchProvider') as mock_search, \ + patch('wxdi.dq_validator.issue_reporting.CamsProvider') as mock_cams: + + reporter = IssueReporter(config) + + # Verify all providers were created with correct config + mock_checks.assert_called_once_with(config) + mock_issues.assert_called_once_with(config) + mock_dimensions.assert_called_once_with(config) + mock_assets.assert_called_once_with(config) + mock_search.assert_called_once_with(config) + mock_cams.assert_called_once_with(config) + + # Verify config is stored + assert reporter.config == config + + # Verify all provider attributes exist + assert hasattr(reporter, 'check_provider') + assert hasattr(reporter, 'issues_provider') + assert hasattr(reporter, 'dimension_provider') + assert hasattr(reporter, 'asset_provider') + assert hasattr(reporter, 'search_provider') + assert hasattr(reporter, 'cams_provider') + + +class TestMapCheckNameToCheckType: + """Test map_check_name_to_check_type static method""" + + def test_format_check(self): + """Test mapping format_check to check type.""" + result = IssueReporter.map_check_name_to_check_type("format_check") + assert result == "format" + + def test_completeness_check(self): + """Test mapping completeness_check to check type.""" + result = IssueReporter.map_check_name_to_check_type("completeness_check") + assert result == "completeness" + + def test_comparison_check(self): + """Test mapping comparison_check to check type.""" + result = IssueReporter.map_check_name_to_check_type("comparison_check") + assert result == "comparison" + + def test_valid_values_check(self): + """Test mapping valid_values_check to check type.""" + result = IssueReporter.map_check_name_to_check_type("valid_values_check") + assert result == "possible_values" + + def test_unknown_check(self): + """Test mapping unknown check name returns None.""" + result = IssueReporter.map_check_name_to_check_type("unknown_check") + assert result is None + + +class TestMapCheckNameToCpdName: + """Test map_check_name_to_cpd_name static method""" + + def test_format_check(self): + """Test mapping format_check to CPD name.""" + result = IssueReporter.map_check_name_to_cpd_name("format_check") + assert result == "Format check" + + def test_completeness_check(self): + """Test mapping completeness_check to CPD name.""" + result = IssueReporter.map_check_name_to_cpd_name("completeness_check") + assert result == "Completeness check" + + def test_case_check(self): + """Test mapping case_check to CPD name.""" + result = IssueReporter.map_check_name_to_cpd_name("case_check") + assert result == "Capitalization style check" + + def test_valid_values_check(self): + """Test mapping valid_values_check to CPD name.""" + result = IssueReporter.map_check_name_to_cpd_name("valid_values_check") + assert result == "Possible values check" + + def test_unknown_check(self): + """Test mapping unknown check name returns None.""" + result = IssueReporter.map_check_name_to_cpd_name("unknown_check") + assert result is None + + +class TestGetCheckFromValidator: + """Test get_check_from_validator static method""" + + def test_get_check_found(self, sample_validator): + """Test getting check from validator when it exists.""" + check = IssueReporter.get_check_from_validator( + sample_validator, + "email", + "format_check" + ) + + assert check is not None + assert isinstance(check, FormatCheck) + assert check.get_check_name() == "format_check" + + def test_get_completeness_check(self, sample_validator): + """Test getting completeness check.""" + check = IssueReporter.get_check_from_validator( + sample_validator, + "name", + "completeness_check" + ) + + assert check is not None + assert isinstance(check, CompletenessCheck) + + def test_get_range_check(self, sample_validator): + """Test getting range check.""" + check = IssueReporter.get_check_from_validator( + sample_validator, + "age", + "range_check" + ) + + assert check is not None + assert isinstance(check, RangeCheck) + + def test_get_comparison_check(self, sample_validator): + """Test getting comparison check.""" + check = IssueReporter.get_check_from_validator( + sample_validator, + "salary", + "comparison_check" + ) + + assert check is not None + assert isinstance(check, ComparisonCheck) + + def test_check_not_found_wrong_column(self, sample_validator): + """Test getting check with wrong column name.""" + check = IssueReporter.get_check_from_validator( + sample_validator, + "nonexistent_column", + "format_check" + ) + + assert check is None + + def test_check_not_found_wrong_check_name(self, sample_validator): + """Test getting check with wrong check name.""" + check = IssueReporter.get_check_from_validator( + sample_validator, + "email", + "nonexistent_check" + ) + + assert check is None + + def test_multiple_checks_on_column(self, sample_validator): + """Test getting specific check when column has multiple checks.""" + # name column has both completeness and length checks + completeness_check = IssueReporter.get_check_from_validator( + sample_validator, + "name", + "completeness_check" + ) + length_check = IssueReporter.get_check_from_validator( + sample_validator, + "name", + "length_check" + ) + + assert completeness_check is not None + assert isinstance(completeness_check, CompletenessCheck) + assert length_check is not None + assert isinstance(length_check, LengthCheck) + + def test_empty_validator(self): + """Test with validator that has no rules.""" + metadata = AssetMetadata('empty_table', [ + ColumnMetadata('col1', DataType.STRING) + ]) + empty_validator = Validator(metadata) + + check = IssueReporter.get_check_from_validator( + empty_validator, + "col1", + "format_check" + ) + + assert check is None + + +class TestGetCheckId: + """Test get_check_id method""" + + def test_get_check_id_success(self, reporter): + """Test successful check ID retrieval.""" + reporter.search_provider.search_dq_check = Mock(return_value={ + "id": "check-id-123", + "native_id": "asset-id/column/check-type", + "type": "format" + }) + + result = reporter.get_check_id( + check_native_id="asset-id/column/check-type", + check_type="format", + project_id="project-123" + ) + + assert result == "check-id-123" + reporter.search_provider.search_dq_check.assert_called_once_with( + native_id="asset-id/column/check-type", + check_type="format", + project_id="project-123", + catalog_id=None + ) + + def test_get_check_id_with_catalog_id(self, reporter): + """Test check ID retrieval with catalog_id.""" + reporter.search_provider.search_dq_check = Mock(return_value={ + "id": "check-id-456" + }) + + result = reporter.get_check_id( + check_native_id="asset-id/column/check-type", + check_type="completeness", + catalog_id="catalog-789" + ) + + assert result == "check-id-456" + reporter.search_provider.search_dq_check.assert_called_once_with( + native_id="asset-id/column/check-type", + check_type="completeness", + project_id=None, + catalog_id="catalog-789" + ) + + def test_get_check_id_not_found(self, reporter): + """Test check ID retrieval when check not found.""" + reporter.search_provider.search_dq_check = Mock(side_effect=ValueError("Not found")) + + result = reporter.get_check_id( + check_native_id="nonexistent", + check_type="format", + project_id="project-123" + ) + + assert result is None + + def test_get_check_id_exception_handling(self, reporter): + """Test that exceptions are caught and None is returned.""" + reporter.search_provider.search_dq_check = Mock(side_effect=Exception("Unexpected error")) + + result = reporter.get_check_id( + check_native_id="asset-id/column/check", + check_type="format", + project_id="project-123" + ) + + assert result is None + + def test_get_check_id_missing_id_in_response(self, reporter): + """Test when response doesn't contain 'id' field.""" + reporter.search_provider.search_dq_check = Mock(return_value={ + "native_id": "asset-id/column/check-type" + # Missing 'id' field + }) + + result = reporter.get_check_id( + check_native_id="asset-id/column/check-type", + check_type="format", + project_id="project-123" + ) + + assert result is None + + +class TestCreateCheck: + """Test create_check method""" + + def test_create_check_format(self, reporter, sample_validator): + """Test creating a format check.""" + # Setup mocks + reporter.dimension_provider.search_dimension = Mock(return_value="dimension-id-123") + reporter.check_provider._create_check_full = Mock(return_value={ + "id": "check-id-456", + "name": "Format check", + "type": "format", + "native_id": "asset-789/format/Validity" + }) + + # Get the format check from validator + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + # Execute + result = reporter.create_check( + asset_id="asset-789", + column_name="email", + check_obj=format_check, + project_id="project-123" + ) + + # Verify - now returns dict + assert isinstance(result, dict) + assert result["id"] == "check-id-456" + + # Verify dimension search was called + reporter.dimension_provider.search_dimension.assert_called_once() + + # Verify check creation was called with correct parameters + reporter.check_provider._create_check_full.assert_called_once() + call_args = reporter.check_provider._create_check_full.call_args + assert call_args[1]['name'] == "Format check" + assert call_args[1]['dimension_id'] == "dimension-id-123" + # When parent_check_id is None, native_id format is: asset_id/check_type/DimensionName + assert "asset-789/format/Validity" in call_args[1]['native_id'] + assert call_args[1]['check_type'] == "format" + assert call_args[1]['project_id'] == "project-123" + + def test_create_check_completeness(self, reporter, sample_validator): + """Test creating a completeness check.""" + reporter.dimension_provider.search_dimension = Mock(return_value="dimension-id-comp") + reporter.check_provider._create_check_full = Mock(return_value={ + "id": "check-id-comp", + "name": "Completeness check", + "type": "completeness", + "native_id": "asset-999/completeness/Completeness" + }) + + completeness_check = IssueReporter.get_check_from_validator( + sample_validator, "name", "completeness_check" + ) + + result = reporter.create_check( + asset_id="asset-999", + column_name="name", + check_obj=completeness_check, + project_id="project-456" + ) + + assert isinstance(result, dict) + assert result["id"] == "check-id-comp" + + call_args = reporter.check_provider._create_check_full.call_args + assert call_args[1]['name'] == "Completeness check" + assert call_args[1]['check_type'] == "completeness" + # When parent_check_id is None, native_id format is: asset_id/check_type/DimensionName + assert "asset-999/completeness/Completeness" in call_args[1]['native_id'] + + def test_create_check_comparison_with_target_column(self, reporter, sample_validator): + """Test creating a comparison check with target column and parent_id.""" + reporter.dimension_provider.search_dimension = Mock(return_value="dimension-id-comp") + reporter.check_provider._create_check_full = Mock(return_value={ + "id": "check-id-comparison", + "name": "Comparison check", + "type": "comparison", + "native_id": "asset-comp/comparison/salary/greaterThan/min_salary" + }) + + comparison_check = IssueReporter.get_check_from_validator( + sample_validator, "salary", "comparison_check" + ) + + result = reporter.create_check( + asset_id="asset-comp", + column_name="salary", + check_obj=comparison_check, + project_id="project-789", + parent_id="parent-check-id" # Need parent_id for column name in native_id + ) + + assert isinstance(result, dict) + assert result["id"] == "check-id-comparison" + + call_args = reporter.check_provider._create_check_full.call_args + # When parent_id is provided, native_id includes column name and operator details + native_id = call_args[1]['native_id'] + assert "asset-comp/comparison/salary/" in native_id + assert "greaterThan" in native_id + assert "min_salary" in native_id + assert call_args[1]['parent_check_id'] == "parent-check-id" + + def test_create_check_with_catalog_id(self, reporter, sample_validator): + """Test creating a check with catalog_id instead of project_id.""" + reporter.dimension_provider.search_dimension = Mock(return_value="dimension-id-cat") + reporter.check_provider._create_check_full = Mock(return_value={ + "id": "check-id-cat", + "name": "Format check", + "type": "format" + }) + + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + result = reporter.create_check( + asset_id="asset-cat", + column_name="email", + check_obj=format_check, + catalog_id="catalog-999" + ) + + assert isinstance(result, dict) + assert result["id"] == "check-id-cat" + + call_args = reporter.check_provider._create_check_full.call_args + assert call_args[1]['catalog_id'] == "catalog-999" + assert call_args[1]['project_id'] is None + + def test_create_check_lowercase_column_name(self, reporter, sample_validator): + """Test that column name is converted to lowercase in native_id when parent_id is provided.""" + reporter.dimension_provider.search_dimension = Mock(return_value="dim-id") + reporter.check_provider._create_check_full = Mock(return_value={ + "id": "check-id", + "name": "Format check", + "type": "format", + "native_id": "asset-id/format/email/" + }) + + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + reporter.create_check( + asset_id="asset-id", + column_name="EMAIL", # Uppercase + check_obj=format_check, + project_id="project-id", + parent_id="parent-id" # Need parent_id for column name to be in native_id + ) + + call_args = reporter.check_provider._create_check_full.call_args + native_id = call_args[1]['native_id'] + # Should be lowercase + assert "/email/" in native_id + assert "/EMAIL/" not in native_id + + +class TestValidateAndPrepareCheckData: + """Test _validate_and_prepare_check_data method""" + + def test_validate_skip_no_failures(self, reporter, sample_validator): + """Test that validation returns None when there are no failures.""" + stats = {'failed': 0, 'total': 100} + + result = reporter._validate_and_prepare_check_data( + column_name="email", + check_name="format_check", + stats=stats, + data_asset_entity=Mock(), + assets_map={}, + validator=sample_validator + ) + + assert result is None + + def test_validate_skip_unmapped_check_type(self, reporter, sample_validator): + """Test that validation returns None for unmapped check types.""" + stats = {'failed': 10, 'total': 100} + + result = reporter._validate_and_prepare_check_data( + column_name="email", + check_name="unknown_check", # Not in mapping + stats=stats, + data_asset_entity=Mock(), + assets_map={}, + validator=sample_validator + ) + + assert result is None + + def test_validate_skip_missing_column_info(self, reporter, sample_validator): + """Test that validation returns None when column info is missing.""" + stats = {'failed': 10, 'total': 100} + data_asset_entity = Mock() + data_asset_entity.column_info = {} # Empty column info + + result = reporter._validate_and_prepare_check_data( + column_name="email", + check_name="format_check", + stats=stats, + data_asset_entity=data_asset_entity, + assets_map={}, + validator=sample_validator + ) + + assert result is None + + def test_validate_skip_missing_column_asset_id(self, reporter, sample_validator): + """Test that validation returns None when column asset ID not found.""" + stats = {'failed': 10, 'total': 100} + data_asset_entity = Mock() + data_asset_entity.column_info = {"email": Mock()} + assets_map = {} # Empty map + + result = reporter._validate_and_prepare_check_data( + column_name="email", + check_name="format_check", + stats=stats, + data_asset_entity=data_asset_entity, + assets_map=assets_map, + validator=sample_validator + ) + + assert result is None + + def test_validate_skip_check_not_in_validator(self, reporter, sample_validator): + """Test that validation returns None when check not found in validator.""" + stats = {'failed': 10, 'total': 100} + data_asset_entity = Mock() + data_asset_entity.column_info = {"email": Mock()} + assets_map = {"email": {"id": "column-asset-id"}} + + result = reporter._validate_and_prepare_check_data( + column_name="email", + check_name="range_check", # email doesn't have range check + stats=stats, + data_asset_entity=data_asset_entity, + assets_map=assets_map, + validator=sample_validator + ) + + assert result is None + + def test_validate_success(self, reporter, sample_validator): + """Test successful validation returns tuple with all data.""" + stats = {'failed': 10, 'total': 100} + data_asset_entity = Mock() + data_asset_entity.column_info = {"email": Mock()} + assets_map = {"email": {"id": "column-asset-id-123", "name": "email"}} + + result = reporter._validate_and_prepare_check_data( + column_name="email", + check_name="format_check", + stats=stats, + data_asset_entity=data_asset_entity, + assets_map=assets_map, + validator=sample_validator + ) + + assert result is not None + check_type, column_id, check_obj, occurrences, total = result + assert check_type == "format" + assert column_id == "column-asset-id-123" + assert isinstance(check_obj, FormatCheck) + assert occurrences == 10 + assert total == 100 + + +class TestCreateCheckAndIssue: + """Test _create_check_and_issue method""" + + def test_create_check_and_issue_success(self, reporter, sample_validator): + """Test successful check and issue creation.""" + # Mock handle_parent to return a parent check dict + reporter.handle_parent = Mock(return_value={"id": "parent-check-id", "type": "format"}) + reporter.create_check = Mock(return_value={ + "id": "check-id-123", + "name": "Format check", + "type": "format" + }) + reporter.issues_provider.create_issue = Mock() + + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column" + } + } + + result = reporter._create_check_and_issue( + asset_id="asset-id", + column_name="email", + column_id="column-asset-id", + check_name="format_check", + check_obj=format_check, + number_of_occurrences=10, + total_records=100, + project_id="project-123", + assets_map=assets_map + ) + + assert result is True + reporter.handle_parent.assert_called_once() + reporter.create_check.assert_called_once() + reporter.issues_provider.create_issue.assert_called_once_with( + dq_check_id="check-id-123", + reported_for_id="column-asset-id", + number_of_occurrences=10, + number_of_tested_records=100, + project_id="project-123", + catalog_id=None + ) + + def test_create_check_and_issue_409_conflict_update_success(self, reporter, sample_validator): + """Test handling 409 conflict by updating existing check.""" + # Mock handle_parent to return a parent check dict + reporter.handle_parent = Mock(return_value={"id": "parent-check-id", "type": "format"}) + # First call raises 409, then get_checks returns existing check + reporter.create_check = Mock( + side_effect=ValueError("409 Conflict: Check already exists") + ) + reporter.check_provider.get_checks = Mock(return_value=[ + {"id": "existing-check-id", "type": "format", "native_id": "asset-id/email/format"} + ]) + reporter.issues_provider.update_issue_metrics = Mock() + + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column" + } + } + + result = reporter._create_check_and_issue( + asset_id="asset-id", + column_name="email", + column_id="column-asset-id", + check_name="format_check", + check_obj=format_check, + number_of_occurrences=10, + total_records=100, + project_id="project-123", + assets_map=assets_map + ) + + assert result is True + reporter.check_provider.get_checks.assert_called_once_with( + dq_asset_id="column-asset-id", + check_type="format", + project_id="project-123" + ) + reporter.issues_provider.update_issue_metrics.assert_called_once() + + def test_create_check_and_issue_409_conflict_check_not_found(self, reporter, sample_validator): + """Test handling 409 conflict when existing check cannot be found.""" + reporter.handle_parent = Mock(return_value=None) + reporter.create_check = Mock( + side_effect=ValueError("409 Conflict: Check already exists") + ) + reporter.check_provider.get_checks = Mock(return_value=[ + {"id": "other-check-id", "type": "completeness"} # Different type + ]) + + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column" + } + } + + result = reporter._create_check_and_issue( + asset_id="asset-id", + column_name="email", + column_id="column-asset-id", + check_name="format_check", + check_obj=format_check, + number_of_occurrences=10, + total_records=100, + project_id="project-123", + assets_map=assets_map + ) + + assert result is False + + def test_create_check_and_issue_non_409_error(self, reporter, sample_validator): + """Test handling non-409 errors.""" + reporter.handle_parent = Mock(return_value=None) + reporter.create_check = Mock( + side_effect=ValueError("500 Internal Server Error") + ) + + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column" + } + } + + result = reporter._create_check_and_issue( + asset_id="asset-id", + column_name="email", + column_id="column-asset-id", + check_name="format_check", + check_obj=format_check, + number_of_occurrences=10, + total_records=100, + project_id="project-123", + assets_map=assets_map + ) + + assert result is False + + def test_create_check_and_issue_unmapped_check_type(self, reporter, sample_validator): + """Test handling unmapped check type.""" + # Create a mock check with unknown type + unknown_check = Mock() + unknown_check.get_check_name = Mock(return_value="unknown_check") + + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column" + } + } + + result = reporter._create_check_and_issue( + asset_id="asset-id", + column_name="email", + column_id="column-asset-id", + check_name="unknown_check", + check_obj=unknown_check, + number_of_occurrences=10, + total_records=100, + project_id="project-123", + assets_map=assets_map + ) + + assert result is False + + +class TestHandleParent: + """Test handle_parent method""" + + def test_handle_parent_found_existing(self, reporter, sample_validator): + """Test finding existing parent check.""" + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + # Mock search_provider to return existing check + reporter.search_provider.search_dq_check = Mock(return_value={ + "id": "parent-check-id", + "native_id": "asset-id/format/Accuracy", + "type": "format" + }) + + result = reporter.handle_parent( + asset_id="asset-id", + check_obj=format_check, + project_id="project-123" + ) + + assert result is not None + assert result["id"] == "parent-check-id" + assert "_newly_created" not in result + reporter.search_provider.search_dq_check.assert_called_once() + + def test_handle_parent_not_found_creates_new(self, reporter, sample_validator): + """Test creating new parent check when not found.""" + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + # Mock search to raise exception (not found) + reporter.search_provider.search_dq_check = Mock( + side_effect=Exception("Check not found") + ) + + # Mock create_check to return new check + reporter.create_check = Mock(return_value={ + "id": "new-parent-check-id", + "native_id": "asset-id/format/Accuracy", + "type": "format" + }) + + result = reporter.handle_parent( + asset_id="asset-id", + check_obj=format_check, + project_id="project-123" + ) + + assert result is not None + assert result["id"] == "new-parent-check-id" + assert result["_newly_created"] is True + reporter.create_check.assert_called_once_with( + asset_id="asset-id", + column_name=None, + check_obj=format_check, + project_id="project-123", + catalog_id=None, + parent_id=None + ) + + def test_handle_parent_creation_fails(self, reporter, sample_validator): + """Test when both search and creation fail - should raise RuntimeError.""" + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + # Mock search to raise exception + reporter.search_provider.search_dq_check = Mock( + side_effect=Exception("Check not found") + ) + + # Mock create_check to also raise exception + reporter.create_check = Mock( + side_effect=Exception("Creation failed") + ) + + # Should raise RuntimeError when parent creation fails + with pytest.raises(RuntimeError) as exc_info: + reporter.handle_parent( + asset_id="asset-id", + check_obj=format_check, + project_id="project-123" + ) + + assert "Failed to create parent check" in str(exc_info.value) + assert "Creation failed" in str(exc_info.value) + + def test_handle_parent_with_catalog_id(self, reporter, sample_validator): + """Test handle_parent with catalog_id.""" + format_check = IssueReporter.get_check_from_validator( + sample_validator, "email", "format_check" + ) + + reporter.search_provider.search_dq_check = Mock(return_value={ + "id": "parent-check-id", + "native_id": "asset-id/format/Validity", + "type": "format" + }) + + result = reporter.handle_parent( + asset_id="asset-id", + check_obj=format_check, + project_id=None, + catalog_id="catalog-123" + ) + + assert result is not None + reporter.search_provider.search_dq_check.assert_called_once_with( + native_id="asset-id/format/Validity", + check_type="format", + project_id=None, + catalog_id="catalog-123", + include_children=False + ) + + +class TestCreateBulkIssues: + """Test create_bulk_issues method""" + + def test_create_bulk_issues_success(self, reporter): + """Test successful bulk issue creation.""" + parent_check = { + "id": "parent-check-id", + "native_id": "asset-id/format/Accuracy", + "type": "format" + } + + child_check = { + "id": "child-check-id", + "native_id": "asset-id/email/format", + "type": "format" + } + + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column", + "native_id": "schema.table.email", + "parent": { + "id": "parent-asset-id" + }, + "weight": 1 + }, + "table": { + "id": "parent-asset-id", + "name": "table", + "type": "data_asset", + "native_id": "schema.table", + "weight": 1 + } + } + + reporter.issues_provider.create_issues_bulk = Mock(return_value={ + "created": 2 + }) + + result = reporter.create_bulk_issues( + parent_check=parent_check, + child_check=child_check, + column_name="email", + assets_map=assets_map, + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + assert result is not None + reporter.issues_provider.create_issues_bulk.assert_called_once() + + # Verify the payload structure + call_args = reporter.issues_provider.create_issues_bulk.call_args + payload = call_args.kwargs["payload"] + + assert len(payload["issues"]) == 2 + assert len(payload["assets"]) == 2 + assert len(payload["existing_checks"]) == 2 + assert payload["issues"][0]["status"] == "aggregation" + assert payload["issues"][1]["status"] == "actual" + + def test_create_bulk_issues_column_not_found(self, reporter): + """Test when column asset not found in assets_map.""" + parent_check = {"id": "parent-check-id"} + child_check = {"id": "child-check-id"} + assets_map = {} + + with pytest.raises(ValueError, match="Column asset not found"): + reporter.create_bulk_issues( + parent_check=parent_check, + child_check=child_check, + column_name="email", + assets_map=assets_map, + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + def test_create_bulk_issues_parent_id_not_found(self, reporter): + """Test when parent asset ID not found in column asset.""" + parent_check = {"id": "parent-check-id"} + child_check = {"id": "child-check-id"} + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column" + # Missing parent + } + } + + with pytest.raises(ValueError, match="Parent asset ID not found"): + reporter.create_bulk_issues( + parent_check=parent_check, + child_check=child_check, + column_name="email", + assets_map=assets_map, + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + def test_create_bulk_issues_parent_asset_not_in_map(self, reporter): + """Test when parent asset not found in assets_map.""" + parent_check = {"id": "parent-check-id"} + child_check = {"id": "child-check-id"} + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column", + "parent": { + "id": "parent-asset-id" + } + } + # Parent asset not in map + } + + with pytest.raises(ValueError, match="Parent asset not found in assets_map"): + reporter.create_bulk_issues( + parent_check=parent_check, + child_check=child_check, + column_name="email", + assets_map=assets_map, + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + def test_create_bulk_issues_parent_native_id_missing(self, reporter): + """Test when parent asset native_id is missing.""" + parent_check = {"id": "parent-check-id"} + child_check = {"id": "child-check-id"} + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column", + "parent": { + "id": "parent-asset-id" + } + }, + "table": { + "id": "parent-asset-id", + "name": "table", + "type": "data_asset" + # Missing native_id + } + } + + with pytest.raises(ValueError, match="Parent asset native_id not found"): + reporter.create_bulk_issues( + parent_check=parent_check, + child_check=child_check, + column_name="email", + assets_map=assets_map, + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + def test_create_bulk_issues_api_failure(self, reporter): + """Test when bulk API call fails.""" + parent_check = { + "id": "parent-check-id", + "native_id": "asset-id/format/Accuracy", + "type": "format" + } + + child_check = { + "id": "child-check-id", + "native_id": "asset-id/email/format", + "type": "format" + } + + assets_map = { + "email": { + "id": "column-asset-id", + "name": "email", + "type": "column", + "native_id": "schema.table.email", + "parent": { + "id": "parent-asset-id" + }, + "weight": 1 + }, + "table": { + "id": "parent-asset-id", + "name": "table", + "type": "data_asset", + "native_id": "schema.table", + "weight": 1 + } + } + + reporter.issues_provider.create_issues_bulk = Mock( + side_effect=Exception("API Error") + ) + + with pytest.raises(Exception, match="API Error"): + reporter.create_bulk_issues( + parent_check=parent_check, + child_check=child_check, + column_name="email", + assets_map=assets_map, + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + +class TestHelperMethods: + """Test refactored helper methods""" + + def test_find_existing_check_success(self, reporter): + """Test successfully finding an existing check.""" + reporter.check_provider.get_checks = Mock(return_value=[ + {"id": "check-1", "type": "format", "native_id": "asset/email/format"}, + {"id": "check-2", "type": "completeness", "native_id": "asset/email/completeness"} + ]) + + result = reporter._find_existing_check( + column_id="column-asset-id", + check_type="format", + project_id="project-123" + ) + + assert result is not None + assert result[0] == "check-1" + assert result[1] == "asset/email/format" + + def test_find_existing_check_not_found(self, reporter): + """Test when check type doesn't match.""" + reporter.check_provider.get_checks = Mock(return_value=[ + {"id": "check-1", "type": "completeness"} + ]) + + result = reporter._find_existing_check( + column_id="column-asset-id", + check_type="format", + project_id="project-123" + ) + + assert result is None + + def test_find_existing_check_api_error(self, reporter): + """Test when API call fails.""" + reporter.check_provider.get_checks = Mock( + side_effect=ValueError("API Error") + ) + + result = reporter._find_existing_check( + column_id="column-asset-id", + check_type="format", + project_id="project-123" + ) + + assert result is None + + def test_update_existing_check_metrics_success(self, reporter): + """Test successful metric update.""" + reporter.issues_provider.update_issue_metrics = Mock() + + result = reporter._update_existing_check_metrics( + existing_check_native_id="asset/email/format", + number_of_occurrences=10, + total_records=100, + column_name="email", + check_type="format", + project_id="project-123" + ) + + assert result is True + reporter.issues_provider.update_issue_metrics.assert_called_once() + + def test_update_existing_check_metrics_failure(self, reporter): + """Test when metric update fails.""" + reporter.issues_provider.update_issue_metrics = Mock( + side_effect=ValueError("Update failed") + ) + + result = reporter._update_existing_check_metrics( + existing_check_native_id="asset/email/format", + number_of_occurrences=10, + total_records=100, + column_name="email", + check_type="format", + project_id="project-123" + ) + + assert result is False + + def test_handle_409_conflict_success(self, reporter): + """Test successful 409 conflict handling.""" + reporter._find_existing_check = Mock(return_value=("check-id", "native-id")) + reporter._update_existing_check_metrics = Mock(return_value=True) + + result = reporter._handle_409_conflict( + column_id="column-asset-id", + check_type="format", + number_of_occurrences=10, + total_records=100, + column_name="email", + check_name="format_check", + asset_id="asset-id", + project_id="project-123" + ) + + assert result is True + + def test_handle_409_conflict_check_not_found(self, reporter): + """Test 409 conflict when check not found.""" + reporter._find_existing_check = Mock(return_value=None) + + result = reporter._handle_409_conflict( + column_id="column-asset-id", + check_type="format", + number_of_occurrences=10, + total_records=100, + column_name="email", + check_name="format_check", + asset_id="asset-id", + project_id="project-123" + ) + + assert result is False + + def test_handle_409_conflict_update_fails_fallback(self, reporter): + """Test 409 conflict when update fails but fallback succeeds.""" + reporter._find_existing_check = Mock(return_value=("check-id", "native-id")) + reporter._update_existing_check_metrics = Mock(return_value=False) + reporter._handle_update_failure = Mock() + + result = reporter._handle_409_conflict( + column_id="column-asset-id", + check_type="format", + number_of_occurrences=10, + total_records=100, + column_name="email", + check_name="format_check", + asset_id="asset-id", + project_id="project-123" + ) + + assert result is True + reporter._handle_update_failure.assert_called_once() + + +class TestHandleUpdateFailure: + """Test _handle_update_failure method""" + + def test_handle_update_failure_issue_not_found_creates_issue(self, reporter): + """Test creating issue when update fails with 'Issue not found'.""" + reporter.get_check_id = Mock(return_value="check-id-123") + reporter.issues_provider.create_issue = Mock() + + error = ValueError("Issue not found for check") + + result = reporter._handle_update_failure( + error=error, + asset_id="asset-id", + check_type="format", + column_name="email", + column_id="column-asset-id", + number_of_occurrences=10, + total_records=100, + project_id="project-123", + check_id="check-id" + ) + + assert result is True + reporter.get_check_id.assert_called_once_with( + check_native_id="asset-id/check-id", + check_type="format", + project_id="project-123" + ) + reporter.issues_provider.create_issue.assert_called_once() + + def test_handle_update_failure_issue_id_not_found(self, reporter): + """Test creating issue when update fails with 'Issue ID not found'.""" + reporter.get_check_id = Mock(return_value="check-id-456") + reporter.issues_provider.create_issue = Mock() + + error = ValueError("Issue ID not found in response") + + result = reporter._handle_update_failure( + error=error, + asset_id="asset-id", + check_type="completeness", + column_name="name", + column_id="column-asset-id", + number_of_occurrences=5, + total_records=50, + project_id="project-123", + check_id="check-id" + ) + + assert result is True + reporter.issues_provider.create_issue.assert_called_once() + + def test_handle_update_failure_check_id_not_found(self, reporter): + """Test when check_id cannot be found.""" + reporter.get_check_id = Mock(return_value=None) + + error = ValueError("Issue not found") + + result = reporter._handle_update_failure( + error=error, + asset_id="asset-id", + check_type="format", + column_name="email", + column_id="column-asset-id", + number_of_occurrences=10, + total_records=100, + project_id="project-123", + check_id="check-id" + ) + + assert result is True + reporter.get_check_id.assert_called_once() + + def test_handle_update_failure_different_error(self, reporter): + """Test handling different error types.""" + error = ValueError("Different error message") + + result = reporter._handle_update_failure( + error=error, + asset_id="asset-id", + check_type="format", + column_name="email", + column_id="column-asset-id", + number_of_occurrences=10, + total_records=100, + project_id="project-123", + check_id="check-id" + ) + + assert result is True + + +class TestHandleExistingCheck: + """Test _handle_existing_check method""" + + def test_handle_existing_check_success(self, reporter): + """Test successful handling of existing check.""" + column_info = Mock() + check = Mock() + check.metadata = Mock() + check.metadata.type = "format" + check.metadata.check_id = "check-id-123" + column_info.column_checks = [check] + + reporter.issues_provider.update_issue_metrics = Mock() + + result = reporter._handle_existing_check( + column_info=column_info, + check_type="format", + asset_id="asset-id", + column_name="email", + column_id="column-asset-id", + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + assert result is True + reporter.issues_provider.update_issue_metrics.assert_called_once_with( + asset_id="asset-id", + check_id="check-id-123", + occurrences=10, + tested_records=100, + column_name="email", + check_type="format", + project_id="project-123", + asset_type="column", + operation="add" + ) + + def test_handle_existing_check_type_mismatch(self, reporter): + """Test when check type doesn't match.""" + column_info = Mock() + check = Mock() + check.metadata = Mock() + check.metadata.type = "completeness" # Different type + check.metadata.check_id = "check-id-123" + column_info.column_checks = [check] + + result = reporter._handle_existing_check( + column_info=column_info, + check_type="format", # Looking for format + asset_id="asset-id", + column_name="email", + column_id="column-asset-id", + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + assert result is False + + def test_handle_existing_check_no_check_id(self, reporter): + """Test when check has no check_id.""" + column_info = Mock() + check = Mock() + check.metadata = Mock() + check.metadata.type = "format" + check.metadata.check_id = None # No check_id + column_info.column_checks = [check] + + result = reporter._handle_existing_check( + column_info=column_info, + check_type="format", + asset_id="asset-id", + column_name="email", + column_id="column-asset-id", + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + assert result is False + + def test_handle_existing_check_update_failure(self, reporter): + """Test handling update failure.""" + column_info = Mock() + check = Mock() + check.metadata = Mock() + check.metadata.type = "format" + check.metadata.check_id = "check-id-123" + column_info.column_checks = [check] + + reporter.issues_provider.update_issue_metrics = Mock( + side_effect=ValueError("Update failed") + ) + reporter._handle_update_failure = Mock(return_value=True) + + result = reporter._handle_existing_check( + column_info=column_info, + check_type="format", + asset_id="asset-id", + column_name="email", + column_id="column-asset-id", + number_of_occurrences=10, + total_records=100, + project_id="project-123" + ) + + assert result is True + reporter._handle_update_failure.assert_called_once() + + +class TestReportIssues: + """Test update_issues method""" + + def test_report_issues_success(self, reporter, sample_validator): + """Test successful update of issues.""" + # Mock dependencies + reporter.cams_provider.get_asset_by_id = Mock() + data_asset = Mock() + data_asset_entity = Mock() + column_info = Mock() + column_info.column_checks = [] + data_asset_entity.column_info = {"email": column_info} + data_asset.entity = data_asset_entity + reporter.cams_provider.get_asset_by_id.return_value = data_asset + + reporter.asset_provider.get_assets = Mock(return_value={ + "assets": [{"name": "email", "id": "column-asset-id"}] + }) + + reporter._create_check_and_issue = Mock(return_value=True) + + # Create combined stats in nested dict format + combined_stats = { + "email": { + "format_check": {"failed": 10, "total": 100} + } + } + + reporter.report_issues( + asset_id="asset-id", + stats=combined_stats, + validator=sample_validator + ) + + reporter._create_check_and_issue.assert_called_once() + + def test_report_issues_with_existing_checks(self, reporter, sample_validator): + """Test update with existing checks.""" + # Mock dependencies + reporter.cams_provider.get_asset_by_id = Mock() + data_asset = Mock() + data_asset_entity = Mock() + column_info = Mock() + check = Mock() + check.metadata = Mock() + check.metadata.type = "format" + check.metadata.check_id = "check-id-123" + column_info.column_checks = [check] + data_asset_entity.column_info = {"email": column_info} + data_asset.entity = data_asset_entity + reporter.cams_provider.get_asset_by_id.return_value = data_asset + + reporter.asset_provider.get_assets = Mock(return_value={ + "assets": [{"name": "email", "id": "column-asset-id"}] + }) + + reporter._handle_existing_check = Mock(return_value=True) + + # Create combined stats in nested dict format + combined_stats = { + "email": { + "format_check": {"failed": 10, "total": 100} + } + } + + reporter.report_issues( + asset_id="asset-id", + stats=combined_stats, + validator=sample_validator + ) + + reporter._handle_existing_check.assert_called_once() + + def test_report_issues_check_not_handled_creates_new(self, reporter, sample_validator): + """Test creating new check when existing check not handled.""" + # Mock dependencies + reporter.cams_provider.get_asset_by_id = Mock() + data_asset = Mock() + data_asset_entity = Mock() + column_info = Mock() + check = Mock() + check.metadata = Mock() + check.metadata.type = "completeness" # Different type + check.metadata.check_id = "check-id-123" + column_info.column_checks = [check] + data_asset_entity.column_info = {"email": column_info} + data_asset.entity = data_asset_entity + reporter.cams_provider.get_asset_by_id.return_value = data_asset + + reporter.asset_provider.get_assets = Mock(return_value={ + "assets": [{"name": "email", "id": "column-asset-id"}] + }) + + reporter._handle_existing_check = Mock(return_value=False) + reporter._create_check_and_issue = Mock(return_value=True) + + # Create combined stats in nested dict format + combined_stats = { + "email": { + "format_check": {"failed": 10, "total": 100} + } + } + + reporter.report_issues( + asset_id="asset-id", + stats=combined_stats, + validator=sample_validator + ) + + reporter._handle_existing_check.assert_called_once() + reporter._create_check_and_issue.assert_called_once() \ No newline at end of file diff --git a/tests/src/dq_validator/test_length_check.py b/tests/src/dq_validator/test_length_check.py new file mode 100644 index 0000000..66c57ed --- /dev/null +++ b/tests/src/dq_validator/test_length_check.py @@ -0,0 +1,307 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for LengthCheck +""" + +import pytest +from wxdi.dq_validator.checks.length_check import LengthCheck +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + + +class TestLengthCheckInitialization: + """Test LengthCheck initialization and parameter validation""" + + def test_init_with_min_length_only(self): + """Test initialization with only min_length""" + check = LengthCheck(min_length=5) + assert check.min_length == 5 + assert check.max_length is None + + def test_init_with_max_length_only(self): + """Test initialization with only max_length""" + check = LengthCheck(max_length=10) + assert check.min_length is None + assert check.max_length == 10 + + def test_init_with_both_lengths(self): + """Test initialization with both min and max length""" + check = LengthCheck(min_length=3, max_length=20) + assert check.min_length == 3 + assert check.max_length == 20 + + def test_init_no_parameters_raises_error(self): + """Test that initialization without parameters raises ValueError""" + with pytest.raises(ValueError) as exc_info: + LengthCheck() + assert "At least one of min_length or max_length must be specified" in str(exc_info.value) + + def test_init_negative_min_length_raises_error(self): + """Test that negative min_length raises ValueError""" + with pytest.raises(ValueError) as exc_info: + LengthCheck(min_length=-1) + assert "min_length cannot be negative" in str(exc_info.value) + + def test_init_negative_max_length_raises_error(self): + """Test that negative max_length raises ValueError""" + with pytest.raises(ValueError) as exc_info: + LengthCheck(max_length=-5) + assert "max_length cannot be negative" in str(exc_info.value) + + def test_init_min_greater_than_max_raises_error(self): + """Test that min_length > max_length raises ValueError""" + with pytest.raises(ValueError) as exc_info: + LengthCheck(min_length=10, max_length=5) + assert "min_length (10) cannot be greater than max_length (5)" in str(exc_info.value) + + def test_get_check_name(self): + """Test get_check_name returns correct name""" + check = LengthCheck(min_length=1) + assert check.get_check_name() == "length_check" + + def test_get_dimension(self): + """Test get_dimension returns correct dimension""" + check = LengthCheck(min_length=1) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + def test_set_dimension(self): + """Test set_dimension changes the dimension""" + check = LengthCheck(min_length=1) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + check.set_dimension(DataQualityDimension.CONSISTENCY) + assert check.get_dimension() == DataQualityDimension.CONSISTENCY + + +class TestLengthCheckStringValidation: + """Test LengthCheck with string values""" + + def test_string_within_range_passes(self): + """Test string within min/max range passes""" + check = LengthCheck(min_length=3, max_length=20) + context = {'column_name': 'username'} + result = check.validate('john_doe', context) + assert result is None + + def test_string_too_short_fails(self): + """Test string shorter than min_length fails""" + check = LengthCheck(min_length=3, max_length=20) + context = {'column_name': 'username'} + result = check.validate('ab', context) + assert result is not None + assert result.column_name == 'username' + assert result.check_name == 'length_check' + assert "length (2) is less than minimum (3)" in result.message + + def test_string_too_long_fails(self): + """Test string longer than max_length fails""" + check = LengthCheck(min_length=3, max_length=20) + context = {'column_name': 'username'} + result = check.validate('a' * 25, context) + assert result is not None + assert "length (25) exceeds maximum (20)" in result.message + + def test_empty_string_with_min_zero_passes(self): + """Test empty string passes when min_length is 0""" + check = LengthCheck(min_length=0, max_length=100) + context = {'column_name': 'optional_field'} + result = check.validate('', context) + assert result is None + + def test_empty_string_with_min_one_fails(self): + """Test empty string fails when min_length is 1""" + check = LengthCheck(min_length=1, max_length=100) + context = {'column_name': 'required_field'} + result = check.validate('', context) + assert result is not None + assert "length (0) is less than minimum (1)" in result.message + + def test_exact_length_required_passes(self): + """Test exact length requirement passes""" + check = LengthCheck(min_length=2, max_length=2) + context = {'column_name': 'country_code'} + result = check.validate('US', context) + assert result is None + + def test_exact_length_required_fails(self): + """Test exact length requirement fails""" + check = LengthCheck(min_length=2, max_length=2) + context = {'column_name': 'country_code'} + result = check.validate('USA', context) + assert result is not None + assert "length (3) exceeds maximum (2)" in result.message + + def test_unicode_characters(self): + """Test Unicode characters are counted correctly""" + check = LengthCheck(min_length=1, max_length=10) + context = {'column_name': 'name'} + result = check.validate('日本語', context) + assert result is None # Length is 3 characters + + def test_whitespace_only_string(self): + """Test whitespace-only string is counted""" + check = LengthCheck(min_length=1, max_length=10) + context = {'column_name': 'text'} + result = check.validate(' ', context) + assert result is None # Length is 3 + + +class TestLengthCheckNonStringValidation: + """Test LengthCheck with non-string values (converted to string)""" + + def test_integer_value_passes(self): + """Test integer value is converted to string""" + check = LengthCheck(min_length=3, max_length=10) + context = {'column_name': 'id'} + result = check.validate(12345, context) + assert result is None # str(12345) = "12345", length = 5 + + def test_integer_value_fails(self): + """Test integer value fails when string length is out of range""" + check = LengthCheck(min_length=1, max_length=5) + context = {'column_name': 'number'} + result = check.validate(123456789, context) + assert result is not None + assert "length (9) exceeds maximum (5)" in result.message + + def test_float_value_passes(self): + """Test float value is converted to string""" + check = LengthCheck(min_length=3, max_length=10) + context = {'column_name': 'amount'} + result = check.validate(123.45, context) + assert result is None # str(123.45) = "123.45", length = 6 + + def test_boolean_true_passes(self): + """Test boolean True is converted to string""" + check = LengthCheck(min_length=4, max_length=5) + context = {'column_name': 'flag'} + result = check.validate(True, context) + assert result is None # str(True) = "True", length = 4 + + def test_boolean_false_passes(self): + """Test boolean False is converted to string""" + check = LengthCheck(min_length=5, max_length=5) + context = {'column_name': 'flag'} + result = check.validate(False, context) + assert result is None # str(False) = "False", length = 5 + + def test_list_value_passes(self): + """Test list value is converted to string""" + check = LengthCheck(min_length=5, max_length=50) + context = {'column_name': 'data'} + result = check.validate([1, 2, 3], context) + assert result is None # str([1, 2, 3]) = "[1, 2, 3]", length = 9 + + def test_dict_value_passes(self): + """Test dict value is converted to string""" + check = LengthCheck(min_length=5, max_length=50) + context = {'column_name': 'config'} + result = check.validate({'a': 1}, context) + assert result is None # str({'a': 1}) = "{'a': 1}", length varies + + +class TestLengthCheckNoneHandling: + """Test LengthCheck with None values""" + + def test_none_value_fails(self): + """Test None value returns error""" + check = LengthCheck(min_length=3, max_length=20) + context = {'column_name': 'username'} + result = check.validate(None, context) + assert result is not None + assert result.column_name == 'username' + assert "is None, cannot check length" in result.message + + +class TestLengthCheckMinLengthOnly: + """Test LengthCheck with only min_length specified""" + + def test_min_length_only_passes(self): + """Test validation passes when only min_length is specified""" + check = LengthCheck(min_length=10) + context = {'column_name': 'description'} + result = check.validate('a' * 100, context) + assert result is None # No max limit + + def test_min_length_only_fails(self): + """Test validation fails when below min_length""" + check = LengthCheck(min_length=10) + context = {'column_name': 'description'} + result = check.validate('short', context) + assert result is not None + assert "length (5) is less than minimum (10)" in result.message + + +class TestLengthCheckMaxLengthOnly: + """Test LengthCheck with only max_length specified""" + + def test_max_length_only_passes(self): + """Test validation passes when only max_length is specified""" + check = LengthCheck(max_length=10) + context = {'column_name': 'code'} + result = check.validate('ABC', context) + assert result is None # No min limit + + def test_max_length_only_fails(self): + """Test validation fails when exceeds max_length""" + check = LengthCheck(max_length=10) + context = {'column_name': 'code'} + result = check.validate('a' * 15, context) + assert result is not None + assert "length (15) exceeds maximum (10)" in result.message + + +class TestLengthCheckEdgeCases: + """Test LengthCheck edge cases""" + + def test_zero_min_length(self): + """Test min_length of 0 is valid""" + check = LengthCheck(min_length=0, max_length=10) + context = {'column_name': 'field'} + result = check.validate('', context) + assert result is None + + def test_zero_max_length(self): + """Test max_length of 0 requires empty string""" + check = LengthCheck(min_length=0, max_length=0) + context = {'column_name': 'field'} + result = check.validate('', context) + assert result is None + + def test_zero_max_length_fails_with_content(self): + """Test max_length of 0 fails with non-empty string""" + check = LengthCheck(min_length=0, max_length=0) + context = {'column_name': 'field'} + result = check.validate('a', context) + assert result is not None + assert "length (1) exceeds maximum (0)" in result.message + + def test_large_min_length(self): + """Test very large min_length""" + check = LengthCheck(min_length=1000) + context = {'column_name': 'field'} + result = check.validate('a' * 999, context) + assert result is not None + assert "length (999) is less than minimum (1000)" in result.message + + def test_repr(self): + """Test __repr__ method""" + check = LengthCheck(min_length=5, max_length=10) + repr_str = repr(check) + assert "LengthCheck" in repr_str + assert "5" in repr_str + assert "10" in repr_str + diff --git a/tests/src/dq_validator/test_pandas_validator.py b/tests/src/dq_validator/test_pandas_validator.py new file mode 100644 index 0000000..63a9ed1 --- /dev/null +++ b/tests/src/dq_validator/test_pandas_validator.py @@ -0,0 +1,535 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for PandasValidator + +Tests the Pandas DataFrame integration including: +- Initialization and configuration +- Summary statistics calculation +- Validation column addition +- Invalid/valid row filtering +- Column expansion +- Chunked processing +- Error handling +""" + +import math +import pytest +import sys +from typing import Dict, Any + +# Check if pandas is available +try: + import pandas as pd + + PANDAS_AVAILABLE = True +except ImportError: + PANDAS_AVAILABLE = False + +from wxdi.dq_validator import ( + AssetMetadata, + ColumnMetadata, + DataType, + Validator, + ValidationRule, +) +from wxdi.dq_validator.checks import ( + LengthCheck, + ValidValuesCheck, + ComparisonCheck, + ComparisonOperator, +) + +if PANDAS_AVAILABLE: + from wxdi.dq_validator.integrations import PandasValidator + + +# Skip all tests if pandas is not available +pytestmark = pytest.mark.skipif(not PANDAS_AVAILABLE, reason="pandas not installed") + + +@pytest.fixture +def sample_metadata(): + """Create sample metadata for testing""" + return AssetMetadata( + table_name="test_table", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("name", DataType.STRING, length=50), + ColumnMetadata("age", DataType.INTEGER), + ColumnMetadata("status", DataType.STRING, length=20), + ], + ) + + +@pytest.fixture +def sample_validator(sample_metadata): + """Create sample validator with rules""" + validator = Validator(sample_metadata) + + # Add validation rules + validator.add_rule( + ValidationRule("name").add_check(LengthCheck(min_length=2, max_length=50)) + ) + validator.add_rule( + ValidationRule("age").add_check( + ComparisonCheck( + operator=ComparisonOperator.GREATER_THAN_OR_EQUAL, target_value=18 + ) + ) + ) + validator.add_rule( + ValidationRule("status").add_check( + ValidValuesCheck(["active", "inactive"], case_sensitive=False) + ) + ) + + return validator + + +@pytest.fixture +def sample_dataframe(): + """Create sample DataFrame for testing""" + return pd.DataFrame( + { + "id": [1, 2, 3, 4, 5], + "name": ["Alice", "B", "Charlie", "David", "Eve"], + "age": [25, 30, 17, 35, 40], + "status": ["active", "ACTIVE", "inactive", "pending", "Active"], + } + ) + + +class TestPandasValidatorInitialization: + """Test PandasValidator initialization""" + + def test_basic_initialization(self, sample_validator): + """Test basic initialization with default parameters""" + pandas_validator = PandasValidator(sample_validator) + + assert pandas_validator.validator == sample_validator + assert pandas_validator.chunk_size == 10000 + assert pandas_validator.column_prefix == "dq_" + + def test_custom_chunk_size(self, sample_validator): + """Test initialization with custom chunk size""" + pandas_validator = PandasValidator(sample_validator, chunk_size=5000) + + assert pandas_validator.chunk_size == 5000 + + def test_custom_column_prefix(self, sample_validator): + """Test initialization with custom column prefix""" + pandas_validator = PandasValidator( + sample_validator, column_prefix="validation_" + ) + + assert pandas_validator.column_prefix == "validation_" + + def test_invalid_chunk_size(self, sample_validator): + """Test initialization with invalid chunk size""" + with pytest.raises(ValueError, match="chunk_size must be positive"): + PandasValidator(sample_validator, chunk_size=0) + + with pytest.raises(ValueError, match="chunk_size must be positive"): + PandasValidator(sample_validator, chunk_size=-100) + + +class TestSummaryStatistics: + """Test summary statistics calculation""" + + def test_basic_summary(self, sample_validator, sample_dataframe): + """Test basic summary statistics""" + pandas_validator = PandasValidator(sample_validator) + summary = pandas_validator.get_summary_statistics(sample_dataframe) + + assert isinstance(summary, dict) + assert "total_rows" in summary + assert "valid_rows" in summary + assert "invalid_rows" in summary + assert "pass_rate" in summary + assert "total_checks" in summary + assert "passed_checks" in summary + assert "failed_checks" in summary + + assert summary["total_rows"] == 5 + assert summary["valid_rows"] + summary["invalid_rows"] == 5 + assert 0 <= summary["pass_rate"] <= 100 + + def test_all_valid_rows(self, sample_validator): + """Test summary with all valid rows""" + df = pd.DataFrame( + { + "id": [1, 2, 3], + "name": ["Alice", "Bob", "Charlie"], + "age": [25, 30, 35], + "status": ["active", "inactive", "active"], + } + ) + + pandas_validator = PandasValidator(sample_validator) + summary = pandas_validator.get_summary_statistics(df) + + assert summary["total_rows"] == 3 + assert summary["valid_rows"] == 3 + assert summary["invalid_rows"] == 0 + assert math.isclose(summary["pass_rate"], 100.0) + + def test_all_invalid_rows(self, sample_validator): + """Test summary with all invalid rows""" + df = pd.DataFrame( + { + "id": [1, 2, 3], + "name": ["A", "B", "C"], # All too short + "age": [15, 16, 17], # All under 18 + "status": ["pending", "deleted", "archived"], # All invalid + } + ) + + pandas_validator = PandasValidator(sample_validator) + summary = pandas_validator.get_summary_statistics(df) + + assert summary["total_rows"] == 3 + assert summary["valid_rows"] == 0 + assert summary["invalid_rows"] == 3 + assert math.isclose(summary["pass_rate"], 0.0, abs_tol=1e-9) + + def test_empty_dataframe(self, sample_validator): + """Test summary with empty DataFrame""" + df = pd.DataFrame(columns=["id", "name", "age", "status"]) + + pandas_validator = PandasValidator(sample_validator) + summary = pandas_validator.get_summary_statistics(df) + + assert summary["total_rows"] == 0 + assert summary["valid_rows"] == 0 + assert summary["invalid_rows"] == 0 + assert math.isclose(summary["pass_rate"], 0.0, abs_tol=1e-9) + + def test_chunked_processing(self, sample_validator): + """Test summary with chunked processing""" + # Create DataFrame larger than chunk size + df = pd.DataFrame( + { + "id": range(1, 101), + "name": ["Alice"] * 100, + "age": [25] * 100, + "status": ["active"] * 100, + } + ) + + pandas_validator = PandasValidator(sample_validator, chunk_size=30) + summary = pandas_validator.get_summary_statistics(df) + + assert summary["total_rows"] == 100 + assert summary["valid_rows"] == 100 + + +class TestAddValidationColumn: + """Test adding validation column to DataFrame""" + + def test_basic_validation_column(self, sample_validator, sample_dataframe): + """Test adding validation column""" + pandas_validator = PandasValidator(sample_validator) + df_validated = pandas_validator.add_validation_column(sample_dataframe) + + # Check that original columns are preserved + assert all(col in df_validated.columns for col in sample_dataframe.columns) + + # Check that validation column is added + assert "dq_validation_result" in df_validated.columns + + # Check that DataFrame has same number of rows + assert len(df_validated) == len(sample_dataframe) + + def test_validation_result_structure(self, sample_validator, sample_dataframe): + """Test structure of validation result""" + pandas_validator = PandasValidator(sample_validator) + df_validated = pandas_validator.add_validation_column(sample_dataframe) + + # Check first validation result + result = df_validated["dq_validation_result"].iloc[0] + + assert isinstance(result, dict) + assert "is_valid" in result + assert "score" in result + assert "pass_rate" in result + assert "total_checks" in result + assert "passed_checks" in result + assert "failed_checks" in result + assert "error_count" in result + assert "errors" in result + + assert isinstance(result["is_valid"], bool) + assert isinstance(result["score"], str) + assert isinstance(result["pass_rate"], float) + assert isinstance(result["errors"], list) + + def test_custom_column_prefix(self, sample_validator, sample_dataframe): + """Test validation column with custom prefix""" + pandas_validator = PandasValidator(sample_validator, column_prefix="val_") + df_validated = pandas_validator.add_validation_column(sample_dataframe) + + assert "val_validation_result" in df_validated.columns + assert "dq_validation_result" not in df_validated.columns + + def test_column_conflict_detection(self, sample_validator): + """Test detection of column name conflicts""" + df = pd.DataFrame( + { + "id": [1, 2], + "name": ["Alice", "Bob"], + "age": [25, 30], + "status": ["active", "inactive"], + "dq_validation_result": ["existing", "data"], # Conflict! + } + ) + + pandas_validator = PandasValidator(sample_validator) + + with pytest.raises(ValueError, match="already exists"): + pandas_validator.add_validation_column(df) + + +class TestInvalidRowFiltering: + """Test filtering invalid rows""" + + def test_get_invalid_rows(self, sample_validator, sample_dataframe): + """Test getting invalid rows""" + pandas_validator = PandasValidator(sample_validator) + invalid_df = pandas_validator.get_invalid_rows(sample_dataframe) + + # Should have validation column + assert "dq_validation_result" in invalid_df.columns + + # All rows should be invalid + for _, row in invalid_df.iterrows(): + assert row["dq_validation_result"]["is_valid"] is False + + def test_get_valid_rows(self, sample_validator, sample_dataframe): + """Test getting valid rows""" + pandas_validator = PandasValidator(sample_validator) + valid_df = pandas_validator.get_valid_rows(sample_dataframe) + + # Should have validation column + assert "dq_validation_result" in valid_df.columns + + # All rows should be valid + for _, row in valid_df.iterrows(): + assert row["dq_validation_result"]["is_valid"] is True + + def test_no_invalid_rows(self, sample_validator): + """Test when there are no invalid rows""" + df = pd.DataFrame( + { + "id": [1, 2, 3], + "name": ["Alice", "Bob", "Charlie"], + "age": [25, 30, 35], + "status": ["active", "inactive", "active"], + } + ) + + pandas_validator = PandasValidator(sample_validator) + invalid_df = pandas_validator.get_invalid_rows(df) + + assert len(invalid_df) == 0 + + def test_no_valid_rows(self, sample_validator): + """Test when there are no valid rows""" + df = pd.DataFrame( + { + "id": [1, 2, 3], + "name": ["A", "B", "C"], + "age": [15, 16, 17], + "status": ["pending", "deleted", "archived"], + } + ) + + pandas_validator = PandasValidator(sample_validator) + valid_df = pandas_validator.get_valid_rows(df) + + assert len(valid_df) == 0 + + +class TestColumnExpansion: + """Test expanding validation column""" + + def test_basic_expansion(self, sample_validator, sample_dataframe): + """Test basic column expansion""" + pandas_validator = PandasValidator(sample_validator) + df_validated = pandas_validator.add_validation_column(sample_dataframe) + df_expanded = pandas_validator.expand_validation_column(df_validated) + + # Check that expanded columns exist + assert "dq_is_valid" in df_expanded.columns + assert "dq_score" in df_expanded.columns + assert "dq_pass_rate" in df_expanded.columns + assert "dq_total_checks" in df_expanded.columns + assert "dq_passed_checks" in df_expanded.columns + assert "dq_failed_checks" in df_expanded.columns + assert "dq_error_count" in df_expanded.columns + assert "dq_errors" in df_expanded.columns + + # Original validation column should be removed + assert "dq_validation_result" not in df_expanded.columns + + # Original columns should be preserved + assert all(col in df_expanded.columns for col in sample_dataframe.columns) + + def test_expansion_with_custom_prefix(self, sample_validator, sample_dataframe): + """Test expansion with custom prefix""" + pandas_validator = PandasValidator(sample_validator, column_prefix="val_") + df_validated = pandas_validator.add_validation_column(sample_dataframe) + df_expanded = pandas_validator.expand_validation_column(df_validated) + + assert "val_is_valid" in df_expanded.columns + assert "val_score" in df_expanded.columns + assert "val_pass_rate" in df_expanded.columns + + def test_expansion_without_validation_column( + self, sample_validator, sample_dataframe + ): + """Test expansion when validation column doesn't exist""" + pandas_validator = PandasValidator(sample_validator) + + with pytest.raises( + ValueError, match="does not contain validation result column" + ): + pandas_validator.expand_validation_column(sample_dataframe) + + +class TestEdgeCases: + """Test edge cases and error handling""" + + def test_dataframe_with_missing_columns(self, sample_validator): + """Test DataFrame missing required columns""" + df = pd.DataFrame( + { + "id": [1, 2], + "name": ["Alice", "Bob"], + # Missing 'age' and 'status' columns + } + ) + + pandas_validator = PandasValidator(sample_validator) + + # Should raise error when trying to validate + with pytest.raises(Exception): + pandas_validator.get_summary_statistics(df) + + def test_dataframe_with_null_values(self, sample_validator): + """Test DataFrame with null values""" + df = pd.DataFrame( + { + "id": [1, 2, 3], + "name": ["Alice", None, "Charlie"], + "age": [25, 30, None], + "status": ["active", "inactive", None], + } + ) + + pandas_validator = PandasValidator(sample_validator) + summary = pandas_validator.get_summary_statistics(df) + + # Should handle nulls gracefully + assert summary["total_rows"] == 3 + assert summary["invalid_rows"] > 0 # Nulls should cause failures + + def test_large_dataframe_chunked_processing(self, sample_validator): + """Test chunked processing with large DataFrame""" + # Create large DataFrame + df = pd.DataFrame( + { + "id": range(1, 10001), + "name": ["Alice"] * 10000, + "age": [25] * 10000, + "status": ["active"] * 10000, + } + ) + + pandas_validator = PandasValidator(sample_validator, chunk_size=1000) + summary = pandas_validator.get_summary_statistics(df) + + assert summary["total_rows"] == 10000 + assert summary["valid_rows"] == 10000 + + def test_string_representation(self, sample_validator): + """Test string representation of validator""" + pandas_validator = PandasValidator(sample_validator, chunk_size=5000) + str_repr = str(pandas_validator) + + assert "PandasValidator" in str_repr + assert "5000" in str_repr + + +class TestIntegrationScenarios: + """Test complete integration scenarios""" + + def test_complete_workflow(self, sample_validator, sample_dataframe): + """Test complete validation workflow""" + pandas_validator = PandasValidator(sample_validator) + + # Step 1: Get summary + summary = pandas_validator.get_summary_statistics(sample_dataframe) + assert summary["total_rows"] == 5 + + # Step 2: Add validation column + df_validated = pandas_validator.add_validation_column(sample_dataframe) + assert "dq_validation_result" in df_validated.columns + + # Step 3: Filter invalid rows + invalid_df = pandas_validator.get_invalid_rows(sample_dataframe) + assert len(invalid_df) == summary["invalid_rows"] + + # Step 4: Expand columns + df_expanded = pandas_validator.expand_validation_column(df_validated) + assert "dq_is_valid" in df_expanded.columns + + def test_multiple_validations_same_dataframe( + self, sample_validator, sample_dataframe + ): + """Test running multiple validations on same DataFrame""" + pandas_validator = PandasValidator(sample_validator) + + # Run validation multiple times + summary1 = pandas_validator.get_summary_statistics(sample_dataframe) + summary2 = pandas_validator.get_summary_statistics(sample_dataframe) + + # Results should be consistent + assert summary1 == summary2 + + def test_validation_with_different_chunk_sizes(self, sample_validator): + """Test that different chunk sizes produce same results""" + df = pd.DataFrame( + { + "id": range(1, 101), + "name": ["Alice"] * 100, + "age": [25] * 100, + "status": ["active"] * 100, + } + ) + + validator1 = PandasValidator(sample_validator, chunk_size=10) + validator2 = PandasValidator(sample_validator, chunk_size=50) + + summary1 = validator1.get_summary_statistics(df) + summary2 = validator2.get_summary_statistics(df) + + # Results should be identical regardless of chunk size + assert summary1 == summary2 + + +if __name__ == "__main__": + pytest.main([__file__, "-v"]) diff --git a/tests/src/dq_validator/test_range_check.py b/tests/src/dq_validator/test_range_check.py new file mode 100644 index 0000000..ed6f8fb --- /dev/null +++ b/tests/src/dq_validator/test_range_check.py @@ -0,0 +1,380 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for RangeCheck +""" + +import pytest +from decimal import Decimal +from datetime import datetime, date +from wxdi.dq_validator.checks.range_check import RangeCheck +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + + +class TestRangeCheckInitialization: + """Test RangeCheck initialization and parameter validation""" + + def test_init_with_min_value_only(self): + """Test initialization with only min_value""" + check = RangeCheck(min_value=10) + assert isinstance(check.min_value, Decimal) + assert check.min_value == Decimal("10") + assert check.max_value is None + + def test_init_with_max_value_only(self): + """Test initialization with only max_value""" + check = RangeCheck(max_value=20) + assert isinstance(check.max_value, Decimal) + assert check.min_value is None + assert check.max_value == Decimal("20") + + def test_init_with_both_values(self): + """Test successful initialization with min and max""" + check = RangeCheck(min_value=10, max_value=20) + assert isinstance(check.min_value, Decimal) + assert isinstance(check.max_value, Decimal) + assert check.min_value == Decimal("10") + assert check.max_value == Decimal("20") + + def test_init_with_no_parameters_raises_error(self): + """Test that missing min or max raises ValueError""" + with pytest.raises(ValueError) as exc_info: + RangeCheck() + assert "At least one of min_value or max_value must be specified" in str( + exc_info.value + ) + + def test_init_min_greater_than_max_raises_error(self): + """Test that min_value > max_value raises ValueError""" + with pytest.raises(ValueError) as exc_info: + RangeCheck(min_value=20, max_value=10) + assert "min_value (20) cannot be greater than max_value (10)" in str( + exc_info.value + ) + + def test_init_incompatible_numeric_and_string_raises_error(self): + """Test initialization failure when min is Decimal and max is string""" + with pytest.raises(TypeError) as exc_info: + RangeCheck(min_value=10, max_value="20") + assert "Incompatible types of min_value and max_value" in str(exc_info.value) + + def test_init_incompatible_datetime_and_decimal_raises_error(self): + """Test initialization failure when min is datetime and max is Decimal""" + with pytest.raises(TypeError) as exc_info: + RangeCheck(min_value=date(2024, 1, 1), max_value=500.5) + assert "Incompatible types of min_value and max_value" in str(exc_info.value) + + def test_init_incompatible_string_and_datetime_raises_error(self): + """Test initialization failure when min is string and max is datetime""" + with pytest.raises(TypeError) as exc_info: + RangeCheck(min_value="2024-01-01", max_value=date(2024, 12, 31)) + assert "Incompatible types of min_value and max_value" in str(exc_info.value) + + def test_get_check_name(self): + """Test get_check_name returns correct name""" + check = RangeCheck(min_value=10, max_value=20) + assert check.get_check_name() == "range_check" + + def test_get_dimension(self): + """Test get_dimension returns correct dimension""" + check = RangeCheck(min_value=10, max_value=20) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + def test_set_dimension(self): + """Test set_dimension changes the dimension""" + check = RangeCheck(min_value=10, max_value=20) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + check.set_dimension(DataQualityDimension.CONSISTENCY) + assert check.get_dimension() == DataQualityDimension.CONSISTENCY + + +class TestRangeCheckNormalization: + """Test normalization logic via class initialization""" + + def test_normalize_numeric_init(self): + """Test normalization of int and float values to Decimal""" + check = RangeCheck(min_value=10, max_value=20.5) + assert isinstance(check.min_value, Decimal) + assert isinstance(check.max_value, Decimal) + assert check.min_value == Decimal("10") + assert check.max_value == Decimal("20.5") + + def test_normalize_decimal_init(self): + """Test normalization of default Decimal values to Decimal""" + check = RangeCheck(min_value=Decimal("10"), max_value=Decimal("20.5")) + assert isinstance(check.min_value, Decimal) + assert isinstance(check.max_value, Decimal) + assert check.min_value == Decimal("10") + assert check.max_value == Decimal("20.5") + + def test_normalize_date_init(self): + """Test normalization of date and default datetime objects to datetime""" + check = RangeCheck( + min_value=date(2024, 1, 1), + max_value=datetime(2024, 12, 31, 23, 59, 59, 999999), + ) + assert isinstance(check.min_value, datetime) + assert isinstance(check.max_value, datetime) + assert check.min_value == datetime(2024, 1, 1, 0, 0, 0) + assert check.max_value == datetime(2024, 12, 31, 23, 59, 59, 999999) + assert check.max_value.microsecond == 999999 + + def test_normalize_string_fallback_init(self): + """Test non-numeric and non-date inputs fallback to string during init""" + check = RangeCheck(min_value="alpha", max_value="omega") + assert isinstance(check.min_value, str) + assert isinstance(check.max_value, str) + assert check.min_value == "alpha" + assert check.max_value == "omega" + + def test_normalization_none_init(self): + """Test normalization of None inputs return None""" + check = RangeCheck(min_value="alpha") + assert isinstance(check.min_value, str) + assert check.min_value == "alpha" + assert check.max_value == None + + +class TestRangeCheckValidation: + """Test validation logic for various scenarios""" + + def test_range_passes(self): + """Test value within range passes""" + check = RangeCheck(min_value=10, max_value=20) + context = {"column_name": "score"} + assert check.validate(15, context) is None + assert check.validate(10, context) is None # Boundary inclusive + assert check.validate(20, context) is None # Boundary inclusive + + def test_less_than_min_fails(self): + """Test value below minimum fails""" + check = RangeCheck(min_value=10, max_value=20) + context = {"column_name": "score"} + result = check.validate(5, context) + assert result is not None + assert "score (5) is less than minimum (10)" in result.message + + def test_greater_than_max_fails(self): + """Test value above maximum fails""" + check = RangeCheck(min_value=10, max_value=20) + context = {"column_name": "score"} + result = check.validate(25, context) + assert result is not None + assert "score (25) is greater than maximum (20)" in result.message + + def test_none_value_fails(self): + """Test None value fails check""" + check = RangeCheck(min_value="alpha", max_value="gamma") + context = {"column_name": "title"} + result = check.validate(None, context) + assert result is not None + assert "title is None, cannot check to be within range" in result.message + + def test_range_empty_string_passes(self): + """Passes because '' is not less than the min '' and not greater than 'z'""" + check = RangeCheck(min_value="", max_value="z") + context = {"column_name": "code"} + result = check.validate("", context) + assert result is None + + def test_range_empty_string_fails(self): + """Fails because '' is lexicographically less than 'a'""" + check = RangeCheck(min_value="a", max_value="z") + context = {"column_name": "code"} + result = check.validate("", context) + assert result is not None + assert "code () is less than minimum (a)" in result.message + + def test_range_whitespace_passes(self): + """Passes because ' ' (two spaces) is 'greater' than ' ' (one space)""" + check = RangeCheck(min_value=" ", max_value="z") + context = {"column_name": "padded_val"} + result = check.validate(" ", context) + assert result is None + + def test_range_whitespace_fails(self): + """Fails because a space is lexicographically less than the character '0'""" + check = RangeCheck(min_value="0", max_value="9") + context = {"column_name": "digit_check"} + result = check.validate(" ", context) + assert result is not None + assert "digit_check ( ) is less than minimum (0)" in result.message + + +class TestRangeCheckDataTypes: + """Test range check with different data types""" + + def test_date_range_passes(self): + """Test validation passes for date value in range""" + check = RangeCheck(date(2024, 1, 1), date(2024, 1, 31)) + context = {"column_name": "created_at"} + result = check.validate(date(2024, 1, 15), context) + assert result is None + + def test_date_range_fails(self): + """Test validation fails for date value out of range""" + check = RangeCheck(date(2024, 1, 1), date(2024, 1, 31)) + context = {"column_name": "created_at"} + result = check.validate(date(2024, 2, 1), context) + assert result is not None + assert ( + "created_at (2024-02-01 00:00:00) is greater than maximum (2024-01-31 00:00:00)" + in result.message + ) + + def test_datetime_range_passes(self): + """Test validation passes for datetime value in range""" + check = RangeCheck(datetime(2024, 1, 1, 0, 0), datetime(2024, 1, 1, 23, 59)) + context = {"column_name": "timestamp"} + result = check.validate(datetime(2024, 1, 1, 12, 0), context) + assert result is None + + def test_datetime_range_fails(self): + """Test validation fails for datetime value out of range""" + check = RangeCheck(datetime(2024, 1, 1, 0, 0), datetime(2024, 1, 1, 12, 0)) + context = {"column_name": "timestamp"} + val = datetime(2024, 1, 1, 15, 0) + result = check.validate(val, context) + assert result is not None + assert ( + "timestamp (2024-01-01 15:00:00) is greater than maximum (2024-01-01 12:00:00)" + in result.message + ) + + def test_string_range_passes(self): + """Test validation passes for string value in range""" + check = RangeCheck("apple", "cherry") + context = {"column_name": "fruit"} + result = check.validate("banana", context) + assert result is None + + def test_string_range_fails(self): + """Test validation fails for string value out of range""" + check = RangeCheck("apple", "cherry") + context = {"column_name": "fruit"} + result = check.validate("date", context) + assert result is not None + assert "fruit (date) is greater than maximum (cherry)" in result.message + + def test_int_range_passes(self): + """Test validation passes for integer value in range""" + check = RangeCheck(1, 10) + context = {"column_name": "count"} + result = check.validate(5, context) + assert result is None + + def test_int_range_fails(self): + """Test validation fails for integer value out of range""" + check = RangeCheck(1, 10) + context = {"column_name": "count"} + result = check.validate(15, context) + assert result is not None + assert "count (15) is greater than maximum (10)" in result.message + + def test_float_range_passes(self): + """Test validation passes for float value in range""" + check = RangeCheck(1.0, 5.5) + context = {"column_name": "rating"} + result = check.validate(3.2, context) + assert result is None + + def test_float_range_fails(self): + """Test validation fails for float value out of range""" + check = RangeCheck(1.0, 5.5) + context = {"column_name": "rating"} + result = check.validate(0.5, context) + assert result is not None + assert "rating (0.5) is less than minimum (1.0)" in result.message + + def test_decimal_range_passes(self): + """Test validation passes for decimal value in range""" + check = RangeCheck(Decimal("10.00"), Decimal("20.00")) + context = {"column_name": "price"} + result = check.validate(Decimal("15.50"), context) + assert result is None + + def test_decimal_range_fails(self): + """Test validation fails for decimal value out of range""" + check = RangeCheck(Decimal("10.00"), Decimal("20.00")) + context = {"column_name": "price"} + result = check.validate(Decimal("25.00"), context) + assert result is not None + assert "price (25.00) is greater than maximum (20.00)" in result.message + + +class TestRangeCheckTypeMismatch: + """Test for type mismatches between values and range boundaries""" + + def test_decimal_range_with_string_value_fails(self): + """Test validation error when comparing a string value against a numeric range""" + check = RangeCheck(min_value=10, max_value=20) + context = {"column_name": "age"} + result = check.validate("25", context) + assert result is not None + assert "Cannot compare str with min_value Decimal" in result.message + + def test_datetime_range_with_decimal_value_fails(self): + """Test validation error when comparing a numeric value against a datetime range""" + check = RangeCheck(min_value=date(2024, 1, 1), max_value=date(2024, 12, 31)) + context = {"column_name": "created_at"} + result = check.validate(500, context) + assert result is not None + assert "Cannot compare Decimal with min_value datetime" in result.message + + def test_string_range_with_datetime_value_fails(self): + """Test validation error when comparing a datetime value against a string range""" + check = RangeCheck(min_value="aaa", max_value="zzz") + context = {"column_name": "code"} + result = check.validate(datetime(2024, 1, 1), context) + assert result is not None + assert "Cannot compare datetime with min_value str" in result.message + + def test_decimal_range_with_datetime_value_fails(self): + """Test validation error when comparing a datetime value against a numeric range""" + check = RangeCheck(min_value=0, max_value=100) + context = {"column_name": "score"} + result = check.validate(datetime(2025, 1, 1), context) + assert result is not None + assert "Cannot compare datetime with min_value Decimal" in result.message + + def test_datetime_range_with_string_value_fails(self): + """Test validation error when comparing a string value against a datetime range""" + check = RangeCheck(min_value=date(2024, 1, 1), max_value=date(2024, 12, 31)) + context = {"column_name": "deadline"} + result = check.validate("2024-06-01", context) + assert result is not None + assert "Cannot compare str with min_value datetime" in result.message + + def test_string_range_with_decimal_value_fails(self): + """Test validation error when comparing a numeric value against a string range""" + check = RangeCheck(min_value="A", max_value="Z") + context = {"column_name": "category_code"} + result = check.validate(10.5, context) + assert result is not None + assert "Cannot compare Decimal with min_value str" in result.message + + +class TestRangeCheckEdgeCases: + """Test edge cases for RangeCheck""" + + def test_repr(self): + """Test __repr__ output""" + check = RangeCheck(5, 15) + repr_str = repr(check) + assert "RangeCheck" in repr_str + assert "min=5" in repr_str + assert "max=15" in repr_str diff --git a/tests/src/dq_validator/test_regex_check.py b/tests/src/dq_validator/test_regex_check.py new file mode 100644 index 0000000..704758d --- /dev/null +++ b/tests/src/dq_validator/test_regex_check.py @@ -0,0 +1,280 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for RegexCheck +""" + +import pytest +import re +from wxdi.dq_validator.checks.regex_check import RegexCheck +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + +class TestRegexCheckInitialization: + """Test RegexCheck initialization and parameter validation""" + + def test_init_with_only_pattern(self): + """Test successful initialization with only a valid pattern""" + pattern = r"^[A-Z]{3}$" + check = RegexCheck(pattern = r"^[A-Z]{3}$") + assert check.pattern == pattern + assert check.case_sensitive is True + assert isinstance(check._compiled_pattern, re.Pattern) + + def test_init_with_only_case_sensitive_raises_error(self): + """Test failed initialization with only a case_sensitive""" + with pytest.raises(ValueError) as exc_info: + RegexCheck(case_sensitive=True) + assert "pattern must be a non-empty string" in str(exc_info.value) + + def test_init_with_both_parameters(self): + """Test successful initialization with only a valid pattern""" + pattern = r"^[A-Z]{3}$" + check = RegexCheck(pattern = r"^[A-Z]{3}$", case_sensitive = False) + assert check.pattern == pattern + assert check.case_sensitive is False + assert isinstance(check._compiled_pattern, re.Pattern) + + def test_init_with_no_parameter_raises_error(self): + """Test failed initialization with no parameter""" + with pytest.raises(ValueError) as exc_info: + RegexCheck() + assert "pattern must be a non-empty string" in str(exc_info.value) + + def test_init_empty_string_raises_error(self): + """Test that empty string pattern raises ValueError""" + with pytest.raises(ValueError) as exc_info: + RegexCheck(pattern="" , case_sensitive = True) + assert "pattern must be a non-empty string" in str(exc_info.value) + + def test_init_invalid_type_raises_error(self): + """Test that non-string pattern raises ValueError""" + with pytest.raises(ValueError) as exc_info: + RegexCheck(pattern=123) + assert "pattern must be a non-empty string" in str(exc_info.value) + + def test_init_invalid_regex_syntax_raises_error(self): + """Test that syntactically incorrect regex raises ValueError""" + invalid_pattern = "[0-9" # Missing closing bracket + with pytest.raises(ValueError) as exc_info: + RegexCheck(pattern=invalid_pattern) + assert f"Invalid regex pattern '{invalid_pattern}'" in str(exc_info.value) + + def test_get_check_name(self): + """Test get_check_name returns correct name""" + check = RegexCheck(pattern=r"^[A-Z]{3}$") + assert check.get_check_name() == "regex_check" + + def test_get_dimension(self): + """Test get_dimension returns correct dimension""" + check = RegexCheck(pattern=r"^[A-Z]{3}$") + assert check.get_dimension() == DataQualityDimension.VALIDITY + + def test_set_dimension(self): + """Test set_dimension changes the dimension""" + check = RegexCheck(pattern=r"^[A-Z]{3}$") + assert check.get_dimension() == DataQualityDimension.VALIDITY + + check.set_dimension(DataQualityDimension.CONSISTENCY) + assert check.get_dimension() == DataQualityDimension.CONSISTENCY + + +class TestRegexCheckValidation: + """Test validation logic for regex matching""" + + def test_regex_match_passes(self): + """Test that value matching the pattern passes""" + # Pattern for a simple email-like structure + check = RegexCheck(pattern=r"^\w+@\w+\.com$") + context = {'column_name': 'email'} + result = check.validate("user@test.com", context) + assert result is None + + def test_regex_no_match_fails(self): + """Test that value not matching the pattern returns ValidationError""" + check = RegexCheck(pattern=r"^\d{3}-\d{2}$") # Format: 000-00 + context = {'column_name': 'code'} + val = "123-A" + result = check.validate(val, context) + assert result is not None + assert r"code ('123-A') does not match pattern '^\d{3}-\d{2}$'" in result.message + + def test_regex_partial_match_passes(self): + """Test that .search() allows partial matches unless anchored""" + # Pattern is not anchored with ^ or $ + check = RegexCheck(pattern=r"cat") + context = {'column_name': 'sentence'} + result = check.validate("the category", context) + assert result is None + + def test_regex_partial_match_fails(self): + """Test that .search() fails when the pattern is nowhere in the string""" + check = RegexCheck(pattern=r"cat") + context = {'column_name': 'sentence'} + result = check.validate("the dog runs", context) + assert result is not None + assert "sentence ('the dog runs') does not match pattern 'cat'" in result.message + + def test_case_sensitive_match_passes(self): + """Test passes when case_sensitive is True (default)""" + check = RegexCheck(pattern=r"^abc$") + context = {'column_name': 'test'} + result = check.validate("abc", context) + assert result is None + + def test_case_sensitive_match_fails(self): + """Test fails when case_sensitive is True (default)""" + check = RegexCheck(pattern=r"^abc$") + context = {'column_name': 'test'} + result = check.validate("ABC", context) + assert result is not None + assert "test ('ABC') does not match pattern '^abc$'" in result.message + + def test_case_insensitive_match_passes(self): + """Test passes when case_sensitive is False""" + check = RegexCheck(pattern=r"^abc$", case_sensitive=False) + context = {'column_name': 'test'} + result = check.validate("abc", context) + assert result is None + result = check.validate("ABC", context) + assert result is None + result = check.validate("aBc", context) + assert result is None + + def test_case_insensitive_match_fails(self): + """Test fails when case_sensitive is False""" + check = RegexCheck(pattern=r"^abc$", case_sensitive=False) + context = {'column_name': 'test'} + result = check.validate("XabcY", context) + assert result is not None + assert "test ('XabcY') does not match pattern '^abc$'" in result.message + + def test_empty_string_passes(self): + """Test that an empty string passes if the pattern allows it (e.g., zero or more)""" + check = RegexCheck(pattern=r"^\d*$") + context = {'column_name': 'optional_id'} + result = check.validate("", context) + assert result is None + + def test_empty_string_fails(self): + """Test that an empty string fails if the pattern requires characters""" + check = RegexCheck(pattern=r"^\d+$") + context = {'column_name': 'required_id'} + result = check.validate("", context) + assert result is not None + assert "does not match pattern" in result.message + + def test_whitespace_string_passes(self): + """Test that whitespace passes if the pattern allows it (Note: .strip() makes this '')""" + check = RegexCheck(pattern=r"^$") + context = {'column_name': 'blank_space'} + result = check.validate(" ", context) + assert result is None + + def test_whitespace_string_fails(self): + """Test that whitespace fails if the pattern expects actual content""" + check = RegexCheck(pattern=r"^\w+$") + context = {'column_name': 'username'} + result = check.validate(" ", context) + assert result is not None + assert "('') does not match pattern" in result.message + + def test_regex_none_value_fails(self): + """Test that None value matching the pattern fails""" + check = RegexCheck(pattern=r"^\w+@\w+\.com$") + context = {'column_name': 'email'} + result = check.validate(None, context) + assert result is not None + assert "email is None, cannot perform regex check" in result.message + + +class TestRegexCheckValueDataTypes: + """Test validation of different data types for the value parameter""" + + def test_validate_integer_passes(self): + """Matches when the integer string representation fits the pattern""" + check = RegexCheck(pattern=r"^\d{3}$") + result = check.validate(123, {'column_name': 'id'}) + assert result is None + + def test_validate_integer_fails(self): + """Fails when the integer string representation doesn't fit (too many digits)""" + check = RegexCheck(pattern=r"^\d{2}$") + result = check.validate(123, {'column_name': 'id'}) + assert result is not None + assert "does not match pattern" in result.message + + def test_validate_float_passes(self): + """Matches a decimal string representation""" + check = RegexCheck(pattern=r"^\d+\.\d+$") + result = check.validate(99.99, {'column_name': 'price'}) + assert result is None + + def test_validate_float_fails(self): + """Fails when float contains unexpected characters (like the dot) for a digit-only pattern""" + check = RegexCheck(pattern=r"^\d+$") + result = check.validate(99.9, {'column_name': 'price'}) + assert result is not None + + def test_validate_boolean_passes(self): + """Matches 'True' or 'False' (Python's capitalized string format)""" + check = RegexCheck(pattern=r"^(True|False)$") + result = check.validate(True, {'column_name': 'active'}) + assert result is None + + def test_validate_boolean_fails(self): + check = RegexCheck(pattern=r"^true$", case_sensitive=True) + result = check.validate(True, {'column_name': 'active'}) + assert result is not None + + def test_validate_list_passes(self): + """Matches the literal string representation of a list""" + check = RegexCheck(pattern=r"^\[.*\]$") + result = check.validate([1, 2], {'column_name': 'tags'}) + assert result is None + + def test_validate_list_fails(self): + """Fails when list contents don't match the specific pattern""" + check = RegexCheck(pattern=r"^\[\d\]$") + result = check.validate([1, 2], {'column_name': 'tags'}) + assert result is not None + + +class TestRegexCheckEdgeCases: + """Test edge cases and special regex scenarios""" + + def test_regex_with_special_characters(self): + """Test regex containing special characters and escape sequences""" + check = RegexCheck(pattern=r"^\$ \d+\.\d{2}$") # Format: $ 10.00 + context = {'column_name': 'price'} + result = check.validate("$ 10.99", context) + assert result is None + result = check.validate("10.99", context) + assert result is not None + + def test_validate_none_custom_message(self): + """Verify the specific error message for None inputs""" + check = RegexCheck(pattern=r"^None$") + result = check.validate(None, {'column_name': 'val'}) + assert "is None, cannot perform regex check" in result.message + + + def test_repr(self): + """Test __repr__ output""" + check = RegexCheck(pattern=r"\d+", case_sensitive=False) + repr_str = repr(check) + assert "RegexCheck" in repr_str + assert "pattern='\\d+'" in repr_str + assert "case_sensitive=False" in repr_str \ No newline at end of file diff --git a/tests/src/dq_validator/test_result_consolidator.py b/tests/src/dq_validator/test_result_consolidator.py new file mode 100644 index 0000000..e6d11d9 --- /dev/null +++ b/tests/src/dq_validator/test_result_consolidator.py @@ -0,0 +1,471 @@ +# Copyright 2026 IBM Corporation +# Licensed under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# See the LICENSE file in the project root for license information. + +""" +Tests for ValidationResultConsolidated utility. +""" + +import pytest +from wxdi.dq_validator.result import ValidationResult +from wxdi.dq_validator.result_consolidator import ValidationResultConsolidated +from wxdi.dq_validator.base import ValidationError +from wxdi.dq_validator import Validator, ValidationRule, AssetMetadata, ColumnMetadata +from wxdi.dq_validator.checks import LengthCheck, CompletenessCheck +from wxdi.dq_validator.metadata import DataType + + +@pytest.fixture +def sample_validator(): + """Create a sample validator for testing""" + metadata = AssetMetadata('test_table', [ + ColumnMetadata('email', DataType.STRING), + ColumnMetadata('name', DataType.STRING) + ]) + validator = Validator(metadata) + # Add multiple checks to each column for comprehensive testing + validator.add_rule(ValidationRule('email').add_check(CompletenessCheck()).add_check(LengthCheck(min_length=5))) + validator.add_rule(ValidationRule('name').add_check(CompletenessCheck()).add_check(LengthCheck(min_length=2))) + return validator + + +class TestValidationResultConsolidated: + """Test ValidationResultConsolidated class""" + + def test_init_with_error_storage(self, sample_validator): + """Test initialization with error storage enabled""" + consolidator = ValidationResultConsolidated(sample_validator, store_errors=True) + assert consolidator.store_errors is True + assert consolidator.total_records == 0 + assert consolidator.valid_records == 0 + assert consolidator.invalid_records == 0 + + def test_init_without_error_storage(self, sample_validator): + """Test initialization with error storage disabled""" + consolidator = ValidationResultConsolidated(sample_validator, store_errors=False) + assert consolidator.store_errors is False + assert consolidator.total_records == 0 + + def test_add_valid_result(self, sample_validator): + """Test adding a valid result""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["test@example.com"], 0) + result.total_checks = 2 + result.passed_checks = 2 + result.failed_checks = 0 + + consolidator.add_result(result) + + assert consolidator.total_records == 1 + assert consolidator.valid_records == 1 + assert consolidator.invalid_records == 0 + + def test_add_invalid_result(self, sample_validator): + """Test adding an invalid result""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["invalid"], 0) + result.total_checks = 2 + result.passed_checks = 1 + result.failed_checks = 1 + + error = ValidationError( + column_name="email", + check_name="completeness_check", + message="Invalid email format", + value="invalid", + expected="valid email" + ) + result.add_error(error) + + consolidator.add_result(result) + + assert consolidator.total_records == 1 + assert consolidator.valid_records == 0 + assert consolidator.invalid_records == 1 + + def test_add_multiple_results(self, sample_validator): + """Test adding multiple results""" + consolidator = ValidationResultConsolidated(sample_validator) + + # Valid result + result1 = ValidationResult(["test@example.com"], 0) + result1.total_checks = 2 + result1.passed_checks = 2 + + # Invalid result + result2 = ValidationResult(["invalid"], 1) + result2.total_checks = 2 + result2.passed_checks = 1 + error = ValidationError("email", "completeness_check", "Invalid", "invalid") + result2.add_error(error) + + consolidator.add_results([result1, result2]) + + assert consolidator.total_records == 2 + assert consolidator.valid_records == 1 + assert consolidator.invalid_records == 1 + + def test_get_overall_statistics(self, sample_validator): + """Test getting overall statistics""" + consolidator = ValidationResultConsolidated(sample_validator) + + # Add some results + for i in range(10): + result = ValidationResult([f"test{i}@example.com"], i) + result.total_checks = 2 + if i < 7: # 7 valid, 3 invalid + result.passed_checks = 2 + else: + result.passed_checks = 1 + error = ValidationError("email", "completeness_check", "Invalid", f"test{i}") + result.add_error(error) + consolidator.add_result(result) + + stats = consolidator.get_overall_statistics() + + assert stats['total_records'] == 10 + assert stats['valid_records'] == 7 + assert stats['invalid_records'] == 3 + assert stats['pass_rate'] == 70.0 + assert stats['total_errors'] == 3 + + def test_get_column_statistics_single(self, sample_validator): + """Test getting statistics for a single column""" + consolidator = ValidationResultConsolidated(sample_validator) + + # Add results with errors on 'email' column + for i in range(5): + result = ValidationResult([f"test{i}"], i) + result.total_checks = 2 + result.passed_checks = 1 + error = ValidationError("email", "completeness_check", "Invalid", f"test{i}") + result.add_error(error) + consolidator.add_result(result) + + stats = consolidator.get_column_statistics('email') + + # Email has 2 checks (completeness_check and length_check), so 5 records * 2 checks = 10 total + assert stats['failed'] == 5 # 5 completeness_check failures + assert stats['passed'] == 5 # 5 length_check passes + assert stats['total'] == 10 # 5 records * 2 checks + + def test_get_column_statistics_all(self, sample_validator): + """Test getting statistics for all columns""" + consolidator = ValidationResultConsolidated(sample_validator) + + # Add errors for different columns + result1 = ValidationResult(["test"], 0) + result1.total_checks = 3 + result1.passed_checks = 1 + result1.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + result1.add_error(ValidationError("name", "length_check", "Too short", "a")) + + consolidator.add_result(result1) + + stats = consolidator.get_column_statistics() + + assert 'email' in stats + assert 'name' in stats + assert stats['email']['failed'] == 1 + assert stats['name']['failed'] == 1 + + def test_get_check_statistics_single(self, sample_validator): + """Test getting statistics for a single check type""" + consolidator = ValidationResultConsolidated(sample_validator) + + # Add results with completeness_check errors + for i in range(3): + result = ValidationResult([f"test{i}"], i) + result.total_checks = 2 + result.passed_checks = 1 + error = ValidationError("email", "completeness_check", "Invalid", f"test{i}") + result.add_error(error) + consolidator.add_result(result) + + stats = consolidator.get_check_statistics('completeness_check') + + # completeness_check is on both email and name columns, so 3 records * 2 columns = 6 total + assert stats['failed'] == 3 # 3 email completeness failures + assert stats['passed'] == 3 # 3 name completeness passes + assert stats['total'] == 6 # 3 records * 2 columns + + def test_get_check_statistics_all(self, sample_validator): + """Test getting statistics for all check types""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["test"], 0) + result.total_checks = 3 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + result.add_error(ValidationError("name", "length_check", "Too short", "a")) + + consolidator.add_result(result) + + stats = consolidator.get_check_statistics() + + assert 'completeness_check' in stats + assert 'length_check' in stats + assert stats['completeness_check']['failed'] == 1 + assert stats['length_check']['failed'] == 1 + + def test_get_combined_statistics_both_specified(self, sample_validator): + """Test getting combined statistics with both column and check specified""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["test"], 0) + result.total_checks = 2 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + + consolidator.add_result(result) + + stats = consolidator.get_combined_statistics('email', 'completeness_check') + + assert stats['failed'] == 1 + assert stats['total'] == 1 + + def test_get_combined_statistics_column_only(self, sample_validator): + """Test getting combined statistics with only column specified""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["test"], 0) + result.total_checks = 3 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + result.add_error(ValidationError("email", "length_check", "Too short", "a")) + + consolidator.add_result(result) + + stats = consolidator.get_combined_statistics(column_name='email') + + assert 'completeness_check' in stats + assert 'length_check' in stats + assert stats['completeness_check']['failed'] == 1 + assert stats['length_check']['failed'] == 1 + + def test_get_combined_statistics_check_only(self, sample_validator): + """Test getting combined statistics with only check specified""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["test"], 0) + result.total_checks = 3 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + result.add_error(ValidationError("name", "completeness_check", "Invalid", "123")) + + consolidator.add_result(result) + + stats = consolidator.get_combined_statistics(check_name='completeness_check') + + assert 'email' in stats + assert 'name' in stats + assert stats['email']['failed'] == 1 + assert stats['name']['failed'] == 1 + + def test_get_combined_statistics_all(self, sample_validator): + """Test getting all combined statistics""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["test"], 0) + result.total_checks = 3 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + result.add_error(ValidationError("name", "length_check", "Too short", "a")) + + consolidator.add_result(result) + + stats = consolidator.get_combined_statistics() + + assert 'email' in stats + assert 'name' in stats + assert 'completeness_check' in stats['email'] + assert 'length_check' in stats['name'] + + def test_get_errors_by_column_with_storage(self, sample_validator): + """Test getting errors by column when storage is enabled""" + consolidator = ValidationResultConsolidated(sample_validator, store_errors=True) + + result = ValidationResult(["test"], 0) + result.total_checks = 2 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + + consolidator.add_result(result) + + errors = consolidator.get_errors_by_column('email') + + assert len(errors) == 1 + assert errors[0]['column'] == 'email' + assert errors[0]['check'] == 'completeness_check' + assert errors[0]['record_index'] == 0 + + def test_get_errors_by_column_without_storage(self, sample_validator): + """Test getting errors by column when storage is disabled""" + consolidator = ValidationResultConsolidated(sample_validator, store_errors=False) + + result = ValidationResult(["test"], 0) + result.total_checks = 2 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + + consolidator.add_result(result) + + with pytest.raises(RuntimeError, match="Error details not available"): + consolidator.get_errors_by_column('email') + + def test_get_errors_by_check_with_storage(self, sample_validator): + """Test getting errors by check when storage is enabled""" + consolidator = ValidationResultConsolidated(sample_validator, store_errors=True) + + result = ValidationResult(["test"], 0) + result.total_checks = 2 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + + consolidator.add_result(result) + + errors = consolidator.get_errors_by_check('completeness_check') + + assert len(errors) == 1 + assert errors[0]['check'] == 'completeness_check' + + def test_get_errors_by_check_without_storage(self, sample_validator): + """Test getting errors by check when storage is disabled""" + consolidator = ValidationResultConsolidated(sample_validator, store_errors=False) + + with pytest.raises(RuntimeError, match="Error details not available"): + consolidator.get_errors_by_check('completeness_check') + + def test_get_errors_by_column_and_check(self, sample_validator): + """Test getting errors by column and check combination""" + consolidator = ValidationResultConsolidated(sample_validator, store_errors=True) + + result = ValidationResult(["test"], 0) + result.total_checks = 3 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + result.add_error(ValidationError("email", "length_check", "Too short", "a")) + + consolidator.add_result(result) + + errors = consolidator.get_errors_by_column_and_check('email', 'completeness_check') + + assert len(errors) == 1 + assert errors[0]['column'] == 'email' + assert errors[0]['check'] == 'completeness_check' + + def test_get_all_errors_with_storage(self, sample_validator): + """Test getting all errors when storage is enabled""" + consolidator = ValidationResultConsolidated(sample_validator, store_errors=True) + + result = ValidationResult(["test"], 0) + result.total_checks = 3 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + result.add_error(ValidationError("name", "length_check", "Too short", "a")) + + consolidator.add_result(result) + + errors = consolidator.get_all_errors() + + assert len(errors) == 2 + + def test_get_all_errors_without_storage(self, sample_validator): + """Test getting all errors when storage is disabled""" + consolidator = ValidationResultConsolidated(sample_validator, store_errors=False) + + with pytest.raises(RuntimeError, match="Error details not available"): + consolidator.get_all_errors() + + def test_get_columns(self, sample_validator): + """Test getting list of validated columns""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["test"], 0) + result.total_checks = 3 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + result.add_error(ValidationError("name", "length_check", "Too short", "a")) + + consolidator.add_result(result) + + columns = consolidator.get_columns() + + assert 'email' in columns + assert 'name' in columns + assert len(columns) == 2 + + def test_get_checks(self, sample_validator): + """Test getting list of executed checks""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["test"], 0) + result.total_checks = 3 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + result.add_error(ValidationError("name", "length_check", "Too short", "a")) + + consolidator.add_result(result) + + checks = consolidator.get_checks() + + assert 'completeness_check' in checks + assert 'length_check' in checks + assert len(checks) == 2 + + def test_to_dict(self, sample_validator): + """Test converting consolidator to dictionary""" + consolidator = ValidationResultConsolidated(sample_validator) + + result = ValidationResult(["test"], 0) + result.total_checks = 2 + result.passed_checks = 1 + result.add_error(ValidationError("email", "completeness_check", "Invalid", "test")) + + consolidator.add_result(result) + + data = consolidator.to_dict() + + assert 'overall' in data + assert 'by_column' in data + assert 'by_check' in data + assert 'combined' in data + assert 'columns' in data + assert 'checks' in data + assert 'error_count' in data + + def test_repr(self, sample_validator): + """Test string representation""" + consolidator = ValidationResultConsolidated(sample_validator) + + result1 = ValidationResult(["test"], 0) + result1.total_checks = 2 + result1.passed_checks = 2 + + result2 = ValidationResult(["invalid"], 1) + result2.total_checks = 2 + result2.passed_checks = 1 + result2.add_error(ValidationError("email", "completeness_check", "Invalid", "invalid")) + + consolidator.add_results([result1, result2]) + + repr_str = repr(consolidator) + + assert 'ValidationResultConsolidated' in repr_str + assert 'records=2' in repr_str + assert 'valid=1' in repr_str + assert 'invalid=1' in repr_str + assert 'errors=1' in repr_str \ No newline at end of file diff --git a/tests/src/dq_validator/test_rule_loader.py b/tests/src/dq_validator/test_rule_loader.py new file mode 100644 index 0000000..94ab8e2 --- /dev/null +++ b/tests/src/dq_validator/test_rule_loader.py @@ -0,0 +1,559 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Test suite for RuleLoader module +""" + +import pytest +import json +from pathlib import Path +from wxdi.dq_validator.rule_loader import RuleLoader +from wxdi.dq_validator.metadata import AssetMetadata, ColumnMetadata, DataType +from wxdi.dq_validator.validator import Validator +from wxdi.dq_validator.provider.response_model import ( + GlossaryTerm, + Metadata, + GlossaryTermEntity, + ExtendedAttributeGroups, +) +from wxdi.dq_validator.provider.constraint_model import ( + DataQualityConstraint, + ConstraintMetadata, + CheckConstraint, + CheckType, +) +from wxdi.dq_validator.provider.data_asset_model import DataAsset +from datetime import datetime + + +class TestRuleLoader: + """Test cases for RuleLoader class""" + + @pytest.fixture + def base_url(self): + """Base URL for API""" + return "https://api.example.com" + + @pytest.fixture + def auth_token(self): + """Authentication token""" + return "test-token-12345" + + @pytest.fixture + def rule_loader(self, base_url, auth_token): + """Create RuleLoader instance""" + return RuleLoader(base_url, auth_token) + + @pytest.fixture + def asset_metadata(self): + """Create sample asset metadata""" + columns = [ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("name", DataType.STRING, length=100), + ColumnMetadata("age", DataType.INTEGER), + ] + return AssetMetadata("users", columns) + + @pytest.fixture + def mock_glossary_term_basic(self): + """Create a basic glossary term without DQ constraints""" + return GlossaryTerm( + metadata=Metadata( + artifact_type="glossary_term", + artifact_id="term-123", + version_id="version-456", + source_repository_id="repo-789", + global_id="global-123", + effective_start_date=datetime(2026, 1, 1), + created_by="user1", + created_at=datetime(2026, 1, 1), + modified_by="user1", + modified_at=datetime(2026, 1, 1), + revision="0", + state="PUBLISHED", + ), + entity=GlossaryTermEntity( + abbreviations=[], + state="PUBLISHED", + default_locale_id="en-US", + reference_copy=False, + steward_ids=[], + steward_group_ids=[], + custom_relationships=[], + custom_attributes=[], + ), + ) + + @pytest.fixture + def mock_glossary_term_with_constraints(self): + """Create a glossary term with DQ constraints""" + # Create completeness constraint + completeness_constraint = DataQualityConstraint( + metadata=ConstraintMetadata(type=CheckType.COMPLETENESS), + origin=[], + check=[CheckConstraint(name="missing_values_allowed", boolean_value=False)], + ) + + # Create range constraint + range_constraint = DataQualityConstraint( + metadata=ConstraintMetadata(type=CheckType.RANGE), + origin=[], + check=[ + CheckConstraint(name="min", numeric_value=0), + CheckConstraint(name="max", numeric_value=120), + ], + ) + + # Create data type constraint + datatype_constraint = DataQualityConstraint( + metadata=ConstraintMetadata(type=CheckType.DATA_TYPE), + origin=[], + check=[ + CheckConstraint(name="data_type", value="integer"), + CheckConstraint(name="length", numeric_value=10), + ], + ) + + return GlossaryTerm( + metadata=Metadata( + artifact_type="glossary_term", + artifact_id="term-123", + version_id="version-456", + source_repository_id="repo-789", + global_id="global-123", + effective_start_date=datetime(2026, 1, 1), + created_by="user1", + created_at=datetime(2026, 1, 1), + modified_by="user1", + modified_at=datetime(2026, 1, 1), + revision="0", + state="PUBLISHED", + ), + entity=GlossaryTermEntity( + abbreviations=[], + state="PUBLISHED", + default_locale_id="en-US", + reference_copy=False, + steward_ids=[], + steward_group_ids=[], + custom_relationships=[], + custom_attributes=[], + extended_attribute_groups=ExtendedAttributeGroups( + dq_constraints=[ + completeness_constraint, + range_constraint, + datatype_constraint, + ] + ), + ), + ) + + def test_rule_loader_initialization(self, base_url, auth_token): + """Test RuleLoader initialization""" + loader = RuleLoader(base_url, auth_token) + assert loader.base_url == base_url + assert loader.auth_token == auth_token + + def test_load_from_glossary_no_version_id( + self, rule_loader, asset_metadata, mocker + ): + """Test load_from_glossary raises ValueError when no version_id""" + # Mock the provider to return a term without version_id + mock_term = mocker.Mock() + mock_term.metadata.version_id = None + + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.GlossaryProvider") + mock_provider.return_value.get_published_artifact_by_id.return_value = mock_term + + with pytest.raises(ValueError, match="No version_id found"): + rule_loader.load_from_glossary("term-123", "age", asset_metadata) + + def test_load_from_glossary_no_constraints( + self, rule_loader, asset_metadata, mock_glossary_term_basic, mocker + ): + """Test load_from_glossary with no DQ constraints""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.GlossaryProvider") + mock_provider.return_value.get_published_artifact_by_id.return_value = ( + mock_glossary_term_basic + ) + mock_provider.return_value.get_term_by_version_id.return_value = ( + mock_glossary_term_basic + ) + + validator = rule_loader.load_from_glossary("term-123", "age", asset_metadata) + + assert isinstance(validator, Validator) + assert validator.metadata == asset_metadata + assert len(validator.rules) == 0 + + def test_load_from_glossary_with_constraints( + self, + rule_loader, + asset_metadata, + mock_glossary_term_basic, + mock_glossary_term_with_constraints, + mocker, + ): + """Test load_from_glossary with DQ constraints""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.GlossaryProvider") + mock_provider.return_value.get_published_artifact_by_id.return_value = ( + mock_glossary_term_basic + ) + mock_provider.return_value.get_term_by_version_id.return_value = ( + mock_glossary_term_with_constraints + ) + + validator = rule_loader.load_from_glossary("term-123", "age", asset_metadata) + + assert isinstance(validator, Validator) + assert validator.metadata == asset_metadata + assert len(validator.rules) == 1 + + rule = validator.rules[0] + assert rule.column_name == "age" + assert len(rule.checks) == 3 + + def test_load_from_glossary_provider_config( + self, rule_loader, asset_metadata, mock_glossary_term_basic, mocker + ): + """Test that GlossaryProvider is initialized with correct config""" + mock_provider_class = mocker.patch("wxdi.dq_validator.rule_loader.GlossaryProvider") + mock_provider_config = mocker.patch("wxdi.dq_validator.rule_loader.ProviderConfig") + + mock_provider_instance = mocker.Mock() + mock_provider_instance.get_published_artifact_by_id.return_value = ( + mock_glossary_term_basic + ) + mock_provider_instance.get_term_by_version_id.return_value = ( + mock_glossary_term_basic + ) + mock_provider_class.return_value = mock_provider_instance + + rule_loader.load_from_glossary("term-123", "age", asset_metadata) + + # Verify ProviderConfig was called with correct parameters + mock_provider_config.assert_called_once_with( + rule_loader.base_url, rule_loader.auth_token + ) + + # Verify GlossaryProvider was initialized with the config + mock_provider_class.assert_called_once() + + def test_load_from_glossary_api_calls( + self, + rule_loader, + asset_metadata, + mock_glossary_term_basic, + mock_glossary_term_with_constraints, + mocker, + ): + """Test that correct API calls are made""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.GlossaryProvider") + mock_instance = mock_provider.return_value + + mock_instance.get_published_artifact_by_id.return_value = ( + mock_glossary_term_basic + ) + mock_instance.get_term_by_version_id.return_value = ( + mock_glossary_term_with_constraints + ) + + rule_loader.load_from_glossary("term-123", "age", asset_metadata) + + # Verify get_published_artifact_by_id was called + mock_instance.get_published_artifact_by_id.assert_called_once_with("term-123") + + # Verify get_term_by_version_id was called with correct parameters + mock_instance.get_term_by_version_id.assert_called_once_with( + "term-123", + "version-456", + {"included_extended_attribute_groups": "dq_constraints"}, + ) + + def test_load_from_glossary_constraint_filtering( + self, + rule_loader, + asset_metadata, + mock_glossary_term_basic, + mocker, + ): + """Test that None checks are filtered out""" + # Create a constraint with unsupported type that returns None from to_check() + unsupported_constraint = DataQualityConstraint( + metadata=ConstraintMetadata(type=CheckType.UNIQUENESS), # Not supported + origin=[], + check=[], + ) + + mock_term_with_none = GlossaryTerm( + metadata=mock_glossary_term_basic.metadata, + entity=GlossaryTermEntity( + abbreviations=[], + state="PUBLISHED", + default_locale_id="en-US", + reference_copy=False, + steward_ids=[], + steward_group_ids=[], + custom_relationships=[], + custom_attributes=[], + extended_attribute_groups=ExtendedAttributeGroups( + dq_constraints=[unsupported_constraint] + ), + ), + ) + + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.GlossaryProvider") + mock_provider.return_value.get_published_artifact_by_id.return_value = ( + mock_glossary_term_basic + ) + mock_provider.return_value.get_term_by_version_id.return_value = ( + mock_term_with_none + ) + + validator = rule_loader.load_from_glossary("term-123", "age", asset_metadata) + + # Should have no rules since the constraint returned None + assert len(validator.rules) == 0 + + +class TestRuleLoaderCams: + """Test cases for RuleLoader.load_from_cams method""" + + @pytest.fixture + def base_url(self): + """Base URL for API""" + return "https://api.example.com" + + @pytest.fixture + def auth_token(self): + """Authentication token""" + return "test-token-12345" + + @pytest.fixture + def project_id(self): + """Project ID""" + return "72d21c1d-499b-4784-a3c7-6f84507f9a20" + + @pytest.fixture + def rule_loader(self, base_url, auth_token): + """Create RuleLoader instance""" + return RuleLoader(base_url, auth_token) + + @pytest.fixture + def data_asset_json(self): + """Load the data asset response JSON""" + json_path = Path(__file__).parent.parent.parent / "data" / "data_asset_response.json" + with open(json_path, "r") as f: + return json.load(f) + + @pytest.fixture + def mock_data_asset(self, data_asset_json): + """Create DataAsset from JSON""" + return DataAsset.from_dict(data_asset_json) + + def test_load_from_cams_with_auto_metadata( + self, rule_loader, project_id, mock_data_asset, mocker + ): + """Test load_from_cams with automatic metadata extraction""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.CamsProvider") + mock_provider.return_value.get_asset_by_id.return_value = mock_data_asset + + validator = rule_loader.load_from_cams( + "6862f3ba-81f5-4122-8286-62bb4c5d6543", project_id + ) + + assert isinstance(validator, Validator) + assert validator.metadata.table_name == "DEPARTMENT" + assert len(validator.metadata.columns) == 5 + + # Check column names + column_names = [col.name for col in validator.metadata.columns] + assert "DEPTNO" in column_names + assert "DEPTNAME" in column_names + assert "MGRNO" in column_names + assert "ADMRDEPT" in column_names + assert "LOCATION" in column_names + + def test_load_from_cams_with_provided_metadata( + self, rule_loader, project_id, mock_data_asset, mocker + ): + """Test load_from_cams with user-provided metadata""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.CamsProvider") + mock_provider.return_value.get_asset_by_id.return_value = mock_data_asset + + # Provide custom metadata + custom_metadata = AssetMetadata( + "CUSTOM_TABLE", + [ + ColumnMetadata("DEPTNO", DataType.STRING, length=3), + ColumnMetadata("DEPTNAME", DataType.STRING, length=36), + ], + ) + + validator = rule_loader.load_from_cams( + "6862f3ba-81f5-4122-8286-62bb4c5d6543", project_id, custom_metadata + ) + + assert isinstance(validator, Validator) + assert validator.metadata.table_name == "CUSTOM_TABLE" + assert len(validator.metadata.columns) == 2 + + def test_load_from_cams_rules_created( + self, rule_loader, project_id, mock_data_asset, mocker + ): + """Test that validation rules are created from column checks""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.CamsProvider") + mock_provider.return_value.get_asset_by_id.return_value = mock_data_asset + + validator = rule_loader.load_from_cams( + "6862f3ba-81f5-4122-8286-62bb4c5d6543", project_id + ) + + # Should have rules for columns with checks + # DEPTNO has 4 checks, MGRNO has 3, ADMRDEPT has 3, DEPTNAME has 3 + # LOCATION has 0 checks + assert len(validator.rules) == 4 + + # Check that rules are for the correct columns + rule_columns = [rule.column_name for rule in validator.rules] + assert "DEPTNO" in rule_columns + assert "MGRNO" in rule_columns + assert "ADMRDEPT" in rule_columns + assert "DEPTNAME" in rule_columns + assert "LOCATION" not in rule_columns # No checks for this column + + def test_load_from_cams_check_types( + self, rule_loader, project_id, mock_data_asset, mocker + ): + """Test that different check types are correctly loaded""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.CamsProvider") + mock_provider.return_value.get_asset_by_id.return_value = mock_data_asset + + validator = rule_loader.load_from_cams( + "6862f3ba-81f5-4122-8286-62bb4c5d6543", project_id + ) + + # Find DEPTNO rule (has format, uniqueness, completeness, data_class checks) + deptno_rule = next(r for r in validator.rules if r.column_name == "DEPTNO") + # Note: uniqueness and data_class checks return None, so only 2 checks should be added + assert len(deptno_rule.checks) == 2 + + # Find DEPTNAME rule (has uniqueness, completeness, case checks) + deptname_rule = next( + r for r in validator.rules if r.column_name == "DEPTNAME" + ) + # Note: uniqueness check returns None, so only 2 checks should be added + assert len(deptname_rule.checks) == 2 + + def test_load_from_cams_provider_config( + self, rule_loader, project_id, mock_data_asset, mocker + ): + """Test that CamsProvider is initialized with correct config""" + mock_provider_class = mocker.patch("wxdi.dq_validator.rule_loader.CamsProvider") + mock_provider_config = mocker.patch("wxdi.dq_validator.rule_loader.ProviderConfig") + + mock_provider_instance = mocker.Mock() + mock_provider_instance.get_asset_by_id.return_value = mock_data_asset + mock_provider_class.return_value = mock_provider_instance + + rule_loader.load_from_cams( + "6862f3ba-81f5-4122-8286-62bb4c5d6543", project_id + ) + + # Verify ProviderConfig was called with correct parameters + mock_provider_config.assert_called_once_with( + rule_loader.base_url, rule_loader.auth_token, project_id=project_id + ) + + def test_load_from_cams_column_metadata_extraction( + self, rule_loader, project_id, mock_data_asset, mocker + ): + """Test that column metadata is correctly extracted from data asset""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.CamsProvider") + mock_provider.return_value.get_asset_by_id.return_value = mock_data_asset + + validator = rule_loader.load_from_cams( + "6862f3ba-81f5-4122-8286-62bb4c5d6543", project_id + ) + + # Check DEPTNO column metadata (uses inferred type) + deptno_col = validator.metadata.get_column("DEPTNO") + assert deptno_col.name == "DEPTNO" + assert deptno_col.data_type == DataType.STRING + assert deptno_col.length == 3 + assert deptno_col.nullable is False + + # Check MGRNO column metadata (uses inferred type INT8 -> INTEGER) + mgrno_col = validator.metadata.get_column("MGRNO") + assert mgrno_col.name == "MGRNO" + assert mgrno_col.data_type == DataType.INTEGER + assert mgrno_col.length == 6 + assert mgrno_col.precision == 3 + assert mgrno_col.nullable is True + + def test_load_from_cams_type_conversion( + self, rule_loader, project_id, mock_data_asset, mocker + ): + """Test CAMS type to DataType conversion""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.CamsProvider") + mock_provider.return_value.get_asset_by_id.return_value = mock_data_asset + + validator = rule_loader.load_from_cams( + "6862f3ba-81f5-4122-8286-62bb4c5d6543", project_id + ) + + # Check various type conversions + # STRING type (from inferred type) + deptno_col = validator.metadata.get_column("DEPTNO") + assert deptno_col.data_type == DataType.STRING + + # INTEGER type (from INT8 inferred type) + mgrno_col = validator.metadata.get_column("MGRNO") + assert mgrno_col.data_type == DataType.INTEGER + + def test_load_from_cams_skips_unsupported_checks( + self, rule_loader, project_id, mock_data_asset, mocker + ): + """Test that unsupported check types are skipped""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.CamsProvider") + mock_provider.return_value.get_asset_by_id.return_value = mock_data_asset + + validator = rule_loader.load_from_cams( + "6862f3ba-81f5-4122-8286-62bb4c5d6543", project_id + ) + + # DEPTNO has uniqueness and data_class checks which return None + # So it should have 2 checks instead of 4 + deptno_rule = next(r for r in validator.rules if r.column_name == "DEPTNO") + assert len(deptno_rule.checks) == 2 + + def test_load_from_cams_column_without_checks( + self, rule_loader, project_id, mock_data_asset, mocker + ): + """Test that columns without checks don't create rules""" + mock_provider = mocker.patch("wxdi.dq_validator.rule_loader.CamsProvider") + mock_provider.return_value.get_asset_by_id.return_value = mock_data_asset + + validator = rule_loader.load_from_cams( + "6862f3ba-81f5-4122-8286-62bb4c5d6543", project_id + ) + + # LOCATION has no checks, so no rule should be created + location_rules = [r for r in validator.rules if r.column_name == "LOCATION"] + assert len(location_rules) == 0 + + +# Made with Bob diff --git a/tests/src/dq_validator/test_spark_validator.py b/tests/src/dq_validator/test_spark_validator.py new file mode 100644 index 0000000..012790a --- /dev/null +++ b/tests/src/dq_validator/test_spark_validator.py @@ -0,0 +1,805 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for SparkValidator + +Tests the PySpark DataFrame integration including: +- Initialization and configuration +- Summary statistics calculation (distributed) +- Validation column addition (UDF-based) +- Invalid/valid row filtering +- Column expansion +- Validation report writing +- Error sampling +- Distributed processing +""" + +import math +import pytest +import sys +from typing import Dict, Any +import tempfile +import shutil + +# Check if pyspark is available +try: + from pyspark.sql import SparkSession + from pyspark.sql.types import StructType, StructField, IntegerType, StringType + + PYSPARK_AVAILABLE = True +except ImportError: + PYSPARK_AVAILABLE = False + +from wxdi.dq_validator import ( + AssetMetadata, + ColumnMetadata, + DataType, + Validator, + ValidationRule, +) +from wxdi.dq_validator.checks import ( + LengthCheck, + ValidValuesCheck, + ComparisonCheck, + ComparisonOperator, +) + +if PYSPARK_AVAILABLE: + from wxdi.dq_validator.integrations import SparkValidator + + +# Skip all tests if pyspark is not available +pytestmark = pytest.mark.skipif(not PYSPARK_AVAILABLE, reason="pyspark not installed") + + +@pytest.fixture(scope="module") +def spark(): + """Create Spark session for testing""" + if not PYSPARK_AVAILABLE: + return None + + # Set Python executable and path for Spark workers + import os + from pathlib import Path + + python_path = sys.executable + os.environ["PYSPARK_PYTHON"] = python_path + os.environ["PYSPARK_DRIVER_PYTHON"] = python_path + + # Add src directory to PYTHONPATH so workers can import dq_validator + src_path = str(Path(__file__).parent.parent / "src") + current_pythonpath = os.environ.get("PYTHONPATH", "") + if current_pythonpath: + os.environ["PYTHONPATH"] = f"{src_path}{os.pathsep}{current_pythonpath}" + else: + os.environ["PYTHONPATH"] = src_path + + spark = ( + SparkSession.builder.appName("DataQualityTests") + .master("local[2]") + .config("spark.sql.shuffle.partitions", "2") + .config("spark.python.worker.reuse", "false") + .config("spark.sql.execution.pyspark.udf.faulthandler.enabled", "true") + .config("spark.python.worker.faulthandler.enabled", "true") + .getOrCreate() + ) + + yield spark + + spark.stop() + + +@pytest.fixture +def sample_metadata(): + """Create sample metadata for testing""" + return AssetMetadata( + table_name="test_table", + columns=[ + ColumnMetadata("id", DataType.INTEGER), + ColumnMetadata("name", DataType.STRING, length=50), + ColumnMetadata("age", DataType.INTEGER), + ColumnMetadata("status", DataType.STRING, length=20), + ], + ) + + +@pytest.fixture +def sample_validator(sample_metadata): + """Create sample validator with rules""" + validator = Validator(sample_metadata) + + # Add validation rules + validator.add_rule( + ValidationRule("name").add_check(LengthCheck(min_length=2, max_length=50)) + ) + validator.add_rule( + ValidationRule("age").add_check( + ComparisonCheck( + operator=ComparisonOperator.GREATER_THAN_OR_EQUAL, target_value=18 + ) + ) + ) + validator.add_rule( + ValidationRule("status").add_check( + ValidValuesCheck(["active", "inactive"], case_sensitive=False) + ) + ) + + return validator + + +@pytest.fixture +def sample_dataframe(spark): + """Create sample Spark DataFrame for testing""" + if not PYSPARK_AVAILABLE: + return None + + data = [ + (1, "Alice", 25, "active"), + (2, "B", 30, "ACTIVE"), + (3, "Charlie", 17, "inactive"), + (4, "David", 35, "pending"), + (5, "Eve", 40, "Active"), + ] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + return spark.createDataFrame(data, schema) + + +class TestSparkValidatorInitialization: + """Test SparkValidator initialization""" + + def test_basic_initialization(self, sample_validator): + """Test basic initialization with default parameters""" + spark_validator = SparkValidator(sample_validator) + + assert spark_validator.validator == sample_validator + assert spark_validator.column_prefix == "dq_" + + def test_custom_column_prefix(self, sample_validator): + """Test initialization with custom column prefix""" + spark_validator = SparkValidator(sample_validator, column_prefix="validation_") + + assert spark_validator.column_prefix == "validation_" + + def test_string_representation(self, sample_validator): + """Test string representation of validator""" + spark_validator = SparkValidator(sample_validator) + str_repr = str(spark_validator) + + assert "SparkValidator" in str_repr + + +class TestSummaryStatistics: + """Test summary statistics calculation with distributed aggregation""" + + def test_basic_summary(self, sample_validator, sample_dataframe): + """Test basic summary statistics""" + spark_validator = SparkValidator(sample_validator) + summary = spark_validator.get_summary_statistics(sample_dataframe) + + assert isinstance(summary, dict) + assert "total_rows" in summary + assert "valid_rows" in summary + assert "invalid_rows" in summary + assert "pass_rate" in summary + assert "total_checks" in summary + assert "passed_checks" in summary + assert "failed_checks" in summary + + assert summary["total_rows"] == 5 + assert summary["valid_rows"] + summary["invalid_rows"] == 5 + assert 0 <= summary["pass_rate"] <= 100 + + def test_all_valid_rows(self, spark, sample_validator): + """Test summary with all valid rows""" + data = [ + (1, "Alice", 25, "active"), + (2, "Bob", 30, "inactive"), + (3, "Charlie", 35, "active"), + ] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + summary = spark_validator.get_summary_statistics(df) + + assert summary["total_rows"] == 3 + assert summary["valid_rows"] == 3 + assert summary["invalid_rows"] == 0 + assert math.isclose(summary["pass_rate"], 100.0) + + def test_all_invalid_rows(self, spark, sample_validator): + """Test summary with all invalid rows""" + data = [ + (1, "A", 15, "pending"), + (2, "B", 16, "deleted"), + (3, "C", 17, "archived"), + ] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + summary = spark_validator.get_summary_statistics(df) + + assert summary["total_rows"] == 3 + assert summary["valid_rows"] == 0 + assert summary["invalid_rows"] == 3 + assert math.isclose(summary["pass_rate"], 0.0, abs_tol=1e-9) + + def test_empty_dataframe(self, spark, sample_validator): + """Test summary with empty DataFrame""" + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + df = spark.createDataFrame([], schema) + + spark_validator = SparkValidator(sample_validator) + summary = spark_validator.get_summary_statistics(df) + + assert summary["total_rows"] == 0 + assert summary["valid_rows"] == 0 + assert summary["invalid_rows"] == 0 + assert math.isclose(summary["pass_rate"], 0.0, abs_tol=1e-9) + + def test_large_dataframe(self, spark, sample_validator): + """Test summary with large DataFrame (distributed processing)""" + # Create large dataset + data = [(i, "Alice", 25, "active") for i in range(1, 1001)] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + summary = spark_validator.get_summary_statistics(df) + + assert summary["total_rows"] == 1000 + assert summary["valid_rows"] == 1000 + + +class TestAddValidationColumn: + """Test adding validation column using Spark UDF""" + + def test_basic_validation_column(self, sample_validator, sample_dataframe): + """Test adding validation column""" + spark_validator = SparkValidator(sample_validator) + df_validated = spark_validator.add_validation_column(sample_dataframe) + + # Check that original columns are preserved + original_cols = sample_dataframe.columns + assert all(col in df_validated.columns for col in original_cols) + + # Check that validation column is added + assert "dq_validation_result" in df_validated.columns + + # Check that DataFrame has same number of rows + assert df_validated.count() == sample_dataframe.count() + + def test_validation_result_structure(self, sample_validator, sample_dataframe): + """Test structure of validation result""" + spark_validator = SparkValidator(sample_validator) + df_validated = spark_validator.add_validation_column(sample_dataframe) + + # Get schema of validation column + validation_col = df_validated.schema["dq_validation_result"] + + # Check that it's a struct type + assert validation_col.dataType.typeName() == "struct" + + # Check field names + field_names = [f.name for f in validation_col.dataType.fields] + assert "is_valid" in field_names + assert "score" in field_names + assert "pass_rate" in field_names + assert "total_checks" in field_names + assert "passed_checks" in field_names + assert "failed_checks" in field_names + assert "error_count" in field_names + assert "errors" in field_names + + def test_custom_column_prefix(self, sample_validator, sample_dataframe): + """Test validation column with custom prefix""" + spark_validator = SparkValidator(sample_validator, column_prefix="val_") + df_validated = spark_validator.add_validation_column(sample_dataframe) + + assert "val_validation_result" in df_validated.columns + assert "dq_validation_result" not in df_validated.columns + + def test_column_conflict_detection(self, spark, sample_validator): + """Test detection of column name conflicts""" + data = [ + (1, "Alice", 25, "active", "existing_data"), + (2, "Bob", 30, "inactive", "existing_data"), + ] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + StructField("dq_validation_result", StringType(), True), # Conflict! + ] + ) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + + with pytest.raises(ValueError, match="already exists"): + spark_validator.add_validation_column(df) + + +class TestInvalidRowFiltering: + """Test filtering invalid rows with distributed operations""" + + def test_get_invalid_rows(self, sample_validator, sample_dataframe): + """Test getting invalid rows""" + spark_validator = SparkValidator(sample_validator) + invalid_df = spark_validator.get_invalid_rows(sample_dataframe) + + # Should have validation column + assert "dq_validation_result" in invalid_df.columns + + # Collect and check all rows are invalid + rows = invalid_df.select("dq_validation_result.is_valid").collect() + for row in rows: + assert row["is_valid"] is False + + def test_get_valid_rows(self, sample_validator, sample_dataframe): + """Test getting valid rows""" + spark_validator = SparkValidator(sample_validator) + valid_df = spark_validator.get_valid_rows(sample_dataframe) + + # Should have validation column + assert "dq_validation_result" in valid_df.columns + + # Collect and check all rows are valid + rows = valid_df.select("dq_validation_result.is_valid").collect() + for row in rows: + assert row["is_valid"] is True + + def test_no_invalid_rows(self, spark, sample_validator): + """Test when there are no invalid rows""" + data = [ + (1, "Alice", 25, "active"), + (2, "Bob", 30, "inactive"), + (3, "Charlie", 35, "active"), + ] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + invalid_df = spark_validator.get_invalid_rows(df) + + assert invalid_df.count() == 0 + + def test_no_valid_rows(self, spark, sample_validator): + """Test when there are no valid rows""" + data = [ + (1, "A", 15, "pending"), + (2, "B", 16, "deleted"), + (3, "C", 17, "archived"), + ] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + valid_df = spark_validator.get_valid_rows(df) + + assert valid_df.count() == 0 + + +class TestColumnExpansion: + """Test expanding validation column""" + + def test_basic_expansion(self, sample_validator, sample_dataframe): + """Test basic column expansion""" + spark_validator = SparkValidator(sample_validator) + df_validated = spark_validator.add_validation_column(sample_dataframe) + df_expanded = spark_validator.expand_validation_column(df_validated) + + # Check that expanded columns exist + assert "dq_is_valid" in df_expanded.columns + assert "dq_score" in df_expanded.columns + assert "dq_pass_rate" in df_expanded.columns + assert "dq_total_checks" in df_expanded.columns + assert "dq_passed_checks" in df_expanded.columns + assert "dq_failed_checks" in df_expanded.columns + assert "dq_error_count" in df_expanded.columns + assert "dq_errors" in df_expanded.columns + + # Original validation column should be removed + assert "dq_validation_result" not in df_expanded.columns + + # Original columns should be preserved + original_cols = sample_dataframe.columns + assert all(col in df_expanded.columns for col in original_cols) + + def test_expansion_with_custom_prefix(self, sample_validator, sample_dataframe): + """Test expansion with custom prefix""" + spark_validator = SparkValidator(sample_validator, column_prefix="val_") + df_validated = spark_validator.add_validation_column(sample_dataframe) + df_expanded = spark_validator.expand_validation_column(df_validated) + + assert "val_is_valid" in df_expanded.columns + assert "val_score" in df_expanded.columns + assert "val_pass_rate" in df_expanded.columns + + def test_expansion_without_validation_column( + self, sample_validator, sample_dataframe + ): + """Test expansion when validation column doesn't exist""" + spark_validator = SparkValidator(sample_validator) + + with pytest.raises( + ValueError, match="does not contain validation result column" + ): + spark_validator.expand_validation_column(sample_dataframe) + + +class TestValidationReport: + """Test validation report writing""" + + def test_write_parquet_report(self, sample_validator, sample_dataframe): + """Test writing validation report in Parquet format""" + spark_validator = SparkValidator(sample_validator) + + with tempfile.TemporaryDirectory() as tmpdir: + output_path = f"{tmpdir}/validation_report" + + spark_validator.write_validation_report( + sample_dataframe, output_path=output_path, format="parquet" + ) + + # Verify report was written + # Note: In real scenario, you'd read back and verify content + + def test_write_csv_report(self, sample_validator, sample_dataframe): + """Test writing validation report in CSV format""" + spark_validator = SparkValidator(sample_validator) + + with tempfile.TemporaryDirectory() as tmpdir: + output_path = f"{tmpdir}/validation_report" + + spark_validator.write_validation_report( + sample_dataframe, output_path=output_path, format="csv" + ) + + # Verify report was written + + +class TestErrorSampling: + """Test error sampling functionality""" + + def test_get_error_sample(self, sample_validator, sample_dataframe): + """Test getting error samples""" + spark_validator = SparkValidator(sample_validator) + error_samples = spark_validator.get_error_sample(sample_dataframe, limit=10) + + assert isinstance(error_samples, list) + assert len(error_samples) <= 10 + + # Check structure of error samples + if len(error_samples) > 0: + sample = error_samples[0] + assert "row" in sample + assert "errors" in sample + assert isinstance(sample["errors"], list) + + def test_error_sample_limit(self, sample_validator, sample_dataframe): + """Test error sample limit""" + spark_validator = SparkValidator(sample_validator) + + # Get small sample + small_sample = spark_validator.get_error_sample(sample_dataframe, limit=2) + assert len(small_sample) <= 2 + + # Get larger sample + large_sample = spark_validator.get_error_sample(sample_dataframe, limit=100) + assert len(large_sample) <= 100 + + def test_no_errors(self, spark, sample_validator): + """Test error sampling when there are no errors""" + data = [(1, "Alice", 25, "active"), (2, "Bob", 30, "inactive")] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + error_samples = spark_validator.get_error_sample(df, limit=10) + + assert len(error_samples) == 0 + + +class TestEdgeCases: + """Test edge cases and error handling""" + + def test_dataframe_with_null_values(self, spark, sample_validator): + """Test DataFrame with null values""" + data = [ + (1, "Alice", 25, "active"), + (2, None, 30, "inactive"), + (3, "Charlie", None, "active"), + ] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + summary = spark_validator.get_summary_statistics(df) + + # Should handle nulls gracefully + assert summary["total_rows"] == 3 + assert summary["invalid_rows"] > 0 # Nulls should cause failures + + def test_distributed_processing_consistency(self, spark, sample_validator): + """Test that distributed processing produces consistent results""" + # Create dataset that will be distributed across partitions + data = [(i, "Alice", 25, "active") for i in range(1, 101)] + + schema = StructType( + [ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ] + ) + + df = spark.createDataFrame(data, schema).repartition(4) + + spark_validator = SparkValidator(sample_validator) + + # Run validation multiple times + summary1 = spark_validator.get_summary_statistics(df) + summary2 = spark_validator.get_summary_statistics(df) + + # Results should be consistent + assert summary1 == summary2 + + +class TestIntegrationScenarios: + """Test complete integration scenarios""" + + def test_complete_workflow(self, sample_validator, sample_dataframe): + """Test complete validation workflow""" + spark_validator = SparkValidator(sample_validator) + + # Step 1: Get summary + summary = spark_validator.get_summary_statistics(sample_dataframe) + assert summary["total_rows"] == 5 + + # Step 2: Add validation column + df_validated = spark_validator.add_validation_column(sample_dataframe) + assert "dq_validation_result" in df_validated.columns + + # Step 3: Filter invalid rows + invalid_df = spark_validator.get_invalid_rows(sample_dataframe) + assert invalid_df.count() == summary["invalid_rows"] + + # Step 4: Expand columns + df_expanded = spark_validator.expand_validation_column(df_validated) + assert "dq_is_valid" in df_expanded.columns + + # Step 5: Get error samples + error_samples = spark_validator.get_error_sample(sample_dataframe, limit=10) + assert isinstance(error_samples, list) + + def test_lazy_evaluation(self, sample_validator, sample_dataframe): + """Test that operations use lazy evaluation""" + spark_validator = SparkValidator(sample_validator) + + # These operations should not trigger computation + df_validated = spark_validator.add_validation_column(sample_dataframe) + invalid_df = spark_validator.get_invalid_rows(sample_dataframe) + + # Only when we call an action (count, collect, etc.) should computation happen + count = invalid_df.count() + assert isinstance(count, int) + + +class TestValidationRowImpl: + """Test the validate_row_impl function execution and row validation logic""" + + def test_validation_result_content(self, sample_validator, sample_dataframe): + """Test that validation results contain correct data for each row""" + spark_validator = SparkValidator(sample_validator) + df_validated = spark_validator.add_validation_column(sample_dataframe) + + # Collect results to verify validation logic execution + results = df_validated.select("name", "age", "dq_validation_result").collect() + + # Verify we have results for all rows + assert len(results) == 5 + + # Check that validation results have expected structure + for row in results: + validation_result = row["dq_validation_result"] + + # Verify all fields are present + assert "is_valid" in validation_result + assert "score" in validation_result + assert "pass_rate" in validation_result + assert "total_checks" in validation_result + assert "passed_checks" in validation_result + assert "failed_checks" in validation_result + assert "error_count" in validation_result + assert "errors" in validation_result + + # Verify types + assert isinstance(validation_result["is_valid"], bool) + assert isinstance(validation_result["score"], str) # Score is a string like "3/3" + assert isinstance(validation_result["pass_rate"], (int, float)) + assert isinstance(validation_result["total_checks"], int) + assert isinstance(validation_result["passed_checks"], int) + assert isinstance(validation_result["failed_checks"], int) + assert isinstance(validation_result["error_count"], int) + assert isinstance(validation_result["errors"], list) + + # Verify logical consistency + assert validation_result["total_checks"] == ( + validation_result["passed_checks"] + validation_result["failed_checks"] + ) + assert validation_result["error_count"] == len(validation_result["errors"]) + + def test_row_conversion_to_list(self, spark, sample_validator): + """Test that Spark Row is correctly converted to list for validation""" + # Create a simple dataframe with known values + data = [ + (1, "Alice", 25, "active"), + (2, "Bob", 30, "inactive"), + ] + + schema = StructType([ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ]) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + df_validated = spark_validator.add_validation_column(df) + + # Collect and verify that validation was performed on list-converted rows + results = df_validated.collect() + + for row in results: + validation_result = row["dq_validation_result"] + # If validation ran successfully, it means list(row) conversion worked + assert validation_result is not None + assert "is_valid" in validation_result + + # Verify that checks were actually executed (not just returning empty results) + assert validation_result["total_checks"] > 0 + + def test_validation_with_errors(self, spark, sample_validator): + """Test that validation errors are properly captured in JSON format""" + # Create data that will fail validation + data = [ + (1, "A", 15, "invalid_status"), # Short name, young age, invalid status + ] + + schema = StructType([ + StructField("id", IntegerType(), True), + StructField("name", StringType(), True), + StructField("age", IntegerType(), True), + StructField("status", StringType(), True), + ]) + + df = spark.createDataFrame(data, schema) + + spark_validator = SparkValidator(sample_validator) + df_validated = spark_validator.add_validation_column(df) + + # Collect and verify error messages + result = df_validated.collect()[0] + validation_result = result["dq_validation_result"] + + # Should have validation errors + assert validation_result["is_valid"] == False + assert validation_result["error_count"] > 0 + assert len(validation_result["errors"]) > 0 + + # Verify errors are JSON strings + import json + for error_str in validation_result["errors"]: + assert isinstance(error_str, str) + # Should be valid JSON + error_dict = json.loads(error_str) + assert isinstance(error_dict, dict) + # Should have expected error fields + assert "column" in error_dict or "check" in error_dict + + +if __name__ == "__main__": + pytest.main([__file__, "-v"]) diff --git a/tests/src/dq_validator/test_valid_values_check.py b/tests/src/dq_validator/test_valid_values_check.py new file mode 100644 index 0000000..fa911a2 --- /dev/null +++ b/tests/src/dq_validator/test_valid_values_check.py @@ -0,0 +1,394 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for ValidValuesCheck +""" + +import pytest +from wxdi.dq_validator.checks.valid_values_check import ValidValuesCheck +from wxdi.dq_validator.data_quality_dimension import DataQualityDimension + + +class TestValidValuesCheckInitialization: + """Test ValidValuesCheck initialization and parameter validation""" + + def test_init_with_string_values(self): + """Test initialization with string values""" + check = ValidValuesCheck(['active', 'inactive', 'pending']) + assert check.valid_values == ['active', 'inactive', 'pending'] + assert check.case_sensitive == False # Default + + def test_init_with_numeric_values(self): + """Test initialization with numeric values""" + check = ValidValuesCheck([1, 2, 3, 4, 5]) + assert check.valid_values == [1, 2, 3, 4, 5] + + def test_init_case_sensitive_true(self): + """Test initialization with case_sensitive=True""" + check = ValidValuesCheck(['Active', 'Inactive'], case_sensitive=True) + assert check.case_sensitive == True + + def test_init_case_sensitive_false(self): + """Test initialization with case_sensitive=False""" + check = ValidValuesCheck(['active', 'inactive'], case_sensitive=False) + assert check.case_sensitive == False + + def test_init_empty_list_raises_error(self): + """Test that empty list raises ValueError""" + with pytest.raises(ValueError) as exc_info: + ValidValuesCheck([]) + assert "valid_values cannot be empty" in str(exc_info.value) + + def test_init_not_list_raises_error(self): + """Test that non-list raises TypeError""" + with pytest.raises(TypeError) as exc_info: + ValidValuesCheck('not_a_list') # type: ignore[arg-type] + assert "valid_values must be a list" in str(exc_info.value) + + def test_init_with_mixed_types(self): + """Test initialization with mixed type values""" + check = ValidValuesCheck([1, 'text', True]) + assert check.valid_values == [1, 'text', True] + + def test_get_check_name(self): + """Test get_check_name returns correct name""" + check = ValidValuesCheck(['a', 'b']) + assert check.get_check_name() == "valid_values_check" + + def test_get_dimension(self): + """Test get_dimension returns correct dimension""" + check = ValidValuesCheck(['a', 'b']) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + def test_set_dimension(self): + """Test set_dimension changes the dimension""" + check = ValidValuesCheck(['a', 'b']) + assert check.get_dimension() == DataQualityDimension.VALIDITY + + check.set_dimension(DataQualityDimension.CONSISTENCY) + assert check.get_dimension() == DataQualityDimension.CONSISTENCY + + +class TestValidValuesCheckCaseSensitive: + """Test ValidValuesCheck with case_sensitive=True""" + + def test_exact_match_passes(self): + """Test exact match passes with case_sensitive=True""" + check = ValidValuesCheck(['active', 'inactive', 'pending'], case_sensitive=True) + context = {'column_name': 'status'} + result = check.validate('active', context) + assert result is None + + def test_case_mismatch_fails(self): + """Test case mismatch fails with case_sensitive=True""" + check = ValidValuesCheck(['active', 'inactive', 'pending'], case_sensitive=True) + context = {'column_name': 'status'} + result = check.validate('Active', context) + assert result is not None + assert result.column_name == 'status' + assert result.check_name == 'valid_values_check' + assert "has invalid value 'Active'" in result.message + + def test_uppercase_mismatch_fails(self): + """Test uppercase mismatch fails with case_sensitive=True""" + check = ValidValuesCheck(['active', 'inactive'], case_sensitive=True) + context = {'column_name': 'status'} + result = check.validate('ACTIVE', context) + assert result is not None + assert "has invalid value 'ACTIVE'" in result.message + + def test_invalid_value_fails(self): + """Test invalid value fails with case_sensitive=True""" + check = ValidValuesCheck(['active', 'inactive'], case_sensitive=True) + context = {'column_name': 'status'} + result = check.validate('archived', context) + assert result is not None + assert "has invalid value 'archived'" in result.message + + +class TestValidValuesCheckCaseInsensitive: + """Test ValidValuesCheck with case_sensitive=False (default)""" + + def test_exact_match_passes(self): + """Test exact match passes with case_sensitive=False""" + check = ValidValuesCheck(['active', 'inactive', 'pending'], case_sensitive=False) + context = {'column_name': 'status'} + result = check.validate('active', context) + assert result is None + + def test_uppercase_match_passes(self): + """Test uppercase match passes with case_sensitive=False""" + check = ValidValuesCheck(['active', 'inactive', 'pending'], case_sensitive=False) + context = {'column_name': 'status'} + result = check.validate('ACTIVE', context) + assert result is None + + def test_titlecase_match_passes(self): + """Test titlecase match passes with case_sensitive=False""" + check = ValidValuesCheck(['active', 'inactive', 'pending'], case_sensitive=False) + context = {'column_name': 'status'} + result = check.validate('Active', context) + assert result is None + + def test_mixed_case_match_passes(self): + """Test mixed case match passes with case_sensitive=False""" + check = ValidValuesCheck(['active', 'inactive'], case_sensitive=False) + context = {'column_name': 'status'} + result = check.validate('AcTiVe', context) + assert result is None + + def test_invalid_value_fails(self): + """Test invalid value fails with case_sensitive=False""" + check = ValidValuesCheck(['active', 'inactive'], case_sensitive=False) + context = {'column_name': 'status'} + result = check.validate('archived', context) + assert result is not None + assert "has invalid value 'archived' (case-insensitive)" in result.message + + def test_default_is_case_insensitive(self): + """Test default behavior is case-insensitive""" + check = ValidValuesCheck(['active', 'inactive']) + context = {'column_name': 'status'} + result = check.validate('ACTIVE', context) + assert result is None + + +class TestValidValuesCheckNumericValues: + """Test ValidValuesCheck with numeric values""" + + def test_valid_integer_passes(self): + """Test valid integer value passes""" + check = ValidValuesCheck([1, 2, 3, 4, 5]) + context = {'column_name': 'priority'} + result = check.validate(3, context) + assert result is None + + def test_invalid_integer_fails(self): + """Test invalid integer value fails""" + check = ValidValuesCheck([1, 2, 3, 4, 5]) + context = {'column_name': 'priority'} + result = check.validate(6, context) + assert result is not None + assert "has invalid value '6'" in result.message + + def test_valid_float_passes(self): + """Test valid float value passes""" + check = ValidValuesCheck([1.0, 2.5, 3.7]) + context = {'column_name': 'rating'} + result = check.validate(2.5, context) + assert result is None + + def test_type_mismatch_string_vs_int_fails(self): + """Test type mismatch between string and int fails""" + check = ValidValuesCheck([1, 2, 3]) + context = {'column_name': 'priority'} + result = check.validate('3', context) + assert result is not None + assert "has invalid value '3'" in result.message + + def test_type_mismatch_int_vs_float_passes(self): + """Test that int and float with same value are considered equal (Python behavior)""" + check = ValidValuesCheck([1, 2, 3]) + context = {'column_name': 'value'} + result = check.validate(1.0, context) + assert result is None # 1 == 1.0 in Python (numeric equality) + + +class TestValidValuesCheckBooleanValues: + """Test ValidValuesCheck with boolean values""" + + def test_valid_true_passes(self): + """Test valid True value passes""" + check = ValidValuesCheck([True, False]) + context = {'column_name': 'is_active'} + result = check.validate(True, context) + assert result is None + + def test_valid_false_passes(self): + """Test valid False value passes""" + check = ValidValuesCheck([True, False]) + context = {'column_name': 'is_active'} + result = check.validate(False, context) + assert result is None + + def test_boolean_case_insensitive_ignored(self): + """Test case_sensitive is ignored for non-string types""" + check = ValidValuesCheck([True], case_sensitive=False) + context = {'column_name': 'flag'} + result = check.validate(True, context) + assert result is None + + +class TestValidValuesCheckNoneHandling: + """Test ValidValuesCheck with None values""" + + def test_none_value_fails(self): + """Test None value returns error""" + check = ValidValuesCheck(['active', 'inactive']) + context = {'column_name': 'status'} + result = check.validate(None, context) + assert result is not None + assert result.column_name == 'status' + assert "is None" in result.message + assert "expected one of" in result.message + + +class TestValidValuesCheckEmptyString: + """Test ValidValuesCheck with empty strings""" + + def test_empty_string_in_valid_values_passes(self): + """Test empty string passes when in valid values""" + check = ValidValuesCheck(['', 'value1', 'value2']) + context = {'column_name': 'optional_field'} + result = check.validate('', context) + assert result is None + + def test_empty_string_not_in_valid_values_fails(self): + """Test empty string fails when not in valid values""" + check = ValidValuesCheck(['value1', 'value2']) + context = {'column_name': 'required_field'} + result = check.validate('', context) + assert result is not None + assert "has invalid value ''" in result.message + + +class TestValidValuesCheckMixedTypes: + """Test ValidValuesCheck with mixed type values""" + + def test_mixed_types_string_passes(self): + """Test string value passes in mixed type list""" + check = ValidValuesCheck([1, 'text', True]) + context = {'column_name': 'mixed_field'} + result = check.validate('text', context) + assert result is None + + def test_mixed_types_int_passes(self): + """Test int value passes in mixed type list""" + check = ValidValuesCheck([1, 'text', True]) + context = {'column_name': 'mixed_field'} + result = check.validate(1, context) + assert result is None + + def test_mixed_types_bool_passes(self): + """Test bool value passes in mixed type list""" + check = ValidValuesCheck([1, 'text', True]) + context = {'column_name': 'mixed_field'} + result = check.validate(True, context) + assert result is None + + def test_mixed_types_invalid_fails(self): + """Test invalid value fails in mixed type list""" + check = ValidValuesCheck([1, 'text', True]) + context = {'column_name': 'mixed_field'} + result = check.validate(2, context) + assert result is not None + + +class TestValidValuesCheckDuplicates: + """Test ValidValuesCheck with duplicate values""" + + def test_duplicates_in_list_allowed(self): + """Test duplicates in valid_values list are allowed""" + check = ValidValuesCheck(['active', 'active', 'inactive']) + context = {'column_name': 'status'} + result = check.validate('active', context) + assert result is None + + def test_duplicates_case_insensitive(self): + """Test duplicates with different cases in case-insensitive mode""" + check = ValidValuesCheck(['Active', 'active', 'ACTIVE'], case_sensitive=False) + context = {'column_name': 'status'} + result = check.validate('active', context) + assert result is None + + +class TestValidValuesCheckErrorMessages: + """Test ValidValuesCheck error message formatting""" + + def test_error_message_includes_column_name(self): + """Test error message includes column name""" + check = ValidValuesCheck(['a', 'b']) + context = {'column_name': 'test_column'} + result = check.validate('c', context) + assert result is not None + assert 'test_column' in result.message + + def test_error_message_includes_invalid_value(self): + """Test error message includes the invalid value""" + check = ValidValuesCheck(['a', 'b']) + context = {'column_name': 'field'} + result = check.validate('invalid', context) + assert result is not None + assert 'invalid' in result.message + + def test_error_message_includes_valid_values(self): + """Test error message includes list of valid values""" + check = ValidValuesCheck(['a', 'b', 'c']) + context = {'column_name': 'field'} + result = check.validate('d', context) + assert result is not None + assert "['a', 'b', 'c']" in result.message + + def test_error_message_case_insensitive_note(self): + """Test error message includes case-insensitive note for strings""" + check = ValidValuesCheck(['active'], case_sensitive=False) + context = {'column_name': 'status'} + result = check.validate('invalid', context) + assert result is not None + assert "(case-insensitive)" in result.message + + def test_error_message_no_case_note_for_case_sensitive(self): + """Test error message has no case note for case-sensitive""" + check = ValidValuesCheck(['active'], case_sensitive=True) + context = {'column_name': 'status'} + result = check.validate('invalid', context) + assert result is not None + assert "(case-insensitive)" not in result.message + + +class TestValidValuesCheckEdgeCases: + """Test ValidValuesCheck edge cases""" + + def test_single_valid_value(self): + """Test with single valid value""" + check = ValidValuesCheck(['only_value']) + context = {'column_name': 'field'} + result = check.validate('only_value', context) + assert result is None + + def test_large_valid_values_list(self): + """Test with large list of valid values""" + large_list = [f'value_{i}' for i in range(1000)] + check = ValidValuesCheck(large_list) + context = {'column_name': 'field'} + result = check.validate('value_500', context) + assert result is None + + def test_special_characters_in_values(self): + """Test with special characters in values""" + check = ValidValuesCheck(['@#$%', '!@#', 'a-b-c']) + context = {'column_name': 'field'} + result = check.validate('@#$%', context) + assert result is None + + def test_repr(self): + """Test __repr__ method""" + check = ValidValuesCheck(['a', 'b', 'c'], case_sensitive=True) + repr_str = repr(check) + assert "ValidValuesCheck" in repr_str + assert "3" in repr_str # Number of values + assert "True" in repr_str # case_sensitive + diff --git a/tests/src/dq_validator/test_version.py b/tests/src/dq_validator/test_version.py new file mode 100644 index 0000000..678e5a2 --- /dev/null +++ b/tests/src/dq_validator/test_version.py @@ -0,0 +1,35 @@ +""" + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +""" +""" +Unit tests for __version__ +""" +import pytest +from packaging.version import parse, Version + +class TestVersions: + test_version = Version('0.5.0') + """ Tests versions embedded in various components """ + def test_wxdi_version(self): + import wxdi + assert TestVersions.test_version < parse(wxdi.__version__) + + def test_wxdi_dqvalidator_version(self): + import wxdi.dq_validator + assert TestVersions.test_version < parse(wxdi.dq_validator.__version__) + + def test_wxdi_dqvalidator_integrations_version(self): + import wxdi.dq_validator.integrations + assert TestVersions.test_version < parse(wxdi.dq_validator.integrations.__version__) \ No newline at end of file diff --git a/tests/src/integration/README.md b/tests/src/integration/README.md new file mode 100644 index 0000000..752c2c1 --- /dev/null +++ b/tests/src/integration/README.md @@ -0,0 +1,221 @@ + +# Integration Tests + +This directory contains integration tests for the data-intelligence-sdk modules that require external service connections. + +## Overview + +Integration tests verify that the SDK works correctly with actual IBM Cloud services. These tests are skipped by default when the required configuration files are not present. + +## Test Files + +### 1. `test_dph_v1.py` +Tests for IBM Data Product Hub API Service (DPH Services). + +**Requirements:** +- Configuration file: `dph_v1.env` +- Valid IBM Cloud API credentials +- Access to IBM Data Product Hub service + +**Tests:** 49 integration tests covering: +- Container initialization +- Data product CRUD operations +- Draft management +- Release management +- Contract terms and documents +- Domain management + +### 2. `test_odcs_generator_collibra.py` +Tests for ODCS generation from Collibra. + +**Requirements:** +- Collibra API credentials +- Sample Collibra data assets + +### 3. `test_odcs_generator_informatica.py` +Tests for ODCS generation from Informatica. + +**Requirements:** +- Informatica API credentials +- Sample Informatica data assets + +### 4. `test_data_product_recommender_integration.py` +Tests for data product recommendation engine. + +**Requirements:** +- Sample query log data files +- Database connection (if testing with live data) + +## Configuration Setup + +### DPH Services Configuration (`dph_v1.env`) + +The `test_dph_v1.py` integration tests require a configuration file named `dph_v1.env` in this directory. + +#### Step 1: Copy the Template + +A template file `dph_v1.env` has been created in this directory. Edit it with your credentials. + +#### Step 2: Get IBM Cloud API Key + +1. Log in to [IBM Cloud](https://cloud.ibm.com/) +2. Navigate to **Manage** > **Access (IAM)** > **API keys** +3. Click **Create** to create a new API key +4. Copy the API key (you won't be able to see it again) + +#### Step 3: Configure the File + +Edit `tests/integration/dph_v1.env` and replace the placeholder values: + +```bash +# Required: Service URL +DATA_PRODUCT_HUB_API_SERVICE_URL=https://api.dataplatform.cloud.ibm.com/ + +# Required: Authentication type +DATA_PRODUCT_HUB_API_SERVICE_AUTH_TYPE=iam + +# Required: Your IBM Cloud API Key +DATA_PRODUCT_HUB_API_SERVICE_APIKEY=your-actual-key-here +``` + +#### Step 4: Verify Permissions + +Ensure your API key has the following permissions: +- Access to IBM Data Product Hub service +- Ability to create/read/update/delete data products +- Access to the target catalog/container + +### Configuration File Format + +The IBM Cloud SDK uses environment variables or configuration files in the following format: + +``` +_URL= +_AUTH_TYPE= +_APIKEY= +``` + +Where `` is the uppercase version of the service name with underscores. + +For DPH Services, the service name is `data_product_hub_api_service`, so variables are prefixed with `DATA_PRODUCT_HUB_API_SERVICE_`. + +## Running Integration Tests + +### Run All Integration Tests + +```bash +# Activate virtual environment +source .venv/bin/activate + +# Run all integration tests +pytest tests/integration/ -v +``` + +### Run Specific Test File + +```bash +# Run only DPH integration tests +pytest tests/integration/test_dph_v1.py -v + +# Run only ODCS Collibra tests +pytest tests/integration/test_odcs_generator_collibra.py -v +``` + +### Run with Coverage + +```bash +pytest tests/integration/ --cov=src/wxdi --cov-report=html +``` + +## Test Behavior + +### Without Configuration +When configuration files are missing, tests are automatically skipped with the message: +``` +SKIPPED [49] External configuration not available, skipping... +``` + +### With Configuration +When valid configuration is provided, tests will: +1. Connect to the actual IBM Cloud service +2. Perform real API operations +3. Validate responses and behavior +4. Clean up resources (where applicable) + +## Security Notes + +⚠️ **IMPORTANT SECURITY CONSIDERATIONS:** + +1. **Never commit credentials**: The `.env` files contain sensitive API keys and should NEVER be committed to version control +2. **Already protected**: The `.gitignore` file already excludes `.env` files +3. **Secure sharing**: Share credentials only through secure channels (password managers, encrypted communication) +4. **Rotate keys**: Regularly rotate API keys, especially if they may have been exposed +5. **Minimal permissions**: Use API keys with the minimum required permissions + +## Troubleshooting + +### Tests are Skipped + +**Problem:** All integration tests show as "skipped" + +**Solution:** +- Verify the configuration file exists: `tests/integration/dph_v1.env` +- Check that the file contains valid credentials +- Ensure the file is in the correct location + +### Authentication Errors + +**Problem:** Tests fail with 401 Unauthorized or 403 Forbidden + +**Solution:** +- Verify your API key is correct and not expired +- Check that your API key has the required permissions +- Ensure the service URL is correct for your region + +### Connection Errors + +**Problem:** Tests fail with connection timeouts or network errors + +**Solution:** +- Verify the service URL is correct +- Check your network connection +- Ensure you can access IBM Cloud services from your network +- Check if there are any firewall restrictions + +### Service-Specific Errors + +**Problem:** Tests fail with service-specific error messages + +**Solution:** +- Check the IBM Data Product Hub service status +- Verify your account has access to the service +- Review the test output for specific error messages +- Consult IBM Cloud documentation for the specific service + +## Additional Resources + +- [IBM Cloud SDK Core Documentation](https://github.com/IBM/python-sdk-core) +- [IBM Cloud Authentication Guide](https://cloud.ibm.com/docs/account?topic=account-userapikey) +- [IBM Data Product Hub Documentation](https://cloud.ibm.com/docs/data-product-hub) +- [pytest Documentation](https://docs.pytest.org/) + +## Support + +For issues related to: +- **SDK functionality**: Open an issue in the data-intelligence-sdk repository +- **IBM Cloud services**: Contact IBM Cloud Support +- **Authentication**: Refer to IBM Cloud IAM documentation \ No newline at end of file diff --git a/tests/src/integration/__init__.py b/tests/src/integration/__init__.py new file mode 100644 index 0000000..2f5430c --- /dev/null +++ b/tests/src/integration/__init__.py @@ -0,0 +1,19 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Integration tests""" + +# This file is only here to get pylint to check the files in this directory diff --git a/tests/src/integration/initial_setup_service.py b/tests/src/integration/initial_setup_service.py new file mode 100644 index 0000000..ecdfddc --- /dev/null +++ b/tests/src/integration/initial_setup_service.py @@ -0,0 +1,137 @@ +# -*- coding: utf-8 -*- +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Integration Tests for DataProductHubApiServiceV1 +""" + +from enum import Enum +from typing import Dict, List +import json + +from ibm_cloud_sdk_core import BaseService, DetailedResponse +from ibm_cloud_sdk_core.authenticators.authenticator import Authenticator +from ibm_cloud_sdk_core.get_authenticator import get_authenticator_from_environment +from ibm_cloud_sdk_core.utils import convert_model + +from wxdi.dph_services.common import get_sdk_headers + +############################################################################## +# Temporary setup Service +############################################################################## + + +class InitialSetupServiceV1(BaseService): + """The Data Product Hub API Service V1 service.""" + + DEFAULT_SERVICE_URL = 'https://api.dataplatform.dev.cloud.ibm.com/v2' + DEFAULT_SERVICE_NAME = 'cams_api_service' + + @classmethod + def new_instance( + cls, + service_name: str = DEFAULT_SERVICE_NAME, + ) -> 'InitialSetupServiceV1': + """ + Return a new client for the Initial Data Product Hub API Service setup using the + specified parameters and external configuration. + """ + authenticator = get_authenticator_from_environment(service_name) + service = cls(authenticator) + service.configure_service(service_name) + return service + + def __init__( + self, + authenticator: Authenticator = None, + ) -> None: + """ + Construct a new client for the Data Product Hub API Service service. + + :param Authenticator authenticator: The authenticator specifies the authentication mechanism. + Get up to date information from https://github.com/IBM/python-sdk-core/blob/main/README.md + about initializing the authenticator of your choice. + """ + BaseService.__init__(self, service_url=self.DEFAULT_SERVICE_URL, authenticator=authenticator) + + def create_data_product_catalog( + self, + **kwargs, + ) -> DetailedResponse: + headers = {} + sdk_headers = get_sdk_headers( + service_name='initial_Setup_service', + service_version='V1', + operation_id='create_data_product_catalog', + ) + headers.update(sdk_headers) + + data = { + 'name': 'Default Data Product Hub', + 'uid': 'ibm-default-hub', + 'subtype': 'ibm_data_product_catalog', + 'generator': 'catalogadmin', + } + data = {k: v for (k, v) in data.items() if v is not None} + data = json.dumps(data) + headers['content-type'] = 'application/json' + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + headers['Accept'] = 'application/json' + + url = '/catalogs' + request = self.prepare_request( + method='POST', + url=url, + headers=headers, + data=data, + ) + + response = self.send(request, **kwargs) + return response + + def delete_data_product_catalog( + self, + id: str, + **kwargs, + ) -> DetailedResponse: + if not id: + raise ValueError('id must be provided') + headers = {} + sdk_headers = get_sdk_headers( + service_name=self.DEFAULT_SERVICE_NAME, + service_version='V1', + operation_id='delete_data_product_catalog', + ) + headers.update(sdk_headers) + + if 'headers' in kwargs: + headers.update(kwargs.get('headers')) + del kwargs['headers'] + + path_param_keys = ['id'] + path_param_values = self.encode_path_vars(id) + path_param_dict = dict(zip(path_param_keys, path_param_values)) + url = '/catalogs/{id}'.format(**path_param_dict) + request = self.prepare_request( + method='DELETE', + url=url, + headers=headers, + ) + + response = self.send(request, **kwargs) + return response diff --git a/tests/src/integration/test_data_product_recommender_integration.py b/tests/src/integration/test_data_product_recommender_integration.py new file mode 100644 index 0000000..00e6eff --- /dev/null +++ b/tests/src/integration/test_data_product_recommender_integration.py @@ -0,0 +1,373 @@ +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Integration tests for Data Product Recommender + +These tests verify end-to-end functionality using sample data files. +""" + +import pytest +import os +import tempfile +import shutil +from pathlib import Path + +from wxdi.data_product_recommender.platforms import ( + SnowflakeQueryParser, + DatabricksQueryParser, + BigQueryQueryParser, + WatsonxDataQueryParser +) +from wxdi.data_product_recommender.recommender import DataProductRecommender + + +class TestSnowflakeIntegration: + """Integration tests for Snowflake platform""" + + @pytest.fixture + def sample_csv_file(self): + """Path to sample Snowflake CSV file""" + return 'input_samples/synthetic_snowflake_business_logs_1000.csv' + + @pytest.fixture + def sample_json_file(self): + """Path to sample Snowflake JSON file""" + return 'input_samples/synthetic_snowflake_business_logs_1000.json' + + @pytest.fixture + def output_dir(self): + """Create temporary output directory""" + temp_dir = tempfile.mkdtemp() + yield temp_dir + shutil.rmtree(temp_dir) + + def test_end_to_end_csv_workflow(self, sample_csv_file, output_dir): + """Test complete workflow with CSV input""" + # Skip if sample file doesn't exist + if not os.path.exists(sample_csv_file): + pytest.skip(f"Sample file not found: {sample_csv_file}") + + # Initialize + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + # Load data + recommender.load_query_logs_from_csv_file(sample_csv_file) + assert recommender.query_logs is not None + assert len(recommender.query_logs) > 0 + + # Calculate metrics + recommender.calculate_metrics() + assert recommender.table_metrics is not None + assert len(recommender.table_metrics) > 0 + + # Generate recommendations + recommendations = recommender.recommend_data_products(num_recommendations=10) + assert 'individual_tables' in recommendations + assert len(recommendations['individual_tables']) > 0 + + # Export markdown + md_file = os.path.join(output_dir, 'test_recommendations.md') + recommender.export_recommendations_markdown(recommendations, md_file) + assert os.path.exists(md_file) + assert os.path.getsize(md_file) > 0 + + # Export JSON + json_file = os.path.join(output_dir, 'test_recommendations.json') + recommender.export_recommendations_json(recommendations, json_file) + assert os.path.exists(json_file) + assert os.path.getsize(json_file) > 0 + + def test_end_to_end_json_workflow(self, sample_json_file, output_dir): + """Test complete workflow with JSON input""" + # Skip if sample file doesn't exist + if not os.path.exists(sample_json_file): + pytest.skip(f"Sample file not found: {sample_json_file}") + + # Initialize + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + # Load data + recommender.load_query_logs_from_json_file(sample_json_file) + assert recommender.query_logs is not None + assert len(recommender.query_logs) > 0 + + # Calculate metrics + recommender.calculate_metrics() + assert recommender.table_metrics is not None + + # Generate recommendations + recommendations = recommender.recommend_data_products(num_recommendations=10) + assert 'individual_tables' in recommendations + + def test_recommendations_with_min_score(self, sample_csv_file): + """Test recommendations with minimum score threshold""" + if not os.path.exists(sample_csv_file): + pytest.skip(f"Sample file not found: {sample_csv_file}") + + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_csv_file(sample_csv_file) + recommender.calculate_metrics() + + # Get recommendations with high threshold + recommendations = recommender.recommend_data_products( + num_recommendations=20, + min_score=70.0 + ) + + # Verify all recommendations meet threshold + for table in recommendations['individual_tables']: + assert table['recommendation_score'] >= 70.0 + + +class TestDatabricksIntegration: + """Integration tests for Databricks platform""" + + @pytest.fixture + def sample_csv_file(self): + """Path to sample Databricks CSV file""" + return 'input_samples/synthetic_databricks_healthcare_logs_1000.csv' + + @pytest.fixture + def sample_json_file(self): + """Path to sample Databricks JSON file""" + return 'input_samples/synthetic_databricks_healthcare_logs_1000.json' + + def test_databricks_csv_workflow(self, sample_csv_file): + """Test Databricks workflow with CSV input""" + if not os.path.exists(sample_csv_file): + pytest.skip(f"Sample file not found: {sample_csv_file}") + + parser = DatabricksQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_csv_file(sample_csv_file) + assert recommender.query_logs is not None + + recommender.calculate_metrics() + assert recommender.table_metrics is not None + + recommendations = recommender.recommend_data_products(num_recommendations=10) + assert 'individual_tables' in recommendations + + def test_databricks_json_workflow(self, sample_json_file): + """Test Databricks workflow with JSON input""" + if not os.path.exists(sample_json_file): + pytest.skip(f"Sample file not found: {sample_json_file}") + + parser = DatabricksQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_json_file(sample_json_file) + assert recommender.query_logs is not None + + recommender.calculate_metrics() + recommendations = recommender.recommend_data_products(num_recommendations=10) + assert 'individual_tables' in recommendations + + +class TestBigQueryIntegration: + """Integration tests for BigQuery platform""" + + @pytest.fixture + def sample_csv_file(self): + """Path to sample BigQuery CSV file""" + return 'input_samples/synthetic_bigquery_retail_logs_1000.csv' + + @pytest.fixture + def sample_json_file(self): + """Path to sample BigQuery JSON file""" + return 'input_samples/synthetic_bigquery_retail_logs_1000.json' + + def test_bigquery_csv_workflow(self, sample_csv_file): + """Test BigQuery workflow with CSV input""" + if not os.path.exists(sample_csv_file): + pytest.skip(f"Sample file not found: {sample_csv_file}") + + parser = BigQueryQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_csv_file(sample_csv_file) + assert recommender.query_logs is not None + + recommender.calculate_metrics() + assert recommender.table_metrics is not None + + recommendations = recommender.recommend_data_products(num_recommendations=10) + assert 'individual_tables' in recommendations + + def test_bigquery_json_workflow(self, sample_json_file): + """Test BigQuery workflow with JSON input""" + if not os.path.exists(sample_json_file): + pytest.skip(f"Sample file not found: {sample_json_file}") + + parser = BigQueryQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_json_file(sample_json_file) + assert recommender.query_logs is not None + + recommender.calculate_metrics() + recommendations = recommender.recommend_data_products(num_recommendations=10) + assert 'individual_tables' in recommendations + + +class TestWatsonxDataIntegration: + """Integration tests for watsonx.data platform""" + + @pytest.fixture + def sample_csv_file(self): + """Path to sample watsonx.data CSV file""" + return 'input_samples/synthetic_watsonx_telecom_logs_1000.csv' + + @pytest.fixture + def sample_json_file(self): + """Path to sample watsonx.data JSON file""" + return 'input_samples/synthetic_watsonx_telecom_logs_1000.json' + + def test_watsonxdata_csv_workflow(self, sample_csv_file): + """Test watsonx.data workflow with CSV input""" + if not os.path.exists(sample_csv_file): + pytest.skip(f"Sample file not found: {sample_csv_file}") + + parser = WatsonxDataQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_csv_file(sample_csv_file) + assert recommender.query_logs is not None + + recommender.calculate_metrics() + assert recommender.table_metrics is not None + + recommendations = recommender.recommend_data_products(num_recommendations=10) + assert 'individual_tables' in recommendations + + def test_watsonxdata_json_workflow(self, sample_json_file): + """Test watsonx.data workflow with JSON input""" + if not os.path.exists(sample_json_file): + pytest.skip(f"Sample file not found: {sample_json_file}") + + parser = WatsonxDataQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_json_file(sample_json_file) + assert recommender.query_logs is not None + + recommender.calculate_metrics() + recommendations = recommender.recommend_data_products(num_recommendations=10) + assert 'individual_tables' in recommendations + + +class TestTableGrouping: + """Integration tests for table grouping functionality""" + + @pytest.fixture + def sample_file(self): + """Path to sample file with table relationships""" + return 'input_samples/synthetic_snowflake_business_logs_1000.csv' + + def test_table_grouping_detection(self, sample_file): + """Test that table groups are detected correctly""" + if not os.path.exists(sample_file): + pytest.skip(f"Sample file not found: {sample_file}") + + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_csv_file(sample_file) + recommender.calculate_metrics() + + recommendations = recommender.recommend_data_products( + num_recommendations=20, + min_frequency_threshold=0.05 # 5% threshold + ) + + # Check if table groups were identified + if 'table_groups' in recommendations and len(recommendations['table_groups']) > 0: + # Verify group structure + for group in recommendations['table_groups']: + assert 'tables' in group + assert 'group_score' in group # Changed from 'score' to 'group_score' + assert len(group['tables']) >= 2 # Groups should have at least 2 tables + + +class TestOutputFormats: + """Integration tests for output format generation""" + + @pytest.fixture + def sample_file(self): + return 'input_samples/synthetic_snowflake_business_logs_1000.csv' + + @pytest.fixture + def output_dir(self): + temp_dir = tempfile.mkdtemp() + yield temp_dir + shutil.rmtree(temp_dir) + + def test_markdown_output_structure(self, sample_file, output_dir): + """Test that markdown output has correct structure""" + if not os.path.exists(sample_file): + pytest.skip(f"Sample file not found: {sample_file}") + + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_csv_file(sample_file) + recommender.calculate_metrics() + recommendations = recommender.recommend_data_products(num_recommendations=10) + + md_file = os.path.join(output_dir, 'test_output.md') + recommender.export_recommendations_markdown(recommendations, md_file) + + # Read and verify markdown content + with open(md_file, 'r') as f: + content = f.read() + assert '# Data Product Recommendations' in content + assert '## Summary Statistics' in content + # Check that content has data product information (tables or metrics) + assert ('ANALYTICS.' in content or 'SALES.' in content or 'PRODUCT.' in content) + + def test_json_output_structure(self, sample_file, output_dir): + """Test that JSON output has correct structure""" + if not os.path.exists(sample_file): + pytest.skip(f"Sample file not found: {sample_file}") + + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + recommender.load_query_logs_from_csv_file(sample_file) + recommender.calculate_metrics() + recommendations = recommender.recommend_data_products(num_recommendations=10) + + json_file = os.path.join(output_dir, 'test_output.json') + recommender.export_recommendations_json(recommendations, json_file) + + # Read and verify JSON structure + import json + with open(json_file, 'r') as f: + data = json.load(f) + assert 'recommendations' in data + assert 'metadata' in data + assert 'generated_at' in data['metadata'] + assert 'total_queries_analyzed' in data['metadata'] + + +if __name__ == '__main__': + pytest.main([__file__, '-v']) + +# Made with Bob diff --git a/tests/src/integration/test_odcs_generator_collibra.py b/tests/src/integration/test_odcs_generator_collibra.py new file mode 100644 index 0000000..c2efdef --- /dev/null +++ b/tests/src/integration/test_odcs_generator_collibra.py @@ -0,0 +1,611 @@ +# -*- coding: utf-8 -*- +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Integration Tests for Collibra ODCS Generator +""" + +import os +import pytest +import tempfile +import yaml +from unittest.mock import Mock, patch, MagicMock +from datetime import datetime +from wxdi.odcs_generator.generate_odcs_from_collibra import ( + CollibraClient, + ODCSGenerator, + parse_arguments, + validate_arguments, + determine_output_file, + write_yaml_file +) + + +class TestCollibraClient: + """Test CollibraClient class""" + + def test_client_initialization(self): + """Test client initialization with valid parameters""" + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + + assert client.base_url == "https://acme.collibra.com" + assert client.auth == ("testuser", "testpass") + assert client.session is not None + + def test_base_url_trailing_slash_removal(self): + """Test that trailing slash is removed from base URL""" + client = CollibraClient( + base_url="https://acme.collibra.com/", + username="testuser", + password="testpass" + ) + + assert client.base_url == "https://acme.collibra.com" + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset(self, mock_get): + """Test asset retrieval""" + mock_response = Mock() + mock_response.json.return_value = { + "id": "test-asset-id", + "displayName": "Test Table", + "type": {"name": "Table"} + } + mock_response.raise_for_status = Mock() + mock_get.return_value = mock_response + + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + + asset = client.get_asset("test-asset-id") + + assert asset["id"] == "test-asset-id" + assert asset["displayName"] == "Test Table" + mock_get.assert_called_once() + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset_attributes(self, mock_get): + """Test asset attributes retrieval""" + mock_response = Mock() + mock_response.json.return_value = { + "results": [ + { + "type": {"name": "Description"}, + "value": "Test description" + }, + { + "type": {"name": "Data Type"}, + "value": "VARCHAR" + } + ] + } + mock_response.raise_for_status = Mock() + mock_get.return_value = mock_response + + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + + attributes = client.get_asset_attributes("test-asset-id") + + assert len(attributes) == 2 + assert attributes[0]["type"]["name"] == "Description" + assert attributes[1]["type"]["name"] == "Data Type" + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset_relations_as_source(self, mock_get): + """Test asset relations retrieval where asset is source""" + mock_response = Mock() + mock_response.json.return_value = { + "results": [ + { + "source": {"id": "asset-1"}, + "target": {"id": "column-1"} + } + ] + } + mock_response.raise_for_status = Mock() + mock_get.return_value = mock_response + + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + + relations = client.get_asset_relations("asset-1", as_source=True) + + assert len(relations) == 1 + assert relations[0]["source"]["id"] == "asset-1" + # Verify correct parameter was used + call_args = mock_get.call_args + assert 'sourceId' in call_args[1]['params'] + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset_relations_as_target(self, mock_get): + """Test asset relations retrieval where asset is target""" + mock_response = Mock() + mock_response.json.return_value = { + "results": [ + { + "source": {"id": "column-1"}, + "target": {"id": "asset-1"} + } + ] + } + mock_response.raise_for_status = Mock() + mock_get.return_value = mock_response + + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + + relations = client.get_asset_relations("asset-1", as_source=False) + + assert len(relations) == 1 + assert relations[0]["target"]["id"] == "asset-1" + # Verify correct parameter was used + call_args = mock_get.call_args + assert 'targetId' in call_args[1]['params'] + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset_tags(self, mock_get): + """Test asset tags retrieval""" + mock_response = Mock() + mock_response.json.return_value = [ + {"name": "PII"}, + {"name": "Sensitive"}, + {"name": "Production"} + ] + mock_response.raise_for_status = Mock() + mock_get.return_value = mock_response + + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + + tags = client.get_asset_tags("test-asset-id") + + assert len(tags) == 3 + assert "PII" in tags + assert "Sensitive" in tags + assert "Production" in tags + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset_tags_error_handling(self, mock_get): + """Test that tag retrieval errors are handled gracefully""" + mock_get.side_effect = Exception("API Error") + + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + + tags = client.get_asset_tags("test-asset-id") + + assert tags == [] + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.post') + def test_get_asset_classifications(self, mock_post): + """Test asset classifications retrieval via GraphQL""" + mock_response = Mock() + mock_response.json.return_value = { + "data": { + "api": { + "asset": { + "id": "test-asset-id", + "classesForAsset": [ + { + "id": "class-1", + "label": "Confidential", + "status": "ACCEPTED" + }, + { + "id": "class-2", + "label": "Public", + "status": "REJECTED" + } + ] + } + } + } + } + mock_response.raise_for_status = Mock() + mock_post.return_value = mock_response + + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + + classifications = client.get_asset_classifications("test-asset-id") + + # Only ACCEPTED classifications should be returned + assert len(classifications) == 1 + assert "Confidential" in classifications + assert "Public" not in classifications + + +class TestODCSGenerator: + """Test ODCSGenerator class""" + + def test_generator_initialization(self): + """Test generator initialization""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + assert generator.client == mock_client + + def test_convert_timestamp(self): + """Test timestamp conversion""" + # Test with valid timestamp (milliseconds) + timestamp_ms = 1609459200000 # 2021-01-01 00:00:00 UTC + result = ODCSGenerator._convert_timestamp(timestamp_ms) + + assert "2021-01-01" in result + assert result.endswith('Z') + + # Test with zero timestamp + result = ODCSGenerator._convert_timestamp(0) + assert result.endswith('Z') + + def test_build_attribute_map(self): + """Test attribute map building""" + attributes = [ + { + "type": {"name": "Description"}, + "value": "Test description" + }, + { + "type": {"name": "Data Type"}, + "value": "VARCHAR" + }, + { + "type": {"name": "Empty"}, + "value": None + } + ] + + attr_map = ODCSGenerator._build_attribute_map(attributes) + + assert attr_map["Description"] == "Test description" + assert attr_map["Data Type"] == "VARCHAR" + assert "Empty" not in attr_map + + def test_logical_type_mapping(self): + """Test logical type mapping""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["text"] == "string" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["whole number"] == "integer" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["decimal number"] == "number" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["date time"] == "timestamp" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["true/false"] == "boolean" + + def test_create_server_definition(self): + """Test server definition creation""" + server = ODCSGenerator._create_server_definition() + + assert "id" in server + assert server["id"].startswith("server-") + assert server["server"] == "CONFIGURE_SERVER_HOSTNAME" + assert server["type"] == "DEFINE_SERVER_TYPE" + + def test_extract_custom_properties(self): + """Test custom properties extraction""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + attr_map = { + "Description": "Test description", # Should be excluded + "Owner": "John Doe", + "Created Date": "2024-01-01", + "Custom Field": "Custom Value" + } + + custom_props = generator._extract_custom_properties(attr_map) + + # Description should be excluded + assert not any(prop["property"] == "description" for prop in custom_props) + + # Other properties should be included (converted to lowercase with underscores) + assert any(prop["property"] == "owner" for prop in custom_props) + assert any(prop["property"] == "created_date" for prop in custom_props) + + @patch.object(CollibraClient, 'get_asset') + @patch.object(CollibraClient, 'get_asset_tags') + @patch.object(CollibraClient, 'get_asset_attributes') + @patch.object(CollibraClient, 'get_asset_relations') + def test_generate_odcs_structure(self, mock_relations, mock_attrs, mock_tags, mock_asset): + """Test ODCS structure generation""" + # Mock asset data + mock_asset.return_value = { + "id": "test-asset-id", + "displayName": "Test Table", + "type": {"name": "Table"}, + "domain": {"name": "Finance"}, + "createdOn": 1609459200000 + } + + mock_tags.return_value = ["PII", "Production"] + mock_attrs.return_value = [ + {"type": {"name": "Description"}, "value": "Test table description"} + ] + mock_relations.return_value = [] + + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + generator = ODCSGenerator(client) + + odcs = generator.generate_odcs("test-asset-id") + + # Verify ODCS structure + assert odcs["id"] == "test-asset-id" + assert odcs["kind"] == "DataContract" + assert odcs["apiVersion"] == "v3.1.0" + assert odcs["domain"] == "Finance" + assert odcs["status"] == "active" + assert "PII" in odcs["tags"] + assert "Production" in odcs["tags"] + + # Verify authoritative definitions + assert len(odcs["description"]["authoritativeDefinitions"]) == 1 + auth_def = odcs["description"]["authoritativeDefinitions"][0] + assert auth_def["type"] == "collibra-asset" + assert "test-asset-id" in auth_def["url"] + + # Verify schema + assert "schema" in odcs + assert isinstance(odcs["schema"], list) + + # Verify servers + assert "servers" in odcs + assert isinstance(odcs["servers"], list) + + +class TestArgumentParsing: + """Test command-line argument parsing""" + + def test_parse_arguments_with_asset_id(self): + """Test parsing with required asset ID""" + with patch('sys.argv', ['script.py', 'test-asset-id']): + args = parse_arguments() + assert args.asset_id == 'test-asset-id' + + def test_parse_arguments_with_all_options(self): + """Test parsing with all command-line options""" + with patch('sys.argv', [ + 'script.py', + 'test-asset-id', + '-o', 'output.yaml', + '--url', 'https://acme.collibra.com', + '-u', 'testuser', + '-p', 'testpass' + ]): + args = parse_arguments() + assert args.asset_id == 'test-asset-id' + assert args.output == 'output.yaml' + assert args.url == 'https://acme.collibra.com' + assert args.username == 'testuser' + assert args.password == 'testpass' + + @patch.dict(os.environ, { + 'COLLIBRA_URL': 'https://env.collibra.com', + 'COLLIBRA_USERNAME': 'envuser', + 'COLLIBRA_PASSWORD': 'envpass' + }) + def test_parse_arguments_from_environment(self): + """Test that environment variables are used as defaults""" + with patch('sys.argv', ['script.py', 'test-asset-id']): + args = parse_arguments() + assert args.url == 'https://env.collibra.com' + assert args.username == 'envuser' + assert args.password == 'envpass' + + +class TestHelperFunctions: + """Test helper functions""" + + def test_determine_output_file_with_custom_output(self): + """Test output file determination with custom output""" + args = Mock() + args.output = 'custom-output.yaml' + + odcs_data = {'name': 'Test Contract'} + + output_file = determine_output_file(args, odcs_data) + assert output_file == 'custom-output.yaml' + + def test_determine_output_file_from_asset_name(self): + """Test output file determination from asset name""" + args = Mock() + args.output = None + + odcs_data = {'name': 'Customer Transactions'} + + output_file = determine_output_file(args, odcs_data) + assert output_file == 'customer-transactions-odcs.yaml' + + def test_determine_output_file_default(self): + """Test output file determination with default""" + args = Mock() + args.output = None + + odcs_data = {} + + output_file = determine_output_file(args, odcs_data) + assert output_file == 'asset-odcs.yaml' + + def test_write_yaml_file(self): + """Test YAML file writing""" + odcs_data = { + "id": "test-id", + "kind": "DataContract", + "apiVersion": "v3.1.0", + "schema": [{"name": "test_table"}], + "servers": [{"id": "server-1", "server": "CONFIGURE_SERVER_HOSTNAME"}] + } + + with tempfile.NamedTemporaryFile(mode='w', suffix='.yaml', delete=False) as f: + temp_file = f.name + + try: + # Write YAML + write_yaml_file(temp_file, odcs_data) + + # Read and verify + with open(temp_file, 'r') as f: + content = f.read() + loaded_data = yaml.safe_load(content) + + assert loaded_data["id"] == "test-id" + assert loaded_data["kind"] == "DataContract" + assert "⚠️" in content # Check for warning comments + finally: + if os.path.exists(temp_file): + os.unlink(temp_file) + + +class TestIntegrationScenarios: + """Integration test scenarios""" + + @patch.dict(os.environ, { + 'COLLIBRA_URL': 'https://acme.collibra.com', + 'COLLIBRA_USERNAME': 'testuser', + 'COLLIBRA_PASSWORD': 'testpass' + }) + @patch.object(CollibraClient, 'get_asset') + @patch.object(CollibraClient, 'get_asset_tags') + @patch.object(CollibraClient, 'get_asset_attributes') + @patch.object(CollibraClient, 'get_asset_relations') + def test_end_to_end_odcs_generation(self, mock_relations, mock_attrs, mock_tags, mock_asset): + """Test end-to-end ODCS generation flow""" + # Mock asset data + mock_asset.return_value = { + "id": "test-id", + "displayName": "CUSTOMER_TABLE", + "type": {"name": "Table"}, + "domain": {"name": "Sales"}, + "createdOn": 1609459200000 + } + + mock_tags.return_value = ["Production"] + + mock_attrs.return_value = [ + {"type": {"name": "Description"}, "value": "Customer information table"}, + {"type": {"name": "Owner"}, "value": "Data Team"} + ] + + # Mock relations with columns + mock_relations.side_effect = [ + [], # as_source + [ # as_target + { + "source": { + "id": "col-1", + "displayName": "customer_id", + "type": {"name": "Column"} + }, + "target": {"id": "test-id"} + } + ] + ] + + # Create client and generator + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpass" + ) + generator = ODCSGenerator(client) + + # Generate ODCS + odcs = generator.generate_odcs("test-id") + + # Verify ODCS structure + assert odcs["id"] == "test-id" + assert odcs["kind"] == "DataContract" + assert odcs["domain"] == "Sales" + assert len(odcs["schema"]) == 1 + assert odcs["schema"][0]["name"] == "CUSTOMER_TABLE" + assert odcs["schema"][0]["description"] == "Customer information table" + + def test_error_handling_invalid_asset_id(self): + """Test error handling for invalid asset ID""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + with pytest.raises(ValueError, match="Asset ID is required"): + generator.generate_odcs("") + + @patch.object(CollibraClient, 'get_asset') + def test_error_handling_api_failure(self, mock_asset): + """Test error handling when API calls fail""" + mock_asset.side_effect = Exception("API Error") + + client = CollibraClient( + base_url="https://acme.collibra.com", + username="testuser", + password="testpswd" + ) + generator = ODCSGenerator(client) + + # Should handle error gracefully + with pytest.raises(Exception): + generator.generate_odcs("test-id") + + +class TestDataTypeMapping: + """Test data type mapping functionality""" + + def test_logical_type_mapping_coverage(self): + """Test that common data types are mapped""" + mapping = ODCSGenerator.LOGICAL_TYPE_MAPPING + + assert mapping["text"] == "string" + assert mapping["whole number"] == "integer" + assert mapping["decimal number"] == "number" + assert mapping["date time"] == "timestamp" + assert mapping["true/false"] == "boolean" + assert mapping["geographical"] == "string" + + def test_numeric_types_list(self): + """Test numeric types list""" + numeric_types = ODCSGenerator.NUMERIC_TYPES + + assert "DECIMAL" in numeric_types + assert "NUMERIC" in numeric_types + assert "NUMBER" in numeric_types + + +if __name__ == '__main__': + pytest.main([__file__, '-v']) + diff --git a/tests/src/integration/test_odcs_generator_informatica.py b/tests/src/integration/test_odcs_generator_informatica.py new file mode 100644 index 0000000..a256e9e --- /dev/null +++ b/tests/src/integration/test_odcs_generator_informatica.py @@ -0,0 +1,389 @@ +# -*- coding: utf-8 -*- +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Integration Tests for Informatica ODCS Generator +""" + +import os +import pytest +import tempfile +import yaml +from unittest.mock import Mock, patch, MagicMock +from wxdi.odcs_generator.generate_odcs_from_informatica import ( + InformaticaClient, + parse_arguments, + validate_arguments, + generate_odcs_yaml, + build_physical_type, + build_column_property, + build_custom_properties, + extract_column_position, + RESOURCE_TYPE_MAPPING, + SYSTEM_ATTRIBUTES_MAPPING +) + + +class TestInformaticaClient: + """Test InformaticaClient class""" + + def test_client_initialization(self): + """Test client initialization with valid parameters""" + client = InformaticaClient( + base_url="https://cdgc.dm-us.informaticacloud.com", + username="testuser", + password="testpass" + ) + + assert client.base_url == "https://cdgc.dm-us.informaticacloud.com" + assert client.username == "testuser" + assert client.password == "testpass" + assert client.region == "dm-us" + assert client.identity_url == "https://dm-us.informaticacloud.com" + + def test_extract_region_from_url(self): + """Test region extraction from various URL formats""" + client = InformaticaClient( + base_url="https://cdgc.dm-us.informaticacloud.com", + username="test", + password="test" + ) + + # Test various URL formats + assert client._extract_region_from_url("https://cdgc.dm-us.informaticacloud.com") == "dm-us" + assert client._extract_region_from_url("https://cdgc.na1.informaticacloud.com") == "na1" + assert client._extract_region_from_url("https://cdgc.eu1.informaticacloud.com/") == "eu1" + + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.requests.post') + def test_get_session_id(self, mock_post): + """Test session ID retrieval""" + mock_response = Mock() + mock_response.json.return_value = {"sessionId": "test-session-id"} + mock_response.raise_for_status = Mock() + mock_post.return_value = mock_response + + client = InformaticaClient( + base_url="https://cdgc.dm-us.informaticacloud.com", + username="testuser", + password="testpass" + ) + + session_data = client.get_session_id() + + assert session_data["sessionId"] == "test-session-id" + mock_post.assert_called_once() + + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.requests.post') + def test_get_auth_token_caching(self, mock_post): + """Test that auth token is cached""" + # Mock session ID call + mock_session_response = Mock() + mock_session_response.json.return_value = {"sessionId": "test-session-id"} + mock_session_response.raise_for_status = Mock() + + # Mock JWT token call + mock_jwt_response = Mock() + mock_jwt_response.json.return_value = {"jwt_token": "test-jwt-token"} + mock_jwt_response.raise_for_status = Mock() + + mock_post.side_effect = [mock_session_response, mock_jwt_response] + + client = InformaticaClient( + base_url="https://cdgc.dm-us.informaticacloud.com", + username="testuser", + password="testpass" + ) + + # First call should make API requests + token1 = client.get_auth_token() + assert token1 == "test-jwt-token" + assert mock_post.call_count == 2 + + # Second call should use cached token + token2 = client.get_auth_token() + assert token2 == "test-jwt-token" + assert mock_post.call_count == 2 # No additional calls + + +class TestHelperFunctions: + """Test helper functions""" + + def test_extract_column_position(self): + """Test column position extraction""" + # Valid position + col_data = {"selfAttributes": {"core.Position": "5"}} + assert extract_column_position(col_data) == 5 + + # Missing position + col_data = {"selfAttributes": {}} + assert extract_column_position(col_data) == 999999 + + # Invalid position + col_data = {"selfAttributes": {"core.Position": "invalid"}} + assert extract_column_position(col_data) == 999999 + + def test_build_physical_type(self): + """Test physical type construction""" + # Type with length + assert build_physical_type("VARCHAR", "255", "") == "VARCHAR(255)" + + # Type with length and scale + assert build_physical_type("DECIMAL", "10", "2") == "DECIMAL(10,2)" + + # Type without length + assert build_physical_type("INTEGER", "", "") == "INTEGER" + + # Type with length but scale is 0 + assert build_physical_type("NUMBER", "18", "0") == "NUMBER(18)" + + def test_build_column_property(self): + """Test column property building""" + column_detail = { + "summary": { + "core.name": "customer_id", + "core.description": "Customer identifier" + }, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "VARCHAR", + "com.infa.odin.models.relational.DatatypeLength": "50", + "com.infa.odin.models.relational.Nullable": "false", + "com.infa.odin.models.relational.PrimaryKeyColumn": "true" + } + } + + prop = build_column_property(column_detail) + + assert prop["name"] == "customer_id" + assert prop["physicalType"] == "VARCHAR(50)" + assert prop["description"] == "Customer identifier" + assert prop["required"] is True + assert prop["primaryKey"] is True + + def test_build_custom_properties(self): + """Test custom properties building""" + table_attrs = { + "core.resourceName": "TestCatalog", + "com.infa.odin.models.relational.NumberOfRows": "1000", + "core.origin": "Production", + "com.infa.odin.models.relational.Owner": "dbo" + } + + custom_props = build_custom_properties(table_attrs) + + assert len(custom_props) == 4 + assert {"property": "Catalog Source Name", "value": "TestCatalog"} in custom_props + assert {"property": "Number of rows", "value": "1000"} in custom_props + assert {"property": "Origin", "value": "Production"} in custom_props + assert {"property": "Schema", "value": "dbo"} in custom_props + + +class TestODCSGeneration: + """Test ODCS YAML generation""" + + def test_generate_odcs_yaml_structure(self): + """Test ODCS YAML structure generation""" + asset_data = { + "core.identity": "test-asset-id", + "summary": { + "core.name": "CUSTOMER_TABLE", + "core.description": "Customer information table" + }, + "selfAttributes": { + "core.name": "customer_table", + "core.businessName": "customer_table", + "com.infa.odin.models.relational.Owner": "dbo", + "core.resourceType": "Snowflake", + "core.resourceName": "TestCatalog" + } + } + + column_details = [ + { + "summary": { + "core.name": "id", + "core.description": "Primary key" + }, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "INTEGER", + "com.infa.odin.models.relational.Nullable": "false", + "com.infa.odin.models.relational.PrimaryKeyColumn": "true", + "core.Position": "1" + } + } + ] + + base_url = "https://cdgc.dm-us.informaticacloud.com" + + odcs = generate_odcs_yaml(asset_data, column_details, base_url) + + # Verify structure + assert odcs["id"] == "test-asset-id" + assert odcs["kind"] == "DataContract" + assert odcs["apiVersion"] == "v3.1.0" + assert odcs["status"] == "active" + + # Verify schema + assert len(odcs["schema"]) == 1 + schema = odcs["schema"][0] + assert schema["name"] == "customer_table" + assert schema["physicalName"] == "dbo/customer_table" + assert schema["physicalType"] == "Table" + assert len(schema["properties"]) == 1 + + # Verify column + column = schema["properties"][0] + assert column["name"] == "id" + assert column["physicalType"] == "INTEGER" + assert column["required"] is True + assert column["primaryKey"] is True + + # Verify servers + assert len(odcs["servers"]) == 1 + server = odcs["servers"][0] + assert server["type"] == "snowflake" + assert server["schema"] == "dbo" + + # Verify authoritative definitions + assert len(odcs["description"]["authoritativeDefinitions"]) == 1 + auth_def = odcs["description"]["authoritativeDefinitions"][0] + assert auth_def["type"] == "informatica-asset" + assert "test-asset-id" in auth_def["url"] + + +class TestResourceTypeMapping: + """Test resource type mapping""" + + def test_resource_type_mapping_coverage(self): + """Test that common database types are mapped""" + assert RESOURCE_TYPE_MAPPING["Snowflake"] == "snowflake" + assert RESOURCE_TYPE_MAPPING["PostgreSQL"] == "postgresql" + assert RESOURCE_TYPE_MAPPING["Oracle"] == "oracle" + assert RESOURCE_TYPE_MAPPING["SqlServer"] == "sqlserver" + assert RESOURCE_TYPE_MAPPING["BigQuery"] == "bigquery" + assert RESOURCE_TYPE_MAPPING["Redshift"] == "redshift" + + +class TestArgumentParsing: + """Test command-line argument parsing""" + + def test_parse_arguments_with_asset_id(self): + """Test parsing with required asset ID""" + with patch('sys.argv', ['script.py', 'test-asset-id']): + args = parse_arguments() + assert args.asset_id == 'test-asset-id' + + def test_parse_arguments_with_all_options(self): + """Test parsing with all command-line options""" + with patch('sys.argv', [ + 'script.py', + 'test-asset-id', + '-o', 'output.yaml', + '--cdgc-url', 'https://cdgc.test.com', + '-u', 'testuser', + '-p', 'testpass' + ]): + args = parse_arguments() + assert args.asset_id == 'test-asset-id' + assert args.output == 'output.yaml' + assert args.cdgc_url == 'https://cdgc.test.com' + assert args.username == 'testuser' + assert args.password == 'testpass' + + +class TestIntegrationScenarios: + """Integration test scenarios""" + + @patch.dict(os.environ, { + 'INFORMATICA_CDGC_URL': 'https://cdgc.dm-us.informaticacloud.com', + 'INFORMATICA_USERNAME': 'testuser', + 'INFORMATICA_PASSWORD': 'testpawd' + }) + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.InformaticaClient') + def test_end_to_end_odcs_generation(self, mock_client_class): + """Test end-to-end ODCS generation flow""" + # Mock client instance + mock_client = Mock() + mock_client.base_url = "https://cdgc.dm-us.informaticacloud.com" + mock_client_class.return_value = mock_client + + # Mock asset data + mock_client.get_asset_details.return_value = { + "core.identity": "test-id", + "summary": {"core.name": "TEST_TABLE"}, + "selfAttributes": { + "core.businessName": "test_table", + "com.infa.odin.models.relational.Owner": "public", + "core.resourceType": "PostgreSQL" + }, + "hierarchy": [{"core.identity": "col-1"}] + } + + # Mock column data + mock_client.get_column_details.return_value = { + "summary": {"core.name": "id"}, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "INTEGER", + "core.Position": "1" + } + } + + # Generate ODCS + asset_data = mock_client.get_asset_details("test-id") + column_ids = [col['core.identity'] for col in asset_data.get('hierarchy', [])] + column_details = [mock_client.get_column_details(col_id) for col_id in column_ids] + + odcs = generate_odcs_yaml(asset_data, column_details, mock_client.base_url) + + # Verify ODCS structure + assert odcs["id"] == "test-id" + assert odcs["kind"] == "DataContract" + assert len(odcs["schema"]) == 1 + assert len(odcs["schema"][0]["properties"]) == 1 + assert odcs["servers"][0]["type"] == "postgresql" + + def test_yaml_file_generation(self): + """Test YAML file writing""" + odcs_data = { + "id": "test-id", + "kind": "DataContract", + "apiVersion": "v3.1.0", + "schema": [{"name": "test_table"}], + "servers": [{"id": "server-1", "server": "CONFIGURE_SERVER_HOSTNAME"}] + } + + with tempfile.NamedTemporaryFile(mode='w', suffix='.yaml', delete=False) as f: + temp_file = f.name + + try: + # Write YAML + yaml_content = yaml.dump(odcs_data, default_flow_style=False, sort_keys=False) + with open(temp_file, 'w') as f: + f.write(yaml_content) + + # Read and verify + with open(temp_file, 'r') as f: + loaded_data = yaml.safe_load(f) + + assert loaded_data["id"] == "test-id" + assert loaded_data["kind"] == "DataContract" + assert loaded_data["apiVersion"] == "v3.1.0" + finally: + if os.path.exists(temp_file): + os.unlink(temp_file) + + +if __name__ == '__main__': + pytest.main([__file__, '-v']) + diff --git a/tests/src/odcs_generator/__init__.py b/tests/src/odcs_generator/__init__.py new file mode 100644 index 0000000..3b445d4 --- /dev/null +++ b/tests/src/odcs_generator/__init__.py @@ -0,0 +1,19 @@ +# coding: utf-8 + +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""ODCS Generator test package""" + +# Made with Bob diff --git a/tests/src/odcs_generator/test_odcs_generator_collibra.py b/tests/src/odcs_generator/test_odcs_generator_collibra.py new file mode 100644 index 0000000..79152a0 --- /dev/null +++ b/tests/src/odcs_generator/test_odcs_generator_collibra.py @@ -0,0 +1,739 @@ +# -*- coding: utf-8 -*- +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Unit Tests for Collibra ODCS Generator +Tests individual functions and methods in isolation +""" + +import pytest +import os +from unittest.mock import Mock, patch +from datetime import datetime +from wxdi.odcs_generator.generate_odcs_from_collibra import ( + CollibraClient, + ODCSGenerator, + determine_output_file +) + + +class TestODCSGeneratorStaticMethods: + """Unit tests for ODCSGenerator static methods""" + + def test_convert_timestamp_with_valid_milliseconds(self): + """Test timestamp conversion with valid milliseconds""" + # 2021-01-01 00:00:00 UTC in milliseconds + timestamp_ms = 1609459200000 + result = ODCSGenerator._convert_timestamp(timestamp_ms) + + assert "2021-01-01" in result + assert result.endswith('Z') + + def test_convert_timestamp_with_zero(self): + """Test timestamp conversion with zero""" + result = ODCSGenerator._convert_timestamp(0) + + # Should return current time + assert result.endswith('Z') + assert 'T' in result + + def test_convert_timestamp_with_none(self): + """Test timestamp conversion with None/falsy value""" + result = ODCSGenerator._convert_timestamp(None) + + # Should return current time + assert result.endswith('Z') + + def test_convert_timestamp_format(self): + """Test that timestamp format is ISO 8601""" + timestamp_ms = 1609459200000 + result = ODCSGenerator._convert_timestamp(timestamp_ms) + + # Should contain date and time separator + assert 'T' in result + # Should end with Z for UTC + assert result.endswith('Z') + + def test_build_attribute_map_empty(self): + """Test building attribute map with empty list""" + result = ODCSGenerator._build_attribute_map([]) + assert result == {} + + def test_build_attribute_map_single_attribute(self): + """Test building attribute map with single attribute""" + attributes = [ + { + "type": {"name": "Description"}, + "value": "Test description" + } + ] + + result = ODCSGenerator._build_attribute_map(attributes) + + assert result["Description"] == "Test description" + + def test_build_attribute_map_multiple_attributes(self): + """Test building attribute map with multiple attributes""" + attributes = [ + {"type": {"name": "Description"}, "value": "Desc1"}, + {"type": {"name": "Data Type"}, "value": "VARCHAR"}, + {"type": {"name": "Owner"}, "value": "John"} + ] + + result = ODCSGenerator._build_attribute_map(attributes) + + assert len(result) == 3 + assert result["Description"] == "Desc1" + assert result["Data Type"] == "VARCHAR" + assert result["Owner"] == "John" + + def test_build_attribute_map_missing_type_name(self): + """Test that attributes without type name are skipped""" + attributes = [ + {"type": {}, "value": "Value1"}, + {"type": {"name": "Valid"}, "value": "Value2"} + ] + + result = ODCSGenerator._build_attribute_map(attributes) + + assert len(result) == 1 + assert result["Valid"] == "Value2" + + def test_build_attribute_map_missing_value(self): + """Test that attributes without value are skipped""" + attributes = [ + {"type": {"name": "Empty"}, "value": None}, + {"type": {"name": "Valid"}, "value": "Value"} + ] + + result = ODCSGenerator._build_attribute_map(attributes) + + assert len(result) == 1 + assert result["Valid"] == "Value" + + def test_build_attribute_map_empty_string_value(self): + """Test that attributes with empty string value are skipped""" + attributes = [ + {"type": {"name": "Empty"}, "value": ""}, + {"type": {"name": "Valid"}, "value": "Value"} + ] + + result = ODCSGenerator._build_attribute_map(attributes) + + assert len(result) == 1 + assert result["Valid"] == "Value" + + def test_create_server_definition(self): + """Test server definition creation""" + server = ODCSGenerator._create_server_definition() + + assert "id" in server + assert server["id"].startswith("server-") + assert len(server["id"]) == 15 # "server-" + 8 hex chars + assert server["server"] == "CONFIGURE_SERVER_HOSTNAME" + assert server["type"] == "DEFINE_SERVER_TYPE" + + def test_create_server_definition_unique_ids(self): + """Test that server definitions have unique IDs""" + server1 = ODCSGenerator._create_server_definition() + server2 = ODCSGenerator._create_server_definition() + + assert server1["id"] != server2["id"] + + +class TestODCSGeneratorExtractCustomProperties: + """Unit tests for _extract_custom_properties method""" + + def test_extract_custom_properties_empty(self): + """Test with empty attribute map""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + result = generator._extract_custom_properties({}) + assert result == [] + + def test_extract_custom_properties_excludes_description(self): + """Test that Description is excluded""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + attr_map = { + "Description": "This should be excluded", + "Owner": "John Doe" + } + + result = generator._extract_custom_properties(attr_map) + + # Description should not be in custom properties + assert not any(prop["property"] == "description" for prop in result) + # Owner should be included (converted to lowercase) + assert any(prop["property"] == "owner" for prop in result) + + def test_extract_custom_properties_multiple(self): + """Test with multiple custom properties""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + attr_map = { + "Owner": "John Doe", + "Created Date": "2024-01-01", + "Status": "Active", + "Custom Field": "Custom Value" + } + + result = generator._extract_custom_properties(attr_map) + + assert len(result) == 4 + properties = {prop["property"]: prop["value"] for prop in result} + # Property names are converted to lowercase with underscores + assert properties["owner"] == "John Doe" + assert properties["created_date"] == "2024-01-01" + assert properties["status"] == "Active" + assert properties["custom_field"] == "Custom Value" + + def test_extract_custom_properties_format(self): + """Test that custom properties have correct format""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + attr_map = {"Owner": "John"} + + result = generator._extract_custom_properties(attr_map) + + assert len(result) == 1 + assert "property" in result[0] + assert "value" in result[0] + assert result[0]["property"] == "owner" # Converted to lowercase + assert result[0]["value"] == "John" + + +class TestODCSGeneratorLogicalTypeMapping: + """Unit tests for logical type mapping""" + + def test_text_to_string(self): + """Test text maps to string""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["text"] == "string" + + def test_whole_number_to_integer(self): + """Test whole number maps to integer""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["whole number"] == "integer" + + def test_decimal_number_to_number(self): + """Test decimal number maps to number""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["decimal number"] == "number" + + def test_date_time_to_timestamp(self): + """Test date time maps to timestamp""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["date time"] == "timestamp" + + def test_boolean_mapping(self): + """Test true/false maps to boolean""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["true/false"] == "boolean" + + def test_geographical_to_string(self): + """Test geographical maps to string""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["geographical"] == "string" + + def test_standard_types(self): + """Test standard type mappings""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["string"] == "string" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["integer"] == "integer" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["number"] == "number" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["date"] == "date" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["time"] == "time" + + def test_complex_types(self): + """Test complex type mappings""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["object"] == "object" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["array"] == "array" + + def test_na_mapping(self): + """Test n/a maps to None""" + assert ODCSGenerator.LOGICAL_TYPE_MAPPING["n/a"] is None + + def test_unmapped_type(self): + """Test that unmapped types return KeyError""" + with pytest.raises(KeyError): + _ = ODCSGenerator.LOGICAL_TYPE_MAPPING["unmapped_type"] + + +class TestODCSGeneratorNumericTypes: + """Unit tests for numeric types list""" + + def test_decimal_in_numeric_types(self): + """Test DECIMAL is in numeric types""" + assert "DECIMAL" in ODCSGenerator.NUMERIC_TYPES + + def test_numeric_in_numeric_types(self): + """Test NUMERIC is in numeric types""" + assert "NUMERIC" in ODCSGenerator.NUMERIC_TYPES + + def test_number_in_numeric_types(self): + """Test NUMBER is in numeric types""" + assert "NUMBER" in ODCSGenerator.NUMERIC_TYPES + + def test_numeric_types_count(self): + """Test expected number of numeric types""" + assert len(ODCSGenerator.NUMERIC_TYPES) == 3 + + +class TestODCSGeneratorExcludedAttributes: + """Unit tests for excluded attributes""" + + def test_description_excluded(self): + """Test that Description is in excluded attributes""" + assert "Description" in ODCSGenerator.EXCLUDED_ATTRIBUTES + + def test_excluded_attributes_is_set(self): + """Test that EXCLUDED_ATTRIBUTES is a set""" + assert isinstance(ODCSGenerator.EXCLUDED_ATTRIBUTES, set) + + +class TestCollibraClientUnit: + """Unit tests for CollibraClient class""" + + def test_client_initialization(self): + """Test client initialization""" + client = CollibraClient( + base_url="https://test.collibra.com", + username="testuser", + password="testpass" + ) + + assert client.base_url == "https://test.collibra.com" + assert client.auth == ("testuser", "testpass") + assert client.session is not None + + def test_base_url_trailing_slash_removed(self): + """Test that trailing slash is removed from base URL""" + client = CollibraClient( + base_url="https://test.collibra.com/", + username="user", + password="pass" + ) + + assert client.base_url == "https://test.collibra.com" + assert not client.base_url.endswith('/') + + def test_headers_json_constant(self): + """Test HEADERS_JSON constant""" + assert CollibraClient.HEADERS_JSON == {"Accept": "application/json"} + + def test_headers_content_json_constant(self): + """Test HEADERS_CONTENT_JSON constant""" + assert CollibraClient.HEADERS_CONTENT_JSON == {"Content-Type": "application/json"} + + def test_default_limit_constant(self): + """Test DEFAULT_LIMIT constant""" + assert CollibraClient.DEFAULT_LIMIT == 1000 + assert isinstance(CollibraClient.DEFAULT_LIMIT, int) + + +class TestDetermineOutputFile: + """Unit tests for determine_output_file function""" + + def test_with_custom_output_specified(self): + """Test with custom output file""" + args = Mock() + args.output = "my-custom-file.yaml" + odcs_data = {"name": "Test Contract"} + + result = determine_output_file(args, odcs_data) + assert result == "my-custom-file.yaml" + + def test_from_contract_name(self): + """Test output file from contract name""" + args = Mock() + args.output = None + odcs_data = {"name": "Customer Transactions"} + + result = determine_output_file(args, odcs_data) + assert result == "customer-transactions-odcs.yaml" + + def test_name_with_spaces(self): + """Test name with spaces is converted to hyphens""" + args = Mock() + args.output = None + odcs_data = {"name": "My Test Contract"} + + result = determine_output_file(args, odcs_data) + assert result == "my-test-contract-odcs.yaml" + assert " " not in result + + def test_name_with_uppercase(self): + """Test name is converted to lowercase""" + args = Mock() + args.output = None + odcs_data = {"name": "CUSTOMER_TABLE"} + + result = determine_output_file(args, odcs_data) + assert result == "customer_table-odcs.yaml" + assert result.islower() or "_" in result + + def test_default_when_no_name(self): + """Test default output file when no name""" + args = Mock() + args.output = None + odcs_data = {} + + result = determine_output_file(args, odcs_data) + assert result == "asset-odcs.yaml" + + def test_name_with_special_characters(self): + """Test name with special characters""" + args = Mock() + args.output = None + odcs_data = {"name": "Test/Table\\Name"} + + result = determine_output_file(args, odcs_data) + # Should handle special characters gracefully + assert result.endswith("-odcs.yaml") + + def test_empty_name(self): + """Test with empty name string""" + args = Mock() + args.output = None + odcs_data = {"name": ""} + + result = determine_output_file(args, odcs_data) + # Should use default when name is empty + assert result.endswith("-odcs.yaml") + + +class TestODCSGeneratorInitialization: + """Unit tests for ODCSGenerator initialization""" + + def test_generator_requires_client(self): + """Test that generator requires a client""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + assert generator.client == mock_client + + def test_generator_stores_client_reference(self): + """Test that generator stores client reference""" + mock_client = Mock(spec=CollibraClient) + mock_client.base_url = "https://test.com" + + generator = ODCSGenerator(mock_client) + + assert generator.client.base_url == "https://test.com" + + +class TestODCSGeneratorValidation: + """Unit tests for ODCSGenerator validation""" + + def test_generate_odcs_requires_asset_id(self): + """Test that generate_odcs requires asset ID""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + with pytest.raises(ValueError, match="Asset ID is required"): + generator.generate_odcs("") + + def test_generate_odcs_rejects_none_asset_id(self): + """Test that generate_odcs rejects None asset ID""" + mock_client = Mock(spec=CollibraClient) + generator = ODCSGenerator(mock_client) + + with pytest.raises((ValueError, TypeError)): + generator.generate_odcs(None) + + def test_generate_odcs_accepts_valid_asset_id(self): + """Test that generate_odcs accepts valid asset ID""" + mock_client = Mock(spec=CollibraClient) + mock_client.get_asset.return_value = { + "id": "test-id", + "displayName": "Test", + "type": {"name": "Table"}, + "domain": {"name": "Domain"}, + "createdOn": 0 + } + mock_client.get_asset_tags.return_value = [] + mock_client.get_asset_attributes.return_value = [] + mock_client.get_asset_relations.return_value = [] + mock_client.base_url = "https://test.com" + + generator = ODCSGenerator(mock_client) + + # Should not raise an error + result = generator.generate_odcs("valid-asset-id") + assert result is not None + + +class TestODCSStructureValidation: + """Unit tests for ODCS structure validation""" + + def test_odcs_has_required_fields(self): + """Test that generated ODCS has required fields""" + mock_client = Mock(spec=CollibraClient) + mock_client.get_asset.return_value = { + "id": "test-id", + "displayName": "Test Table", + "type": {"name": "Table"}, + "domain": {"name": "Finance"}, + "createdOn": 1609459200000 + } + mock_client.get_asset_tags.return_value = [] + mock_client.get_asset_attributes.return_value = [] + mock_client.get_asset_relations.return_value = [] + mock_client.base_url = "https://test.com" + + generator = ODCSGenerator(mock_client) + odcs = generator.generate_odcs("test-id") + + # Check required ODCS fields + required_fields = [ + 'id', 'kind', 'apiVersion', 'domain', 'dataProduct', + 'version', 'name', 'status', 'contractCreatedTs', + 'description', 'tags', 'schema', 'servers' + ] + + for field in required_fields: + assert field in odcs, f"Missing required field: {field}" + + def test_odcs_kind_is_data_contract(self): + """Test that ODCS kind is DataContract""" + mock_client = Mock(spec=CollibraClient) + mock_client.get_asset.return_value = { + "id": "test-id", + "displayName": "Test", + "type": {"name": "Table"}, + "domain": {"name": "Domain"}, + "createdOn": 0 + } + mock_client.get_asset_tags.return_value = [] + mock_client.get_asset_attributes.return_value = [] + mock_client.get_asset_relations.return_value = [] + mock_client.base_url = "https://test.com" + + generator = ODCSGenerator(mock_client) + odcs = generator.generate_odcs("test-id") + + assert odcs["kind"] == "DataContract" + + def test_odcs_api_version(self): + """Test that ODCS has correct API version""" + mock_client = Mock(spec=CollibraClient) + mock_client.get_asset.return_value = { + "id": "test-id", + "displayName": "Test", + "type": {"name": "Table"}, + "domain": {"name": "Domain"}, + "createdOn": 0 + } + mock_client.get_asset_tags.return_value = [] + mock_client.get_asset_attributes.return_value = [] + mock_client.get_asset_relations.return_value = [] + mock_client.base_url = "https://test.com" + + generator = ODCSGenerator(mock_client) + odcs = generator.generate_odcs("test-id") + + assert odcs["apiVersion"] == "v3.1.0" + + +class TestCollibraClientAPIMethods: + """Unit tests for CollibraClient API methods with mocked responses""" + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset_success(self, mock_get): + """Test successful asset retrieval""" + mock_response = Mock() + mock_response.json.return_value = { + "id": "asset-123", + "displayName": "Test Table", + "type": {"name": "Table"}, + "domain": {"name": "Finance"} + } + mock_response.raise_for_status = Mock() + mock_get.return_value = mock_response + + client = CollibraClient( + "https://test.collibra.com", + "testuser", + "testpass" + ) + + result = client.get_asset("asset-123") + + assert result["id"] == "asset-123" + assert result["displayName"] == "Test Table" + mock_get.assert_called_once() + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset_attributes_success(self, mock_get): + """Test successful asset attributes retrieval""" + mock_response = Mock() + mock_response.json.return_value = { + "results": [ + {"type": {"name": "Description"}, "value": "Test description"}, + {"type": {"name": "Owner"}, "value": "John Doe"} + ] + } + mock_response.raise_for_status = Mock() + mock_get.return_value = mock_response + + client = CollibraClient( + "https://test.collibra.com", + "testuser", + "testpass" + ) + + result = client.get_asset_attributes("asset-123") + + assert len(result) == 2 + assert result[0]["type"]["name"] == "Description" + mock_get.assert_called_once() + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset_tags_success(self, mock_get): + """Test successful asset tags retrieval""" + mock_response = Mock() + mock_response.json.return_value = [ + {"name": "PII"}, + {"name": "Sensitive"} + ] + mock_response.raise_for_status = Mock() + mock_get.return_value = mock_response + + client = CollibraClient( + "https://test.collibra.com", + "testuser", + "testpass" + ) + + result = client.get_asset_tags("asset-123") + + assert len(result) == 2 + assert result[0] == "PII" + mock_get.assert_called_once() + + @patch('wxdi.odcs_generator.generate_odcs_from_collibra.requests.Session.get') + def test_get_asset_relations_success(self, mock_get): + """Test successful asset relations retrieval""" + mock_response = Mock() + mock_response.json.return_value = { + "results": [ + { + "type": {"role": "target"}, + "target": { + "id": "col-1", + "name": "customer_id", + "type": {"name": "Column"} + } + } + ] + } + mock_response.raise_for_status = Mock() + mock_get.return_value = mock_response + + client = CollibraClient( + "https://test.collibra.com", + "testuser", + "testpass" + ) + + result = client.get_asset_relations("asset-123") + + assert len(result) == 1 + assert result[0]["target"]["name"] == "customer_id" + mock_get.assert_called_once() + +class TestParseAndValidateArgumentsCollibra: + """Unit tests for Collibra argument parsing and validation""" + + @patch('sys.argv', ['script.py', 'asset-123', '--url', 'https://test.collibra.com', '-u', 'user', '-p', 'pass']) + def test_parse_arguments_all_provided(self): + """Test parsing with all arguments provided""" + from wxdi.odcs_generator.generate_odcs_from_collibra import parse_arguments + + args = parse_arguments() + + assert args.asset_id == 'asset-123' + assert args.url == 'https://test.collibra.com' + assert args.username == 'user' + assert args.password == 'pass' + + @patch('sys.argv', ['script.py', 'asset-123', '-o', 'output.yaml']) + @patch.dict(os.environ, { + 'COLLIBRA_URL': 'https://env.collibra.com', + 'COLLIBRA_USERNAME': 'envuser', + 'COLLIBRA_PASSWORD': 'envpass' # pragma: allowlist secret + }) + def test_parse_arguments_from_env(self): + """Test parsing with environment variables""" + from wxdi.odcs_generator.generate_odcs_from_collibra import parse_arguments + + args = parse_arguments() + + assert args.asset_id == 'asset-123' + assert args.output == 'output.yaml' + assert args.url == 'https://env.collibra.com' + assert args.username == 'envuser' + assert args.password == 'envpass' + + def test_validate_arguments_missing_url(self): + """Test validation fails with missing URL""" + from wxdi.odcs_generator.generate_odcs_from_collibra import validate_arguments + + args = Mock() + args.url = None + args.username = 'user' + args.password = 'pass' + + with pytest.raises(SystemExit): + validate_arguments(args) + + def test_validate_arguments_missing_username(self): + """Test validation fails with missing username""" + from wxdi.odcs_generator.generate_odcs_from_collibra import validate_arguments + + args = Mock() + args.url = 'https://test.com' + args.username = None + args.password = 'pass' + + with pytest.raises(SystemExit): + validate_arguments(args) + + def test_validate_arguments_missing_password(self): + """Test validation fails with missing password""" + from wxdi.odcs_generator.generate_odcs_from_collibra import validate_arguments + + args = Mock() + args.url = 'https://test.com' + args.username = 'user' + args.password = None + + with pytest.raises(SystemExit): + validate_arguments(args) + + def test_validate_arguments_all_present(self): + """Test validation passes with all arguments""" + from wxdi.odcs_generator.generate_odcs_from_collibra import validate_arguments + + args = Mock() + args.url = 'https://test.com' + args.username = 'user' + args.password = 'pass' + + # Should not raise an exception + validate_arguments(args) + + +if __name__ == '__main__': + pytest.main([__file__, '-v']) + diff --git a/tests/src/odcs_generator/test_odcs_generator_informatica.py b/tests/src/odcs_generator/test_odcs_generator_informatica.py new file mode 100644 index 0000000..c55ce14 --- /dev/null +++ b/tests/src/odcs_generator/test_odcs_generator_informatica.py @@ -0,0 +1,1060 @@ +# -*- coding: utf-8 -*- +# Copyright 2026 IBM Corporation +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Unit Tests for Informatica ODCS Generator +Tests individual functions and methods in isolation +""" + +import pytest +import os +from unittest.mock import Mock, patch +from wxdi.odcs_generator.generate_odcs_from_informatica import ( + InformaticaClient, + extract_column_position, + build_physical_type, + build_column_property, + build_custom_properties, + generate_odcs_yaml, + determine_output_file, + RESOURCE_TYPE_MAPPING, + SYSTEM_ATTRIBUTES_MAPPING +) + + +class TestExtractColumnPosition: + """Unit tests for extract_column_position function""" + + def test_valid_position_as_string(self): + """Test extraction with valid position as string""" + col_data = {"selfAttributes": {"core.Position": "5"}} + assert extract_column_position(col_data) == 5 + + def test_valid_position_as_int(self): + """Test extraction with valid position as integer""" + col_data = {"selfAttributes": {"core.Position": 10}} + assert extract_column_position(col_data) == 10 + + def test_missing_self_attributes(self): + """Test with missing selfAttributes""" + col_data = {} + assert extract_column_position(col_data) == 999999 + + def test_missing_position_key(self): + """Test with missing Position key""" + col_data = {"selfAttributes": {}} + assert extract_column_position(col_data) == 999999 + + def test_none_position_value(self): + """Test with None position value""" + col_data = {"selfAttributes": {"core.Position": None}} + assert extract_column_position(col_data) == 999999 + + def test_invalid_position_string(self): + """Test with invalid position string""" + col_data = {"selfAttributes": {"core.Position": "invalid"}} + assert extract_column_position(col_data) == 999999 + + def test_empty_position_string(self): + """Test with empty position string""" + col_data = {"selfAttributes": {"core.Position": ""}} + assert extract_column_position(col_data) == 999999 + + def test_negative_position(self): + """Test with negative position""" + col_data = {"selfAttributes": {"core.Position": "-1"}} + assert extract_column_position(col_data) == -1 + + +class TestBuildPhysicalType: + """Unit tests for build_physical_type function""" + + def test_type_without_length(self): + """Test type without length or scale""" + assert build_physical_type("INTEGER", "", "") == "INTEGER" + assert build_physical_type("TIMESTAMP", None, None) == "TIMESTAMP" + + def test_type_with_length_only(self): + """Test type with length but no scale""" + assert build_physical_type("VARCHAR", "255", "") == "VARCHAR(255)" + assert build_physical_type("CHAR", "10", None) == "CHAR(10)" + + def test_type_with_length_and_scale(self): + """Test type with both length and scale""" + assert build_physical_type("DECIMAL", "10", "2") == "DECIMAL(10,2)" + assert build_physical_type("NUMBER", "18", "4") == "NUMBER(18,4)" + + def test_type_with_length_and_zero_scale(self): + """Test type with length and scale of 0""" + assert build_physical_type("NUMBER", "18", "0") == "NUMBER(18)" + assert build_physical_type("DECIMAL", "10", "0") == "DECIMAL(10)" + + def test_type_with_empty_strings(self): + """Test with empty strings for length and scale""" + assert build_physical_type("VARCHAR", "", "") == "VARCHAR" + + def test_various_data_types(self): + """Test with various common data types""" + assert build_physical_type("BIGINT", "", "") == "BIGINT" + assert build_physical_type("TEXT", "", "") == "TEXT" + assert build_physical_type("BOOLEAN", "", "") == "BOOLEAN" + assert build_physical_type("DATE", "", "") == "DATE" + + +class TestBuildColumnProperty: + """Unit tests for build_column_property function""" + + def test_basic_column_property(self): + """Test building basic column property""" + column_detail = { + "summary": { + "core.name": "customer_id" + }, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "INTEGER" + } + } + + prop = build_column_property(column_detail) + + assert prop["name"] == "customer_id" + assert prop["physicalType"] == "INTEGER" + assert prop["required"] is False # Default when Nullable not specified (nullable=true by default) + + def test_column_with_description(self): + """Test column with description""" + column_detail = { + "summary": { + "core.name": "email", + "core.description": "Customer email address" + }, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "VARCHAR", + "com.infa.odin.models.relational.DatatypeLength": "255" + } + } + + prop = build_column_property(column_detail) + + assert prop["name"] == "email" + assert prop["physicalType"] == "VARCHAR(255)" + assert prop["description"] == "Customer email address" + + def test_nullable_column(self): + """Test nullable column""" + column_detail = { + "summary": {"core.name": "middle_name"}, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "VARCHAR", + "com.infa.odin.models.relational.DatatypeLength": "50", + "com.infa.odin.models.relational.Nullable": "true" + } + } + + prop = build_column_property(column_detail) + + assert prop["required"] is False + + def test_non_nullable_column(self): + """Test non-nullable column""" + column_detail = { + "summary": {"core.name": "id"}, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "INTEGER", + "com.infa.odin.models.relational.Nullable": "false" + } + } + + prop = build_column_property(column_detail) + + assert prop["required"] is True + + def test_primary_key_column(self): + """Test primary key column""" + column_detail = { + "summary": {"core.name": "id"}, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "INTEGER", + "com.infa.odin.models.relational.PrimaryKeyColumn": "true" + } + } + + prop = build_column_property(column_detail) + + assert prop.get("primaryKey") is True + + def test_non_primary_key_column(self): + """Test non-primary key column""" + column_detail = { + "summary": {"core.name": "name"}, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "VARCHAR", + "com.infa.odin.models.relational.PrimaryKeyColumn": "false" + } + } + + prop = build_column_property(column_detail) + + assert "primaryKey" not in prop or prop.get("primaryKey") is False + + def test_decimal_column_with_scale(self): + """Test decimal column with precision and scale""" + column_detail = { + "summary": {"core.name": "price"}, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "DECIMAL", + "com.infa.odin.models.relational.DatatypeLength": "10", + "com.infa.odin.models.relational.DatatypeScale": "2" + } + } + + prop = build_column_property(column_detail) + + assert prop["physicalType"] == "DECIMAL(10,2)" + + def test_column_without_description(self): + """Test that description is not added when empty""" + column_detail = { + "summary": {"core.name": "col1"}, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "INTEGER" + } + } + + prop = build_column_property(column_detail) + + assert "description" not in prop + + +class TestBuildCustomProperties: + """Unit tests for build_custom_properties function""" + + def test_empty_attributes(self): + """Test with empty attributes""" + assert build_custom_properties({}) == [] + + def test_single_attribute(self): + """Test with single attribute""" + attrs = {"core.resourceName": "TestCatalog"} + props = build_custom_properties(attrs) + + assert len(props) == 1 + assert props[0]["property"] == "Catalog Source Name" + assert props[0]["value"] == "TestCatalog" + + def test_multiple_attributes(self): + """Test with multiple attributes""" + attrs = { + "core.resourceName": "TestCatalog", + "com.infa.odin.models.relational.NumberOfRows": "1000", + "core.origin": "Production" + } + props = build_custom_properties(attrs) + + assert len(props) == 3 + property_names = [p["property"] for p in props] + assert "Catalog Source Name" in property_names + assert "Number of rows" in property_names + assert "Origin" in property_names + + def test_unmapped_attributes_ignored(self): + """Test that unmapped attributes are ignored""" + attrs = { + "core.resourceName": "TestCatalog", + "unmapped.attribute": "SomeValue" + } + props = build_custom_properties(attrs) + + assert len(props) == 1 + assert props[0]["property"] == "Catalog Source Name" + + def test_all_system_attributes(self): + """Test with all system attributes""" + attrs = { + "core.resourceName": "Catalog1", + "com.infa.odin.models.relational.NumberOfRows": "5000", + "core.origin": "Prod", + "com.infa.odin.models.relational.Owner": "dbo", + "core.sourceCreatedBy": "admin", + "core.sourceCreatedOn": "2024-01-01", + "core.sourceModifiedBy": "user1", + "core.sourceModifiedOn": "2024-02-01" + } + props = build_custom_properties(attrs) + + assert len(props) == 8 + + +class TestGenerateOdcsYaml: + """Unit tests for generate_odcs_yaml function""" + + def test_basic_odcs_structure(self): + """Test basic ODCS structure generation""" + asset_data = { + "core.identity": "test-id", + "summary": {"core.name": "TEST_TABLE"}, + "selfAttributes": { + "core.businessName": "test_table", + "com.infa.odin.models.relational.Owner": "public" + } + } + column_details = [] + base_url = "https://cdgc.test.com" + + odcs = generate_odcs_yaml(asset_data, column_details, base_url) + + assert odcs["id"] == "test-id" + assert odcs["kind"] == "DataContract" + assert odcs["apiVersion"] == "v3.1.0" + assert odcs["status"] == "active" + assert "contractCreatedTs" in odcs + + def test_schema_with_columns(self): + """Test schema generation with columns""" + asset_data = { + "core.identity": "test-id", + "summary": {"core.name": "CUSTOMER"}, + "selfAttributes": { + "core.name": "customer", + "com.infa.odin.models.relational.Owner": "dbo" + } + } + column_details = [ + { + "summary": {"core.name": "id"}, + "selfAttributes": { + "com.infa.odin.models.relational.Datatype": "INTEGER", + "core.Position": "1" + } + } + ] + base_url = "https://cdgc.test.com" + + odcs = generate_odcs_yaml(asset_data, column_details, base_url) + + assert len(odcs["schema"]) == 1 + assert odcs["schema"][0]["name"] == "customer" + assert len(odcs["schema"][0]["properties"]) == 1 + assert odcs["schema"][0]["properties"][0]["name"] == "id" + + def test_physical_name_construction(self): + """Test physical name construction with schema""" + asset_data = { + "core.identity": "test-id", + "summary": {"core.name": "TABLE1"}, + "selfAttributes": { + "core.name": "table1", + "com.infa.odin.models.relational.Owner": "schema1" + } + } + column_details = [] + base_url = "https://cdgc.test.com" + + odcs = generate_odcs_yaml(asset_data, column_details, base_url) + + assert odcs["schema"][0]["physicalName"] == "schema1/table1" + + def test_server_type_mapping(self): + """Test server type mapping from resource type""" + asset_data = { + "core.identity": "test-id", + "summary": {"core.name": "TABLE1"}, + "selfAttributes": { + "core.businessName": "table1", + "core.resourceType": "Snowflake", + "com.infa.odin.models.relational.Owner": "public" + } + } + column_details = [] + base_url = "https://cdgc.test.com" + + odcs = generate_odcs_yaml(asset_data, column_details, base_url) + + assert odcs["servers"][0]["type"] == "snowflake" + + def test_authoritative_definition(self): + """Test authoritative definition URL""" + asset_data = { + "core.identity": "asset-123", + "summary": {"core.name": "TABLE1"}, + "selfAttributes": {"core.businessName": "table1"} + } + column_details = [] + base_url = "https://cdgc.test.com" + + odcs = generate_odcs_yaml(asset_data, column_details, base_url) + + auth_defs = odcs["description"]["authoritativeDefinitions"] + assert len(auth_defs) == 1 + assert auth_defs[0]["type"] == "informatica-asset" + assert "asset-123" in auth_defs[0]["url"] + + +class TestDetermineOutputFile: + """Unit tests for determine_output_file function""" + + def test_with_custom_output(self): + """Test with custom output file specified""" + args = Mock() + args.output = "custom-file.yaml" + odcs_data = {"name": "Test"} + + result = determine_output_file(args, odcs_data) + assert result == "custom-file.yaml" + + def test_from_asset_name(self): + """Test output file from asset name""" + args = Mock() + args.output = None + odcs_data = {"name": "Customer Transactions"} + + result = determine_output_file(args, odcs_data) + assert result == "customer-transactions-odcs.yaml" + + def test_default_name(self): + """Test default output file name""" + args = Mock() + args.output = None + odcs_data = {} + + result = determine_output_file(args, odcs_data) + assert result == "asset-odcs.yaml" + + def test_name_with_special_characters(self): + """Test name with special characters""" + args = Mock() + args.output = None + odcs_data = {"name": "Test Table (Production)"} + + result = determine_output_file(args, odcs_data) + assert result == "test-table-(production)-odcs.yaml" + + +class TestResourceTypeMapping: + """Unit tests for RESOURCE_TYPE_MAPPING constant""" + + def test_common_databases_mapped(self): + """Test that common database types are mapped""" + assert RESOURCE_TYPE_MAPPING["Snowflake"] == "snowflake" + assert RESOURCE_TYPE_MAPPING["PostgreSQL"] == "postgresql" + assert RESOURCE_TYPE_MAPPING["Oracle"] == "oracle" + assert RESOURCE_TYPE_MAPPING["SqlServer"] == "sqlserver" + + def test_cloud_databases_mapped(self): + """Test that cloud database types are mapped""" + assert RESOURCE_TYPE_MAPPING["BigQuery"] == "bigquery" + assert RESOURCE_TYPE_MAPPING["Redshift"] == "redshift" + assert RESOURCE_TYPE_MAPPING["Databricks"] == "databricks" + assert RESOURCE_TYPE_MAPPING["Synapse"] == "synapse" + + def test_mapping_count(self): + """Test that expected number of mappings exist""" + assert len(RESOURCE_TYPE_MAPPING) >= 13 + + +class TestSystemAttributesMapping: + """Unit tests for SYSTEM_ATTRIBUTES_MAPPING constant""" + + def test_core_attributes_mapped(self): + """Test that core attributes are mapped""" + assert "core.resourceName" in SYSTEM_ATTRIBUTES_MAPPING + assert "core.origin" in SYSTEM_ATTRIBUTES_MAPPING + + def test_relational_attributes_mapped(self): + """Test that relational model attributes are mapped""" + assert "com.infa.odin.models.relational.NumberOfRows" in SYSTEM_ATTRIBUTES_MAPPING + assert "com.infa.odin.models.relational.Owner" in SYSTEM_ATTRIBUTES_MAPPING + + def test_timestamp_attributes_mapped(self): + """Test that timestamp attributes are mapped""" + assert "core.sourceCreatedOn" in SYSTEM_ATTRIBUTES_MAPPING + assert "core.sourceModifiedOn" in SYSTEM_ATTRIBUTES_MAPPING + + def test_user_attributes_mapped(self): + """Test that user attributes are mapped""" + assert "core.sourceCreatedBy" in SYSTEM_ATTRIBUTES_MAPPING + assert "core.sourceModifiedBy" in SYSTEM_ATTRIBUTES_MAPPING + + +class TestInformaticaClientUnit: + """Unit tests for InformaticaClient class methods""" + + def test_extract_region_from_standard_url(self): + """Test region extraction from standard URL""" + client = InformaticaClient( + "https://cdgc.dm-us.informaticacloud.com", + "user", + "pass" + ) + assert client.region == "dm-us" + + def test_extract_region_from_various_formats(self): + """Test region extraction from various URL formats""" + test_cases = [ + ("https://cdgc.na1.informaticacloud.com", "na1"), + ("https://cdgc.eu1.informaticacloud.com/", "eu1"), + ("https://cdgc.ap1.informaticacloud.com", "ap1"), + ] + + for url, expected_region in test_cases: + client = InformaticaClient(url, "user", "pass") + assert client.region == expected_region + + def test_identity_url_construction(self): + """Test identity URL construction""" + client = InformaticaClient( + "https://cdgc.dm-us.informaticacloud.com", + "user", + "pass" + ) + assert client.identity_url == "https://dm-us.informaticacloud.com" + + def test_base_url_normalization(self): + """Test that base URL trailing slash is removed""" + client = InformaticaClient( + "https://cdgc.test.com/", + "user", + "pass" + ) + assert client.base_url == "https://cdgc.test.com" + + +class TestInformaticaClientAPIMethods: + """Unit tests for InformaticaClient API methods with mocked responses""" + + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.requests.post') + def test_get_session_id_success(self, mock_post): + """Test successful session ID retrieval""" + mock_response = Mock() + mock_response.json.return_value = {"sessionId": "test-session-123"} + mock_response.raise_for_status = Mock() + mock_post.return_value = mock_response + + client = InformaticaClient( + "https://cdgc.dm-us.informaticacloud.com", + "testuser", + "testpass" + ) + + result = client.get_session_id() + + assert result["sessionId"] == "test-session-123" + mock_post.assert_called_once() + + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.requests.post') + def test_get_auth_token_success(self, mock_post): + """Test successful auth token retrieval""" + # Mock session ID call + mock_session_response = Mock() + mock_session_response.json.return_value = {"sessionId": "session-123"} + mock_session_response.raise_for_status = Mock() + + # Mock JWT token call + mock_jwt_response = Mock() + mock_jwt_response.json.return_value = {"jwt_token": "jwt-token-456"} + mock_jwt_response.raise_for_status = Mock() + + mock_post.side_effect = [mock_session_response, mock_jwt_response] + + client = InformaticaClient( + "https://cdgc.dm-us.informaticacloud.com", + "testuser", + "testpass" + ) + + token = client.get_auth_token() + + assert token == "jwt-token-456" + assert mock_post.call_count == 2 + + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.requests.post') + def test_get_auth_token_caching(self, mock_post): + """Test that auth token is cached""" + mock_session_response = Mock() + mock_session_response.json.return_value = {"sessionId": "session-123"} + mock_session_response.raise_for_status = Mock() + + mock_jwt_response = Mock() + mock_jwt_response.json.return_value = {"jwt_token": "jwt-token-456"} + mock_jwt_response.raise_for_status = Mock() + + mock_post.side_effect = [mock_session_response, mock_jwt_response] + + client = InformaticaClient( + "https://cdgc.dm-us.informaticacloud.com", + "testuser", + "testpass" + ) + + # First call + token1 = client.get_auth_token() + # Second call should use cached token + token2 = client.get_auth_token() + + assert token1 == token2 + # Should only call API twice (session + jwt), not four times + assert mock_post.call_count == 2 + + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.requests.get') + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.requests.post') + def test_get_asset_details_success(self, mock_post, mock_get): + """Test successful asset details retrieval""" + # Mock auth calls + mock_session_response = Mock() + mock_session_response.json.return_value = {"sessionId": "session-123"} + mock_session_response.raise_for_status = Mock() + + mock_jwt_response = Mock() + mock_jwt_response.json.return_value = {"jwt_token": "jwt-token-456"} + mock_jwt_response.raise_for_status = Mock() + + mock_post.side_effect = [mock_session_response, mock_jwt_response] + + # Mock asset details call + mock_asset_response = Mock() + mock_asset_response.json.return_value = { + "core.identity": "asset-123", + "summary": {"core.name": "TEST_TABLE"} + } + mock_asset_response.raise_for_status = Mock() + mock_get.return_value = mock_asset_response + + client = InformaticaClient( + "https://cdgc.dm-us.informaticacloud.com", + "testuser", + "testpass" + ) + + result = client.get_asset_details("asset-123") + + assert result["core.identity"] == "asset-123" + assert result["summary"]["core.name"] == "TEST_TABLE" + mock_get.assert_called_once() + + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.requests.get') + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.requests.post') + def test_get_column_details_success(self, mock_post, mock_get): + """Test successful column details retrieval""" + # Mock auth calls + mock_session_response = Mock() + mock_session_response.json.return_value = {"sessionId": "session-123"} + mock_session_response.raise_for_status = Mock() + + mock_jwt_response = Mock() + mock_jwt_response.json.return_value = {"jwt_token": "jwt-token-456"} + mock_jwt_response.raise_for_status = Mock() + + mock_post.side_effect = [mock_session_response, mock_jwt_response] + + # Mock column details call + mock_column_response = Mock() + mock_column_response.json.return_value = { + "summary": {"core.name": "customer_id"}, + "selfAttributes": {"com.infa.odin.models.relational.Datatype": "INTEGER"} + } + mock_column_response.raise_for_status = Mock() + mock_get.return_value = mock_column_response + + client = InformaticaClient( + "https://cdgc.dm-us.informaticacloud.com", + "testuser", + "testpass" + ) + + result = client.get_column_details("column-456") + + assert result["summary"]["core.name"] == "customer_id" + mock_get.assert_called_once() + + +class TestParseAndValidateArguments: + """Unit tests for argument parsing and validation""" + + @patch('sys.argv', ['script.py', 'asset-123', '--cdgc-url', 'https://test.com', '-u', 'user', '-p', 'pass']) + def test_parse_arguments_all_provided(self): + """Test parsing with all arguments provided""" + from wxdi.odcs_generator.generate_odcs_from_informatica import parse_arguments + + args = parse_arguments() + + assert args.asset_id == 'asset-123' + assert args.cdgc_url == 'https://test.com' + assert args.username == 'user' + assert args.password == 'pass' + + @patch('sys.argv', ['script.py', 'asset-123', '-o', 'output.yaml']) + @patch.dict(os.environ, { + 'INFORMATICA_CDGC_URL': 'https://env.com', + 'INFORMATICA_USERNAME': 'envuser', + 'INFORMATICA_PASSWORD': 'envpass' # pragma: allowlist secret + }) + def test_parse_arguments_from_env(self): + """Test parsing with environment variables""" + from wxdi.odcs_generator.generate_odcs_from_informatica import parse_arguments + + args = parse_arguments() + + assert args.asset_id == 'asset-123' + assert args.output == 'output.yaml' + assert args.cdgc_url == 'https://env.com' + assert args.username == 'envuser' + assert args.password == 'envpass' + + def test_validate_arguments_missing_url(self): + """Test validation fails with missing URL""" + from wxdi.odcs_generator.generate_odcs_from_informatica import validate_arguments + + args = Mock() + args.cdgc_url = None + args.username = 'user' + args.password = 'pass' + + with pytest.raises(SystemExit): + validate_arguments(args) + + def test_validate_arguments_missing_username(self): + """Test validation fails with missing username""" + from wxdi.odcs_generator.generate_odcs_from_informatica import validate_arguments + + args = Mock() + args.cdgc_url = 'https://test.com' + args.username = None + args.password = 'pass' + + with pytest.raises(SystemExit): + validate_arguments(args) + + def test_validate_arguments_missing_password(self): + """Test validation fails with missing password""" + from wxdi.odcs_generator.generate_odcs_from_informatica import validate_arguments + + args = Mock() + args.cdgc_url = 'https://test.com' + args.username = 'user' + args.password = None + + with pytest.raises(SystemExit): + validate_arguments(args) + + def test_validate_arguments_all_present(self): + """Test validation passes with all arguments""" + from wxdi.odcs_generator.generate_odcs_from_informatica import validate_arguments + + args = Mock() + args.cdgc_url = 'https://test.com' + args.username = 'user' + args.password = 'pass' + + # Should not raise an exception + + +class TestGenerateODCSYAML: + """Test cases for generate_odcs_yaml function""" + + def test_generate_odcs_yaml_with_description(self): + """Test ODCS generation with table description""" + asset_data = { + 'core.identity': 'table-123', + 'summary': { + 'core.name': 'test_table', + 'core.description': 'Test table description' + }, + 'selfAttributes': { + 'core.businessName': 'TestTable', + 'com.infa.odin.models.relational.Owner': 'test_schema', + 'core.resourceType': 'Snowflake' + } + } + + column_details = [ + { + 'core.identity': 'col-1', + 'summary': {'core.name': 'id'}, + 'selfAttributes': { + 'com.infa.odin.models.relational.Datatype': 'INTEGER', + 'com.infa.odin.models.relational.Nullable': 'false', + 'com.infa.odin.models.relational.PrimaryKeyColumn': 'true' + } + } + ] + + result = generate_odcs_yaml(asset_data, column_details, 'https://test.com') + + assert result['id'] == 'table-123' + assert result['schema'][0]['description'] == 'Test table description' + assert result['schema'][0]['properties'][0]['primaryKey'] is True + assert result['servers'][0]['type'] == 'snowflake' + + def test_generate_odcs_yaml_without_description(self): + """Test ODCS generation without table description""" + asset_data = { + 'core.identity': 'table-456', + 'summary': {'core.name': 'test_table'}, + 'selfAttributes': { + 'core.businessName': 'TestTable', + 'com.infa.odin.models.relational.Owner': 'test_schema' + } + } + + column_details = [] + + result = generate_odcs_yaml(asset_data, column_details, 'https://test.com') + + assert 'description' not in result['schema'][0] + + +class TestHelperFunctions: + """Test cases for helper functions""" + + def test_add_server_warning_comments(self): + """Test _add_server_warning_comments function""" + from wxdi.odcs_generator.generate_odcs_from_informatica import _add_server_warning_comments + + modified_lines = [] + _add_server_warning_comments(modified_lines, ' - id: server-123') + + assert len(modified_lines) == 5 + assert '⚠️ MANUAL CONFIGURATION REQUIRED' in modified_lines[1] + + def test_add_inline_comment_server(self): + """Test _add_inline_comment_if_needed for server field""" + from wxdi.odcs_generator.generate_odcs_from_informatica import _add_inline_comment_if_needed + + line = ' server: CONFIGURE_SERVER_HOSTNAME' + result = _add_inline_comment_if_needed(line) + assert '⚠️ UPDATE' in result + assert 'prod.snowflake.acme.com' in result + + def test_add_inline_comment_type(self): + """Test _add_inline_comment_if_needed for type field""" + from wxdi.odcs_generator.generate_odcs_from_informatica import _add_inline_comment_if_needed + + line = ' type: CONFIGURE_SERVER_TYPE' + result = _add_inline_comment_if_needed(line) + assert '⚠️ UPDATE' in result + assert 'snowflake' in result + + def test_add_inline_comment_schema(self): + """Test _add_inline_comment_if_needed for schema field""" + from wxdi.odcs_generator.generate_odcs_from_informatica import _add_inline_comment_if_needed + + line = ' schema: CONFIGURE_SCHEMA_NAME' + result = _add_inline_comment_if_needed(line) + assert '⚠️ UPDATE' in result + assert 'public' in result + + def test_add_inline_comment_no_change(self): + """Test _add_inline_comment_if_needed for normal line""" + from wxdi.odcs_generator.generate_odcs_from_informatica import _add_inline_comment_if_needed + + line = ' name: test' + result = _add_inline_comment_if_needed(line) + assert result == line + + def test_determine_output_file_with_arg(self): + """Test determine_output_file with output argument""" + args = Mock() + args.output = 'custom.yaml' + odcs_data = {'name': 'test'} + + result = determine_output_file(args, odcs_data) + assert result == 'custom.yaml' + + def test_determine_output_file_default(self): + """Test determine_output_file with default naming""" + args = Mock() + args.output = None + odcs_data = {'name': 'Test Table'} + + result = determine_output_file(args, odcs_data) + assert result == 'test-table-odcs.yaml' + + +class TestWriteYAMLFile: + """Test cases for write_yaml_file function""" + + @patch('builtins.open', create=True) + @patch('builtins.print') + def test_write_yaml_file(self, mock_print, mock_open_func): + """Test write_yaml_file function""" + from wxdi.odcs_generator.generate_odcs_from_informatica import write_yaml_file + + mock_file = Mock() + mock_open_func.return_value.__enter__.return_value = mock_file + + odcs_data = { + 'id': 'test-123', + 'servers': [ + { + 'id': 'server-1', + 'server': 'CONFIGURE_SERVER_HOSTNAME', + 'type': 'CONFIGURE_SERVER_TYPE', + 'schema': 'CONFIGURE_SCHEMA_NAME' + } + ] + } + + write_yaml_file('test.yaml', odcs_data) + + mock_open_func.assert_called_once_with('test.yaml', 'w') + mock_print.assert_called() + + +class TestMainFunction: + """Test cases for main function""" + + @patch('sys.argv', ['script.py', 'asset-123', '--cdgc-url', 'https://test.com', + '-u', 'user', '-p', 'pass']) + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.InformaticaClient') + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.write_yaml_file') + @patch('builtins.print') + def test_main_success(self, mock_print, mock_write, mock_client_class): + """Test main function success path""" + from wxdi.odcs_generator.generate_odcs_from_informatica import main + + # Mock client instance + mock_client = Mock() + mock_client.base_url = 'https://test.com' + mock_client.get_asset_details.return_value = { + 'core.identity': 'asset-123', + 'summary': {'core.name': 'test_table'}, + 'selfAttributes': {'core.businessName': 'TestTable'}, + 'hierarchy': [{'core.identity': 'col-1'}] + } + mock_client.get_column_details.return_value = { + 'core.identity': 'col-1', + 'summary': {'core.name': 'id'}, + 'selfAttributes': { + 'com.infa.odin.models.relational.Datatype': 'INTEGER', + 'com.infa.odin.models.relational.Position': '1' + } + } + mock_client_class.return_value = mock_client + + main() + + mock_client.get_asset_details.assert_called_once_with('asset-123') + mock_write.assert_called_once() + + @patch('sys.argv', ['script.py', 'asset-123', '--cdgc-url', 'https://test.com', + '-u', 'user', '-p', 'pass']) + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.InformaticaClient') + @patch('sys.exit') + def test_main_http_401_error(self, mock_exit, mock_client_class): + """Test main function with 401 HTTP error""" + from wxdi.odcs_generator.generate_odcs_from_informatica import main + import requests + + mock_client = Mock() + mock_response = Mock() + mock_response.status_code = 401 + http_error = requests.exceptions.HTTPError(response=mock_response) + mock_client.get_asset_details.side_effect = http_error + mock_client_class.return_value = mock_client + + main() + + mock_exit.assert_called_once_with(1) + + @patch('sys.argv', ['script.py', 'asset-123', '--cdgc-url', 'https://test.com', + '-u', 'user', '-p', 'pass']) + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.InformaticaClient') + @patch('sys.exit') + def test_main_http_404_error(self, mock_exit, mock_client_class): + """Test main function with 404 HTTP error""" + from wxdi.odcs_generator.generate_odcs_from_informatica import main + import requests + + mock_client = Mock() + mock_response = Mock() + mock_response.status_code = 404 + http_error = requests.exceptions.HTTPError(response=mock_response) + mock_client.get_asset_details.side_effect = http_error + mock_client_class.return_value = mock_client + + main() + + mock_exit.assert_called_once_with(1) + + @patch('sys.argv', ['script.py', 'asset-123', '--cdgc-url', 'https://test.com', + '-u', 'user', '-p', 'pass']) + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.InformaticaClient') + @patch('sys.exit') + def test_main_connection_error(self, mock_exit, mock_client_class): + """Test main function with connection error""" + from wxdi.odcs_generator.generate_odcs_from_informatica import main + import requests + + mock_client = Mock() + mock_client.get_asset_details.side_effect = requests.exceptions.ConnectionError() + mock_client_class.return_value = mock_client + + main() + + mock_exit.assert_called_once_with(1) + + @patch('sys.argv', ['script.py', 'asset-123', '--cdgc-url', 'https://test.com', + '-u', 'user', '-p', 'pass']) + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.InformaticaClient') + @patch('sys.exit') + def test_main_timeout_error(self, mock_exit, mock_client_class): + """Test main function with timeout error""" + from wxdi.odcs_generator.generate_odcs_from_informatica import main + import requests + + mock_client = Mock() + mock_client.get_asset_details.side_effect = requests.exceptions.Timeout() + mock_client_class.return_value = mock_client + + main() + + mock_exit.assert_called_once_with(1) + + @patch('sys.argv', ['script.py', 'asset-123', '--cdgc-url', 'https://test.com', + '-u', 'user', '-p', 'pass']) + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.InformaticaClient') + @patch('sys.exit') + def test_main_key_error(self, mock_exit, mock_client_class): + """Test main function with KeyError""" + from wxdi.odcs_generator.generate_odcs_from_informatica import main + + mock_client = Mock() + mock_client.get_asset_details.side_effect = KeyError('missing_field') + mock_client_class.return_value = mock_client + + main() + + mock_exit.assert_called_once_with(1) + + @patch('sys.argv', ['script.py', 'asset-123', '--cdgc-url', 'https://test.com', + '-u', 'user', '-p', 'pass']) + @patch('wxdi.odcs_generator.generate_odcs_from_informatica.InformaticaClient') + @patch('sys.exit') + def test_main_unexpected_error(self, mock_exit, mock_client_class): + """Test main function with unexpected error""" + from wxdi.odcs_generator.generate_odcs_from_informatica import main + + mock_client = Mock() + mock_client.get_asset_details.side_effect = ValueError('unexpected error') + mock_client_class.return_value = mock_client + + main() + + mock_exit.assert_called_once_with(1) + + +if __name__ == '__main__': + pytest.main([__file__, '-v']) + From e40250eb3745788abeeb15b3e39e10ed57511150 Mon Sep 17 00:00:00 2001 From: Koichi Nishitani Date: Tue, 21 Apr 2026 21:35:57 +0900 Subject: [PATCH 17/33] chore: Sync with enterprise (manual) on 2026-04-21 (#8) update build script to version Signed-off-by: Koichi Nishitani Signed-off-by: Greeshma Rajendran --- .github/workflows/build.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 1096af0..c4263e2 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -25,7 +25,7 @@ jobs: uses: ./.github/workflows/verify.yaml publish-release: - if: contains(fromJSON('["main", "sandbox"]'), github.ref_name) && github.event.pull_request.action == 'closed' && github.event.pull_request.merged + if: contains(fromJSON('["main", "sandbox"]'), github.ref_name) && (github.event_name == 'workflow_dispatch' || github.event.action == 'closed' && github.event.pull_request.merged) needs: build-and-verify name: semantic-release runs-on: ubuntu-latest From af359473317dde0e99359b969fc98b76092972a5 Mon Sep 17 00:00:00 2001 From: Koichi Nishitani Date: Tue, 21 Apr 2026 23:41:46 +0900 Subject: [PATCH 18/33] chore: fix semantic release build (#9) Signed-off-by: Koichi Nishitani Signed-off-by: Greeshma Rajendran --- .github/workflows/build.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index c4263e2..e331bab 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -35,7 +35,7 @@ jobs: with: persist-credentials: false fetch-depth: 0 - ref: github.ref + ref: ${{ github.ref }} - name: Setup Node.js uses: actions/setup-node@v4 From 0423492c064085fe4601ac01da3a78fd8e703580 Mon Sep 17 00:00:00 2001 From: Koichi Nishitani Date: Wed, 22 Apr 2026 14:55:20 +0900 Subject: [PATCH 19/33] chore: use push for semantic-release and update action versions (#10) Signed-off-by: Koichi Nishitani Signed-off-by: Greeshma Rajendran --- .github/workflows/build.yaml | 13 ++++++++----- .github/workflows/deploy-package.yaml | 10 +++++----- .github/workflows/docs.yml | 8 ++++---- .github/workflows/verify.yaml | 8 ++++---- 4 files changed, 21 insertions(+), 18 deletions(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index e331bab..ef4d3db 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -13,8 +13,11 @@ name: build on: + push: + branches: + - main + - sandbox pull_request: - types: ['opened', 'synchronize', 'reopened', 'closed'] branches: - main - sandbox @@ -25,25 +28,25 @@ jobs: uses: ./.github/workflows/verify.yaml publish-release: - if: contains(fromJSON('["main", "sandbox"]'), github.ref_name) && (github.event_name == 'workflow_dispatch' || github.event.action == 'closed' && github.event.pull_request.merged) + if: contains(fromJSON('["main", "sandbox"]'), github.ref_name) && (github.event_name != 'pull_request') needs: build-and-verify name: semantic-release runs-on: ubuntu-latest steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@v6 with: persist-credentials: false fetch-depth: 0 ref: ${{ github.ref }} - name: Setup Node.js - uses: actions/setup-node@v4 + uses: actions/setup-node@v6 with: node-version: 24 - name: Set up Python - uses: actions/setup-python@v5 + uses: actions/setup-python@v6 with: python-version: "3.14" diff --git a/.github/workflows/deploy-package.yaml b/.github/workflows/deploy-package.yaml index 7c0be1a..50ff69f 100644 --- a/.github/workflows/deploy-package.yaml +++ b/.github/workflows/deploy-package.yaml @@ -28,10 +28,10 @@ jobs: needs: build steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@v6 - name: Set up Python - uses: actions/setup-python@v5 + uses: actions/setup-python@v6 with: python-version: "3.14" @@ -53,12 +53,12 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@v6 with: ref: 'refs/heads/sandbox' - name: Set up Python - uses: actions/setup-python@v5 + uses: actions/setup-python@v6 with: python-version: "3.14" @@ -86,7 +86,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout main - uses: actions/checkout@v4 + uses: actions/checkout@v6 with: persist-credentials: false fetch-depth: 0 diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 4444513..a22c199 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -36,10 +36,10 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@v6 - name: Set up Python - uses: actions/setup-python@v5 + uses: actions/setup-python@v6 with: python-version: '3.14' cache: 'pip' @@ -59,7 +59,7 @@ jobs: sphinx-build -b html -W --keep-going . _build/html - name: Upload artifact - uses: actions/upload-pages-artifact@v3 + uses: actions/upload-pages-artifact@v5 with: path: docs/_build/html @@ -77,6 +77,6 @@ jobs: steps: - name: Deploy to GitHub Pages id: deployment - uses: actions/deploy-pages@v4 + uses: actions/deploy-pages@v5 # Made with Bob diff --git a/.github/workflows/verify.yaml b/.github/workflows/verify.yaml index 8348e6d..3386a57 100644 --- a/.github/workflows/verify.yaml +++ b/.github/workflows/verify.yaml @@ -22,10 +22,10 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@v6 - name: Setup Python - uses: actions/setup-python@v5 + uses: actions/setup-python@v6 with: python-version: "3.14" @@ -49,10 +49,10 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@v6 - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v5 + uses: actions/setup-python@v6 with: python-version: ${{ matrix.python-version }} From 1b1e9c53f07bbd75433d7ab2f06ef22db2c8d97a Mon Sep 17 00:00:00 2001 From: semantic-release-bot Date: Wed, 22 Apr 2026 05:58:24 +0000 Subject: [PATCH 20/33] Update version 1.0.0 -> 2.0.0 Signed-off-by: Greeshma Rajendran --- .bumpversion.toml | 2 +- README.md | 6 +++--- docs/chapters/01_welcome/installation.rst | 6 +++--- docs/conf.py | 4 ++-- setup.py | 2 +- src/wxdi/version.py | 2 +- 6 files changed, 11 insertions(+), 11 deletions(-) diff --git a/.bumpversion.toml b/.bumpversion.toml index 9c72852..d9dbdad 100644 --- a/.bumpversion.toml +++ b/.bumpversion.toml @@ -12,7 +12,7 @@ # limitations under the License. [tool.bumpversion] -current_version = "1.0.0" +current_version = "2.0.0" commit = true message = "Update version {current_version} -> {new_version}" diff --git a/README.md b/README.md index ae0c783..9040d1a 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ See the License for the specific language governing permissions and limitations under the License. --> -# IBM watsonx.data intelligence SDK Version 1.0.0 +# IBM watsonx.data intelligence SDK Version 2.0.0 A comprehensive Python SDK for data intelligence operations including: - **Data Quality Validation**: Validate streaming data records, Pandas DataFrames, and PySpark DataFrames @@ -295,7 +295,7 @@ container_response = dph_service.initialize( # Create a data product data_product = dph_service.create_data_product( drafts=[{ - 'version': '1.0.0', + 'version': '2.0.0', 'name': 'My Data Product', 'description': 'A sample data product', 'asset': { @@ -1186,5 +1186,5 @@ For issues, questions, or contributions, please open an issue on GitHub. - pytest-cov >= 4.0.0 - pytest-mock >= 3.7.0 - black >= 26.3.1 -- mypy >= 1.0.0 +- mypy >= 2.0.0 diff --git a/docs/chapters/01_welcome/installation.rst b/docs/chapters/01_welcome/installation.rst index 2e55d91..0ca04d3 100644 --- a/docs/chapters/01_welcome/installation.rst +++ b/docs/chapters/01_welcome/installation.rst @@ -100,7 +100,7 @@ To verify that the SDK is installed correctly: >>> import wxdi.dq_validator >>> from wxdi.common.auth import AuthProvider >>> print(dq_validator.__version__) - 1.0.0 + 2.0.0 Versioning ---------- @@ -116,7 +116,7 @@ Version numbers follow the format ``MAJOR.MINOR.PATCH``: Current Version ~~~~~~~~~~~~~~~ -The current version of the SDK is **1.0.0**. +The current version of the SDK is **2.0.0**. Checking Your Version ~~~~~~~~~~~~~~~~~~~~~ @@ -133,7 +133,7 @@ Or programmatically: >>> import wxdi.dq_validator >>> print(dq_validator.__version__) - 1.0.0 + 2.0.0 Upgrading --------- diff --git a/docs/conf.py b/docs/conf.py index 363a656..917b9ea 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -38,9 +38,9 @@ # the built documents. # # The short X.Y version. -version = "1.0.0" +version = "2.0.0" # The full version, including alpha/beta/rc tags. -release = "1.0.0" +release = "2.0.0" # -- General configuration --------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration diff --git a/setup.py b/setup.py index 0f2301a..5c928a1 100644 --- a/setup.py +++ b/setup.py @@ -27,7 +27,7 @@ setup( name="data-intelligence-sdk", - version='1.0.0', + version='2.0.0', author="IBM", author_email="Data_Intelligence_SDK@wwpdl.vnet.ibm.com", description="A Python SDK for IBM watsonx.data intelligence that provides data quality validation for streaming records and DataFrames, " \ diff --git a/src/wxdi/version.py b/src/wxdi/version.py index 9ab7ad4..52a54fa 100644 --- a/src/wxdi/version.py +++ b/src/wxdi/version.py @@ -16,4 +16,4 @@ """ Version of IBM watsonx.data intelligence SDK """ -__version__ = '1.0.0' \ No newline at end of file +__version__ = '2.0.0' \ No newline at end of file From 507f69a014babcf2fea8e0fa12ad932b5365f899 Mon Sep 17 00:00:00 2001 From: semantic-release-bot Date: Wed, 22 Apr 2026 05:58:24 +0000 Subject: [PATCH 21/33] chore(release): 2.0.0-rc.1 release notes # [2.0.0-rc.1](https://github.com/IBM/data-intelligence-sdk/compare/v1.0.0...v2.0.0-rc.1) (2026-04-22) ### Perf * Sync from enterprise 4e91aee (via .ignore) on 2026-04-17 ([#7](https://github.com/IBM/data-intelligence-sdk/issues/7)) ([eeeffba](https://github.com/IBM/data-intelligence-sdk/commit/eeeffba17ce0e399446099d47352f7738259feae)) ### BREAKING CHANGES * Bug fixes for Data Quality and addition of Data Product Hub modules Signed-off-by: Koichi Nishitani * fix relative path of Actions scripts Signed-off-by: Koichi Nishitani * fix script condition Signed-off-by: Koichi Nishitani * fix script condition again Signed-off-by: Koichi Nishitani * add missing Makefile and fix documentation Signed-off-by: Koichi Nishitani * add dq tests and upgrade python Signed-off-by: Koichi Nishitani * try to extend build timeout Signed-off-by: Koichi Nishitani * upgrade to python 3.10 as 3.9 is EOL Signed-off-by: Koichi Nishitani * modify constraint_model to use custom StrEnum in python 3.10 Signed-off-by: Koichi Nishitani * add pylint Signed-off-by: Koichi Nishitani * skip linting until ready Signed-off-by: Koichi Nishitani * make sure to build before checking Signed-off-by: Koichi Nishitani Signed-off-by: Greeshma Rajendran --- CHANGELOG.md | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index aa2844f..a42bb2b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,61 @@ +# [2.0.0-rc.1](https://github.com/IBM/data-intelligence-sdk/compare/v1.0.0...v2.0.0-rc.1) (2026-04-22) + + +### Perf + +* Sync from enterprise 4e91aee (via .ignore) on 2026-04-17 ([#7](https://github.com/IBM/data-intelligence-sdk/issues/7)) ([eeeffba](https://github.com/IBM/data-intelligence-sdk/commit/eeeffba17ce0e399446099d47352f7738259feae)) + + +### BREAKING CHANGES + +* Bug fixes for Data Quality and addition of Data Product Hub modules + +Signed-off-by: Koichi Nishitani + +* fix relative path of Actions scripts + +Signed-off-by: Koichi Nishitani + +* fix script condition + +Signed-off-by: Koichi Nishitani + +* fix script condition again + +Signed-off-by: Koichi Nishitani + +* add missing Makefile and fix documentation + +Signed-off-by: Koichi Nishitani + +* add dq tests and upgrade python + +Signed-off-by: Koichi Nishitani + +* try to extend build timeout + +Signed-off-by: Koichi Nishitani + +* upgrade to python 3.10 as 3.9 is EOL + +Signed-off-by: Koichi Nishitani + +* modify constraint_model to use custom StrEnum in python 3.10 + +Signed-off-by: Koichi Nishitani + +* add pylint + +Signed-off-by: Koichi Nishitani + +* skip linting until ready + +Signed-off-by: Koichi Nishitani + +* make sure to build before checking + +Signed-off-by: Koichi Nishitani + # [1.0.0](https://github.com/IBM/data-intelligence-sdk/compare/v0.5.3...v1.0.0) (2026-03-17) From 8b2c9ffb95af2d640bd232326414dde8e8109186 Mon Sep 17 00:00:00 2001 From: Koichi Nishitani Date: Thu, 23 Apr 2026 18:15:17 +0900 Subject: [PATCH 22/33] chore: enable rc versions (#12) Signed-off-by: Koichi Nishitani Signed-off-by: Greeshma Rajendran --- .bumpversion.toml | 21 +++++++++++- .github/workflows/deploy-package.yaml | 46 +++++++++++++-------------- 2 files changed, 43 insertions(+), 24 deletions(-) diff --git a/.bumpversion.toml b/.bumpversion.toml index d9dbdad..86b135e 100644 --- a/.bumpversion.toml +++ b/.bumpversion.toml @@ -12,9 +12,28 @@ # limitations under the License. [tool.bumpversion] -current_version = "2.0.0" +current_version = "2.0.0-rc.1" commit = true message = "Update version {current_version} -> {new_version}" +parse = """(?x) + (?P0|[1-9]\\d*)\\. + (?P0|[1-9]\\d*)\\. + (?P0|[1-9]\\d*) + (?: + - # dash separator for pre-release section + (?Prc|stable) # pre-release label + \\.(?P0|[1-9]\\d*) # pre-release version number + )? # pre-release section is optional +""" + +serialize = [ + "{major}.{minor}.{patch}-{pre_label}.{pre_num}", + "{major}.{minor}.{patch}", +] + +[tool.bumpversion.parts.pre_label] +values = ["rc", "stable"] +optional_value = "stable" [[tool.bumpversion.files]] filename = "src/wxdi/version.py" diff --git a/.github/workflows/deploy-package.yaml b/.github/workflows/deploy-package.yaml index 50ff69f..4a8dced 100644 --- a/.github/workflows/deploy-package.yaml +++ b/.github/workflows/deploy-package.yaml @@ -79,26 +79,26 @@ jobs: name: Build and deploy documentation uses: ./.github/workflows/docs.yml - update-sandbox: - if: startsWith(github.ref, 'refs/tags/') && !contains(github.ref, '-rc.') - name: Update sandbox branch - needs: build-documentation - runs-on: ubuntu-latest - steps: - - name: Checkout main - uses: actions/checkout@v6 - with: - persist-credentials: false - fetch-depth: 0 - ref: 'refs/heads/main' - - - name: Also get the sandbox branch - run: | - git remote set-branches --add origin sandbox - git fetch - - name: Checkout main - run: git checkout sandbox - - name: Merge main into sandbox - run: git merge --signoff main - - name: Push changes to sandbox - run: git push -v origin +# update-sandbox: +# if: startsWith(github.ref, 'refs/tags/') && !contains(github.ref, '-rc.') +# name: Update sandbox branch +# needs: build-documentation +# runs-on: ubuntu-latest +# steps: +# - name: Checkout main +# uses: actions/checkout@v6 +# with: +# persist-credentials: false +# fetch-depth: 0 +# ref: 'refs/heads/main' +# +# - name: Also get the sandbox branch +# run: | +# git remote set-branches --add origin sandbox +# git fetch +# - name: Checkout main +# run: git checkout sandbox +# - name: Merge main into sandbox +# run: git merge --signoff main +# - name: Push changes to sandbox +# run: git push -v origin From bf6079a61b887847c45ed8022c00ea14293a3250 Mon Sep 17 00:00:00 2001 From: Koichi Nishitani Date: Fri, 24 Apr 2026 00:57:43 +0900 Subject: [PATCH 23/33] chore: trigger versioning Signed-off-by: Koichi Nishitani Signed-off-by: Greeshma Rajendran --- .bumpversion.toml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.bumpversion.toml b/.bumpversion.toml index 86b135e..1d0c8e7 100644 --- a/.bumpversion.toml +++ b/.bumpversion.toml @@ -12,9 +12,10 @@ # limitations under the License. [tool.bumpversion] -current_version = "2.0.0-rc.1" +current_version = "1.0.0" commit = true message = "Update version {current_version} -> {new_version}" +ignore_missing_version = true parse = """(?x) (?P0|[1-9]\\d*)\\. (?P0|[1-9]\\d*)\\. From da6144b0a6120ebc11732ebacb29d419e5f08d13 Mon Sep 17 00:00:00 2001 From: semantic-release-bot Date: Thu, 23 Apr 2026 16:11:42 +0000 Subject: [PATCH 24/33] Update version 1.0.0 -> 2.0.0 Signed-off-by: Greeshma Rajendran --- .bumpversion.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.bumpversion.toml b/.bumpversion.toml index 1d0c8e7..e50d6af 100644 --- a/.bumpversion.toml +++ b/.bumpversion.toml @@ -12,7 +12,7 @@ # limitations under the License. [tool.bumpversion] -current_version = "1.0.0" +current_version = "2.0.0" commit = true message = "Update version {current_version} -> {new_version}" ignore_missing_version = true From 777ce8dc234037a64d7694d0b79955db40bb8804 Mon Sep 17 00:00:00 2001 From: semantic-release-bot Date: Thu, 23 Apr 2026 16:11:42 +0000 Subject: [PATCH 25/33] chore(release): 2.0.0 release notes # [2.0.0](https://github.com/IBM/data-intelligence-sdk/compare/v1.0.0...v2.0.0) (2026-04-23) * Upgrade to version 2.0.0 ([#13](https://github.com/IBM/data-intelligence-sdk/issues/13)) ([41f25ce](https://github.com/IBM/data-intelligence-sdk/commit/41f25cefc349a9d2add7d99bc94945d123104ca1)), closes [#7](https://github.com/IBM/data-intelligence-sdk/issues/7) ### BREAKING CHANGES * Bug fixes for Data Quality and addition of Data Product Hub modules Signed-off-by: Koichi Nishitani * fix relative path of Actions scripts Signed-off-by: Koichi Nishitani * fix script condition Signed-off-by: Koichi Nishitani * fix script condition again Signed-off-by: Koichi Nishitani * add missing Makefile and fix documentation Signed-off-by: Koichi Nishitani * add dq tests and upgrade python Signed-off-by: Koichi Nishitani * try to extend build timeout Signed-off-by: Koichi Nishitani * upgrade to python 3.10 as 3.9 is EOL Signed-off-by: Koichi Nishitani * modify constraint_model to use custom StrEnum in python 3.10 Signed-off-by: Koichi Nishitani * add pylint Signed-off-by: Koichi Nishitani * skip linting until ready Signed-off-by: Koichi Nishitani * make sure to build before checking Signed-off-by: Koichi Nishitani Signed-off-by: Greeshma Rajendran --- CHANGELOG.md | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index a42bb2b..2aff2e4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,59 @@ +# [2.0.0](https://github.com/IBM/data-intelligence-sdk/compare/v1.0.0...v2.0.0) (2026-04-23) + + +* Upgrade to version 2.0.0 ([#13](https://github.com/IBM/data-intelligence-sdk/issues/13)) ([41f25ce](https://github.com/IBM/data-intelligence-sdk/commit/41f25cefc349a9d2add7d99bc94945d123104ca1)), closes [#7](https://github.com/IBM/data-intelligence-sdk/issues/7) + + +### BREAKING CHANGES + +* Bug fixes for Data Quality and addition of Data Product Hub modules + +Signed-off-by: Koichi Nishitani + +* fix relative path of Actions scripts + +Signed-off-by: Koichi Nishitani + +* fix script condition + +Signed-off-by: Koichi Nishitani + +* fix script condition again + +Signed-off-by: Koichi Nishitani + +* add missing Makefile and fix documentation + +Signed-off-by: Koichi Nishitani + +* add dq tests and upgrade python + +Signed-off-by: Koichi Nishitani + +* try to extend build timeout + +Signed-off-by: Koichi Nishitani + +* upgrade to python 3.10 as 3.9 is EOL + +Signed-off-by: Koichi Nishitani + +* modify constraint_model to use custom StrEnum in python 3.10 + +Signed-off-by: Koichi Nishitani + +* add pylint + +Signed-off-by: Koichi Nishitani + +* skip linting until ready + +Signed-off-by: Koichi Nishitani + +* make sure to build before checking + +Signed-off-by: Koichi Nishitani + # [2.0.0-rc.1](https://github.com/IBM/data-intelligence-sdk/compare/v1.0.0...v2.0.0-rc.1) (2026-04-22) From 68edc598613dfd418661ad227a42a4dfcd273483 Mon Sep 17 00:00:00 2001 From: Koichi Nishitani Date: Fri, 24 Apr 2026 11:27:38 +0900 Subject: [PATCH 26/33] chore: enable workflow_call documentation publish Signed-off-by: Koichi Nishitani Signed-off-by: Greeshma Rajendran --- .github/workflows/docs.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index a22c199..6cf2651 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -65,7 +65,7 @@ jobs: deploy: # Only deploy on push to main/master, not on PRs - if: github.event_name == 'push' && github.ref == 'refs/heads/main' + if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main' environment: name: github-pages From b102de5f99989a685e42a4774845a515d34617f8 Mon Sep 17 00:00:00 2001 From: Greeshma Rajendran Date: Wed, 6 May 2026 17:28:28 +0530 Subject: [PATCH 27/33] [skip ci] dph documentation update, docs(api): updated DPH documentation (#14) Signed-off-by: Greeshma Rajendran --- docs/api/data_product_recommender/index.rst | 68 ++ docs/api/dph_services/core.rst | 57 ++ docs/api/dph_services/index.rst | 108 +++ docs/api/index.rst | 33 + docs/api/odcs_generator/index.rst | 53 ++ docs/chapters/05_dph_services/examples.rst | 514 +++++++++++++++ docs/chapters/05_dph_services/index.rst | 115 ++++ docs/chapters/05_dph_services/overview.rst | 266 ++++++++ docs/chapters/05_dph_services/usage_guide.rst | 614 ++++++++++++++++++ .../collibra_integration.rst | 113 ++++ docs/chapters/06_odcs_generator/examples.rst | 98 +++ docs/chapters/06_odcs_generator/index.rst | 189 ++++++ .../informatica_integration.rst | 93 +++ docs/chapters/06_odcs_generator/overview.rst | 345 ++++++++++ .../07_data_product_recommender/examples.rst | 100 +++ .../07_data_product_recommender/index.rst | 111 ++++ .../07_data_product_recommender/overview.rst | 65 ++ .../usage_guide.rst | 82 +++ .../index.rst | 216 +++--- docs/index.rst | 19 +- 20 files changed, 3149 insertions(+), 110 deletions(-) create mode 100644 docs/api/data_product_recommender/index.rst create mode 100644 docs/api/dph_services/core.rst create mode 100644 docs/api/dph_services/index.rst create mode 100644 docs/api/odcs_generator/index.rst create mode 100644 docs/chapters/05_dph_services/examples.rst create mode 100644 docs/chapters/05_dph_services/index.rst create mode 100644 docs/chapters/05_dph_services/overview.rst create mode 100644 docs/chapters/05_dph_services/usage_guide.rst create mode 100644 docs/chapters/06_odcs_generator/collibra_integration.rst create mode 100644 docs/chapters/06_odcs_generator/examples.rst create mode 100644 docs/chapters/06_odcs_generator/index.rst create mode 100644 docs/chapters/06_odcs_generator/informatica_integration.rst create mode 100644 docs/chapters/06_odcs_generator/overview.rst create mode 100644 docs/chapters/07_data_product_recommender/examples.rst create mode 100644 docs/chapters/07_data_product_recommender/index.rst create mode 100644 docs/chapters/07_data_product_recommender/overview.rst create mode 100644 docs/chapters/07_data_product_recommender/usage_guide.rst rename docs/chapters/{05_future_modules => 08_future_modules}/index.rst (96%) diff --git a/docs/api/data_product_recommender/index.rst b/docs/api/data_product_recommender/index.rst new file mode 100644 index 0000000..713e0f5 --- /dev/null +++ b/docs/api/data_product_recommender/index.rst @@ -0,0 +1,68 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _api_data_product_recommender: + +Data Product Recommender Reference +=================================== + +Class reference for the Data Product Recommender module. + +Core Classes +------------ + +.. currentmodule:: wxdi.data_product_recommender.recommender + +.. autoclass:: DataProductRecommender + :members: + :undoc-members: + :show-inheritance: + +Platform Parsers +---------------- + +.. currentmodule:: wxdi.data_product_recommender.platforms + +.. autoclass:: SnowflakeQueryParser + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: DatabricksQueryParser + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: BigQueryQueryParser + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: WatsonxDataQueryParser + :members: + :undoc-members: + :show-inheritance: + +Base Classes +------------ + +.. currentmodule:: wxdi.data_product_recommender.base + +.. autoclass:: QueryLogParser + :members: + :undoc-members: + :show-inheritance: + +.. Made with Bob diff --git a/docs/api/dph_services/core.rst b/docs/api/dph_services/core.rst new file mode 100644 index 0000000..2009b39 --- /dev/null +++ b/docs/api/dph_services/core.rst @@ -0,0 +1,57 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _api_dph_services_core: + +Core Classes +============ + +Main service class and data models for Data Product Hub Services. + +DphV1 Service +------------- + +.. currentmodule:: wxdi.dph_services + +.. autoclass:: DphV1 + :members: + :undoc-members: + :show-inheritance: + :inherited-members: + +Common Models +------------- + +.. autoclass:: wxdi.dph_services.common.DataProduct + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: wxdi.dph_services.common.DataProductDraft + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: wxdi.dph_services.common.ContractTerms + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: wxdi.dph_services.common.Domain + :members: + :undoc-members: + :show-inheritance: + +.. Made with Bob \ No newline at end of file diff --git a/docs/api/dph_services/index.rst b/docs/api/dph_services/index.rst new file mode 100644 index 0000000..d9adaa6 --- /dev/null +++ b/docs/api/dph_services/index.rst @@ -0,0 +1,108 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _api_dph_services: + +DPH Services API +================ + +API reference for the Data Product Hub Services module. + +.. toctree:: + :maxdepth: 2 + + core + +Main Service Class +------------------ + +.. currentmodule:: wxdi.dph_services + +.. autoclass:: DphV1 + :members: + :undoc-members: + :show-inheritance: + :special-members: __init__ + +Container Operations +-------------------- + +.. automethod:: DphV1.initialize +.. automethod:: DphV1.get_initialize_status +.. automethod:: DphV1.get_service_id_credentials +.. automethod:: DphV1.manage_api_keys + +Data Product Operations +----------------------- + +.. automethod:: DphV1.create_data_product +.. automethod:: DphV1.list_data_products +.. automethod:: DphV1.list_data_products_with_pager +.. automethod:: DphV1.get_data_product +.. automethod:: DphV1.update_data_product +.. automethod:: DphV1.delete_data_product + +Draft Operations +---------------- + +.. automethod:: DphV1.create_data_product_draft +.. automethod:: DphV1.list_data_product_drafts +.. automethod:: DphV1.get_data_product_draft +.. automethod:: DphV1.update_data_product_draft +.. automethod:: DphV1.delete_data_product_draft +.. automethod:: DphV1.publish_data_product_draft + +Release Operations +------------------ + +.. automethod:: DphV1.list_data_product_releases +.. automethod:: DphV1.get_data_product_release +.. automethod:: DphV1.update_data_product_release +.. automethod:: DphV1.retire_data_product_release + +Contract Terms Operations +------------------------- + +.. automethod:: DphV1.create_draft_contract_terms_document +.. automethod:: DphV1.get_data_product_draft_contract_terms +.. automethod:: DphV1.update_draft_contract_terms_document +.. automethod:: DphV1.delete_draft_contract_terms_document + +Domain Operations +----------------- + +.. automethod:: DphV1.list_data_product_domains +.. automethod:: DphV1.create_data_product_domain +.. automethod:: DphV1.create_data_product_subdomain +.. automethod:: DphV1.get_domain +.. automethod:: DphV1.update_data_product_domain +.. automethod:: DphV1.delete_domain + +Asset Visualization Operations +------------------------------- + +.. automethod:: DphV1.create_data_asset_visualization +.. automethod:: DphV1.reinitiate_data_asset_visualization + +Contract Template Operations +---------------------------- + +.. automethod:: DphV1.create_contract_template +.. automethod:: DphV1.list_data_product_contract_template +.. automethod:: DphV1.get_contract_template +.. automethod:: DphV1.update_data_product_contract_template +.. automethod:: DphV1.delete_data_product_contract_template + +.. Made with Bob \ No newline at end of file diff --git a/docs/api/index.rst b/docs/api/index.rst index bab9877..6455d0f 100644 --- a/docs/api/index.rst +++ b/docs/api/index.rst @@ -27,6 +27,9 @@ This API reference documentation is auto-generated from the source code docstrin common/index dq_validator/index + dph_services/index + odcs_generator/index + data_product_recommender/index Module Organization ------------------- @@ -50,6 +53,36 @@ In-memory data quality validation: * :ref:`REST API Providers` - IBM Cloud Pak for Data integration * :ref:`Result Consolidation` - Result aggregation and analysis +DPH Services Module +~~~~~~~~~~~~~~~~~~~ + +Data Product Hub API client: + +* :ref:`DphV1 Service` - Main service class for Data Product Hub operations +* Container, data product, draft, release, and domain management +* Contract terms and template operations +* Asset visualization + +ODCS Generator Module +~~~~~~~~~~~~~~~~~~~~~ + +Generate Open Data Contract Standard files: + +* :ref:`Collibra Integration` - CollibraClient and ODCSGenerator classes +* :ref:`Informatica Integration` - InformaticaClient and ODCSGenerator classes +* ODCS v3.1.0 compliant YAML generation +* Command-line and Python interfaces + +Data Product Recommender Module +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Query log analysis tool for data product recommendations: + +* :ref:`DataProductRecommender` - Core recommendation engine +* Platform-specific query log parsers (Snowflake, Databricks, BigQuery, watsonx.data) +* Scoring and ranking algorithms +* CLI and Python interfaces + Navigation Tips --------------- diff --git a/docs/api/odcs_generator/index.rst b/docs/api/odcs_generator/index.rst new file mode 100644 index 0000000..d7e3203 --- /dev/null +++ b/docs/api/odcs_generator/index.rst @@ -0,0 +1,53 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _api_odcs_generator: + +ODCS Generator Reference +======================== + +Class reference for the ODCS Generator module. + +Collibra Integration +-------------------- + +.. currentmodule:: wxdi.odcs_generator.generate_odcs_from_collibra + +.. autoclass:: CollibraClient + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: ODCSGenerator + :members: + :undoc-members: + :show-inheritance: + +Informatica Integration +----------------------- + +.. currentmodule:: wxdi.odcs_generator.generate_odcs_from_informatica + +.. autoclass:: InformaticaClient + :members: + :undoc-members: + :show-inheritance: + +.. autoclass:: ODCSGenerator + :members: + :undoc-members: + :show-inheritance: + +.. Made with Bob diff --git a/docs/chapters/05_dph_services/examples.rst b/docs/chapters/05_dph_services/examples.rst new file mode 100644 index 0000000..2d277c9 --- /dev/null +++ b/docs/chapters/05_dph_services/examples.rst @@ -0,0 +1,514 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _dph_services_examples: + +Examples +======== + +Complete examples demonstrating common use cases for the Data Product Hub Services module. + +Complete Workflow Example +-------------------------- + +This example demonstrates the complete lifecycle of a data product: + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + from ibm_cloud_sdk_core import ApiException + + # Initialize service + authenticator = IAMAuthenticator('your-api-key') + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + + # Step 1: Initialize container + print("Initializing container...") + init_response = dph_service.initialize( + include=['delivery_methods', 'data_product_samples', 'domains_multi_industry'] + ) + container_id = init_response.result['container']['id'] + print(f"Container initialized: {container_id}") + + # Step 2: Create a domain + print("\nCreating domain...") + domain = dph_service.create_data_product_domain( + name='Customer Analytics', + description='Customer behavior and analytics data products', + container={'id': container_id} + ) + domain_id = domain.result['id'] + print(f"Domain created: {domain_id}") + + # Step 3: Create data product with draft + print("\nCreating data product...") + data_product = dph_service.create_data_product( + drafts=[{ + 'version': '1.0.0', + 'name': 'Customer Purchase History', + 'description': 'Historical customer purchase data for analytics', + 'asset': { + 'id': 'asset-12345', + 'container': {'id': container_id} + }, + 'domain': { + 'id': domain_id, + 'name': 'Customer Analytics' + }, + 'parts_out': [{ + 'asset': { + 'id': 'asset-12345', + 'container': {'id': container_id} + }, + 'delivery_methods': [{ + 'id': 'delivery-method-001', + 'container': {'id': container_id} + }] + }] + }] + ) + + product_id = data_product.result['id'] + draft_id = data_product.result['drafts'][0]['id'] + print(f"Data product created: {product_id}") + print(f"Draft created: {draft_id}") + + # Step 4: Add contract terms + print("\nAdding contract terms...") + contract_terms = dph_service.get_data_product_draft_contract_terms( + data_product_id=product_id, + draft_id=draft_id + ) + + terms_id = contract_terms.result['id'] + + doc = dph_service.create_draft_contract_terms_document( + data_product_id=product_id, + draft_id=draft_id, + contract_terms_id=terms_id, + type='terms_and_conditions', + name='Data Usage Terms', + url='https://example.com/terms.pdf' + ) + print(f"Contract document added: {doc.result['id']}") + + # Step 5: Publish the draft + print("\nPublishing draft...") + release = dph_service.publish_data_product_draft( + data_product_id=product_id, + draft_id=draft_id + ) + release_id = release.result['id'] + print(f"Release published: {release_id}") + + # Step 6: Create a new version + print("\nCreating new version...") + new_draft = dph_service.create_data_product_draft( + data_product_id=product_id, + asset={'id': 'asset-12345', 'container': {'id': container_id}}, + version='1.1.0', + name='Customer Purchase History v1.1', + description='Enhanced with additional purchase metrics' + ) + new_draft_id = new_draft.result['id'] + print(f"New draft created: {new_draft_id}") + + print("\n✅ Complete workflow executed successfully!") + +Batch Operations Example +------------------------- + +Create multiple data products efficiently: + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + + # Initialize service + authenticator = IAMAuthenticator('your-api-key') + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + + # Define multiple data products + products_to_create = [ + { + 'name': 'Customer Demographics', + 'description': 'Customer demographic information', + 'asset_id': 'asset-001' + }, + { + 'name': 'Transaction History', + 'description': 'Historical transaction records', + 'asset_id': 'asset-002' + }, + { + 'name': 'Product Catalog', + 'description': 'Complete product catalog data', + 'asset_id': 'asset-003' + } + ] + + # Create all products + created_products = [] + + for product_info in products_to_create: + try: + product = dph_service.create_data_product( + drafts=[{ + 'version': '1.0.0', + 'name': product_info['name'], + 'description': product_info['description'], + 'asset': { + 'id': product_info['asset_id'], + 'container': {'id': 'container-123'} + } + }] + ) + created_products.append(product.result) + print(f"✅ Created: {product_info['name']}") + except Exception as e: + print(f"❌ Failed to create {product_info['name']}: {e}") + + print(f"\nTotal products created: {len(created_products)}") + +Pagination Example +------------------ + +Handle large datasets with pagination: + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + + # Initialize service + authenticator = IAMAuthenticator('your-api-key') + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + + # Method 1: Using pager (recommended) + print("Fetching all data products using pager...") + all_products = [] + pager = dph_service.list_data_products_with_pager(limit=50) + + for page in pager: + all_products.extend(page['data_products']) + print(f"Fetched {len(page['data_products'])} products...") + + print(f"Total products: {len(all_products)}") + + # Method 2: Manual pagination + print("\nManual pagination example...") + all_products_manual = [] + start = None + + while True: + response = dph_service.list_data_products( + limit=50, + start=start + ) + + products = response.result['data_products'] + all_products_manual.extend(products) + + # Check if there are more pages + if 'next' not in response.result or not response.result['next']: + break + + # Extract start token from next link + start = response.result['next'].get('start') + + print(f"Total products (manual): {len(all_products_manual)}") + +Error Handling Example +---------------------- + +Robust error handling for production use: + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + from ibm_cloud_sdk_core import ApiException + import time + + # Initialize service + authenticator = IAMAuthenticator('your-api-key') + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + + def create_data_product_with_retry(dph_service, drafts, max_retries=3): + """Create data product with retry logic""" + for attempt in range(max_retries): + try: + response = dph_service.create_data_product(drafts=drafts) + return response + except ApiException as e: + if e.code == 429: # Rate limit + wait_time = 2 ** attempt + print(f"Rate limited. Waiting {wait_time}s before retry...") + time.sleep(wait_time) + elif e.code >= 500: # Server error + if attempt < max_retries - 1: + print(f"Server error. Retrying... (attempt {attempt + 1})") + time.sleep(2 ** attempt) + else: + raise + elif e.code == 404: + print(f"Resource not found: {e.message}") + raise + elif e.code == 401: + print("Authentication failed. Check your credentials.") + raise + elif e.code == 403: + print("Insufficient permissions.") + raise + else: + print(f"API Error {e.code}: {e.message}") + raise + + raise Exception("Max retries exceeded") + + # Use the retry function + try: + drafts = [{ + 'version': '1.0.0', + 'name': 'Test Product', + 'description': 'Test description', + 'asset': {'id': 'asset-123', 'container': {'id': 'container-456'}} + }] + + product = create_data_product_with_retry(dph_service, drafts) + print(f"✅ Product created: {product.result['id']}") + except Exception as e: + print(f"❌ Failed to create product: {e}") + +Search and Filter Example +-------------------------- + +Find specific data products: + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + + # Initialize service + authenticator = IAMAuthenticator('your-api-key') + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + + # Get all products and filter + all_products = [] + pager = dph_service.list_data_products_with_pager(limit=100) + + for page in pager: + all_products.extend(page['data_products']) + + # Filter by name pattern + customer_products = [ + p for p in all_products + if 'customer' in p['name'].lower() + ] + print(f"Found {len(customer_products)} customer-related products") + + # Filter by domain + analytics_products = [ + p for p in all_products + if p.get('domain', {}).get('name') == 'Customer Analytics' + ] + print(f"Found {len(analytics_products)} analytics products") + + # Filter by version + v1_products = [ + p for p in all_products + if p['version'].startswith('1.') + ] + print(f"Found {len(v1_products)} v1.x products") + +Contract Template Example +-------------------------- + +Create and use contract templates: + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + + # Initialize service + authenticator = IAMAuthenticator('your-api-key') + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + + # Create a reusable contract template + template = dph_service.create_contract_template( + name='Standard Data Sharing Agreement', + description='Standard terms for internal data sharing', + contract_terms_documents=[ + { + 'type': 'terms_and_conditions', + 'name': 'Terms and Conditions', + 'url': 'https://example.com/standard-terms.pdf' + }, + { + 'type': 'sla', + 'name': 'Service Level Agreement', + 'url': 'https://example.com/sla.pdf' + } + ] + ) + + template_id = template.result['id'] + print(f"Template created: {template_id}") + + # Use template when creating data products + data_product = dph_service.create_data_product( + drafts=[{ + 'version': '1.0.0', + 'name': 'Sales Data', + 'description': 'Monthly sales data', + 'asset': {'id': 'asset-123', 'container': {'id': 'container-456'}}, + 'contract_terms': { + 'template_id': template_id + } + }] + ) + + print(f"Data product created with template: {data_product.result['id']}") + +Domain Hierarchy Example +------------------------ + +Create and manage domain hierarchies: + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + + # Initialize service + authenticator = IAMAuthenticator('your-api-key') + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + + # Create parent domain + parent_domain = dph_service.create_data_product_domain( + name='Customer Data', + description='All customer-related data products', + container={'id': 'container-123'} + ) + parent_id = parent_domain.result['id'] + print(f"Parent domain created: {parent_id}") + + # Create subdomains + subdomains = [ + {'name': 'Demographics', 'description': 'Customer demographic data'}, + {'name': 'Behavior', 'description': 'Customer behavior analytics'}, + {'name': 'Transactions', 'description': 'Customer transaction history'} + ] + + for subdomain_info in subdomains: + subdomain = dph_service.create_data_product_subdomain( + domain_id=parent_id, + name=subdomain_info['name'], + description=subdomain_info['description'] + ) + print(f"Subdomain created: {subdomain_info['name']}") + + # List all domains with hierarchy + domains = dph_service.list_data_product_domains(limit=100) + + for domain in domains.result['domains']: + print(f"\n{domain['name']}") + if 'subdomains' in domain: + for subdomain in domain['subdomains']: + print(f" └─ {subdomain['name']}") + +Monitoring and Reporting Example +--------------------------------- + +Generate reports on data product usage: + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + from collections import defaultdict + from datetime import datetime + + # Initialize service + authenticator = IAMAuthenticator('your-api-key') + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + + # Collect all data products + all_products = [] + pager = dph_service.list_data_products_with_pager(limit=100) + + for page in pager: + all_products.extend(page['data_products']) + + # Generate statistics + stats = { + 'total_products': len(all_products), + 'by_domain': defaultdict(int), + 'by_version': defaultdict(int), + 'by_state': defaultdict(int) + } + + for product in all_products: + # Count by domain + domain_name = product.get('domain', {}).get('name', 'Unknown') + stats['by_domain'][domain_name] += 1 + + # Count by version + version = product.get('version', 'Unknown') + major_version = version.split('.')[0] if '.' in version else version + stats['by_version'][f"v{major_version}.x"] += 1 + + # Count by state + state = product.get('state', 'Unknown') + stats['by_state'][state] += 1 + + # Print report + print("=" * 50) + print("DATA PRODUCT REPORT") + print("=" * 50) + print(f"\nTotal Data Products: {stats['total_products']}") + + print("\nBy Domain:") + for domain, count in sorted(stats['by_domain'].items()): + print(f" {domain}: {count}") + + print("\nBy Version:") + for version, count in sorted(stats['by_version'].items()): + print(f" {version}: {count}") + + print("\nBy State:") + for state, count in sorted(stats['by_state'].items()): + print(f" {state}: {count}") + +See Also +-------- + +- :ref:`dph_services_usage` - Detailed usage guide +- :ref:`api_dph_services` - API reference +- :ref:`dph_services_overview` - Architecture overview + +.. Made with Bob diff --git a/docs/chapters/05_dph_services/index.rst b/docs/chapters/05_dph_services/index.rst new file mode 100644 index 0000000..59a1455 --- /dev/null +++ b/docs/chapters/05_dph_services/index.rst @@ -0,0 +1,115 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _dph_services: + +Data Product Hub Services +========================== + +Python client library for IBM Data Product Hub API, providing programmatic access to data product management, container operations, contract terms, and asset visualization. + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + overview + usage_guide + examples + +Overview +-------- + +The ``dph_services`` module provides a complete Python SDK for interacting with IBM Data Product Hub services. It enables developers to programmatically manage the entire data product lifecycle, from initialization to publication and retirement. + +Key Features +------------ + +**Container Management** + Initialize and configure data product containers with delivery methods, samples, and domain structures. + +**Data Product Lifecycle** + Create, update, publish, and retire data products with full version control and draft management. + +**Contract Terms** + Manage contract terms, documents, and templates for data product agreements. + +**Asset Visualization** + Create and manage data asset visualizations for better data discovery. + +**Domain Organization** + Organize data products into domains and subdomains for better categorization. + +**Release Management** + Handle data product releases with versioning and retirement capabilities. + +Quick Start +----------- + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + + # Initialize authenticator + authenticator = IAMAuthenticator('your-api-key') + + # Create service instance + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + + # Initialize container + response = dph_service.initialize( + include=['delivery_methods', 'data_product_samples', 'domains_multi_industry'] + ) + + # Create a data product + data_product = dph_service.create_data_product( + drafts=[{ + 'version': '1.0.0', + 'name': 'Customer Analytics Data Product', + 'description': 'Comprehensive customer analytics dataset', + 'asset': { + 'id': 'asset-123', + 'container': {'id': 'container-456'} + } + }] + ) + +Use Cases +--------- + +**Data Product Onboarding** + Automate the creation and configuration of new data products in your data marketplace. + +**Lifecycle Automation** + Build workflows that automatically promote drafts to releases based on quality checks. + +**Contract Management** + Programmatically manage data sharing agreements and terms of use. + +**Catalog Integration** + Integrate with data catalogs to automatically create data products from existing assets. + +**Governance Workflows** + Implement approval workflows and governance policies for data product publication. + +Next Steps +---------- + +- :ref:`dph_services_usage` - Detailed usage guide with examples +- :ref:`dph_services_examples` - Practical code examples +- :ref:`api_dph_services` - Complete API reference + +.. Made with Bob diff --git a/docs/chapters/05_dph_services/overview.rst b/docs/chapters/05_dph_services/overview.rst new file mode 100644 index 0000000..ba2042e --- /dev/null +++ b/docs/chapters/05_dph_services/overview.rst @@ -0,0 +1,266 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _dph_services_overview: + +DPH Services Overview +===================== + +The Data Product Hub Services module provides a comprehensive Python SDK for managing data products in IBM Data Product Hub. + +Architecture +------------ + +The module is built on top of the IBM Cloud SDK Core and provides: + +- **Type-safe API**: Full type hints for better IDE support +- **Error handling**: Comprehensive exception handling with detailed error messages +- **Pagination support**: Built-in pagination for large result sets +- **Authentication**: Seamless integration with IBM Cloud authentication + +Core Components +--------------- + +Container Management +~~~~~~~~~~~~~~~~~~~~ + +Containers are the foundation of Data Product Hub, providing: + +- Delivery method configurations +- Sample data products +- Domain structures +- Service credentials + +Data Products +~~~~~~~~~~~~~ + +Data products represent packaged data assets with: + +- Metadata and descriptions +- Version control +- Asset references +- Domain associations +- Contract terms + +Drafts and Releases +~~~~~~~~~~~~~~~~~~~ + +**Drafts**: Work-in-progress versions that can be edited and updated + +**Releases**: Published versions that are immutable and available for consumption + +Contract Terms +~~~~~~~~~~~~~~ + +Legal and business terms governing data product usage: + +- Terms and conditions documents +- Service level agreements +- Usage restrictions +- Compliance requirements + +Domains +~~~~~~~ + +Organizational structure for categorizing data products: + +- Top-level domains (e.g., Finance, Marketing) +- Subdomains for finer categorization +- Multi-industry domain support + +Data Flow +--------- + +1. **Initialize Container**: Set up the data product hub environment +2. **Create Draft**: Define a new data product version +3. **Add Contract Terms**: Attach legal and business terms +4. **Publish Draft**: Convert draft to a release +5. **Manage Lifecycle**: Update, version, or retire releases + +Authentication +-------------- + +The module supports multiple authentication methods: + +**IAM Authentication** (Recommended) + +.. code-block:: python + + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + + authenticator = IAMAuthenticator('your-api-key') + dph_service = DphV1(authenticator=authenticator) + +**Bearer Token Authentication** + +.. code-block:: python + + from ibm_cloud_sdk_core.authenticators import BearerTokenAuthenticator + + authenticator = BearerTokenAuthenticator('your-bearer-token') + dph_service = DphV1(authenticator=authenticator) + +**Cloud Pak for Data Authentication** + +.. code-block:: python + + from ibm_cloud_sdk_core.authenticators import CloudPakForDataAuthenticator + + authenticator = CloudPakForDataAuthenticator( + username='your-username', + password='your-password', + url='https://your-cpd-instance.com' + ) + dph_service = DphV1(authenticator=authenticator) + +Error Handling +-------------- + +The SDK uses standard IBM Cloud SDK exceptions: + +.. code-block:: python + + from ibm_cloud_sdk_core import ApiException + + try: + response = dph_service.get_data_product(data_product_id='invalid-id') + except ApiException as e: + print(f"Error Code: {e.code}") + print(f"Error Message: {e.message}") + print(f"HTTP Status: {e.http_response.status_code}") + +Common error codes: + +- **400**: Bad Request - Invalid parameters +- **401**: Unauthorized - Authentication failed +- **403**: Forbidden - Insufficient permissions +- **404**: Not Found - Resource doesn't exist +- **409**: Conflict - Resource already exists +- **500**: Internal Server Error - Service error + +Best Practices +-------------- + +**Use Pagination for Large Datasets** + +.. code-block:: python + + # Use pager for automatic pagination + pager = dph_service.list_data_products_with_pager(limit=50) + for page in pager: + for product in page['data_products']: + process_product(product) + +**Implement Retry Logic** + +.. code-block:: python + + import time + from ibm_cloud_sdk_core import ApiException + + def create_with_retry(dph_service, drafts, max_retries=3): + for attempt in range(max_retries): + try: + return dph_service.create_data_product(drafts=drafts) + except ApiException as e: + if e.code == 429 or e.code >= 500: # Rate limit or server error + if attempt < max_retries - 1: + time.sleep(2 ** attempt) # Exponential backoff + continue + raise + +**Validate Before Publishing** + +.. code-block:: python + + def validate_draft(draft): + """Validate draft before publishing""" + required_fields = ['name', 'version', 'asset', 'domain'] + for field in required_fields: + if field not in draft or not draft[field]: + raise ValueError(f"Missing required field: {field}") + return True + +**Use JSON Patch for Updates** + +.. code-block:: python + + # Efficient updates using JSON Patch + patch_operations = [ + {'op': 'replace', 'path': '/description', 'value': 'Updated description'}, + {'op': 'add', 'path': '/tags/-', 'value': 'new-tag'} + ] + + dph_service.update_data_product( + data_product_id=product_id, + json_patch_instructions=patch_operations + ) + +Performance Considerations +-------------------------- + +**Batch Operations** + +When creating multiple data products, consider batching to reduce API calls: + +.. code-block:: python + + # Create multiple drafts in a single data product + dph_service.create_data_product( + drafts=[draft1, draft2, draft3] + ) + +**Caching** + +Cache frequently accessed data to reduce API calls: + +.. code-block:: python + + from functools import lru_cache + + @lru_cache(maxsize=100) + def get_domain_cached(domain_id): + return dph_service.get_domain(domain_id=domain_id) + +**Parallel Processing** + +Use concurrent requests for independent operations: + +.. code-block:: python + + from concurrent.futures import ThreadPoolExecutor + + def get_product(product_id): + return dph_service.get_data_product(data_product_id=product_id) + + with ThreadPoolExecutor(max_workers=5) as executor: + products = list(executor.map(get_product, product_ids)) + +Requirements +------------ + +- Python 3.8 or higher +- ibm-cloud-sdk-core >= 3.16.7 +- requests >= 2.32.4 +- python-dateutil >= 2.5.3 + +See Also +-------- + +- :ref:`dph_services_usage` - Detailed usage guide +- :ref:`dph_services_examples` - Code examples +- :ref:`api_dph_services` - API reference + +.. Made with Bob diff --git a/docs/chapters/05_dph_services/usage_guide.rst b/docs/chapters/05_dph_services/usage_guide.rst new file mode 100644 index 0000000..af9b340 --- /dev/null +++ b/docs/chapters/05_dph_services/usage_guide.rst @@ -0,0 +1,614 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _dph_services_usage: + +Usage Guide +=========== + +This guide provides detailed instructions for using the Data Product Hub Services module. + +Installation +------------ + +Install the data-intelligence-sdk package: + +.. code-block:: bash + + pip install -e . + +Or install from PyPI (when available): + +.. code-block:: bash + + pip install data-intelligence-sdk + +Setup and Configuration +----------------------- + +Initialize the Service +~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + from wxdi.dph_services import DphV1 + from ibm_cloud_sdk_core.authenticators import IAMAuthenticator + + # Create authenticator + authenticator = IAMAuthenticator('your-api-key') + + # Initialize service + dph_service = DphV1(authenticator=authenticator) + dph_service.set_service_url('https://your-dph-instance.com') + +Environment Variables +~~~~~~~~~~~~~~~~~~~~~ + +You can also configure using environment variables: + +.. code-block:: bash + + export DPH_APIKEY=your-api-key + export DPH_URL=https://your-dph-instance.com + +.. code-block:: python + + from wxdi.dph_services import DphV1 + + # Automatically uses environment variables + dph_service = DphV1.new_instance() + +Container Operations +-------------------- + +Initialize Container +~~~~~~~~~~~~~~~~~~~~ + +Initialize a new container with default settings: + +.. code-block:: python + + response = dph_service.initialize( + include=[ + 'delivery_methods', + 'data_product_samples', + 'domains_multi_industry' + ] + ) + + print(f"Container ID: {response.result['container']['id']}") + +Check Initialization Status +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + status = dph_service.get_initialize_status() + + if status.result['status'] == 'SUCCEEDED': + print("Container is ready") + elif status.result['status'] == 'IN_PROGRESS': + print("Initialization in progress") + else: + print(f"Status: {status.result['status']}") + +Get Service Credentials +~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + credentials = dph_service.get_service_id_credentials() + print(f"Service ID: {credentials.result['service_id']}") + +Data Product Management +----------------------- + +Create a Data Product +~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + data_product = dph_service.create_data_product( + drafts=[{ + 'version': '1.0.0', + 'name': 'Customer Analytics Dataset', + 'description': 'Comprehensive customer behavior analytics', + 'asset': { + 'id': 'asset-123', + 'container': {'id': 'container-456'} + }, + 'domain': { + 'id': 'domain-789', + 'name': 'Customer Analytics' + }, + 'parts_out': [{ + 'asset': { + 'id': 'asset-123', + 'container': {'id': 'container-456'} + }, + 'delivery_methods': [{ + 'id': 'method-001', + 'container': {'id': 'container-456'} + }] + }] + }] + ) + + product_id = data_product.result['id'] + print(f"Created data product: {product_id}") + +List Data Products +~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + # List with pagination + response = dph_service.list_data_products(limit=50) + + for product in response.result['data_products']: + print(f"- {product['name']} (v{product['version']})") + + # Use pager for all results + all_products = [] + pager = dph_service.list_data_products_with_pager(limit=50) + + for page in pager: + all_products.extend(page['data_products']) + +Get Data Product Details +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + product = dph_service.get_data_product(data_product_id=product_id) + + print(f"Name: {product.result['name']}") + print(f"Version: {product.result['version']}") + print(f"Description: {product.result['description']}") + print(f"Status: {product.result['state']}") + +Update Data Product +~~~~~~~~~~~~~~~~~~~ + +Use JSON Patch operations for updates: + +.. code-block:: python + + updated = dph_service.update_data_product( + data_product_id=product_id, + json_patch_instructions=[ + { + 'op': 'replace', + 'path': '/description', + 'value': 'Updated comprehensive customer analytics' + }, + { + 'op': 'add', + 'path': '/tags/-', + 'value': 'analytics' + } + ] + ) + +Delete Data Product +~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + dph_service.delete_data_product(data_product_id=product_id) + print("Data product deleted") + +Draft Management +---------------- + +Create a Draft +~~~~~~~~~~~~~~ + +.. code-block:: python + + draft = dph_service.create_data_product_draft( + data_product_id=product_id, + asset={'id': 'asset-123', 'container': {'id': 'container-456'}}, + version='1.1.0', + name='Customer Analytics Dataset v1.1', + description='Enhanced version with additional metrics' + ) + + draft_id = draft.result['id'] + +List Drafts +~~~~~~~~~~~ + +.. code-block:: python + + drafts = dph_service.list_data_product_drafts( + data_product_id=product_id, + limit=50 + ) + + for draft in drafts.result['drafts']: + print(f"- Draft {draft['version']}: {draft['state']}") + +Update Draft +~~~~~~~~~~~~ + +.. code-block:: python + + updated_draft = dph_service.update_data_product_draft( + data_product_id=product_id, + draft_id=draft_id, + json_patch_instructions=[ + { + 'op': 'replace', + 'path': '/description', + 'value': 'Updated draft description' + } + ] + ) + +Publish Draft +~~~~~~~~~~~~~ + +.. code-block:: python + + # Publish draft to create a release + release = dph_service.publish_data_product_draft( + data_product_id=product_id, + draft_id=draft_id + ) + + print(f"Published release: {release.result['id']}") + +Delete Draft +~~~~~~~~~~~~ + +.. code-block:: python + + dph_service.delete_data_product_draft( + data_product_id=product_id, + draft_id=draft_id + ) + +Contract Terms Management +------------------------- + +Create Contract Terms Document +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + doc = dph_service.create_draft_contract_terms_document( + data_product_id=product_id, + draft_id=draft_id, + contract_terms_id=terms_id, + type='terms_and_conditions', + name='Terms and Conditions', + url='https://example.com/terms.pdf', + attachment={ + 'id': 'attachment-123' + } + ) + +Get Contract Terms +~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + terms = dph_service.get_data_product_draft_contract_terms( + data_product_id=product_id, + draft_id=draft_id + ) + + for doc in terms.result['documents']: + print(f"- {doc['name']}: {doc['type']}") + +Update Contract Terms Document +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + updated_doc = dph_service.update_draft_contract_terms_document( + data_product_id=product_id, + draft_id=draft_id, + contract_terms_id=terms_id, + document_id=doc_id, + json_patch_instructions=[ + { + 'op': 'replace', + 'path': '/url', + 'value': 'https://example.com/updated-terms.pdf' + } + ] + ) + +Delete Contract Terms Document +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + dph_service.delete_draft_contract_terms_document( + data_product_id=product_id, + draft_id=draft_id, + contract_terms_id=terms_id, + document_id=doc_id + ) + +Release Management +------------------ + +List Releases +~~~~~~~~~~~~~ + +.. code-block:: python + + releases = dph_service.list_data_product_releases( + data_product_id=product_id, + state=['available', 'retired'] + ) + + for release in releases.result['releases']: + print(f"- v{release['version']}: {release['state']}") + +Get Release Details +~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + release = dph_service.get_data_product_release( + data_product_id=product_id, + release_id=release_id + ) + + print(f"Version: {release.result['version']}") + print(f"State: {release.result['state']}") + print(f"Published: {release.result['created_at']}") + +Update Release +~~~~~~~~~~~~~~ + +.. code-block:: python + + updated_release = dph_service.update_data_product_release( + data_product_id=product_id, + release_id=release_id, + json_patch_instructions=[ + { + 'op': 'replace', + 'path': '/description', + 'value': 'Updated release description' + } + ] + ) + +Retire Release +~~~~~~~~~~~~~~ + +.. code-block:: python + + retired = dph_service.retire_data_product_release( + data_product_id=product_id, + release_id=release_id + ) + + print(f"Release retired: {retired.result['state']}") + +Domain Management +----------------- + +List Domains +~~~~~~~~~~~~ + +.. code-block:: python + + domains = dph_service.list_data_product_domains(limit=50) + + for domain in domains.result['domains']: + print(f"- {domain['name']}: {domain['description']}") + +Create Domain +~~~~~~~~~~~~~ + +.. code-block:: python + + domain = dph_service.create_data_product_domain( + name='Customer Analytics', + description='Customer-related data products and analytics', + container={'id': 'container-123'} + ) + + domain_id = domain.result['id'] + +Create Subdomain +~~~~~~~~~~~~~~~~ + +.. code-block:: python + + subdomain = dph_service.create_data_product_subdomain( + domain_id=domain_id, + name='Customer Segmentation', + description='Customer segmentation and clustering datasets' + ) + +Get Domain +~~~~~~~~~~ + +.. code-block:: python + + domain = dph_service.get_domain(domain_id=domain_id) + + print(f"Name: {domain.result['name']}") + print(f"Subdomains: {len(domain.result.get('subdomains', []))}") + +Update Domain +~~~~~~~~~~~~~ + +.. code-block:: python + + updated_domain = dph_service.update_data_product_domain( + domain_id=domain_id, + json_patch_instructions=[ + { + 'op': 'replace', + 'path': '/description', + 'value': 'Updated domain description' + } + ] + ) + +Delete Domain +~~~~~~~~~~~~~ + +.. code-block:: python + + dph_service.delete_domain(domain_id=domain_id) + +Asset Visualization +------------------- + +Create Visualization +~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + visualization = dph_service.create_data_asset_visualization( + container={'id': 'container-123'}, + assets=[ + {'id': 'asset-1', 'container': {'id': 'container-123'}}, + {'id': 'asset-2', 'container': {'id': 'container-123'}}, + {'id': 'asset-3', 'container': {'id': 'container-123'}} + ] + ) + + print(f"Visualization created: {visualization.result['id']}") + +Reinitiate Visualization +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + reinitiated = dph_service.reinitiate_data_asset_visualization( + container={'id': 'container-123'}, + assets=[ + {'id': 'asset-1', 'container': {'id': 'container-123'}}, + {'id': 'asset-4', 'container': {'id': 'container-123'}} + ] + ) + +Contract Templates +------------------ + +Create Template +~~~~~~~~~~~~~~~ + +.. code-block:: python + + template = dph_service.create_contract_template( + name='Standard Data Sharing Agreement', + description='Standard terms for data product sharing', + contract_terms_documents=[{ + 'type': 'terms_and_conditions', + 'name': 'Standard Terms', + 'url': 'https://example.com/standard-terms.pdf' + }] + ) + + template_id = template.result['id'] + +List Templates +~~~~~~~~~~~~~~ + +.. code-block:: python + + templates = dph_service.list_data_product_contract_template(limit=50) + + for template in templates.result['contract_templates']: + print(f"- {template['name']}") + +Get Template +~~~~~~~~~~~~ + +.. code-block:: python + + template = dph_service.get_contract_template( + contract_template_id=template_id + ) + +Update Template +~~~~~~~~~~~~~~~ + +.. code-block:: python + + updated_template = dph_service.update_data_product_contract_template( + contract_template_id=template_id, + json_patch_instructions=[ + { + 'op': 'replace', + 'path': '/description', + 'value': 'Updated template description' + } + ] + ) + +Delete Template +~~~~~~~~~~~~~~~ + +.. code-block:: python + + dph_service.delete_data_product_contract_template( + contract_template_id=template_id + ) + +Advanced Topics +--------------- + +Custom Headers +~~~~~~~~~~~~~~ + +Add custom headers to requests: + +.. code-block:: python + + response = dph_service.get_data_product( + data_product_id=product_id, + headers={ + 'Custom-Header': 'value', + 'X-Request-ID': 'unique-id' + } + ) + +Timeout Configuration +~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + dph_service.set_http_config({ + 'timeout': 60 # 60 seconds + }) + +Disable SSL Verification (Development Only) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: python + + dph_service.set_disable_ssl_verification(True) + +See Also +-------- + +- :ref:`dph_services_examples` - Complete code examples +- :ref:`api_dph_services` - API reference +- :ref:`dph_services_overview` - Architecture overview + +.. Made with Bob diff --git a/docs/chapters/06_odcs_generator/collibra_integration.rst b/docs/chapters/06_odcs_generator/collibra_integration.rst new file mode 100644 index 0000000..d5fe3fa --- /dev/null +++ b/docs/chapters/06_odcs_generator/collibra_integration.rst @@ -0,0 +1,113 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _odcs_generator_collibra: + +Collibra Integration +==================== + +Generate ODCS files from Collibra data catalog assets. + +Overview +-------- + +The Collibra integration extracts metadata from Collibra assets and generates ODCS v3.1.0 compliant YAML files. + +Features +-------- + +- ✅ Automatic metadata extraction via REST API +- ✅ Column discovery through asset relations +- ✅ Data type mapping (logical and physical) +- ✅ Classification support via GraphQL API +- ✅ Tag integration at asset and column levels +- ✅ Custom attribute preservation + +Installation +------------ + +.. code-block:: bash + + pip install -e . + +Configuration +------------- + +Environment Variables +~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: bash + + export COLLIBRA_URL="https://your-instance.collibra.com" + export COLLIBRA_USERNAME="your_username" + export COLLIBRA_PASSWORD="your_password" + +Required Permissions +~~~~~~~~~~~~~~~~~~~~ + +- Read access to assets +- Read access to attributes +- Read access to relations +- Access to GraphQL API +- Read access to tags + +Usage +----- + +Command Line +~~~~~~~~~~~~ + +.. code-block:: bash + + python -m wxdi.odcs_generator.generate_odcs_from_collibra + +With options: + +.. code-block:: bash + + python -m wxdi.odcs_generator.generate_odcs_from_collibra \ + --output my-contract.yaml \ + --url https://collibra.com \ + --username myuser \ + --password mypass + +Python API +~~~~~~~~~~ + +.. code-block:: python + + from wxdi.odcs_generator.generate_odcs_from_collibra import CollibraClient, ODCSGenerator + + # Initialize client + client = CollibraClient( + base_url="https://your-instance.collibra.com", + username="your_username", + password="your_password" + ) + + # Create generator + generator = ODCSGenerator(client) + + # Generate ODCS + odcs_data = generator.generate_odcs("asset-id") + + # Save to file + generator.save_to_yaml(odcs_data, "output.yaml") + +See Also +-------- + +- :ref:`odcs_generator_examples` - Complete examples +- :ref:`api_odcs_generator` - API reference diff --git a/docs/chapters/06_odcs_generator/examples.rst b/docs/chapters/06_odcs_generator/examples.rst new file mode 100644 index 0000000..37c6f58 --- /dev/null +++ b/docs/chapters/06_odcs_generator/examples.rst @@ -0,0 +1,98 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _odcs_generator_examples: + +Examples +======== + +Complete examples for ODCS Generator module. + +Collibra Example +---------------- + +.. code-block:: python + + from wxdi.odcs_generator.generate_odcs_from_collibra import CollibraClient, ODCSGenerator + + # Initialize client + client = CollibraClient( + base_url="https://your-instance.collibra.com", + username="your_username", + password="your_password" + ) + + # Create generator + generator = ODCSGenerator(client) + + # Generate ODCS + odcs_data = generator.generate_odcs("019a57f9-62d2-7aa0-9f22-4fa2cea1180b") + + # Customize + odcs_data['dataProduct'] = 'Customer Data Product' + odcs_data['version'] = '2.0.0' + + # Save to file + generator.save_to_yaml(odcs_data, "customer-data-odcs.yaml") + +Informatica Example +------------------- + +.. code-block:: python + + from wxdi.odcs_generator.generate_odcs_from_informatica import InformaticaClient, ODCSGenerator + + # Initialize client + client = InformaticaClient( + base_url="https://your-informatica-instance.com", + username="your_username", + password="your_password" + ) + + # Create generator + generator = ODCSGenerator(client) + + # Generate ODCS + odcs_data = generator.generate_odcs("asset-id-123") + + # Save to file + generator.save_to_yaml(odcs_data, "output.yaml") + +Batch Processing +---------------- + +.. code-block:: python + + from wxdi.odcs_generator.generate_odcs_from_collibra import CollibraClient, ODCSGenerator + + client = CollibraClient(base_url, username, password) + generator = ODCSGenerator(client) + + asset_ids = ['id1', 'id2', 'id3'] + + for asset_id in asset_ids: + try: + odcs_data = generator.generate_odcs(asset_id) + generator.save_to_yaml(odcs_data, f"{asset_id}-odcs.yaml") + print(f"✅ Generated ODCS for {asset_id}") + except Exception as e: + print(f"❌ Failed for {asset_id}: {e}") + +See Also +-------- + +- :ref:`odcs_generator_collibra` - Collibra integration +- :ref:`odcs_generator_informatica` - Informatica integration +- :ref:`api_odcs_generator` - API reference diff --git a/docs/chapters/06_odcs_generator/index.rst b/docs/chapters/06_odcs_generator/index.rst new file mode 100644 index 0000000..31b1ce5 --- /dev/null +++ b/docs/chapters/06_odcs_generator/index.rst @@ -0,0 +1,189 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _odcs_generator: + +ODCS Generator +============== + +Tools to automatically generate ODCS (Open Data Contract Standard) v3.1.0 compliant YAML files from data catalog metadata. + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + overview + collibra_integration + informatica_integration + examples + +Overview +-------- + +The ``odcs_generator`` module provides automated generation of Open Data Contract Standard (ODCS) files from enterprise data catalogs. It extracts metadata from catalog systems and transforms it into standardized data contracts. + +Key Features +------------ + +**Multi-Catalog Support** + Generate ODCS files from Collibra and Informatica CDGC data catalogs. + +**Automatic Metadata Extraction** + Fetch asset details, attributes, relations, and classifications automatically. + +**Column Discovery** + Automatically discover and document table columns through catalog relations. + +**Data Type Mapping** + Intelligent mapping of catalog data types to ODCS standard types. + +**Classification Support** + Extract and include data classifications and sensitivity labels. + +**ODCS v3.1.0 Compliance** + Generate fully compliant ODCS YAML files ready for use. + +Quick Start +----------- + +**Collibra Integration** + +.. code-block:: python + + from wxdi.odcs_generator.generate_odcs_from_collibra import CollibraClient, ODCSGenerator + + # Initialize client + client = CollibraClient( + base_url="https://your-instance.collibra.com", + username="your_username", + password="your_password" + ) + + # Create generator + generator = ODCSGenerator(client) + + # Generate ODCS + odcs_data = generator.generate_odcs("asset-id") + + # Save to file + generator.save_to_yaml(odcs_data, "output.yaml") + +**Informatica Integration** + +.. code-block:: python + + from wxdi.odcs_generator.generate_odcs_from_informatica import InformaticaClient, ODCSGenerator + + # Initialize client + client = InformaticaClient( + base_url="https://your-informatica-instance.com", + username="your_username", + password="your_password" + ) + + # Create generator + generator = ODCSGenerator(client) + + # Generate ODCS + odcs_data = generator.generate_odcs("asset-id") + +Use Cases +--------- + +**Data Contract Automation** + Automatically generate data contracts from existing catalog metadata. + +**Catalog Migration** + Export catalog metadata to standardized ODCS format for migration. + +**Documentation Generation** + Create comprehensive data documentation from catalog assets. + +**Compliance Reporting** + Generate standardized contracts for compliance and governance. + +**Data Product Onboarding** + Accelerate data product creation with automated contract generation. + +Supported Catalogs +------------------ + +**Collibra** + - Asset metadata extraction + - Column discovery via relations + - Data classifications via GraphQL + - Tag integration + - Custom attributes + +**Informatica CDGC** + - Asset metadata extraction + - Column schema discovery + - System attributes + - Technical metadata + - Business glossary terms + +What is ODCS? +------------- + +The Open Data Contract Standard (ODCS) is an open-source specification for defining data contracts. It provides: + +- **Standardized Format**: Common structure for data contracts across organizations +- **Schema Definition**: Detailed column-level metadata and constraints +- **Quality Rules**: Data quality expectations and validation rules +- **Service Level Agreements**: Performance and availability commitments +- **Governance**: Data ownership, stewardship, and compliance information + +ODCS v3.1.0 Structure +~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: yaml + + id: unique-contract-id + kind: DataContract + apiVersion: v3.1.0 + domain: domain-name + dataProduct: product-name + version: 1.0.0 + name: contract-name + status: active + description: + authoritativeDefinitions: + - type: source-system + url: source-url + schema: + - id: table-id + name: table-name + columns: + - id: column-id + name: column-name + logicalType: string + physicalType: VARCHAR(255) + description: column description + classification: PII + quality: + - id: rule-id + name: rule-name + type: completeness + column: column-name + +Next Steps +---------- + +- :ref:`odcs_generator_collibra` - Collibra integration guide +- :ref:`odcs_generator_informatica` - Informatica integration guide +- :ref:`odcs_generator_examples` - Complete code examples +- :ref:`api_odcs_generator` - API reference + +.. Made with Bob diff --git a/docs/chapters/06_odcs_generator/informatica_integration.rst b/docs/chapters/06_odcs_generator/informatica_integration.rst new file mode 100644 index 0000000..735691e --- /dev/null +++ b/docs/chapters/06_odcs_generator/informatica_integration.rst @@ -0,0 +1,93 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _odcs_generator_informatica: + +Informatica Integration +======================== + +Generate ODCS files from Informatica CDGC (Cloud Data Governance and Catalog) assets. + +Overview +-------- + +The Informatica integration extracts metadata from Informatica CDGC and generates ODCS v3.1.0 compliant YAML files. + +Features +-------- + +- ✅ Asset metadata extraction via REST API +- ✅ Column schema discovery +- ✅ System attribute handling +- ✅ Technical metadata extraction +- ✅ Business glossary term integration + +Installation +------------ + +.. code-block:: bash + + pip install -e . + +Configuration +------------- + +Environment Variables +~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: bash + + export INFORMATICA_URL="https://your-informatica-instance.com" + export INFORMATICA_USERNAME="your_username" + export INFORMATICA_PASSWORD="your_password" + +Usage +----- + +Command Line +~~~~~~~~~~~~ + +.. code-block:: bash + + python -m wxdi.odcs_generator.generate_odcs_from_informatica + +Python API +~~~~~~~~~~ + +.. code-block:: python + + from wxdi.odcs_generator.generate_odcs_from_informatica import InformaticaClient, ODCSGenerator + + # Initialize client + client = InformaticaClient( + base_url="https://your-informatica-instance.com", + username="your_username", + password="your_password" + ) + + # Create generator + generator = ODCSGenerator(client) + + # Generate ODCS + odcs_data = generator.generate_odcs("asset-id") + + # Save to file + generator.save_to_yaml(odcs_data, "output.yaml") + +See Also +-------- + +- :ref:`odcs_generator_examples` - Complete examples +- :ref:`api_odcs_generator` - API reference diff --git a/docs/chapters/06_odcs_generator/overview.rst b/docs/chapters/06_odcs_generator/overview.rst new file mode 100644 index 0000000..e75c68d --- /dev/null +++ b/docs/chapters/06_odcs_generator/overview.rst @@ -0,0 +1,345 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _odcs_generator_overview: + +ODCS Generator Overview +======================= + +The ODCS Generator module automates the creation of Open Data Contract Standard (ODCS) v3.1.0 compliant YAML files from enterprise data catalog metadata. + +Architecture +------------ + +The module uses a modular architecture with catalog-specific clients and a common generator: + +.. code-block:: text + + ┌─────────────────────────────────────────┐ + │ ODCS Generator Module │ + ├─────────────────────────────────────────┤ + │ │ + │ ┌──────────────┐ ┌───────────────┐ │ + │ │ Collibra │ │ Informatica │ │ + │ │ Client │ │ Client │ │ + │ └──────┬───────┘ └───────┬───────┘ │ + │ │ │ │ + │ └───────┬───────────┘ │ + │ │ │ + │ ┌───────▼────────┐ │ + │ │ ODCS Generator │ │ + │ └───────┬────────┘ │ + │ │ │ + │ ┌───────▼────────┐ │ + │ │ YAML Output │ │ + │ └────────────────┘ │ + └─────────────────────────────────────────┘ + +Core Components +--------------- + +Catalog Clients +~~~~~~~~~~~~~~~ + +**CollibraClient** + - REST API integration + - GraphQL API for classifications + - Asset, attribute, and relation extraction + - Tag and classification support + +**InformaticaClient** + - REST API integration + - Asset metadata extraction + - Column schema discovery + - System attribute handling + +ODCS Generator +~~~~~~~~~~~~~~ + +The generator transforms catalog metadata into ODCS format: + +1. **Metadata Extraction**: Fetch asset details from catalog +2. **Column Discovery**: Identify and extract column information +3. **Type Mapping**: Convert catalog types to ODCS types +4. **Classification Mapping**: Extract data classifications +5. **YAML Generation**: Create compliant ODCS YAML file + +Data Type Mapping +----------------- + +Logical Type Mapping +~~~~~~~~~~~~~~~~~~~~ + +Catalog types are mapped to ODCS logical types: + +.. list-table:: + :header-rows: 1 + :widths: 30 30 40 + + * - Catalog Type + - ODCS Logical Type + - Description + * - text, string, varchar + - string + - Text data + * - whole number, int, integer + - integer + - Whole numbers + * - decimal number, float, double + - number + - Decimal numbers + * - date time, timestamp + - timestamp + - Date and time + * - true/false, boolean + - boolean + - Boolean values + * - geographical, geo + - string + - Geographic data + +Physical Type Mapping +~~~~~~~~~~~~~~~~~~~~~ + +Physical types preserve database-specific information: + +- ``VARCHAR(255)`` - Variable character with length +- ``DECIMAL(10,2)`` - Decimal with precision and scale +- ``NUMBER(18,4)`` - Numeric with precision and scale +- ``TIMESTAMP(6)`` - Timestamp with precision + +Classification Support +---------------------- + +The generator extracts and maps data classifications: + +**Collibra Classifications** + - Extracted via GraphQL API + - Mapped to ODCS classification field + - Supports custom classification schemes + +**Informatica Classifications** + - Extracted from asset attributes + - Mapped to ODCS tags and classifications + - Supports data sensitivity labels + +Common Classifications +~~~~~~~~~~~~~~~~~~~~~~ + +- **PII** - Personally Identifiable Information +- **PHI** - Protected Health Information +- **Confidential** - Confidential business data +- **Public** - Publicly available data +- **Internal** - Internal use only + +ODCS Structure +-------------- + +Generated ODCS files follow this structure: + +Contract Metadata +~~~~~~~~~~~~~~~~~ + +.. code-block:: yaml + + id: unique-contract-id + kind: DataContract + apiVersion: v3.1.0 + domain: domain-name + dataProduct: product-name + version: 1.0.0 + name: contract-name + status: active + contractCreatedTs: 2026-04-16T06:00:00Z + +Description Section +~~~~~~~~~~~~~~~~~~~ + +.. code-block:: yaml + + description: + purpose: Purpose of the data + authoritativeDefinitions: + - type: collibra-asset + url: https://collibra.com/asset/123 + limitations: Usage limitations + usage: Intended usage + +Schema Section +~~~~~~~~~~~~~~ + +.. code-block:: yaml + + schema: + - id: table-id + name: table_name + physicalName: PHYSICAL_TABLE_NAME + physicalType: table + description: Table description + tags: + - customer-data + - analytics + columns: + - id: column-id + name: column_name + logicalType: string + physicalType: VARCHAR(255) + description: Column description + isNullable: false + isPrimaryKey: false + classification: PII + tags: + - sensitive + +Quality Section +~~~~~~~~~~~~~~~ + +.. code-block:: yaml + + quality: + - id: rule-001 + name: completeness-check + type: completeness + column: customer_id + dimension: completeness + threshold: 0.95 + +Service Level Agreement +~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: yaml + + sla: + interval: daily + uptime: 99.9% + responseTime: 100ms + +Best Practices +-------------- + +**Validate Catalog Connectivity** + +.. code-block:: python + + try: + client = CollibraClient(base_url, username, password) + # Test connection + asset = client.get_asset("test-id") + except Exception as e: + print(f"Connection failed: {e}") + +**Handle Missing Metadata** + +.. code-block:: python + + # Provide defaults for missing fields + odcs_data = generator.generate_odcs( + asset_id, + defaults={ + 'dataProduct': 'Default Product', + 'version': '1.0.0', + 'status': 'draft' + } + ) + +**Batch Processing** + +.. code-block:: python + + asset_ids = ['id1', 'id2', 'id3'] + + for asset_id in asset_ids: + try: + odcs_data = generator.generate_odcs(asset_id) + generator.save_to_yaml(odcs_data, f"{asset_id}-odcs.yaml") + except Exception as e: + print(f"Failed for {asset_id}: {e}") + +**Customize Output** + +.. code-block:: python + + # Generate ODCS + odcs_data = generator.generate_odcs(asset_id) + + # Customize before saving + odcs_data['dataProduct'] = 'My Data Product' + odcs_data['version'] = '2.0.0' + odcs_data['quality'] = [ + { + 'id': 'custom-rule', + 'name': 'Custom Quality Rule', + 'type': 'accuracy' + } + ] + + # Save customized ODCS + generator.save_to_yaml(odcs_data, 'custom-odcs.yaml') + +Error Handling +-------------- + +Common errors and solutions: + +**Authentication Errors** + +.. code-block:: python + + from requests.exceptions import HTTPError + + try: + client = CollibraClient(url, username, password) + except HTTPError as e: + if e.response.status_code == 401: + print("Invalid credentials") + elif e.response.status_code == 403: + print("Insufficient permissions") + +**Asset Not Found** + +.. code-block:: python + + try: + odcs_data = generator.generate_odcs(asset_id) + except ValueError as e: + print(f"Asset not found: {e}") + +**Missing Columns** + +.. code-block:: python + + odcs_data = generator.generate_odcs(asset_id) + + if not odcs_data.get('schema', [{}])[0].get('columns'): + print("Warning: No columns found for asset") + +Requirements +------------ + +- Python 3.8 or higher +- requests >= 2.32.4 +- pyyaml >= 5.4.0 +- urllib3 >= 2.6.3 +- python-dateutil >= 2.5.3 + +See Also +-------- + +- :ref:`odcs_generator_collibra` - Collibra integration +- :ref:`odcs_generator_informatica` - Informatica integration +- :ref:`odcs_generator_examples` - Code examples +- :ref:`api_odcs_generator` - API reference + +.. Made with Bob diff --git a/docs/chapters/07_data_product_recommender/examples.rst b/docs/chapters/07_data_product_recommender/examples.rst new file mode 100644 index 0000000..ae39ed9 --- /dev/null +++ b/docs/chapters/07_data_product_recommender/examples.rst @@ -0,0 +1,100 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _data_product_recommender_examples: + +Examples +======== + +Basic Example +------------- + +.. code-block:: python + + from wxdi.data_product_recommender.platforms import SnowflakeQueryParser + from wxdi.data_product_recommender.recommender import DataProductRecommender + + # Initialize with Snowflake parser + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + # Load and analyze query logs + recommender.load_query_logs_from_csv_file('query_logs.csv') + recommender.calculate_metrics() + + # Get top 20 recommendations + recommendations = recommender.recommend_data_products(num_recommendations=20) + + # Export to Markdown + recommender.export_recommendations_markdown(recommendations, 'output/recommendations.md') + + # Export to JSON + recommender.export_recommendations_json(recommendations, 'output/recommendations.json') + +Multi-Platform Example +---------------------- + +.. code-block:: python + + from wxdi.data_product_recommender.platforms import ( + SnowflakeQueryParser, + DatabricksQueryParser, + BigQueryQueryParser + ) + from wxdi.data_product_recommender.recommender import DataProductRecommender + + # Snowflake + snowflake_parser = SnowflakeQueryParser() + snowflake_recommender = DataProductRecommender(snowflake_parser) + snowflake_recommender.load_query_logs_from_csv_file('snowflake_logs.csv') + + # Databricks + databricks_parser = DatabricksQueryParser() + databricks_recommender = DataProductRecommender(databricks_parser) + databricks_recommender.load_query_logs_from_csv_file('databricks_logs.csv') + + # BigQuery + bigquery_parser = BigQueryQueryParser() + bigquery_recommender = DataProductRecommender(bigquery_parser) + bigquery_recommender.load_query_logs_from_csv_file('bigquery_logs.csv') + +Custom Scoring Weights +---------------------- + +.. code-block:: python + + from wxdi.data_product_recommender.platforms import SnowflakeQueryParser + from wxdi.data_product_recommender.recommender import DataProductRecommender + + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + # Customize scoring weights + recommender.weights = { + 'query_count': 0.5, # Emphasize query volume + 'user_diversity': 0.3, # Moderate user diversity + 'recency': 0.1, # Less emphasis on recency + 'consistency': 0.1 # Less emphasis on consistency + } + + recommender.load_query_logs_from_csv_file('query_logs.csv') + recommender.calculate_metrics() + recommendations = recommender.recommend_data_products(num_recommendations=20) + +See Also +-------- + +- :ref:`data_product_recommender_usage` - Usage guide +- :ref:`api_data_product_recommender` - API reference diff --git a/docs/chapters/07_data_product_recommender/index.rst b/docs/chapters/07_data_product_recommender/index.rst new file mode 100644 index 0000000..44eb21f --- /dev/null +++ b/docs/chapters/07_data_product_recommender/index.rst @@ -0,0 +1,111 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _data_product_recommender: + +Data Product Recommender +========================= + +Analyze database query logs to identify high-value tables and logical groupings for data product prioritization. + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + overview + usage_guide + examples + +Overview +-------- + +The ``data_product_recommender`` module analyzes query log files to identify which tables should be prioritized as data products in a data marketplace. + +Key Features +------------ + +**Multi-Platform Support** + Supports Snowflake, Databricks, BigQuery, and watsonx.data query log formats. + +**File-Based Input** + Works with CSV and JSON query log files (no direct database connection required). + +**Intelligent Scoring** + Combines query frequency, user diversity, recency, and consistency metrics. + +**Table Grouping** + Identifies tables frequently used together for logical data product groupings. + +**Multiple Output Formats** + Generates both Markdown (human-readable) and JSON (agent-consumable) reports. + +**CLI and Python API** + Use from command line or integrate into applications. + +Quick Start +----------- + +Command Line +~~~~~~~~~~~~ + +.. code-block:: bash + + python -m wxdi.data_product_recommender.cli \ + --platform snowflake \ + --input-file query_logs.csv \ + --output output \ + --num-recommendations 20 + +Python API +~~~~~~~~~~ + +.. code-block:: python + + from wxdi.data_product_recommender.platforms import SnowflakeQueryParser + from wxdi.data_product_recommender.recommender import DataProductRecommender + + # Initialize + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + # Load and analyze + recommender.load_query_logs_from_csv_file('query_logs.csv') + recommender.calculate_metrics() + recommendations = recommender.recommend_data_products(num_recommendations=20) + + # Export + recommender.export_recommendations_markdown(recommendations, 'output/recommendations.md') + +Use Cases +--------- + +**Accelerate Data Product Onboarding** + Leverage existing usage patterns rather than starting from scratch. + +**Identify High-Value Assets** + Find tables with demonstrated business value through real usage. + +**Discover Logical Groupings** + Identify tables commonly used together for cohesive data products. + +**Prioritize Catalog Promotion** + Focus efforts on tables with highest user demand and diversity. + +Next Steps +---------- + +- :ref:`data_product_recommender_usage` - Detailed usage guide +- :ref:`data_product_recommender_examples` - Code examples +- :ref:`api_data_product_recommender` - API reference diff --git a/docs/chapters/07_data_product_recommender/overview.rst b/docs/chapters/07_data_product_recommender/overview.rst new file mode 100644 index 0000000..7b46a50 --- /dev/null +++ b/docs/chapters/07_data_product_recommender/overview.rst @@ -0,0 +1,65 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _data_product_recommender_overview: + +Overview +======== + +The Data Product Recommender analyzes query logs to identify high-value tables for data product creation. + +Scoring Methodology +------------------- + +Individual Table Scoring (0-100) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- **37.5%** Query Count - Volume of usage +- **37.5%** User Diversity - Breadth of usage across teams +- **15%** Recency - Recent activity +- **10%** Consistency - Regular usage patterns + +Table Group Scoring (0-100) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +- **30%** Cohesion - How tightly tables are connected +- **20%** Usage - Relative usage compared to other groups +- **15%** User Reach - Percentage of users querying the group +- **20%** Recency - Recent activity across tables +- **10%** Consistency - Regular usage patterns +- **5%** Size - Number of tables in the group + +Star Rating Scale +~~~~~~~~~~~~~~~~~ + +- ⭐⭐⭐⭐⭐ **Excellent (80-100)**: Implement immediately +- ⭐⭐⭐⭐ **Good (60-79)**: Medium priority +- ⭐⭐⭐ **Fair (40-59)**: Consider splitting or implement later +- ⭐⭐ **Weak (20-39)**: Reconsider grouping +- ⭐ **Poor (0-19)**: Do not implement + +Platform Support +---------------- + +- ✅ **Snowflake** - Export from SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY +- ✅ **Databricks** - Export from system.query.history +- ✅ **BigQuery** - Export from INFORMATION_SCHEMA.JOBS_BY_PROJECT +- ✅ **watsonx.data** - Export from system.runtime.queries + +See Also +-------- + +- :ref:`data_product_recommender_usage` - Usage guide +- :ref:`data_product_recommender_examples` - Examples diff --git a/docs/chapters/07_data_product_recommender/usage_guide.rst b/docs/chapters/07_data_product_recommender/usage_guide.rst new file mode 100644 index 0000000..e638e48 --- /dev/null +++ b/docs/chapters/07_data_product_recommender/usage_guide.rst @@ -0,0 +1,82 @@ +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _data_product_recommender_usage: + +Usage Guide +=========== + +Installation +------------ + +.. code-block:: bash + + pip install -e . + +CLI Usage +--------- + +.. code-block:: bash + + python -m wxdi.data_product_recommender.cli \ + --platform snowflake \ + --input-file query_logs.csv \ + --output output \ + --num-recommendations 20 \ + --min-score 60.0 + +Options +~~~~~~~ + +- ``--platform`` - Database platform (snowflake, databricks, bigquery, watsonxdata) +- ``--input-file`` - Path to CSV or JSON query log file +- ``--output`` - Output directory (default: output) +- ``--output-format`` - Output format: markdown or json (default: markdown) +- ``--num-recommendations`` - Number of recommendations (default: 20) +- ``--min-score`` - Minimum score threshold 0-100 + +Python API +---------- + +.. code-block:: python + + from wxdi.data_product_recommender.platforms import SnowflakeQueryParser + from wxdi.data_product_recommender.recommender import DataProductRecommender + + # Initialize + parser = SnowflakeQueryParser() + recommender = DataProductRecommender(parser) + + # Load query logs + recommender.load_query_logs_from_csv_file('query_logs.csv') + + # Calculate metrics + recommender.calculate_metrics() + + # Get recommendations + recommendations = recommender.recommend_data_products( + num_recommendations=20, + min_score=60.0 + ) + + # Export results + recommender.export_recommendations_markdown(recommendations, 'output/recommendations.md') + recommender.export_recommendations_json(recommendations, 'output/recommendations.json') + +See Also +-------- + +- :ref:`data_product_recommender_examples` - Complete examples +- :ref:`api_data_product_recommender` - API reference diff --git a/docs/chapters/05_future_modules/index.rst b/docs/chapters/08_future_modules/index.rst similarity index 96% rename from docs/chapters/05_future_modules/index.rst rename to docs/chapters/08_future_modules/index.rst index 9eb46fb..38136f0 100644 --- a/docs/chapters/05_future_modules/index.rst +++ b/docs/chapters/08_future_modules/index.rst @@ -1,108 +1,108 @@ -.. - Copyright 2026 IBM Corporation - - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - -.. _future_modules: - -Future Modules -============== - -The ``IBM watsonx.data intelligence SDK`` is designed with a modular architecture that allows different teams to contribute specialized functionality while sharing common components like authentication. - -Architecture for Extensibility -------------------------------- - -The SDK's modular design enables: - -* **Independent Development**: Teams can develop modules independently -* **Shared Infrastructure**: All modules use common authentication and configuration -* **Consistent API**: Modules follow the same design patterns -* **Easy Integration**: New modules integrate seamlessly with existing ones - -Adding New Modules ------------------- - -Teams adding new modules should: - -1. **Use Common Authentication**: Leverage the ``common.auth`` module for authentication -2. **Follow Naming Conventions**: Use clear, descriptive module names -3. **Provide Documentation**: Include comprehensive documentation following this structure -4. **Include Examples**: Provide working code examples -5. **Add Tests**: Include unit and integration tests - -Documentation Structure for New Modules ----------------------------------------- - -When adding a new module, create documentation following this pattern: - -.. code-block:: text - - docs/chapters/0X_module_name/ - ├── index.rst # Module overview - ├── core_concepts.rst # Key concepts - ├── usage.rst # Usage guide - ├── examples.rst # Code examples - └── api_reference.rst # API documentation - -API Reference Structure -~~~~~~~~~~~~~~~~~~~~~~~ - -Add API reference documentation: - -.. code-block:: text - - docs/api/module_name/ - ├── index.rst # API overview - ├── classes.rst # Main classes - └── utilities.rst # Utility functions - -Planned Modules ---------------- - -While specific modules are still being defined, potential areas include: - -* Data profiling and statistics -* Data lineage tracking -* Data catalog integration -* Additional data quality features -* Custom analytics capabilities - -Contact -------- - -If your team is planning to add a module to the SDK: - -* Review the existing module structure (``dq_validator``) -* Follow the authentication patterns in ``common.auth`` -* Coordinate with the SDK maintainers -* Submit documentation along with your module - -For questions or to propose a new module: - -* Email: Data_Intelligence_SDK@wwpdl.vnet.ibm.com -* GitHub: Open an issue or discussion - -Contributing ------------- - -See the CONTRIBUTING.md file in the repository for detailed guidelines on: - -* Code style and standards -* Testing requirements -* Documentation requirements -* Pull request process - -We look forward to growing the SDK with contributions from teams across IBM! - -.. Made with Bob +.. + Copyright 2026 IBM Corporation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + +.. _future_modules: + +Future Modules +============== + +The ``IBM watsonx.data intelligence SDK`` is designed with a modular architecture that allows different teams to contribute specialized functionality while sharing common components like authentication. + +Architecture for Extensibility +------------------------------- + +The SDK's modular design enables: + +* **Independent Development**: Teams can develop modules independently +* **Shared Infrastructure**: All modules use common authentication and configuration +* **Consistent API**: Modules follow the same design patterns +* **Easy Integration**: New modules integrate seamlessly with existing ones + +Adding New Modules +------------------ + +Teams adding new modules should: + +1. **Use Common Authentication**: Leverage the ``common.auth`` module for authentication +2. **Follow Naming Conventions**: Use clear, descriptive module names +3. **Provide Documentation**: Include comprehensive documentation following this structure +4. **Include Examples**: Provide working code examples +5. **Add Tests**: Include unit and integration tests + +Documentation Structure for New Modules +---------------------------------------- + +When adding a new module, create documentation following this pattern: + +.. code-block:: text + + docs/chapters/0X_module_name/ + ├── index.rst # Module overview + ├── core_concepts.rst # Key concepts + ├── usage.rst # Usage guide + ├── examples.rst # Code examples + └── api_reference.rst # API documentation + +API Reference Structure +~~~~~~~~~~~~~~~~~~~~~~~ + +Add API reference documentation: + +.. code-block:: text + + docs/api/module_name/ + ├── index.rst # API overview + ├── classes.rst # Main classes + └── utilities.rst # Utility functions + +Planned Modules +--------------- + +While specific modules are still being defined, potential areas include: + +* Data profiling and statistics +* Data lineage tracking +* Data catalog integration +* Additional data quality features +* Custom analytics capabilities + +Contact +------- + +If your team is planning to add a module to the SDK: + +* Review the existing module structure (``dq_validator``) +* Follow the authentication patterns in ``common.auth`` +* Coordinate with the SDK maintainers +* Submit documentation along with your module + +For questions or to propose a new module: + +* Email: Data_Intelligence_SDK@wwpdl.vnet.ibm.com +* GitHub: Open an issue or discussion + +Contributing +------------ + +See the CONTRIBUTING.md file in the repository for detailed guidelines on: + +* Code style and standards +* Testing requirements +* Documentation requirements +* Pull request process + +We look forward to growing the SDK with contributions from teams across IBM! + +.. Made with Bob diff --git a/docs/index.rst b/docs/index.rst index 151d37b..428f467 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -16,12 +16,15 @@ IBM watsonx.data intelligence SDK for Python ============================================ -The ``IBM watsonx.data intelligence SDK`` for Python is a comprehensive toolkit for data intelligence operations, providing modular components for data quality validation, authentication, and more. +The ``IBM watsonx.data intelligence SDK`` for Python is a comprehensive toolkit for data intelligence operations, providing modular components for data quality validation, data product management, ODCS generation, and intelligent recommendations. This SDK is designed with a modular architecture, allowing different teams to contribute specialized functionality while sharing common components like authentication. Currently, the SDK includes: * **Common Modules**: Shared authentication and configuration for all SDK modules * **DQ Validator**: In-memory data quality validation for streaming data, Pandas DataFrames, and PySpark DataFrames +* **DPH Services**: Python client for IBM Data Product Hub API +* **ODCS Generator**: Generate Open Data Contract Standard files from data catalogs +* **Data Product Recommender**: Analyze query logs to identify high-value data products The ``IBM watsonx.data intelligence SDK`` is supported on Python 3.8+. @@ -31,6 +34,15 @@ Key Features **Data Quality Validation** Comprehensive validation framework with 9 check types, support for array-based records and DataFrames, and integration with IBM Cloud Pak for Data. +**Data Product Hub Integration** + Complete Python SDK for managing data products, drafts, releases, contract terms, and domains. + +**ODCS Generation** + Automated generation of ODCS v3.1.0 compliant YAML files from Collibra and Informatica catalogs. + +**Intelligent Recommendations** + Query log analysis to identify high-value tables and logical groupings for data product prioritization. + **Multi-Environment Authentication** Unified authentication supporting IBM Cloud, AWS Cloud, Government Cloud, and on-premises deployments. @@ -48,7 +60,10 @@ Key Features chapters/02_overview/index chapters/03_common_modules/index chapters/04_dq_validator/index - chapters/05_future_modules/index + chapters/05_dph_services/index + chapters/06_odcs_generator/index + chapters/07_data_product_recommender/index + chapters/08_future_modules/index api/index .. Made with Bob From 3dd9fdf055f3a194d0ca05ec8276694550129ad7 Mon Sep 17 00:00:00 2001 From: Greeshma Rajendran Date: Fri, 8 May 2026 18:50:49 +0530 Subject: [PATCH 28/33] docs(api): Fixed documentation warnings Signed-off-by: Greeshma Rajendran --- docs/api/dph_services/core.rst | 23 ---------- docs/api/dph_services/index.rst | 73 ++----------------------------- docs/api/odcs_generator/index.rst | 5 --- src/wxdi/dph_services/dph_v1.py | 4 +- 4 files changed, 5 insertions(+), 100 deletions(-) diff --git a/docs/api/dph_services/core.rst b/docs/api/dph_services/core.rst index 2009b39..aa92c32 100644 --- a/docs/api/dph_services/core.rst +++ b/docs/api/dph_services/core.rst @@ -31,27 +31,4 @@ DphV1 Service :show-inheritance: :inherited-members: -Common Models -------------- - -.. autoclass:: wxdi.dph_services.common.DataProduct - :members: - :undoc-members: - :show-inheritance: - -.. autoclass:: wxdi.dph_services.common.DataProductDraft - :members: - :undoc-members: - :show-inheritance: - -.. autoclass:: wxdi.dph_services.common.ContractTerms - :members: - :undoc-members: - :show-inheritance: - -.. autoclass:: wxdi.dph_services.common.Domain - :members: - :undoc-members: - :show-inheritance: - .. Made with Bob \ No newline at end of file diff --git a/docs/api/dph_services/index.rst b/docs/api/dph_services/index.rst index d9adaa6..c42ff65 100644 --- a/docs/api/dph_services/index.rst +++ b/docs/api/dph_services/index.rst @@ -28,6 +28,9 @@ API reference for the Data Product Hub Services module. Main Service Class ------------------ +The DphV1 class provides access to all Data Product Hub Services operations. +For detailed API reference including all methods, see :ref:`api_dph_services_core`. + .. currentmodule:: wxdi.dph_services .. autoclass:: DphV1 @@ -35,74 +38,6 @@ Main Service Class :undoc-members: :show-inheritance: :special-members: __init__ - -Container Operations --------------------- - -.. automethod:: DphV1.initialize -.. automethod:: DphV1.get_initialize_status -.. automethod:: DphV1.get_service_id_credentials -.. automethod:: DphV1.manage_api_keys - -Data Product Operations ------------------------ - -.. automethod:: DphV1.create_data_product -.. automethod:: DphV1.list_data_products -.. automethod:: DphV1.list_data_products_with_pager -.. automethod:: DphV1.get_data_product -.. automethod:: DphV1.update_data_product -.. automethod:: DphV1.delete_data_product - -Draft Operations ----------------- - -.. automethod:: DphV1.create_data_product_draft -.. automethod:: DphV1.list_data_product_drafts -.. automethod:: DphV1.get_data_product_draft -.. automethod:: DphV1.update_data_product_draft -.. automethod:: DphV1.delete_data_product_draft -.. automethod:: DphV1.publish_data_product_draft - -Release Operations ------------------- - -.. automethod:: DphV1.list_data_product_releases -.. automethod:: DphV1.get_data_product_release -.. automethod:: DphV1.update_data_product_release -.. automethod:: DphV1.retire_data_product_release - -Contract Terms Operations -------------------------- - -.. automethod:: DphV1.create_draft_contract_terms_document -.. automethod:: DphV1.get_data_product_draft_contract_terms -.. automethod:: DphV1.update_draft_contract_terms_document -.. automethod:: DphV1.delete_draft_contract_terms_document - -Domain Operations ------------------ - -.. automethod:: DphV1.list_data_product_domains -.. automethod:: DphV1.create_data_product_domain -.. automethod:: DphV1.create_data_product_subdomain -.. automethod:: DphV1.get_domain -.. automethod:: DphV1.update_data_product_domain -.. automethod:: DphV1.delete_domain - -Asset Visualization Operations -------------------------------- - -.. automethod:: DphV1.create_data_asset_visualization -.. automethod:: DphV1.reinitiate_data_asset_visualization - -Contract Template Operations ----------------------------- - -.. automethod:: DphV1.create_contract_template -.. automethod:: DphV1.list_data_product_contract_template -.. automethod:: DphV1.get_contract_template -.. automethod:: DphV1.update_data_product_contract_template -.. automethod:: DphV1.delete_data_product_contract_template + :no-index: .. Made with Bob \ No newline at end of file diff --git a/docs/api/odcs_generator/index.rst b/docs/api/odcs_generator/index.rst index d7e3203..2b45be6 100644 --- a/docs/api/odcs_generator/index.rst +++ b/docs/api/odcs_generator/index.rst @@ -45,9 +45,4 @@ Informatica Integration :undoc-members: :show-inheritance: -.. autoclass:: ODCSGenerator - :members: - :undoc-members: - :show-inheritance: - .. Made with Bob diff --git a/src/wxdi/dph_services/dph_v1.py b/src/wxdi/dph_services/dph_v1.py index a05cec9..d7908ff 100644 --- a/src/wxdi/dph_services/dph_v1.py +++ b/src/wxdi/dph_services/dph_v1.py @@ -1392,9 +1392,7 @@ def replace_data_product_draft_contract_terms( contract. :param List[ContractTemplateSLA] sla: (optional) Service Level Agreement details. - :param List[ContractTemplateSupportAndCommunication] - support_and_communication: (optional) Support and communication details for - the contract. + :param List[ContractTemplateSupportAndCommunication] support_and_communication: (optional) Support and communication details for the contract. :param List[ContractTemplateCustomProperty] custom_properties: (optional) Custom properties that are not part of the standard contract. :param ContractTest contract_test: (optional) Contains the contract test From cedccef2073bb9a34ea7e59b5ccb0782279656cd Mon Sep 17 00:00:00 2001 From: Greeshma Rajendran Date: Fri, 8 May 2026 18:59:53 +0530 Subject: [PATCH 29/33] chore: updated secrets Signed-off-by: Greeshma Rajendran --- .secrets.baseline | 62 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 61 insertions(+), 1 deletion(-) diff --git a/.secrets.baseline b/.secrets.baseline index e1737e6..9098ff2 100644 --- a/.secrets.baseline +++ b/.secrets.baseline @@ -3,7 +3,7 @@ "files": "^.secrets.baseline$", "lines": null }, - "generated_at": "2026-04-09T04:39:13Z", + "generated_at": "2026-05-08T13:27:00Z", "plugins_used": [ { "name": "AWSKeyDetector" @@ -301,6 +301,66 @@ "verified_result": null } ], + "docs/chapters/05_dph_services/overview.rst": [ + { + "hashed_secret": "45d676e7c6ab44cf4b8fa366ef2d8fccd3e6d6e6", + "is_secret": false, + "is_verified": false, + "line_number": 123, + "type": "Secret Keyword", + "verified_result": null + } + ], + "docs/chapters/05_dph_services/usage_guide.rst": [ + { + "hashed_secret": "45d676e7c6ab44cf4b8fa366ef2d8fccd3e6d6e6", + "is_secret": false, + "is_verified": false, + "line_number": 63, + "type": "Secret Keyword", + "verified_result": null + } + ], + "docs/chapters/06_odcs_generator/collibra_integration.rst": [ + { + "hashed_secret": "ddc861617551c5e789c290865300f615e6f51cc7", + "is_secret": false, + "is_verified": false, + "line_number": 97, + "type": "Secret Keyword", + "verified_result": null + } + ], + "docs/chapters/06_odcs_generator/examples.rst": [ + { + "hashed_secret": "ddc861617551c5e789c290865300f615e6f51cc7", + "is_secret": false, + "is_verified": false, + "line_number": 61, + "type": "Secret Keyword", + "verified_result": null + } + ], + "docs/chapters/06_odcs_generator/index.rst": [ + { + "hashed_secret": "ddc861617551c5e789c290865300f615e6f51cc7", + "is_secret": false, + "is_verified": false, + "line_number": 93, + "type": "Secret Keyword", + "verified_result": null + } + ], + "docs/chapters/06_odcs_generator/informatica_integration.rst": [ + { + "hashed_secret": "0eb9a6a3306220b901c7b4920cd9896899f219be", + "is_secret": false, + "is_verified": false, + "line_number": 77, + "type": "Secret Keyword", + "verified_result": null + } + ], "examples/auth_provider_usage.py": [ { "hashed_secret": "df5cc5832dc34a455c18662ac84587ea19cf2435", From 042bd42b403f2e4d5211ce22baed33f24fadd816 Mon Sep 17 00:00:00 2001 From: Greeshma Rajendran Date: Mon, 11 May 2026 11:26:33 +0530 Subject: [PATCH 30/33] chore: updated secrets Signed-off-by: Greeshma Rajendran --- .secrets.baseline | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.secrets.baseline b/.secrets.baseline index 9098ff2..182558b 100644 --- a/.secrets.baseline +++ b/.secrets.baseline @@ -3,7 +3,7 @@ "files": "^.secrets.baseline$", "lines": null }, - "generated_at": "2026-05-08T13:27:00Z", + "generated_at": "2026-05-08T13:46:12Z", "plugins_used": [ { "name": "AWSKeyDetector" From d33695c0dd5e31673cc7cdc6110a615102f6c8e1 Mon Sep 17 00:00:00 2001 From: Greeshma Rajendran Date: Mon, 11 May 2026 14:37:13 +0530 Subject: [PATCH 31/33] chore: updated secrets Signed-off-by: Greeshma Rajendran --- .secrets.baseline | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/.secrets.baseline b/.secrets.baseline index 182558b..6a81058 100644 --- a/.secrets.baseline +++ b/.secrets.baseline @@ -3,7 +3,7 @@ "files": "^.secrets.baseline$", "lines": null }, - "generated_at": "2026-05-08T13:46:12Z", + "generated_at": "2026-05-11T09:05:52Z", "plugins_used": [ { "name": "AWSKeyDetector" @@ -313,7 +313,7 @@ ], "docs/chapters/05_dph_services/usage_guide.rst": [ { - "hashed_secret": "45d676e7c6ab44cf4b8fa366ef2d8fccd3e6d6e6", + "hashed_secret": "11fa7c37d697f30e6aee828b4426a10f83ab2380", "is_secret": false, "is_verified": false, "line_number": 63, @@ -323,7 +323,7 @@ ], "docs/chapters/06_odcs_generator/collibra_integration.rst": [ { - "hashed_secret": "ddc861617551c5e789c290865300f615e6f51cc7", + "hashed_secret": "564e340cd48437d2dfe876ee154cc99dc4d0d137", "is_secret": false, "is_verified": false, "line_number": 97, @@ -333,7 +333,7 @@ ], "docs/chapters/06_odcs_generator/examples.rst": [ { - "hashed_secret": "ddc861617551c5e789c290865300f615e6f51cc7", + "hashed_secret": "564e340cd48437d2dfe876ee154cc99dc4d0d137", "is_secret": false, "is_verified": false, "line_number": 61, @@ -343,7 +343,7 @@ ], "docs/chapters/06_odcs_generator/index.rst": [ { - "hashed_secret": "ddc861617551c5e789c290865300f615e6f51cc7", + "hashed_secret": "564e340cd48437d2dfe876ee154cc99dc4d0d137", "is_secret": false, "is_verified": false, "line_number": 93, @@ -353,7 +353,7 @@ ], "docs/chapters/06_odcs_generator/informatica_integration.rst": [ { - "hashed_secret": "0eb9a6a3306220b901c7b4920cd9896899f219be", + "hashed_secret": "564e340cd48437d2dfe876ee154cc99dc4d0d137", "is_secret": false, "is_verified": false, "line_number": 77, From 387ec322cc264d590e565ee708b7b170e1ea20b6 Mon Sep 17 00:00:00 2001 From: Greeshma Rajendran Date: Mon, 11 May 2026 16:10:57 +0530 Subject: [PATCH 32/33] removing changes Signed-off-by: Greeshma Rajendran --- .bumpversion.toml | 3 +- .github/workflows/docs.yml | 2 +- CHANGELOG.md | 56 -------------------------------------- 3 files changed, 2 insertions(+), 59 deletions(-) diff --git a/.bumpversion.toml b/.bumpversion.toml index e50d6af..86b135e 100644 --- a/.bumpversion.toml +++ b/.bumpversion.toml @@ -12,10 +12,9 @@ # limitations under the License. [tool.bumpversion] -current_version = "2.0.0" +current_version = "2.0.0-rc.1" commit = true message = "Update version {current_version} -> {new_version}" -ignore_missing_version = true parse = """(?x) (?P0|[1-9]\\d*)\\. (?P0|[1-9]\\d*)\\. diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 6cf2651..a22c199 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -65,7 +65,7 @@ jobs: deploy: # Only deploy on push to main/master, not on PRs - if: github.event_name != 'pull_request' && github.ref == 'refs/heads/main' + if: github.event_name == 'push' && github.ref == 'refs/heads/main' environment: name: github-pages diff --git a/CHANGELOG.md b/CHANGELOG.md index 2aff2e4..a42bb2b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,59 +1,3 @@ -# [2.0.0](https://github.com/IBM/data-intelligence-sdk/compare/v1.0.0...v2.0.0) (2026-04-23) - - -* Upgrade to version 2.0.0 ([#13](https://github.com/IBM/data-intelligence-sdk/issues/13)) ([41f25ce](https://github.com/IBM/data-intelligence-sdk/commit/41f25cefc349a9d2add7d99bc94945d123104ca1)), closes [#7](https://github.com/IBM/data-intelligence-sdk/issues/7) - - -### BREAKING CHANGES - -* Bug fixes for Data Quality and addition of Data Product Hub modules - -Signed-off-by: Koichi Nishitani - -* fix relative path of Actions scripts - -Signed-off-by: Koichi Nishitani - -* fix script condition - -Signed-off-by: Koichi Nishitani - -* fix script condition again - -Signed-off-by: Koichi Nishitani - -* add missing Makefile and fix documentation - -Signed-off-by: Koichi Nishitani - -* add dq tests and upgrade python - -Signed-off-by: Koichi Nishitani - -* try to extend build timeout - -Signed-off-by: Koichi Nishitani - -* upgrade to python 3.10 as 3.9 is EOL - -Signed-off-by: Koichi Nishitani - -* modify constraint_model to use custom StrEnum in python 3.10 - -Signed-off-by: Koichi Nishitani - -* add pylint - -Signed-off-by: Koichi Nishitani - -* skip linting until ready - -Signed-off-by: Koichi Nishitani - -* make sure to build before checking - -Signed-off-by: Koichi Nishitani - # [2.0.0-rc.1](https://github.com/IBM/data-intelligence-sdk/compare/v1.0.0...v2.0.0-rc.1) (2026-04-22) From 4eddaae61f4fa026181af2790767a0793f7d6966 Mon Sep 17 00:00:00 2001 From: Greeshma Rajendran Date: Mon, 11 May 2026 16:18:20 +0530 Subject: [PATCH 33/33] chore : adding DCO Signed-off-by: Greeshma Rajendran --- docs/chapters/05_dph_services/usage_guide.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/chapters/05_dph_services/usage_guide.rst b/docs/chapters/05_dph_services/usage_guide.rst index af9b340..6e4fbc0 100644 --- a/docs/chapters/05_dph_services/usage_guide.rst +++ b/docs/chapters/05_dph_services/usage_guide.rst @@ -611,4 +611,3 @@ See Also - :ref:`api_dph_services` - API reference - :ref:`dph_services_overview` - Architecture overview -.. Made with Bob