-
Notifications
You must be signed in to change notification settings - Fork 123
Fix: normalize rows_affected to handle string values from dbt-databricks #915
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix: normalize rows_affected to handle string values from dbt-databricks #915
Conversation
When using dbt-databricks with view_update_via_alter enabled, skipped views return rows_affected as a string '-1' instead of an integer. This causes Spark's VALUES clause to fail with INVALID_INLINE_TABLE.INCOMPATIBLE_TYPES_IN_INLINE_TABLE when building bulk INSERTs for dbt_run_results. The fix adds a normalize_rows_affected macro that: - Converts string '-1' to null (skipped operations have no meaningful row count) - Converts other string numbers to integers (defensive handling) - Passes through normal integers unchanged Fixes: elementary-data/elementary#2089 Linear: CORE-267 Co-Authored-By: Yosef Arbiv <yosef.arbiv@gmail.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
👋 @devin-ai-integration[bot] |
📝 WalkthroughWalkthroughAdds a normalization macro for Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
Comment |
Tests the following cases:
- None input returns None
- Integer inputs pass through unchanged (123, 0, -1)
- String '-1' returns None (skipped operations)
- Other string numbers are converted to integers ('123', '0', '456')
Co-Authored-By: Yosef Arbiv <yosef.arbiv@gmail.com>
The log_macro_results helper already applies tojson() to the result, so the test macro should return the raw value instead of tojson(result). Co-Authored-By: Yosef Arbiv <yosef.arbiv@gmail.com>
When the macro returns None, log_macro_results doesn't log anything, so run_operation returns an empty list. The test now handles this case by checking if the result list is empty before parsing. Co-Authored-By: Yosef Arbiv <yosef.arbiv@gmail.com>
Summary
When using dbt-databricks with
use_materialization_v2andview_update_via_alterenabled, skipped views returnrows_affectedas a string"-1"instead of an integer. This causes Spark's VALUES clause to fail withINVALID_INLINE_TABLE.INCOMPATIBLE_TYPES_IN_INLINE_TABLEwhen building bulk INSERTs fordbt_run_results, since other rows have integer values.This PR adds a
normalize_rows_affectedmacro that:"-1"tonull(skipped operations have no meaningful row count)Fixes: elementary-data/elementary#2089
Linear: CORE-267
Review & Testing Checklist for Human
| intJinja filter behavior: If a non-numeric string (other than "-1") is passed, the filter may fail or return 0. Consider whether additional validation is needed.view_update_via_alterenabled and verify the on-run-end hook completes successfully with mixed view/model results.Notes
Summary by CodeRabbit
Bug Fixes
Tests
✏️ Tip: You can customize this high-level summary in your review settings.