fix(connectors): replace overloaded InvalidRecord with distinct error variants#3194
Open
atharvalade wants to merge 5 commits intoapache:masterfrom
Open
fix(connectors): replace overloaded InvalidRecord with distinct error variants#3194atharvalade wants to merge 5 commits intoapache:masterfrom
atharvalade wants to merge 5 commits intoapache:masterfrom
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #3194 +/- ##
=============================================
- Coverage 74.10% 58.49% -15.62%
Complexity 943 943
=============================================
Files 1164 1163 -1
Lines 103048 92923 -10125
Branches 80081 69957 -10124
=============================================
- Hits 76368 54351 -22017
- Misses 23995 35971 +11976
+ Partials 2685 2601 -84
🚀 New features to boost your workflow:
|
slbotbm
reviewed
Apr 29, 2026
Removed redundant phrasing from SchemaMismatch error documentation.
hubcio
requested changes
May 4, 2026
| #[error("Write failure: {0}")] | ||
| WriteFailure(String), | ||
| /// A catalog or transaction-level failure (e.g. applying or committing an | ||
| /// Iceberg transaction). Callers may retry on transient catalog outages. |
Contributor
There was a problem hiding this comment.
doc "callers may retry on transient catalog outages" is misleading. action.apply() at router/mod.rs:213 is in-memory transaction prep - deterministic failures (invalid partition spec, schema validation) cannot be retried. only tx.commit(catalog) at router/mod.rs:222 hits the network. suggest dropping the retry claim, or splitting into ApplyError (deterministic) vs CommitError (transient-eligible).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #3176
Rationale
Error::InvalidRecordwas used for five unrelated failure modes in the Iceberg sink'swrite_datafunction, making it impossible for callers to distinguish schema mismatches from I/O failures from catalog outages.What changed?
The Iceberg sink mapped Arrow schema conversion errors, Parquet write failures, and Iceberg catalog transaction failures all to
Error::InvalidRecord. Callers could not programmatically decide whether to fix a table definition, skip a corrupt message, or retry a catalog outage.Three new SDK error variants —
SchemaMismatch(String),WriteFailure(String),CatalogError(String)— replace the overloadedInvalidRecordat the appropriate call sites.InvalidRecordis preserved only for the genuine record-batch deserialization error.Local Execution
AI Usage