Skip to content

feat: avro support applying field-ids based on name mapping #127

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
zeroshade merged 17 commits into from
Aug 21, 2025
Merged

feat: avro support applying field-ids based on name mapping #127

zeroshade merged 17 commits into from
Aug 21, 2025
+785 −11

Conversation

@MisterRaindrop
Copy link
Contributor

@MisterRaindrop MisterRaindrop commented Jun 22, 2025 •
edited
Loading

see issue #128

@MisterRaindrop MisterRaindrop mentioned this pull request Jun 22, 2025
Closed
@MisterRaindrop MisterRaindrop changed the title avro support applying field-ids based on name mapping feat: avro support applying field-ids based on name mapping Jun 22, 2025
wgtmac
wgtmac reviewed Jun 24, 2025
View reviewed changes
mapleFU
mapleFU reviewed Jun 24, 2025
View reviewed changes
mapleFU
mapleFU reviewed Jun 24, 2025
View reviewed changes
Copy link
Member

@mapleFU mapleFU left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a style pass

wgtmac
wgtmac requested changes Jun 30, 2025
View reviewed changes
@MisterRaindrop MisterRaindrop force-pushed the rain/support_field_ids_mapping_2 branch from 46391f0 to 7ddeb06 Compare July 5, 2025 09:01
@MisterRaindrop
Copy link
Contributor Author

MisterRaindrop commented Jul 7, 2025 •
edited
Loading

@wgtmac hi please help me review it again when you’re free.

wgtmac
wgtmac requested changes Jul 8, 2025
View reviewed changes
wgtmac
wgtmac requested changes Jul 14, 2025
View reviewed changes
Copy link
Member

@wgtmac wgtmac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update! I have left some comments. Let me know when all test cases are good.

@MisterRaindrop
Copy link
Contributor Author

MisterRaindrop commented Jul 15, 2025

@wgtmac hi, I update code and all test is ok

@MisterRaindrop MisterRaindrop force-pushed the rain/support_field_ids_mapping_2 branch from 13aef21 to a19781e Compare July 18, 2025 02:25
wgtmac
wgtmac reviewed Jul 21, 2025
View reviewed changes
@MisterRaindrop
Copy link
Contributor Author

MisterRaindrop commented Jul 22, 2025

Thanks a lot for reviewing my changes , I’ve updated the code based on your suggestions, please check if there are any other suggestions for this PR? @wgtmac

wgtmac
wgtmac reviewed Jul 22, 2025
View reviewed changes
wgtmac
wgtmac reviewed Jul 22, 2025
View reviewed changes
@MisterRaindrop MisterRaindrop force-pushed the rain/support_field_ids_mapping_2 branch from 9ad7bcf to c5a71ad Compare July 27, 2025 03:36
MisterRaindrop and others added 13 commits July 28, 2025 21:09
@MisterRaindrop
support applying field-ids based on name mapping
97bc935
@MisterRaindrop
fix build error and set name_mapping to ReaderOptions
79bb90e
@MisterRaindrop @wgtmac
Update src/iceberg/file_reader.h
0bc3f5a 
Co-authored-by: Gang Wu <ustcwg@gmail.com>
@MisterRaindrop @wgtmac
Update src/iceberg/avro/avro_reader.cc
31cc997 
Co-authored-by: Gang Wu <ustcwg@gmail.com>
@MisterRaindrop @wgtmac
Update src/iceberg/avro/avro_reader.cc
2f2e4fd 
Co-authored-by: Gang Wu <ustcwg@gmail.com>
@MisterRaindrop
Refactor the applying field-ids on name mapping code.
bb91f9d
@MisterRaindrop
fix applying field-ids on name mapping code
b4d110f
@MisterRaindrop @wgtmac
Update src/iceberg/avro/avro_schema_util.cc
472fe6e 
Co-authored-by: Gang Wu <ustcwg@gmail.com>
@MisterRaindrop @wgtmac
Update src/iceberg/avro/avro_schema_util.cc
be71cef 
Co-authored-by: Gang Wu <ustcwg@gmail.com>
@MisterRaindrop @wgtmac
Update src/iceberg/avro/avro_schema_util.cc
7c922d4 
Co-authored-by: Gang Wu <ustcwg@gmail.com>
@MisterRaindrop @wgtmac
Update src/iceberg/avro/avro_schema_util.cc
2482b6c 
Co-authored-by: Gang Wu <ustcwg@gmail.com>
@MisterRaindrop
fix name mapping and schema
aa5b725
@MisterRaindrop
update code
bd8638f
@MisterRaindrop MisterRaindrop force-pushed the rain/support_field_ids_mapping_2 branch from c5a71ad to 94cee21 Compare July 28, 2025 14:00
@MisterRaindrop
Copy link
Contributor Author

MisterRaindrop commented Aug 1, 2025

Thanks review PR, I have to fix field id, and test case is ok, please check if there are any other suggestions? @wgtmac

wgtmac
wgtmac reviewed Aug 4, 2025
View reviewed changes
@wgtmac
Copy link
Member

wgtmac commented Aug 11, 2025

Is it ready to review again? @MisterRaindrop

@MisterRaindrop
Copy link
Contributor Author

MisterRaindrop commented Aug 11, 2025

please review it thanks

wgtmac
wgtmac requested changes Aug 11, 2025
View reviewed changes
Copy link
Member

@wgtmac wgtmac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I just found some issues with the attributes. Others look good to me.

@MisterRaindrop
Copy link
Contributor Author

MisterRaindrop commented Aug 11, 2025

I will fix it very thanks

@MisterRaindrop
Copy link
Contributor Author

MisterRaindrop commented Aug 14, 2025 •
edited
Loading

@wgtmac can start to review it, thanks

wgtmac
wgtmac reviewed Aug 17, 2025
View reviewed changes
@MisterRaindrop
Copy link
Contributor Author

MisterRaindrop commented Aug 19, 2025

@wgtmac It's ready to review

wgtmac
wgtmac approved these changes Aug 19, 2025
View reviewed changes
Copy link
Member

@wgtmac wgtmac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @MisterRaindrop!

@wgtmac
Copy link
Member

wgtmac commented Aug 19, 2025

@Fokko @zeroshade Could you please take a look? I think this is ready to merge.

@zeroshade zeroshade merged commit 82a1cd6 into apache:main Aug 21, 2025
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zeroshade zeroshade zeroshade approved these changes

@wgtmac wgtmac wgtmac approved these changes

+1 more reviewer

@mapleFU mapleFU mapleFU left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@MisterRaindrop @wgtmac @zeroshade @mapleFU