Skip to content

fix: Use thread context classloader for Iceberg class loading in CometScanRule#4609

Open
chern wants to merge 1 commit into
apache:mainfrom
chern:classloader-fix
Open

fix: Use thread context classloader for Iceberg class loading in CometScanRule#4609
chern wants to merge 1 commit into
apache:mainfrom
chern:classloader-fix

Conversation

@chern

@chern chern commented Jun 8, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Closes #3737.

Rationale for this change

Similar change to #3738.

Comet uses Class.forName(className) to reflectively load Iceberg classes.
The single-argument Class.forName uses the classloader of the calling class, not the thread context classloader.

When the Comet JAR is placed in Spark's jars/ folder, it is loaded by the system classloader.
However, Iceberg classes may be bundled inside the user's application jar (as uber jar), which is loaded by Spark's application classloader .
Since parent classloaders cannot see classes loaded by child classloaders, Class.forName from Comet's code fails with ClassNotFoundException.

What changes are included in this PR?

  • Replaces bare Class.forName() calls in CometScanRule with the IcebergReflection.loadClass() helper method.

How are these changes tested?

  • Verified in a deployment where the Comet jar is in Spark's jars/ folder and Iceberg is bundled in the application's uber jar; ran a few SQL queries that involved reading from Iceberg.

@mbutrovich mbutrovich self-requested a review June 8, 2026 20:02
@chern chern changed the title Use thread context classloader for Iceberg class loading in CometScanRule fix: Use thread context classloader for Iceberg class loading in CometScanRule Jun 8, 2026
@karuppayya

Copy link
Copy Markdown
Contributor

@mbutrovich Do we need a sperate issue tracking this or reuse old?

@mbutrovich

Copy link
Copy Markdown
Contributor

@mbutrovich Do we need a sperate issue tracking this or reuse old?

No preference.

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @chern!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Iceberg class loading fails with ClassNotFoundException when Comet JAR is on system classpath

3 participants