Skip to content

fix: log Comet execution errors before Spark can obscure them#3955

Closed
andygrove wants to merge 1 commit intoapache:mainfrom
andygrove:wrap-comet-errors
Closed

fix: log Comet execution errors before Spark can obscure them#3955
andygrove wants to merge 1 commit intoapache:mainfrom
andygrove:wrap-comet-errors

Conversation

@andygrove
Copy link
Copy Markdown
Member

@andygrove andygrove commented Apr 15, 2026

Which issue does this PR close?

Closes #.

Rationale for this change

When a NullPointerException or AssertionError occurs in Comet execution code, Spark's QueryExecution.toInternalError catches it and wraps it in a generic [INTERNAL_ERROR] The "collect" action failed. You hit a bug in Spark or the Spark plugins you use. message. This hides the original Comet context (which operator failed, what the root cause was), making it very difficult for users to debug or report issues.

What changes are included in this PR?

Adds error logging in the CometPlan trait using the template method pattern:

  • CometPlan now overrides doExecuteColumnar, doExecute, and doExecuteBroadcast with a withErrorLogging wrapper that catches exceptions, logs them with Comet-specific context (operator class name and method), and re-throws the original exception.
  • All Comet operators now override doExecuteCometColumnar, doExecuteComet, and doExecuteCometBroadcast instead of the Spark methods directly.
  • The original exception is preserved and re-thrown unchanged, but the Comet-specific context is captured in the logs before Spark has a chance to obscure it.

How are these changes tested?

Verified manually by injecting a NullPointerException in CometNativeExec.doExecuteCometColumnar and running CometExecSuite. Confirmed that the error is logged with the Comet operator name and original cause before Spark wraps it.

@andygrove andygrove changed the title fix: wrap Comet execution errors to prevent Spark from obscuring them fix: wrap Comet execution errors to prevent Spark from obscuring them [do not merge] Apr 16, 2026
@andygrove andygrove changed the title fix: wrap Comet execution errors to prevent Spark from obscuring them [do not merge] fix: log Comet execution errors before Spark can obscure them Apr 16, 2026
@andygrove andygrove closed this Apr 16, 2026
@andygrove andygrove deleted the wrap-comet-errors branch April 16, 2026 16:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant