Fix kernel attribute query when cuLaunchKernel uses CUkernel handles by reoLantern · Pull Request #521 · accel-sim/accel-sim-framework

reoLantern · 2025-12-29T18:12:09Z

Problem

On newer CUDA drivers, cuLaunchKernel can receive a CUkernel handle instead of a CUfunction.
In that case cuFuncGetAttribute() returns CUDA_ERROR_INVALID_HANDLE, so tracer headers
show garbage values (shmem/nregs/binary version).

Solution

Add a minimal fallback: when cuFuncGetAttribute() returns INVALID_HANDLE, query the
attribute via cuKernelGetAttribute() using the same handle and current device.

Changes

Add helper get_attr_with_kernel_fallback(...) in tracer_tool.cu
Use it for NUM_REGS, SHARED_SIZE_BYTES, and BINARY_VERSION

Testing

Rebuilt tracer_tool and generated traces; header fields now report valid values.

Copilot

Pull request overview

This PR addresses a compatibility issue with newer CUDA drivers where cuLaunchKernel may receive CUkernel handles instead of CUfunction handles, causing cuFuncGetAttribute() to fail with CUDA_ERROR_INVALID_HANDLE and resulting in garbage values in trace headers.

Key Changes:

Introduces a helper function get_attr_with_kernel_fallback() that attempts to query attributes via cuFuncGetAttribute() first, then falls back to cuKernelGetAttribute() if an INVALID_HANDLE error occurs
Replaces direct cuFuncGetAttribute() calls with the new helper for NUM_REGS, SHARED_SIZE_BYTES, and BINARY_VERSION attributes
Removes explicit error checking via CUDA_SAFECALL macros in favor of silent error handling with fallback

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-29T18:15:11Z

+  if (res == CUDA_ERROR_INVALID_HANDLE) {
+    CUdevice dev = 0;
+    if (cuCtxGetDevice(&dev) == CUDA_SUCCESS) {
+      cuKernelGetAttribute(&value, attr, (CUkernel)func, dev);
+    }
+  }


The return value of cuKernelGetAttribute is not checked. If this call fails, value might remain uninitialized or contain incorrect data, which would then be returned and used in trace headers. The original code used CUDA_SAFECALL which would have caught such errors. Consider checking the return value and handling failures appropriately, or at least logging when the fallback also fails.

Suggested change

if (res == CUDA_ERROR_INVALID_HANDLE) {

CUdevice dev = 0;

if (cuCtxGetDevice(&dev) == CUDA_SUCCESS) {

cuKernelGetAttribute(&value, attr, (CUkernel)func, dev);

}

}

if (res == CUDA_SUCCESS) {

return value;

}

if (res == CUDA_ERROR_INVALID_HANDLE) {

CUdevice dev = 0;

CUresult dev_res = cuCtxGetDevice(&dev);

if (dev_res == CUDA_SUCCESS) {

CUresult kern_res = cuKernelGetAttribute(&value, attr, (CUkernel)func, dev);

if (kern_res != CUDA_SUCCESS) {

fprintf(stderr,

"tracer_tool: cuKernelGetAttribute failed (attr=%d) with error %d\n",

(int)attr, (int)kern_res);

}

} else {

fprintf(stderr,

"tracer_tool: cuCtxGetDevice failed with error %d while getting attribute %d\n",

(int)dev_res, (int)attr);

}

} else {

fprintf(stderr,

"tracer_tool: cuFuncGetAttribute failed (attr=%d) with error %d\n",

(int)attr, (int)res);

}

/* On any failure, return the default value (0) */

Copilot · 2025-12-29T18:15:12Z

+    if (cuCtxGetDevice(&dev) == CUDA_SUCCESS) {
+      cuKernelGetAttribute(&value, attr, (CUkernel)func, dev);
+    }
+  }


The function should document its behavior when both cuFuncGetAttribute and cuKernelGetAttribute fail. Currently, it silently returns 0 (the initialized value) without any indication that the attribute query failed. This differs from the original implementation which used CUDA_SAFECALL to enforce error handling. Consider adding a comment explaining that 0 is returned on failure, or adding optional error logging.

Suggested change

}

}

/* If both attribute queries fail, this function returns 0 (the default

* initialized value) without reporting an error. Callers should treat 0

* as a possible "attribute unavailable" result. */

Copilot · 2025-12-29T18:15:12Z

+    if (cuCtxGetDevice(&dev) == CUDA_SUCCESS) {
+      cuKernelGetAttribute(&value, attr, (CUkernel)func, dev);


If cuCtxGetDevice fails, the function skips the cuKernelGetAttribute call and returns 0. However, it's unclear whether this is the intended behavior. Consider checking if cuCtxGetDevice can legitimately fail in scenarios where cuFuncGetAttribute returns INVALID_HANDLE, and whether silently returning 0 is appropriate in such cases.

Suggested change

if (cuCtxGetDevice(&dev) == CUDA_SUCCESS) {

cuKernelGetAttribute(&value, attr, (CUkernel)func, dev);

CUresult ctx_res = cuCtxGetDevice(&dev);

if (ctx_res == CUDA_SUCCESS) {

cuKernelGetAttribute(&value, attr, (CUkernel)func, dev);

} else {

fprintf(stderr,

"get_attr_with_kernel_fallback: cuCtxGetDevice failed with "

"error code %d\n",

(int)ctx_res);

William-An · 2026-01-21T01:50:36Z

This happens on driver versions for running CUDA 13+, right?

reoLantern · 2026-01-21T01:56:44Z

Yes, I am using CUDA 13.0.

reoLantern · 2026-03-11T08:53:20Z

@JRPan Hi, can you merge this?

JRPan · 2026-03-11T17:45:48Z

Why not just replace the call completely?

JRPan · 2026-03-24T03:46:46Z

Thanks.

Tested & Works.

MICRO 2025 tracer was written against CUDA < 12. Under CUDA 13 runtime, cuLaunchKernel may pass a CUkernel handle (not CUfunction), making cuFuncGetAttribute return CUDA_ERROR_INVALID_HANDLE and leave the output at 0. Observed locally: binary_version = 0, cascade into `cuobjdump -arch=sm_0` failure in enhanced_tracer(). Cherry-picked helper from accel-sim upstream (3016c65, PR accel-sim#521): static int get_attr_with_kernel_fallback(CUfunction, CUfunction_attribute) - tries cuFuncGetAttribute first - on CUDA_ERROR_INVALID_HANDLE, retries with cuKernelGetAttribute Replaces 3 cuFuncGetAttribute call sites in enter_kernel_launch path (nregs, shmem_static_nbytes, binary_version). Diff vs MICRO 2025's upstream tracer_tool.cu: ~20 lines added for this helper + 3 call sites changed. Still byte-identical on all other paths. Canary on local H100 hits a separate unrelated issue (MICRO 2025 get_opcode_map has no entry for sm_90 — Hopper isn't in their paper's validation set). Real canary deferred to 2070S server (sm_75 Turing, which MICRO 2025 supports). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

tracer_nvbit: fall back to cuKernelGetAttribute for kernel attrs

4e2bd5d

Copilot AI review requested due to automatic review settings December 29, 2025 18:12

Copilot started reviewing on behalf of reoLantern December 29, 2025 18:12 View session

Copilot AI reviewed Dec 29, 2025

View reviewed changes

JRPan requested review from William-An and cesar-avalos3 January 21, 2026 01:39

Merge branch 'dev' into official

88d0647

William-An and others added 2 commits January 20, 2026 21:01

format tracer tool code

cfe1a92

Merge branch 'dev' into official

56f22ce

Merge branch 'accel-sim:dev' into official

f786677

JRPan merged commit 3016c65 into accel-sim:dev Mar 24, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix kernel attribute query when cuLaunchKernel uses CUkernel handles#521

Fix kernel attribute query when cuLaunchKernel uses CUkernel handles#521
JRPan merged 5 commits intoaccel-sim:devfrom
reoLantern:official

reoLantern commented Dec 29, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Dec 29, 2025

Uh oh!

Copilot AI Dec 29, 2025

Uh oh!

Copilot AI Dec 29, 2025

Uh oh!

William-An commented Jan 21, 2026

Uh oh!

reoLantern commented Jan 21, 2026

Uh oh!

reoLantern commented Mar 11, 2026

Uh oh!

JRPan commented Mar 11, 2026

Uh oh!

JRPan commented Mar 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

-  if (res == CUDA_ERROR_INVALID_HANDLE) {
-    CUdevice dev = 0;
-    if (cuCtxGetDevice(&dev) == CUDA_SUCCESS) {
-      cuKernelGetAttribute(&value, attr, (CUkernel)func, dev);
-    }
-  }
+  if (res == CUDA_SUCCESS) {
+    return value;
+  }
+  if (res == CUDA_ERROR_INVALID_HANDLE) {
+    CUdevice dev = 0;
+    CUresult dev_res = cuCtxGetDevice(&dev);
+    if (dev_res == CUDA_SUCCESS) {
+      CUresult kern_res = cuKernelGetAttribute(&value, attr, (CUkernel)func, dev);
+      if (kern_res != CUDA_SUCCESS) {
+        fprintf(stderr,
+                "tracer_tool: cuKernelGetAttribute failed (attr=%d) with error %d\n",
+                (int)attr, (int)kern_res);
+      }
+    } else {
+      fprintf(stderr,
+              "tracer_tool: cuCtxGetDevice failed with error %d while getting attribute %d\n",
+              (int)dev_res, (int)attr);
+    }
+  } else {
+    fprintf(stderr,
+            "tracer_tool: cuFuncGetAttribute failed (attr=%d) with error %d\n",
+            (int)attr, (int)res);
+  }
+  /* On any failure, return the default value (0) */

-  }
+  }
+  /* If both attribute queries fail, this function returns 0 (the default
+   * initialized value) without reporting an error. Callers should treat 0
+   * as a possible "attribute unavailable" result. */

		if (cuCtxGetDevice(&dev) == CUDA_SUCCESS) {
		cuKernelGetAttribute(&value, attr, (CUkernel)func, dev);

Conversation

reoLantern commented Dec 29, 2025

Problem

Solution

Changes

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

William-An commented Jan 21, 2026

Uh oh!

reoLantern commented Jan 21, 2026

Uh oh!

reoLantern commented Mar 11, 2026

Uh oh!

JRPan commented Mar 11, 2026

Uh oh!

JRPan commented Mar 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants