TensorRT 2:4 sparsity is not applied with Q/DQ quantization

[uamsam]

I’m testing INT8 quantization using ASP 2:4 sparse model.

• Case A: Export sparse FP32 ONNX without QuantizeLinear/DequantizeLinear(Q/DQ) and build INT8 model using Polygraphy (with sparsity enabled).
  -> TensorRT layer info indicates sparsity-enabled tactics/kernels are selected for some layers.

• Case B: Starting from the same sparse model, export an ONNX with Q/DQ using pytorch-quantization and build with: trtexec --int8 --sparsity=enable
  -> Layer info still shows HasSparseWeights=1 for some layers, but sparsity-enabled tactics/kernels do not appear to be selected.

Question:
Is this difference between no Q/DQ ONNX vs Q/DQ ONNX expected for 2:4 sparsity?
If not, are there recommended export/build settings for Q/DQ ONNX to enable sparsity tactics ?

Environment: Jetson (JetPack 6.1.2 / TensorRT 8.6.1)

[Saad-Mallebhari]

@uamsam
The behavior you're seeing is likely expected, and the interaction between Q/DQ nodes and sparsity tactic selection is a known constraint.

When Q/DQ nodes are present, TensorRT's tactic selector operates under additional precision and layout constraints imposed by the quantization scheme. Sparsity-optimized kernels have specific requirements around weight layout and precision , Q/DQ graphs can prevent the weight folding and fusion that sparse tactics rely on, and explicit INT8 pipelines may restrict the tactic pool to candidates that don't include sparse kernels. This can happen even when HasSparseWeights=1 is correctly detected.

In short: HasSparseWeights=1 confirms the weight pattern is valid for 2:4 sparsity, but it doesn't guarantee a sparse tactic will be selected if no sparse tactic satisfies all the Q/DQ constraints simultaneously.

A few things worth trying to narrow this down:

1. Check available sparse tactics explicitly - build with --profilingVerbosity=detailed and inspect whether any sparse tactics appear as candidates for the affected layers, or whether they're being rejected entirely:

trtexec --onnx=model.onnx --int8 --sparsity=enable \
        --profilingVerbosity=detailed \
        --exportLayerInfo=layer_info.json

2. Check TRT 8.6 on Jetson support - combined INT8 Q/DQ + 2:4 sparsity may have limitations on this version. Reviewing the TensorRT 8.6 release notes for Jetson regarding simultaneous Q/DQ and sparsity support is worth confirming before assuming it's a configuration issue.