Vulkan backend produces all-zero outputs on PowerVR GPU (Pixel 10 Pro)

[abdelaziz-mahdy]

Summary

The Vulkan backend produces all-zero outputs on a PowerVR D-Series GPU (Google Pixel 10 Pro). The same model files work correctly on macOS via MoltenVK and on Android via XNNPACK.

Environment

• Device: Google Pixel 10 Pro
• GPU: PowerVR D-Series DXT-48-1536 MC1 (maxImageDimension3D = 2048)
• ExecuTorch: Built from source, branch fix/vulkan-texture-ubo-budget (my UBO budget fix from PR Fix Vulkan texture tensor UBO budget overflow #17294, rebased on upstream/main as of Feb 7 2026, commit ba2516cefa)
• NDK: 28.2.13676358 (Clang 19.0.1)
• Build: Release, arm64-v8a, XNNPACK + Vulkan backends

What I Observe

Platform	Backend	YOLO (v8)	MobileNet
macOS	XNNPACK	Correct	Correct
macOS	Vulkan (MoltenVK)	Correct	Correct
Android	XNNPACK	Correct	Correct
Android	Vulkan (PowerVR)	All zeros	NaN values

• YOLO: Output tensor shape [1, 84, 8400] is correct, but all confidence values are exactly 0.0
• MobileNet: Output contains NaN values
• Both texture_limits: (2048, 2048, 2048) and storage_type_override: BUFFER produce the same zero results

What I Found by Adding Tracing

I added __android_log_print traces to VulkanBackend.cpp, ComputeGraph.cpp, and StagingBuffer.cpp to narrow things down. Key findings:

1. GPU is PowerVR — no support in ExecuTorch

GPU: name='powervr d-series dxt-48-1536 mc1', max3D=2048

ExecuTorch's Vulkan backend only handles Adreno, Mali, NVIDIA, and SwiftShader. There is zero PowerVR-specific handling.

2. Output texture extents exceed maxImageDimension3D

output_tensor[3] extents=(2100,64,1)  max3D=2048 EXCEEDS=1
output_tensor[4] extents=(2100,4,16)  max3D=2048 EXCEEDS=1

2100 = 8400 anchors / 4 (texel packing). The detection head outputs also exceed the limit. This is only checked in #ifdef VULKAN_DEBUG builds (Tensor.cpp:632-660), so release builds silently hit undefined behavior.

I exported with texture_limits: (2048, 2048, 2048) in VulkanPartitioner, but this only controls which ops get delegated — it doesn't account for texel packing that turns dimension 8400 into texture extent 2100.

3. Even in-limit tensors are zero

Output tensors 0-2 have extents within the 2048 limit (e.g., (80,80,16), (40,40,32), (20,20,64)), but they are also all zeros. This suggests either intermediate tensors also exceed limits, or one bad texture corrupts the entire command buffer state on PowerVR.

4. Execution mechanics work fine

• 332 nodes encode and submit, fence waits successfully
• Staging buffers have correct memory flags (HOST_VISIBLE | HOST_COHERENT | DEVICE_LOCAL)
• Input data is valid (verified non-zero values in staging buffer)
• GPU "completes" work but staging buffers read back all zeros

5. Single command buffer didn't help

I tried forcing all dispatches into one command buffer (setting execute_threshold_node_count to UINT32_MAX). Same result — all zeros.

Related

• PR Fix Vulkan texture tensor UBO budget overflow #17294 — my fix for a separate UBO budget crash (uniform data allocation has exceeded tensor uniform buffer size). That fix prevents a crash but does not affect the zero-output issue.
• cases.py:1464-1470 — there's an existing TODO noting Android arm64 failures where "writes from the first or second shader dispatch being 'ignored'" which matches my symptoms exactly.

Questions

1. Is PowerVR expected to work at all with the Vulkan backend? Or is it currently untested/unsupported?
2. Could the texture_limits partitioner option be made aware of texel packing so it avoids delegating ops whose packed extents exceed maxImageDimension3D?
3. Should the texture extent check in Tensor.cpp:632-660 be enabled in release builds (not just debug)?
4. Any other suggestions for debugging this? Happy to add more traces or test patches.

cc @SS-JIA @manuelcandales @digantdesai @cbilgin

[SS-JIA]

Is PowerVR expected to work at all with the Vulkan backend? Or is it currently untested/unsupported?

It is expected to work - however, we don't have any devices with PowerVR GPUs on hand to test with which is why we haven't caught this issue earlier, unfortunately.

Could the texture_limits partitioner option be made aware of texel packing so it avoids delegating ops whose packed extents exceed maxImageDimension3D?

This should be happening already - when you exported with texture_limits: (2048, 2048, 2048), did you still observe the texture limits being exceeded?

Should the texture extent check in Tensor.cpp:632-660 be enabled in release builds (not just debug)?

Yes, I'll look into enabling this.

Any other suggestions for debugging this? Happy to add more traces or test patches.

I would like to focus on MobileNet's NaN output first, since iirc MobileNet shouldn't be using any large tensors that would exceed texture limits. Would you be able to create a test script that extracts progressively larger parts of the mobileNet model, executes it on device, and then checks for NaN in the outputs? You can see #17224 as a reference, which is a script I created with AI to debug another model. The script executes the model on the host device, but the mobilenet script should instead push the model file and execute via adb.

The intention is to try to see exactly "when" the NaN values get produced - I am suspecting that there is a specific shader/operator that is producing incorrect outputs, so I want to try to narrow down which operator that could be.

Thanks in advance!

[abdelaziz-mahdy]

@SS-JIA Thanks for the quick response and for self-assigning!

Progressive Model Slicing Results

Following your suggestion (and referencing #17224), I created a progressive debug script that exports increasingly larger slices of MobileNetV3-Small with the Vulkan backend, then runs them on the Pixel 10 Pro.

Important note on export tooling: I initially hit a crash (Missing operator: et_vk.binary_add_with_clamp.default) which turned out to be caused by exporting models with pip ExecuTorch 1.0.1 — that release contains fusion ops from PR #14960 that were reverted in PR #15066. After switching to ExecuTorch from source (main branch, commit 4930e7ce92), the crash went away and all ops register correctly. Just flagging this in case others hit it.

Export Setup

All models exported from source (main, 4930e7ce92) with:

VulkanPartitioner(compile_options={
    "texture_limits": (2048, 2048, 2048),
    "force_fp16": True,
})

Deterministic input: value[i] = (i % 256) / 255.0 (non-zero, [0..1] range).

Results

#	Model Slice	Layers	Output Shape	Elements	Result
1	features[0:1]	Conv2d(3→16, 3x3, s2) + BN + Hardswish	[1, 16, 112, 112]	200,704	ALL ZEROS
2	features[0:2]	+ InvertedResidual(16→16)	[1, 16, 56, 56]	50,176	ALL ZEROS
3	features[0:4]	+ 2 more InvertedResidual blocks	[1, 24, 28, 28]	18,816	ALL ZEROS
4	features[0:7]	+ 3 more blocks (introduces SE)	[1, 40, 14, 14]	7,840	ALL NaN (7,840/7,840)
5	features[0:9]	+ 2 more blocks	[1, 48, 14, 14]	9,408	ALL NaN (9,408/9,408)
6	features[0:13]	All 13 feature blocks	[1, 576, 7, 7]	28,224	ALL NaN (28,224/28,224)
7	features + avgpool	+ AdaptiveAvgPool2d + Flatten	[1, 576]	576	ALL NaN
8	classifier only	Linear(576→1024→1000)	[1, 1000]	1,000	ALL NaN
9	full model	Complete MobileNetV3-Small	[1, 1000]	1,000	ALL NaN

Analysis

The failure starts at the very first layer. features[0:1] is just:

Conv2d(3, 16, kernel_size=3, stride=2, padding=1) → BatchNorm2d(16) → Hardswish

It outputs 200,704 elements, all exactly 0.0.

The VK_TRACE logs for this slice show these ops all registered successfully:

build_graph: op='et_vk.prepack.default' registered=1       # weight
build_graph: op='et_vk.prepack.default' registered=1       # bias
build_graph: op='aten.convolution.default' registered=1    # ← the conv
build_graph: op='dim_order_ops._to_dim_order_copy.default' registered=1
build_graph: op='dim_order_ops._to_dim_order_copy.default' registered=1
build_graph: op='aten.clone.default' registered=1
build_graph: op='aten.add.Tensor' registered=1              # hardswish: x+3
build_graph: op='aten.clamp.default' registered=1           # clamp(_, 0, 6)
build_graph: op='aten.clamp.default' registered=1
build_graph: op='aten.mul.Tensor' registered=1              # x * _
build_graph: op='aten.clone.default' registered=1
build_graph: op='aten.div.Tensor' registered=1              # _ / 6

Since hardswish(0) = 0 * clamp(0+3, 0, 6) / 6 = 0, the zeros propagate correctly through hardswish. The root issue is aten.convolution.default itself returning all zeros.

The ALL ZEROS → ALL NaN transition at features[0:7] coincides with the introduction of Squeeze-and-Excite blocks, which involve aten.mean.dim (global avg pool of all-zeros → 0) followed by division, producing NaN.

Regarding texture_limits

when you exported with texture_limits: (2048, 2048, 2048), did you still observe the texture limits being exceeded?

For these MobileNet slices, tensor dimensions are well within the 2048 limit (largest is [1, 576, 7, 7]). The texture limit issue from my original report was specific to YOLO's [1, 84, 8400] output where packing turns 8400 → 2100 texels. The MobileNet issue is purely about the convolution shader producing wrong output — no texture limits involved.

Confirmation: XNNPACK works perfectly

Same device, same models — all 39 standard XNNPACK integration tests pass on the Pixel 10 Pro. The issue is exclusively in the Vulkan backend.

Scripts

I created a Python export script (mobilenet_debug.py) following the pattern in #17224. It supports --test-host and --test-device modes:

mobilenet_debug.py (click to expand)

# Progressive MobileNet V3 Small debug script for Vulkan backend.
# Exports partial models to find which operator produces NaN/incorrect outputs.
#
# Usage:
#   # Export all partial models (saves .pte files to output dir)
#   python mobilenet_debug.py --output-dir /path/to/output
#
#   # Export and test on host (no device needed)
#   python mobilenet_debug.py --test-host
#
#   # Export and test on Android device via adb
#   python mobilenet_debug.py --test-device

import argparse
import logging
import os
import subprocess
import tempfile

import torch
import torch.nn as nn
import torchvision.models as models

FORMAT = "[%(levelname)s %(asctime)s %(filename)s:%(lineno)s] %(message)s"
logging.basicConfig(level=logging.INFO, format=FORMAT)


class FeaturesSlice(nn.Module):
    """First N feature blocks of MobileNet V3 Small."""
    def __init__(self, model, end_idx):
        super().__init__()
        self.features = nn.Sequential(*list(model.features.children())[:end_idx])

    def forward(self, x):
        return self.features(x)


class FeaturesOnly(nn.Module):
    """All 13 feature blocks (full backbone)."""
    def __init__(self, model):
        super().__init__()
        self.features = model.features

    def forward(self, x):
        return self.features(x)


class FeaturesAndPool(nn.Module):
    """Features + adaptive avg pool."""
    def __init__(self, model):
        super().__init__()
        self.features = model.features
        self.avgpool = model.avgpool

    def forward(self, x):
        x = self.features(x)
        x = self.avgpool(x)
        return torch.flatten(x, 1)


class ClassifierOnly(nn.Module):
    """Just the classifier (takes flattened 576-d input)."""
    def __init__(self, model):
        super().__init__()
        self.classifier = model.classifier

    def forward(self, x):
        return self.classifier(x)


class FullModel(nn.Module):
    """Complete MobileNet V3 Small."""
    def __init__(self, model):
        super().__init__()
        self.features = model.features
        self.avgpool = model.avgpool
        self.classifier = model.classifier

    def forward(self, x):
        x = self.features(x)
        x = self.avgpool(x)
        x = torch.flatten(x, 1)
        x = self.classifier(x)
        return x


def export_partial(name, model, example_inputs, output_dir, vulkan_options=None):
    """Export a partial model to .pte with Vulkan delegate."""
    from executorch.backends.vulkan.partitioner.vulkan_partitioner import VulkanPartitioner
    from executorch.exir import to_edge_transform_and_lower
    from executorch.extension.export_util.utils import save_pte_program
    from torch.export import export

    model.eval()
    safe_name = name.lower().replace(" ", "_").replace("+", "_")

    logging.info(f"Exporting: {name}")

    program = export(model, example_inputs, strict=True)

    if vulkan_options is None:
        vulkan_options = {
            "texture_limits": (2048, 2048, 2048),
            "force_fp16": True,
        }

    edge_program = to_edge_transform_and_lower(
        program,
        partitioner=[VulkanPartitioner(vulkan_options)],
    )

    exec_prog = edge_program.to_executorch()

    filename = f"mobilenet_debug_{safe_name}_vulkan"
    os.makedirs(output_dir, exist_ok=True)
    save_pte_program(exec_prog, filename, output_dir)
    filepath = os.path.join(output_dir, f"{filename}.pte")
    logging.info(f"  Saved: {filepath} ({os.path.getsize(filepath) / 1024:.0f} KB)")

    return exec_prog, filepath


def main():
    parser = argparse.ArgumentParser(description="MobileNet V3 Small Vulkan debug")
    parser.add_argument("--output-dir", default="./mobilenet_debug_models")
    parser.add_argument("--test-host", action="store_true")
    parser.add_argument("--test-device", action="store_true")
    args = parser.parse_args()

    logging.info("Loading MobileNet V3 Small (pretrained)...")
    base_model = models.mobilenet_v3_small(weights="DEFAULT").eval()

    example_input = torch.randn(1, 3, 224, 224)

    with torch.no_grad():
        features_out = base_model.features(example_input)
        pooled = base_model.avgpool(features_out)
        flattened = torch.flatten(pooled, 1)

    test_cases = [
        ("features_0_1", FeaturesSlice(base_model, 1), (example_input,)),
        ("features_0_2", FeaturesSlice(base_model, 2), (example_input,)),
        ("features_0_4", FeaturesSlice(base_model, 4), (example_input,)),
        ("features_0_7", FeaturesSlice(base_model, 7), (example_input,)),
        ("features_0_9", FeaturesSlice(base_model, 9), (example_input,)),
        ("features_all", FeaturesOnly(base_model), (example_input,)),
        ("features_pool", FeaturesAndPool(base_model), (example_input,)),
        ("classifier_only", ClassifierOnly(base_model), (flattened,)),
        ("full_model", FullModel(base_model), (example_input,)),
    ]

    for name, model, inputs in test_cases:
        model.eval()
        exec_prog, pte_path = export_partial(name, model, inputs, args.output_dir)

    logging.info(f"Exported {len(test_cases)} models to {args.output_dir}")


if __name__ == "__main__":
    with torch.no_grad():
        main()

Raw Test Logs

features[0:1] — First conv block (ALL ZEROS)

Testing: features_0_1 (First conv block only)
  Asset: assets/debug_models/mobilenet_debug_features_0_1_vulkan.pte
  Input shape: [1, 3, 224, 224]

VK_TRACE build_graph: op='et_vk.prepack.default' registered=1
VK_TRACE build_graph: op='et_vk.prepack.default' registered=1
VK_TRACE build_graph: op='aten.convolution.default' registered=1
VK_TRACE build_graph: op='dim_order_ops._to_dim_order_copy.default' registered=1
VK_TRACE build_graph: op='dim_order_ops._to_dim_order_copy.default' registered=1
VK_TRACE build_graph: op='aten.clone.default' registered=1
VK_TRACE build_graph: op='aten.add.Tensor' registered=1
VK_TRACE build_graph: op='aten.clamp.default' registered=1
VK_TRACE build_graph: op='aten.clamp.default' registered=1
VK_TRACE build_graph: op='aten.mul.Tensor' registered=1
VK_TRACE build_graph: op='aten.clone.default' registered=1
VK_TRACE build_graph: op='aten.div.Tensor' registered=1

  Model loaded successfully (status: 0)
  Forward pass completed (status: 0)
  Output shape: [1, 16, 112, 112]
  Total elements: 200704
  First 8 values: [0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000]
  NaN count: 0
  Inf count: 0
  Zero count: 200704
  All zero: true
  Min: 0.0, Max: 0.0
  RESULT: FAIL - All zeros

features[0:2] — First 2 blocks (ALL ZEROS)

Testing: features_0_2 (First 2 blocks)
  Input shape: [1, 3, 224, 224]

VK_TRACE build_graph: op='et_vk.prepack.default' registered=1  (x4)
VK_TRACE build_graph: op='aten.convolution.default' registered=1
VK_TRACE build_graph: op='dim_order_ops._to_dim_order_copy.default' registered=1  (x4)
VK_TRACE build_graph: op='aten.clone.default' registered=1
VK_TRACE build_graph: op='aten.add.Tensor' registered=1
VK_TRACE build_graph: op='aten.clamp.default' registered=1  (x2)
VK_TRACE build_graph: op='aten.mul.Tensor' registered=1
VK_TRACE build_graph: op='aten.clone.default' registered=1
VK_TRACE build_graph: op='aten.div.Tensor' registered=1
VK_TRACE build_graph: op='et_vk.conv_with_clamp.default' registered=1
VK_TRACE build_graph: op='aten.mean.dim' registered=1
VK_TRACE build_graph: op='et_vk.conv_with_clamp.default' registered=1
VK_TRACE build_graph: op='aten.convolution.default' registered=1  (x2)

  Output shape: [1, 16, 56, 56]
  Total elements: 50176
  First 8 values: [0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000]
  Zero count: 50176
  All zero: true
  RESULT: FAIL - All zeros

features[0:4] — First 4 blocks (ALL ZEROS)

Testing: features_0_4 (First 4 blocks)
  Output shape: [1, 24, 28, 28]
  Total elements: 18816
  First 8 values: [0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000, 0.000000]
  Zero count: 18816
  All zero: true
  RESULT: FAIL - All zeros

features[0:7] — First 7 blocks with SE (ALL NaN)

Testing: features_0_7 (First 7 blocks)
  Output shape: [1, 40, 14, 14]
  Total elements: 7840
  First 8 values: [NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN]
  NaN count: 7840
  Zero count: 0
  Min: Infinity, Max: -Infinity
  RESULT: FAIL - NaN detected

features[0:9] through full_model (ALL NaN)

features_0_9:   [1, 48, 14, 14]  — NaN: 9408/9408
features_all:   [1, 576, 7, 7]   — NaN: 28224/28224
features_pool:  [1, 576]          — NaN: 576/576
classifier_only:[1, 1000]         — NaN: 1000/1000
full_model:     [1, 1000]         — NaN: 1000/1000

Summary

The issue narrows down to aten.convolution.default producing all-zero output on PowerVR. This is a standard 3x3 convolution with no large tensors — the smallest possible test case. All ops register and execute without error; the runtime reports success but the output buffer contains only zeros.

Happy to run any additional traces or test patches you'd like. I have the device available and can iterate quickly.

[abdelaziz-mahdy]

Following up on my previous comment, I ran more targeted tests to narrow down the issue. I also opened a draft PR (#17323) with initial PowerVR support work (device detection, workgroup tuning, robustBufferAccess).

All tests below were run with the changes from #17323 applied, built from source against main.

Minimal Single-Op Tests

I exported 13 minimal single-operator models to isolate exactly which operations fail. Each model contains a single op (e.g. one Conv2d, one Linear, one Add) and runs with a known constant input so I can verify the output.

Model	Type	Expected	Actual	Status
bare_add (x+1.0)	FP16	2.0	2.0	PASS
bare_half (x*0.5)	FP16	0.5	0.5	PASS
zero_conv (all-zero weights, no bias)	FP16	0.0	0.5	FAIL (+0.5)
identity_conv (1x1, input=5.0)	FP16	5.0	5.5	FAIL (+0.5)
bare_conv2d (3->4, 3x3, no bias)	FP16	0.8403	1.3400	FAIL (+0.5)
bare_conv2d_bias (3->4, 3x3, bias=0.5)	FP16	1.3403	1.3400	PASS*
bare_linear (16->4)	FP16	1.6	2.0996	FAIL (+0.5)
bare_conv2d_fp32 (3->4, 3x3, no bias)	FP32	0.8403	0.8403	PASS
bare_conv2d_large (3->16, 224x224)	FP16	0.9941	1.4940	FAIL (+0.5)
bare_conv2d_ones (weights=1.0)	FP16	22.6875	23.1875	FAIL (+0.5)
bare_conv2d_dw (depthwise, groups=4)	FP16	0.8403	1.3395	FAIL (+0.5)
bare_conv2d_pw (pointwise 1x1, w=0.5)	FP16	1.5	2.0	FAIL (+0.5)
bare_conv2d_xnnpack (CPU reference)	FP32	0.8403	0.8403	PASS

*bare_conv2d_bias: the output is identical to bare_conv2d (no bias) — the explicit bias value (0.5) appears to be ignored or overwritten.

Observations

• FP32 works perfectly — bare_conv2d_fp32 matches the expected output exactly
• FP16 conv/linear ops have a consistent +0.5 offset — every FP16 conv and linear test shows exactly +0.5
• Non-conv FP16 ops work — bare_add and bare_half produce correct results
• The offset is deterministic — same +0.5 every run, not random
• Explicit bias is ignored — bare_conv2d_bias (bias=0.5) produces the same output as bare_conv2d (no bias)
• All conv variants affected — regular, depthwise, pointwise, and linear all show +0.5

MobileNet Progressive Slicing

All MobileNet slices produce ALL NaN:

Slice	Output Shape	Result
features[0:1] (Conv+BN+Hardswish)	[1,16,112,112]	ALL NaN
features[0:2]	[1,16,56,56]	ALL NaN
features[0:4]	[1,24,28,28]	ALL NaN
features[0:7] (introduces SE)	[1,40,14,14]	ALL NaN
features[0:9]	[1,48,14,14]	ALL NaN
features_all	[1,576,7,7]	ALL NaN
features_pool	[1,576]	ALL NaN
classifier_only (Linear only)	[1,1000]	ALL NaN
full_model	[1,1000]	ALL NaN

The NaN likely cascades from the +0.5 bias offset through batch normalization (incorrect mean/variance → division by near-zero → NaN).

Draft PR

I opened #17323 as a draft with initial PowerVR support:

• PowerVR device detection (POWERVR in DeviceType enum)
• Conservative workgroup sizes (32 instead of 64) for TBDR architecture
• robustBufferAccess enabled
• Optimal tiling forced
• vmaFlushAllocation added to set_staging_zeros() and cast_and_copy_from() (correctness fix, didn't resolve the +0.5 issue)

The core FP16 +0.5 offset issue is still unresolved — I suspect it's in the FP16 prepack/texture path but haven't pinpointed it yet.

[SS-JIA]

@abdelaziz-mahdy

I see, from the operator test it seems that there is currently an issue with FP16 inference on the GPU. However, when you execute/export the mobilenet model, does it use fp16 inference? If so, have you tried with fp32 inference? How are you exporting the model?

[abdelaziz-mahdy]

@SS-JIA

Yes, the previous MobileNet tests were exported with force_fp16: True. Following your suggestion, I exported and tested FP32 MobileNet as well.

Export Setup

FP32 models exported from source (main, 4930e7ce92) with:

VulkanPartitioner(compile_options={
    "texture_limits": (2048, 2048, 2048),
    # no force_fp16
})

Real Image Test (cat.jpg)

I tested all three backends with a real cat image, preprocessed with standard ImageNet normalization (224x224, mean/std):

Backend	Result	Details
XNNPACK (CPU)	PASS	Top-1: class 281 (tabby) at 37.1%, 4 cat classes in top-5
FP32 Vulkan	FAIL	All 1000 outputs = NaN
FP16 Vulkan	FAIL	All 1000 outputs = NaN

FP32 Vulkan also produces NaN with real image input on PowerVR.

Bare Single-Op Tests Still Pass

The confusing part is that bare single-operator FP32 conv tests pass:

Test	Type	Expected	Actual	Status
bare_conv2d_fp32 (3->4, 3x3)	FP32	0.8403	0.8403	PASS
bare_conv2d_xnnpack (CPU ref)	FP32	0.8403	0.8403	PASS

So a single FP32 Conv2d works, but a full FP32 MobileNet model produces NaN. The failure happens somewhere in the composition of multiple operators.

FP32 Progressive Slices (uniform input = 1.0)

Slice	Output	Result
features[0:1] (Conv+BN+Hardswish)	Near-zero (expected for uniform input through BN)	No NaN
features[0:4] (4 blocks)	Mean=165.3, range [61.2, 175.5]	Non-zero, no NaN
full model	All zeros (1000/1000)	No NaN but all zeros

With uniform 1.0 input, FP32 partial slices don't NaN. But with real image input, the full model does NaN. This suggests the issue may be input-dependent or triggered by specific value ranges that a real image produces but uniform input doesn't.

Summary

• FP32 and FP16 Vulkan both fail on PowerVR for full MobileNet with real image input
• Bare single-op FP32 conv works correctly
• The failure is somewhere in the multi-operator pipeline, possibly triggered by specific value ranges
• XNNPACK (CPU) works perfectly on the same device

Happy to run any additional tests or patches.

[abdelaziz-mahdy]

Update: Root Cause Found — Hardswish/Hardsigmoid Decomposition

TL;DR

The NaN is caused by aten.hardswish and aten.hardsigmoid being decomposed into primitive ops (mul/add/clamp/div with constant tensors) before reaching the Vulkan backend. The Vulkan backend has native GLSL shaders for both ops that work correctly on PowerVR, but they never get used because the ops are decomposed during to_edge().

Root Cause

PyTorch's default decomposition table decomposes:

• hardswish(x) → x * clamp(x + 3, 0, 6) / 6
• hardsigmoid(x) → clamp(x + 3, 0, 6) / 6

The decomposed path loads constant scalars (3 and 6) via dim_order_ops._to_dim_order_copy (buffer→texture conversion). On PowerVR GPUs, this produces Inf values, which cascade into NaN through the rest of the network.

The Vulkan backend already has native GLSL shaders for both ops in activations.h, registered via VK_REGISTER_OP in UnaryOp.cpp. However, vulkan_partitioner.py's ops_not_to_decompose list only contained upsample_nearest2d.vec — missing hardswish, hardsigmoid, and other ops with native shaders that get decomposed (hardshrink, silu).

How I Found It

Isolation testing — exported 4 minimal single-op models:

Model	Decomposed Path	Native Shader Path
Conv2d only	OK (no NaN)	N/A
Hardswish only	NaN (7403/200704, first4=[Inf,Inf,Inf,Inf])	PERFECT MATCH (maxDiff=0.000000)
Conv2d + Hardswish	ALL NaN	0 NaN
Full MobileNet V3	ALL 1000 NaN	0 NaN (FP32 and FP16)

Fix

PR: #17361

Adds hardswish, hardsigmoid, hardshrink, silu to ops_not_to_decompose in vulkan_partitioner.py, and registers hardswish/hardsigmoid in op_registry.py.

Remaining Issue

After fixing the NaN, the full MobileNet outputs are numerically correct (no NaN) but all near-zero. This appears to be a separate convolution accuracy issue on PowerVR where small per-layer errors in aten.convolution compound through 13 feature blocks. Still investigating this separately.