[ET Device Support] Add ExecutorchBackendConfig flags for skipping H2D/D2H copies

[Gasoonjia]

Stack from ghstack (oldest at bottom):

• [ET Device Support] Add device tensor helper functions to TensorPtr API #18761
• [ET Device Support] make device support config method-based #19582
• -> [ET Device Support] Add ExecutorchBackendConfig flags for skipping H2D/D2H copies #18760
• [ET Device Support] PropagateDevicePass inserts H2D/D2H copy ops at delegate boundaries #18730
• [ET Device Support] Define et_copy runtime h2d and d2h copy ops #18729
• [ET Device Support] Define AOT device copy ops registry #18728
• [ET Device Support] CudaAllocator: device memory allocator for CUDA backend #18477
• [ET Device Support] Module: allocate device memory for planned buffers #18476
• [ET Device Support] MemoryManager: add per-buffer device metadata #18475

Add skip_h2d_for_method_inputs and skip_d2h_for_method_outputs config
flags to ExecutorchBackendConfig. These control whether PropagateDevicePass
skips inserting H2D/D2H copy ops at method I/O boundaries:

• skip_h2d_for_method_inputs: user provides GPU tensor directly
• skip_d2h_for_method_outputs: output stays on device for cross-method pipelines

Differential Revision: D99636778

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/18760

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• Run pull request jobs on OSDC runners in shadow mode

❌ 70 New Failures, 1 Cancelled Job, 1 Unrelated Failure, 4 Unclassified Failures

As of commit 3a15470 with merge base 3cc1ba9 ():

NEW FAILURES - The following jobs have failed:

• Build Presets / apple (pybind) / build (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 2
• Build Presets / linux (pybind, linux.2xlarge, executorch-ubuntu-22.04-clang12) / build (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• Build Presets / linux (pybind, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / build (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• Build Presets / windows (pybind) / build (gh)
  Process completed with exit code 1.
• pull / android / build-android (gh)
  Process completed with exit code 2.
• pull / test-coreml-bc-macos (macos-m1-stable) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / test-coreml-bc-macos (macos-m2-stable) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux (fp32, xnnpack+custom+qe, linux.arm64.2xlarge, executorch-ubuntu-22.04-gc... / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04... / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu... / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux (fp32, xnnpack+quantize_kv, linux.arm64.2xlarge, executorch-ubuntu-22.04-... / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-llama-runner-linux-android / linux-job (gh)
  RuntimeError: Command docker exec -t b60a27fab392e930a30faecee600109ac767b1b31eb0c98665dbb38b3cb0fb24 /exec failed with exit code 2
• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• pull / test-lora-linux / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-lora-multimethod-linux / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-mcu-cortex-m-backend / linux-job (gh)
  undefined reference to executorch::runtime::get_device_allocator(executorch::runtime::etensor::DeviceType)'`
• pull / test-models-linux (add_mul, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t b83eae712c4ad45f7501c8030822ab4f3f3acaff4196821445279413d8dce142 /exec failed with exit code 1
• pull / test-models-linux (add, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 5660b3daf456960e02badc30806e8562a3fd5d50b40dd0651a7ed416a166e202 /exec failed with exit code 1
• pull / test-models-linux (emformer_join, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t 48273c080f90e6a56b06d4daa67d62ff0c39fd32864e1772b32bc987a7303ef9 /exec failed with exit code 1
• pull / test-models-linux (emformer_transcribe, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 10b2b60e4921b8435fcfd853a6f999c62ab8804e35625ea4dc0e2763d179f3f7 /exec failed with exit code 1
• pull / test-models-linux (ic3, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 40f0a3684b7194a9c96c0e7344ceedecaa6eea1d39abf9c032b6c65c5e311231 /exec failed with exit code 1
• pull / test-models-linux (ic4, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t 987bf96c7a86fc5f2e435bf6e3d426dcdd4e2f787bef64cf09475929d32187bf /exec failed with exit code 1
• pull / test-models-linux (linear, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d6190ea023da80e08eaeacb584ab07d2bbdeb0d4758529c065ffedf5619ac9b8 /exec failed with exit code 1
• pull / test-models-linux (mobilebert, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d80457dac17d2176813709cff3af7dad60b3c9d032ac115e13b137e76b7ce7b9 /exec failed with exit code 1
• pull / test-models-linux (mv2, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 2669731a4c0a1ef5d56fc5bfc64c8c84bda3793b68d5a061beead6ccabda7bfc /exec failed with exit code 1
• pull / test-models-linux (resnet18, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 1b12cf4f8b12436f935e1abc800be64992c1ad8358a03a4fb7ef5036189c53be /exec failed with exit code 1
• pull / test-models-linux (resnet50, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 3070076add71ef094009955efee71b6adebd56251f0a7ea3ef70f51819bf4a4b /exec failed with exit code 1
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
  RuntimeError: Command docker exec -t 7ea38e37d76c7f439c76ded57657c78a2eb1c833580b67ad09cb2dca371b19f0 /exec failed with exit code 1
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
  RuntimeError: Command docker exec -t b881bf616f51e41bb28187150e0ffcf725f09a71fe3b52d4820cf191d6f80809 /exec failed with exit code 1
• pull / test-models-linux-basic (mv3, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
  RuntimeError: Command docker exec -t e3365359f947c91577f9775af4fbb0c7adb93312a26488591375d1babdf2bf8d /exec failed with exit code 1
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, buck2, linux.2xlarge, executorch-u... / linux-job (gh)
  RuntimeError: Command docker exec -t 94b3b4b5ab10a6d514de85b71fa71c36a3fe43fd6ddf73c6228d2f9db303309f /exec failed with exit code 1
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.2xlarge, executorch-u... / linux-job (gh)
  RuntimeError: Command docker exec -t a773dfe998f871ec55b7e99d2594b0e15c57f3e4cce6e1826edc57bff683b6e8 /exec failed with exit code 1
• pull / test-models-linux-basic (vit, xnnpack-quantization-delegation, cmake, linux.arm64.2xlarge, execut... / linux-job (gh)
  RuntimeError: Command docker exec -t 533d352a358a4586b72de5ba42a1757ca678553eee93cd1d3495f19e46bd3670 /exec failed with exit code 1
• pull / test-moshi-linux / linux-job (gh)
  test_mimi
• pull / test-multimodal-linux (gemma3-4b) / linux-job (gh)
  RuntimeError: Command docker exec -t 095b2550a6367e9d92cf4825be1c1a8a826e9f2ca31091f8debd3dc5ea24122c /exec failed with exit code 1
• pull / test-openvino-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 5c7a6911a30b41fe8ae444b0642e0b12efb9fb65686fce2f46b3d1217e85fc10 /exec failed with exit code 1
• pull / test-parakeet-xnnpack-linux / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• pull / test-phi-3-mini-runner-linux / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• pull / test-qnn-delegate-linux / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• pull / test-qnn-models-linux (dl3) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• pull / test-qnn-models-linux (mv2) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• pull / test-qnn-models-linux (mv3) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• pull / test-qnn-passes-linux / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• pull / test-qnn-python-imports-linux / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• pull / test-qnn-testsuite-linux / test-backend-linux (qnn, models) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• pull / test-qnn-testsuite-linux / test-backend-linux (qnn, operators) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• pull / test-sqnr-static-llm-qnn-linux (smollm2_135m) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• pull / test-static-llama-qnn-linux (stories_110m) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• pull / test-static-llama-qnn-linux (stories_260k_bc) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• pull / test-voxtral-realtime-xnnpack-linux / linux-job (gh)
  AttributeError: '_OpNamespace' 'llama' object has no attribute 'sdpa_with_kv_cache'
• pull / test-vulkan-models-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 4207ee8773a4ab6c94e518a043e20e71dd866adf8913964ff201f4c4425afc71 /exec failed with exit code 1
• pull / test-vulkan-operators-linux / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• pull / unittest / linux / linux-job (gh)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• pull / unittest / macos / macos-job (gh)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• pull / unittest / windows / windows-job (gh)
  Process completed with exit code 1.
• pull / unittest-arm-backend-with-no-deps (test_pytest_ops_no_target) / linux-job (gh)
  RuntimeError: Command docker exec -t 210cac9899a08714921642bb7821ac494171e20448ca10c3d3c2cfac0a800884 /exec failed with exit code 1
• pull / unittest-buck / linux / linux-job (gh)
  kernels/portable/cpu/test/op__device_copy_test.cpp:95:28: error: variable type '(anonymous namespace)::MockDeviceAllocator' is an abstract class
• pull / unittest-buck / macos / macos-job (gh)
  kernels/portable/cpu/test/op__device_copy_test.cpp:95:28: error: variable type 'MockDeviceAllocator' is an abstract class
• pull / unittest-editable / linux / linux-job (gh)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• pull / unittest-editable / macos / macos-job (gh)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• pull / unittest-editable / windows / windows-job (gh)
  Process completed with exit code 1.
• pull / unittest-nxp-neutron / linux-job (gh)
  RuntimeError: Command docker exec -t 7e92e35fa8fc3fe8ab987620862c56adaedbf17c177a1f12a855071bac170cf3 /exec failed with exit code 1
• pull / unittest-wasm-bindings (--enable-etdump) / linux-job (gh)
  RuntimeError: Command docker exec -t b5173285c468a186bce369d4daee020d49bac20dde96cb35308e9fa44ef4e750 /exec failed with exit code 2
• pull / unittest-wasm-bindings / linux-job (gh)
  RuntimeError: Command docker exec -t 61209befa70a8c488e4c1b71fba9d2afbf607a82648ee23631a262fa9b717294 /exec failed with exit code 2
• Test QNN Backend / test-qnn / test-backend-linux (qnn, models) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`
• Test QNN Backend / test-qnn / test-backend-linux (qnn, operators) / linux-job (gh)
  undefined reference to executorch::runtime::DeviceMemoryBuffer::create(unsigned long, executorch::runtime::etensor::DeviceType, signed char, unsigned long)'`

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Check Labels / Check labels (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: GraphQL query
• periodic / test-models-linux (buck2, mv3, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t e1ca3c9bee461570fe07c90c0f6d2ec5ff8134b35c53d7080d467897016d356a /exec failed with exit code 1
• periodic / test-models-linux (cmake, mv3, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t 784edf7349f66f942b587e213e5b15790cab7e295b7e379ba72a0930db3bfa0c /exec failed with exit code 1
• periodic / test-models-linux (cmake, vit, xnnpack-quantization-delegation, linux.2xlarge, 90) / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t 49653a7aa0b8e33aa2484076db9218d5a186868e6a1eab31b7f0b1cadc94896b /exec failed with exit code 1

CANCELLED JOB - The following job was cancelled. Please retry:

• pull / unittest-arm-backend-with-no-deps (test_pytest_ops_tosa) / linux-job (gh)
  ##[error]The operation was canceled.

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• pull / unittest-arm-backend-with-no-deps (test_pytest_models_tosa) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[digantdesai]

Review automatically exported from Phabricator review in Meta.

[JacobSzwejbka]

Review automatically exported from Phabricator review in Meta.