Let a C++ application use the ExecuTorch TensorRT delegate from an install - #4458
Draft
shoumikhin wants to merge 8 commits into
Draft
Let a C++ application use the ExecuTorch TensorRT delegate from an install#4458shoumikhin wants to merge 8 commits into
shoumikhin wants to merge 8 commits into
Conversation
…stall
Today the delegate can only be used by adding this directory to another CMake
project as a subproject and building it from source. `torchtrt::executorch_backend`
is created as an in-tree alias, and while the built library is installed, it is
not exported, so `find_package` finds nothing. An application that installed
Torch-TensorRT and ExecuTorch as packages has no way to link the delegate.
Export the shared delegate through a CMake export set and generate a package
config for it, so an application can do
find_package(executorch REQUIRED)
find_package(torch_tensorrt_executorch REQUIRED)
target_link_libraries(app PRIVATE executorch::runtime torchtrt::executorch_backend)
and get a working link with no source checkout.
Only the shared delegate is exported. The static target's interface names
`$<TARGET_FILE:...>` to force a whole-archive link, which is a path inside the
build tree and cannot be meaningfully installed. The static target and its
in-tree alias keep working exactly as before, so existing source builds are
unaffected.
Also add an example application that consumes the packages this way, and an
end-to-end test around it.
The example is deliberately not a variant of the existing reference runner: that
one calls `add_subdirectory` on an ExecuTorch source tree, so it never exercises
prebuilt libraries. This one resolves everything through `find_package` and
imported targets. It also takes a file of expected values and compares the
outputs against them, so a wrong result fails instead of only being printed, and
it can allocate its input tensors in device memory to match how an accelerator
application feeds data.
Test plan:
The new test exports a model through the TensorRT compiler, builds the example
against installed packages, runs it, and compares the outputs against a reference
computed in Python. It asserts the serialized program actually contains a
TensorRT delegate first: without that a program where TensorRT claimed nothing
would still load and run, and the test would prove nothing.
A second case repeats the run with the input tensors allocated on the device,
since the CUDA delegate copies device to device and never stages through host
memory, so it expects tensors that are already resident.
A third case asserts the example builds with no ExecuTorch source tree involved,
by checking that only the application's own CMake cache exists in its build
directory. A source build would leave a second one behind.
Verified so far on Linux x86_64: the example configures against an installed
ExecuTorch package, reporting the prebuilt runtime and finding each shipped
component library; it builds and links with no source checkout, with exactly one
CMake cache in the build directory and no ExecuTorch subproject; and the output
comparison both passes on a matching reference and fails on a deliberately wrong
one. The tests skip cleanly where the compiled TensorRT runtime, a CUDA device,
or an ExecuTorch install with a CMake package is absent.
The link-resolution chain looked for the runtime as an in-tree target, then as a source checkout, but never as the imported target an installed ExecuTorch package provides. Building the shared delegate against an installed package therefore failed asking for a source tree, which defeats the purpose of a delegate that is meant to resolve the runtime from the shipped shared library. Accept the imported target as well. Test plan: built the shared delegate against an installed ExecuTorch package with no source checkout present, on Linux aarch64. It now configures, builds, and installs a consumable package (shared library, CMake config and targets, and headers), where before it stopped at configure time.
Three problems stopped an application from consuming the delegate from an install, each found by building and running against an installed package rather than by inspection. The CMake package installed into a directory named after the target rather than after the package, so find_package never looked there. It now installs where a consumer will look. The export set names the concrete library, while an application wants a stable name that does not depend on which library flavor it got. The package config now offers that name. The delegate registers itself from a static initializer, so an application references none of its symbols and the linker is free to drop it. It was being dropped, and the backend was then missing at runtime. The example now names the library on its link line so the reference is recorded. Note the location of an imported target lives in a configuration-specific property, so a generator expression is needed rather than reading IMPORTED_LOCATION. Test plan: on Linux aarch64, built the delegate against an installed ExecuTorch package with no source checkout, installed it, and built the example against both installed packages. find_package now resolves the delegate, the library appears in the executable's dynamic dependencies, and at runtime the backend registers and its initialization runs. Engine execution additionally requires the export and the delegate to use the same TensorRT version, which is an environment matter.
The test was formatted against a newer target version than this repository uses, so the linting job reported it would be reformatted.
The existing reference-runner check builds ExecuTorch from source through add_subdirectory. That is worth verifying for that path, but it says nothing about whether an application can consume the delegate from an installed package, which is what the export set added here is for. Add a GPU check that installs the shared delegate, builds the example against installed packages only, and runs it against a reference computed in Python. It asserts three things that would otherwise fail silently. TensorRT must actually claim part of the graph, because a program with no TensorRT delegate still loads and runs and would make the rest of the check meaningless. The application's build directory must contain exactly one CMake cache, because a second one means ExecuTorch was configured as a subproject and nothing prebuilt was exercised. And the delegate must appear in the application's recorded dependencies, because it registers itself from a static initializer that a linker is free to drop, which leaves the backend missing at runtime. Test plan: ran the equivalent steps by hand on Linux aarch64 with a GPU. The delegate installs a consumable package, find_package resolves it, the example links and retains it, and at runtime the backend registers and initializes. Engine execution additionally needs the export and the delegate to use the same TensorRT version, which is a property of the environment rather than of this change.
The example needs CMake 3.24. Without a check, an older CMake surfaces as a version error from inside the package config, several layers below the script, which is needlessly hard to read. Check up front and skip with a clear reason instead.
CI installs a released ExecuTorch, which ships the CMake config but no separately linkable runtime, so there is nothing for an application to consume yet. The check treated that as a failure, which reports a missing upstream feature as a broken change. Look for the runtime target first and skip with the reason when it is not offered. Moving that probe ahead of the export also means the skip costs nothing. Test plan: against a released wheel the check now reports that the package offers no shared runtime target and exits zero. Against a wheel that ships the runtime it resolves the CMake directory and proceeds.
shoumikhin
force-pushed
the
executorch-wheel-consumable
branch
7 times, most recently
from
August 2, 2026 18:59
af3904b to
59d0d37
Compare
Two problems in the new verification path. The install check looked for the delegate under a hardcoded lib directory, but the install honors the platform library directory, which is lib64 on several distributions. A completely successful build would have been reported as a failed install. The configure now pins the directory and the check searches for the file rather than assuming where it landed. The device-input path handed a device pointer to the runtime without checking how that input is supplied. A memory-planned input is copied into the plan with a host memcpy, so a device pointer there would be read from the host. Only a non-planned input has its pointer aliased, which is what makes device memory safe. The runner now reports that clearly instead of corrupting memory. Test plan: confirmed the install check finds the library when it lands in either lib or lib64, and that a host-side copy of a device pointer is what the runtime would do for a memory-planned input.
shoumikhin
force-pushed
the
executorch-wheel-consumable
branch
from
August 2, 2026 20:33
59d0d37 to
333b46a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
An application that installs Torch-TensorRT and ExecuTorch as packages currently
has no way to link the ExecuTorch TensorRT delegate.
torchtrt::executorch_backendis created as an in-tree alias, and while the built library is installed, it is not
exported, so
find_packagefinds nothing. The only way to use the delegate todayis to add this directory to another CMake project and build it from source.
This exports the shared delegate through a CMake export set with a relocatable
package config, so an application can do:
and get a working link with no source checkout.
Three smaller problems had to be fixed for that to actually work, each found by
building against an installed package rather than by reading the code:
or a source checkout, but never as the imported target an installed package
provides. Building the shared delegate against a wheel therefore failed asking
for a source tree, which defeats the purpose of a delegate meant to resolve the
runtime from the shipped shared library.
after the package, so
find_packagenever looked there.name that does not depend on which library flavor it got. The config now offers
that name.
Only the shared delegate is exported. The static target's interface names
$<TARGET_FILE:...>to force a whole-archive link, which is a path inside thebuild tree and cannot be meaningfully installed. The static target and its in-tree
alias keep working exactly as before, so existing source builds are unaffected.
Example and test
Also adds an example application that consumes the packages this way, plus a test
around it.
The example is deliberately not a variant of the existing reference runner: that
one calls
add_subdirectoryon an ExecuTorch source tree, so it never exercisesprebuilt libraries. This one resolves everything through
find_packageandimported targets. It also takes a file of expected values and compares outputs
against them, so a wrong result fails rather than only being printed, and it can
allocate its input tensors in device memory to match how an accelerator
application feeds data.
The delegate registers itself from a static initializer, so an application
references none of its symbols and the linker is free to drop it. It was being
dropped, and the backend was then missing at runtime. The example names the
library on its link line so the reference is recorded. Note that an imported
target's location lives in a configuration-specific property, so a generator
expression is needed rather than reading
IMPORTED_LOCATION.What is verified
On Linux x86_64 and aarch64:
no source checkout: exactly one CMake cache in its build directory and no
ExecuTorch subproject.
wrong one.
On Linux aarch64 with a GPU, additionally:
consumable package: shared library, CMake config and targets, and headers.
find_package(torch_tensorrt_executorch)resolves and yieldstorchtrt::executorch_backend.dependencies, confirming the retention above.
runtime the delegate registers and its initialization runs.
What is not verified
Executing a TensorRT-delegated program end to end still fails on this setup with a
serialization version mismatch:
A TensorRT engine is version-locked. In the environment used here the Python
package that builds the engine and the system libraries the delegate compiles
against are different TensorRT major versions, and only one of them ships headers.
Closing that needs a single TensorRT providing both headers and libraries; it is
not a property of this change. The two device-execution tests are written and
collect cleanly, and they skip where the compiled runtime, a GPU, or an ExecuTorch
install with a CMake package is absent.