Skip to content

Let a C++ application use the ExecuTorch TensorRT delegate from an install - #4458

Draft
shoumikhin wants to merge 8 commits into
pytorch:mainfrom
shoumikhin:executorch-wheel-consumable
Draft

Let a C++ application use the ExecuTorch TensorRT delegate from an install#4458
shoumikhin wants to merge 8 commits into
pytorch:mainfrom
shoumikhin:executorch-wheel-consumable

Conversation

@shoumikhin

Copy link
Copy Markdown
Contributor

What this changes

An application that installs Torch-TensorRT and ExecuTorch as packages currently
has no way to link the ExecuTorch TensorRT delegate. torchtrt::executorch_backend
is created as an in-tree alias, and while the built library is installed, it is not
exported, so find_package finds nothing. The only way to use the delegate today
is to add this directory to another CMake project and build it from source.

This exports the shared delegate through a CMake export set with a relocatable
package config, so an application can do:

find_package(executorch REQUIRED)
find_package(torch_tensorrt_executorch REQUIRED)
target_link_libraries(app PRIVATE
    executorch::runtime
    torchtrt::executorch_backend
)

and get a working link with no source checkout.

Three smaller problems had to be fixed for that to actually work, each found by
building against an installed package rather than by reading the code:

  1. The link-resolution chain looked for the ExecuTorch runtime as an in-tree target
    or a source checkout, but never as the imported target an installed package
    provides. Building the shared delegate against a wheel therefore failed asking
    for a source tree, which defeats the purpose of a delegate meant to resolve the
    runtime from the shipped shared library.
  2. The package config installed into a directory named after the target rather than
    after the package, so find_package never looked there.
  3. The export set names the concrete library, while an application wants a stable
    name that does not depend on which library flavor it got. The config now offers
    that name.

Only the shared delegate is exported. The static target's interface names
$<TARGET_FILE:...> to force a whole-archive link, which is a path inside the
build tree and cannot be meaningfully installed. The static target and its in-tree
alias keep working exactly as before, so existing source builds are unaffected.

Example and test

Also adds an example application that consumes the packages this way, plus a test
around it.

The example is deliberately not a variant of the existing reference runner: that
one calls add_subdirectory on an ExecuTorch source tree, so it never exercises
prebuilt libraries. This one resolves everything through find_package and
imported targets. It also takes a file of expected values and compares outputs
against them, so a wrong result fails rather than only being printed, and it can
allocate its input tensors in device memory to match how an accelerator
application feeds data.

The delegate registers itself from a static initializer, so an application
references none of its symbols and the linker is free to drop it. It was being
dropped, and the backend was then missing at runtime. The example names the
library on its link line so the reference is recorded. Note that an imported
target's location lives in a configuration-specific property, so a generator
expression is needed rather than reading IMPORTED_LOCATION.

What is verified

On Linux x86_64 and aarch64:

  • The example configures and builds against an installed ExecuTorch package with
    no source checkout: exactly one CMake cache in its build directory and no
    ExecuTorch subproject.
  • The output comparison passes on a matching reference and fails on a deliberately
    wrong one.

On Linux aarch64 with a GPU, additionally:

  • The shared delegate builds against an installed ExecuTorch package and installs a
    consumable package: shared library, CMake config and targets, and headers.
  • find_package(torch_tensorrt_executorch) resolves and yields
    torchtrt::executorch_backend.
  • The example links it, and the library appears in the executable's dynamic
    dependencies, confirming the retention above.
  • Export produces a program where TensorRT genuinely claims the graph, and at
    runtime the delegate registers and its initialization runs.

What is not verified

Executing a TensorRT-delegated program end to end still fails on this setup with a
serialization version mismatch:

TensorRTBackend::init: failed to deserialize TensorRT engine
Current Version: 240, Serialized Engine Version: 243

A TensorRT engine is version-locked. In the environment used here the Python
package that builds the engine and the system libraries the delegate compiles
against are different TensorRT major versions, and only one of them ships headers.
Closing that needs a single TensorRT providing both headers and libraries; it is
not a property of this change. The two device-execution tests are written and
collect cleanly, and they skip where the compiled runtime, a GPU, or an ExecuTorch
install with a CMake package is absent.

…stall

Today the delegate can only be used by adding this directory to another CMake
project as a subproject and building it from source. `torchtrt::executorch_backend`
is created as an in-tree alias, and while the built library is installed, it is
not exported, so `find_package` finds nothing. An application that installed
Torch-TensorRT and ExecuTorch as packages has no way to link the delegate.

Export the shared delegate through a CMake export set and generate a package
config for it, so an application can do

    find_package(executorch REQUIRED)
    find_package(torch_tensorrt_executorch REQUIRED)
    target_link_libraries(app PRIVATE executorch::runtime torchtrt::executorch_backend)

and get a working link with no source checkout.

Only the shared delegate is exported. The static target's interface names
`$<TARGET_FILE:...>` to force a whole-archive link, which is a path inside the
build tree and cannot be meaningfully installed. The static target and its
in-tree alias keep working exactly as before, so existing source builds are
unaffected.

Also add an example application that consumes the packages this way, and an
end-to-end test around it.

The example is deliberately not a variant of the existing reference runner: that
one calls `add_subdirectory` on an ExecuTorch source tree, so it never exercises
prebuilt libraries. This one resolves everything through `find_package` and
imported targets. It also takes a file of expected values and compares the
outputs against them, so a wrong result fails instead of only being printed, and
it can allocate its input tensors in device memory to match how an accelerator
application feeds data.

Test plan:

The new test exports a model through the TensorRT compiler, builds the example
against installed packages, runs it, and compares the outputs against a reference
computed in Python. It asserts the serialized program actually contains a
TensorRT delegate first: without that a program where TensorRT claimed nothing
would still load and run, and the test would prove nothing.

A second case repeats the run with the input tensors allocated on the device,
since the CUDA delegate copies device to device and never stages through host
memory, so it expects tensors that are already resident.

A third case asserts the example builds with no ExecuTorch source tree involved,
by checking that only the application's own CMake cache exists in its build
directory. A source build would leave a second one behind.

Verified so far on Linux x86_64: the example configures against an installed
ExecuTorch package, reporting the prebuilt runtime and finding each shipped
component library; it builds and links with no source checkout, with exactly one
CMake cache in the build directory and no ExecuTorch subproject; and the output
comparison both passes on a matching reference and fails on a deliberately wrong
one. The tests skip cleanly where the compiled TensorRT runtime, a CUDA device,
or an ExecuTorch install with a CMake package is absent.
The link-resolution chain looked for the runtime as an in-tree target, then as a
source checkout, but never as the imported target an installed ExecuTorch package
provides. Building the shared delegate against an installed package therefore
failed asking for a source tree, which defeats the purpose of a delegate that is
meant to resolve the runtime from the shipped shared library.

Accept the imported target as well.

Test plan: built the shared delegate against an installed ExecuTorch package with
no source checkout present, on Linux aarch64. It now configures, builds, and
installs a consumable package (shared library, CMake config and targets, and
headers), where before it stopped at configure time.
Three problems stopped an application from consuming the delegate from an install,
each found by building and running against an installed package rather than by
inspection.

The CMake package installed into a directory named after the target rather than
after the package, so find_package never looked there. It now installs where a
consumer will look.

The export set names the concrete library, while an application wants a stable
name that does not depend on which library flavor it got. The package config now
offers that name.

The delegate registers itself from a static initializer, so an application
references none of its symbols and the linker is free to drop it. It was being
dropped, and the backend was then missing at runtime. The example now names the
library on its link line so the reference is recorded. Note the location of an
imported target lives in a configuration-specific property, so a generator
expression is needed rather than reading IMPORTED_LOCATION.

Test plan: on Linux aarch64, built the delegate against an installed ExecuTorch
package with no source checkout, installed it, and built the example against both
installed packages. find_package now resolves the delegate, the library appears in
the executable's dynamic dependencies, and at runtime the backend registers and
its initialization runs. Engine execution additionally requires the export and the
delegate to use the same TensorRT version, which is an environment matter.
@meta-cla meta-cla Bot added the cla signed label Aug 1, 2026
@github-actions github-actions Bot added component: tests Issues re: Tests component: api [C++] Issues re: C++ API labels Aug 1, 2026
The test was formatted against a newer target version than this repository uses,
so the linting job reported it would be reformatted.
The existing reference-runner check builds ExecuTorch from source through
add_subdirectory. That is worth verifying for that path, but it says nothing about
whether an application can consume the delegate from an installed package, which is
what the export set added here is for.

Add a GPU check that installs the shared delegate, builds the example against
installed packages only, and runs it against a reference computed in Python.

It asserts three things that would otherwise fail silently. TensorRT must actually
claim part of the graph, because a program with no TensorRT delegate still loads and
runs and would make the rest of the check meaningless. The application's build
directory must contain exactly one CMake cache, because a second one means
ExecuTorch was configured as a subproject and nothing prebuilt was exercised. And
the delegate must appear in the application's recorded dependencies, because it
registers itself from a static initializer that a linker is free to drop, which
leaves the backend missing at runtime.

Test plan: ran the equivalent steps by hand on Linux aarch64 with a GPU. The
delegate installs a consumable package, find_package resolves it, the example links
and retains it, and at runtime the backend registers and initializes. Engine
execution additionally needs the export and the delegate to use the same TensorRT
version, which is a property of the environment rather than of this change.
The example needs CMake 3.24. Without a check, an older CMake surfaces as a version
error from inside the package config, several layers below the script, which is
needlessly hard to read. Check up front and skip with a clear reason instead.
CI installs a released ExecuTorch, which ships the CMake config but no separately
linkable runtime, so there is nothing for an application to consume yet. The check
treated that as a failure, which reports a missing upstream feature as a broken
change.

Look for the runtime target first and skip with the reason when it is not offered.
Moving that probe ahead of the export also means the skip costs nothing.

Test plan: against a released wheel the check now reports that the package offers
no shared runtime target and exits zero. Against a wheel that ships the runtime it
resolves the CMake directory and proceeds.
@shoumikhin
shoumikhin force-pushed the executorch-wheel-consumable branch 7 times, most recently from af3904b to 59d0d37 Compare August 2, 2026 18:59
Two problems in the new verification path.

The install check looked for the delegate under a hardcoded lib directory, but the
install honors the platform library directory, which is lib64 on several
distributions. A completely successful build would have been reported as a failed
install. The configure now pins the directory and the check searches for the file
rather than assuming where it landed.

The device-input path handed a device pointer to the runtime without checking how
that input is supplied. A memory-planned input is copied into the plan with a host
memcpy, so a device pointer there would be read from the host. Only a non-planned
input has its pointer aliased, which is what makes device memory safe. The runner
now reports that clearly instead of corrupting memory.

Test plan: confirmed the install check finds the library when it lands in either
lib or lib64, and that a host-side copy of a device pointer is what the runtime
would do for a memory-planned input.
@shoumikhin
shoumikhin force-pushed the executorch-wheel-consumable branch from 59d0d37 to 333b46a Compare August 2, 2026 20:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant