Skip to content

fix: optimize attention decomposition - #4448

Open
fs-eire wants to merge 3 commits into
pytorch:mainfrom
fs-eire:attention-arange-decomposition-fixes
Open

fix: optimize attention decomposition#4448
fs-eire wants to merge 3 commits into
pytorch:mainfrom
fs-eire:attention-arange-decomposition-fixes

Conversation

@fs-eire

@fs-eire fs-eire commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Description

Align attention decomposition behavior with the ONNX graph.

This PR does the following:

  • Restrict attention-mask Select choices to scalars while preserving mask polarity and -inf semantics.
  • Preserve static arange operations as TensorRT LINSPACE Fill layers.
  • Add regression coverage for attention masks and static arange conversion.

With this change, Torch-TensorRT is able to generate optimized engine with a pattern that an attention using a causal mask AND a padding mask, when decompose_attention=True.

Fixes # (issue)

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • My code follows the style guidelines of this project (You can use the linters)
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas and hacks
  • I have made corresponding changes to the documentation
  • I have added tests to verify my fix or my feature
  • New and existing unit tests pass locally with my changes
  • I have added the relevant labels to my PR in so that relevant reviewers are notified

fs-eire added 3 commits July 29, 2026 23:53
Restore the original -inf value and invalid-mask polarity while retaining scalar Select inputs required for compact TensorRT MHA masks.
@meta-cla meta-cla Bot added the cla signed label Jul 30, 2026
@github-actions github-actions Bot added component: tests Issues re: Tests component: lowering Issues re: The lowering / preprocessing passes component: conversion Issues re: Conversion stage component: core Issues re: The core compiler component: converters Issues re: Specific op converters component: api [Python] Issues re: Python API component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels Jul 30, 2026
@github-actions
github-actions Bot requested a review from lanluo-nvidia July 30, 2026 07:06
@narendasan
narendasan requested review from micwill755 and zewenli98 and removed request for lanluo-nvidia July 31, 2026 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed component: api [Python] Issues re: Python API component: conversion Issues re: Conversion stage component: converters Issues re: Specific op converters component: core Issues re: The core compiler component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths component: lowering Issues re: The lowering / preprocessing passes component: tests Issues re: Tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant