Fix mouse exploration on Windows Terminal

[DataTriny]

Link to issue number:

Closes #20448

Summary of the issue:

Mouse exploration doesn't work on Windows Terminal due to the window hierarchy.

Description of user facing changes:

Hovering the mouse over a Windows Terminal window now reports the text (the whole line by default). Controls of the tab bar and scrollbars also work.

Description of developer facing changes:

Adds a dedicated app module for this application as well as a new TextRange class to make sure hovering over a window doesn't try to report the entire text buffer.

Description of development approach:

Testing strategy:

Opened a Windows Terminal window and moved the mouse around. Also tried with an instance with a large buffer.

Known issues with pull request:

None

Code Review Checklist:

• Documentation:
  • Change log entry
  • User Documentation
  • Developer / Technical Documentation
  • Context sensitive help for GUI changes
• Testing:
  • Unit tests
  • System (end to end) tests
  • Manual testing
• UX of all users considered:
  • Speech
  • Braille
  • Low Vision
  • Different web browsers
  • Localization in other languages / culture than English
• API is compatible with existing add-ons.
• Security precautions taken.

[josephsl]

Hi,

Can someone test this on a touchscreen computer to make sure the fix also applies with that input modality?

Thanks.

[hwf1324]

I didn't want to test by building a self-signed launcher and installing it, so I used the following app module code for testing:

# scratchpad\appModules\windowsterminal.py

"""Windows Terminal appModule"""

import appModuleHandler
import winUser


class AppModule(appModuleHandler.AppModule):

	def isGoodUIAWindow(self, hwnd) -> bool:
		if winUser.getClassName(hwnd) == "CASCADIA_HOSTING_WINDOW_CLASS":
			return True
		return super().isGoodUIAWindow(hwnd)

The conclusion is that touch input can correctly read the text on the terminal. Furthermore, strangely enough, unlike the mouse, touch input can read the text line by line.

Additionally, the following COM error is raised when touching the tab title text control:

error executing script: <bound method GlobalCommands.script_touch_explore of <globalCommands.GlobalCommands object at 0x000001C0D1461A90>> with gesture '移动'
Traceback (most recent call last):
  File "scriptHandler.pyc", line 300, in executeScript
  File "globalCommands.pyc", line 4722, in script_touch_explore
  File "screenExplorer.pyc", line 81, in moveTo
  File "NVDAObjects\UIA\__init__.pyc", line 1113, in compareEndPoints
_ctypes.COMError: (-2147024809, '参数错误。', (None, None, None, 0, None))

[hwf1324]

The error above may be related to WinUI 3; additionally, it is impossible to navigate to the minimize, maximize, and close buttons via touch exploration.

[DataTriny]

I don't get any exception when hovering over a tab title with the mouse. The name of the Close/Maximize/Minimize buttons is not reported with or without this patch, but the Narrator also fails here so maybe not something we can fix on the client side.

[carlos-zamora]

The error above may be related to WinUI 3; additionally, it is impossible to navigate to the minimize, maximize, and close buttons via touch exploration.

FYI: Windows Terminal is built using WinUI 2 and XAML Islands, not WinUI 3.

I forget why exactly we replaced the minimize, maximize, and close buttons, but the code for that in the Terminal repo is here: https://github.com/microsoft/terminal/blob/main/src/cascadia/TerminalApp/MinMaxCloseControl.xaml

I'm noticing that they are marked as AutomationProperties.AccessibilityView="Raw" though. Could that be related/relevant here?

[seanbudd]

Is this still being worked on?

[seanbudd]

Thanks @DataTriny