Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve accessibility of the Terminal View by making Reading Mode navigation more intuitive and add accessibility actions for the context menu #4104

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

alzwded
Copy link

@alzwded alzwded commented Aug 15, 2024

tl;dr it's a lot easier to read the output of the last command
and to get to the copy/paste/more... actions

Problem

It's a bit yiffy to navigate the wall of text in the terminal view. As things
stand right now with content description, accessibility services see the
TerminalView as an opaque blob with a wall-of-text description. Since it's
a description, there is nothing locking you in the control when using reading
controls (like you would be in a TextView).

Additionally, if I read the documentation of setContentDescription right,
it is really meant to offer a description of non-textual elements, or of
irritatingly opaque content; not to be the content itself. This is probably
irrelevant, as the goal was to make it easier to get to the output of the
last command.

Solution

It is also nigh impossible to notice the popup menu (copy/paste/more...)
if you don't... see it. It ends up at the end of the control stack and it
is not announced (if I understand correctly, you have to issue an accessibility
focus event, but the accessibility of the selection handle thing is a whole
different problem to the one I'm trying to solve here).

The goal was to make the TerminalView behave more like a text view:

  • have reading controls (by line, word or line) available regardless of
    what the system guesses, within the terminal view itself
  • don't accidently leave the terminal view when using reading controls
    too much (e.g. by word)
  • ability to easily jump to end, by swiping up/back with reading controls
    after activating the view (since it no longer "escapes")
  • note, reading controls are different to basic navigation; basic navigation
    let's you flick between the view, soft keys (CTRL, ALT etc), copy/paste/menu
    popup, your soft keyboard, etc; reading controls allow you to navigate
    a wall of text within a control, i.e. the terminal view
  • added accessibility actions to open the More... menu and copy and paste
    actions; note, these are apparent if TalkBack is running

Code Changes

Actual changes, on TerminalView:

  • explicitly mark the view as important to accessibility, because it does
    things which are important to accessibility, and it is the main focus of
    the whole app. The documentation of that method asks to set the flag to
    YES if a view does any of the things below;
  • call sendAccessibilityEvent(TYPE_VIEW_TEXT_CHANGED) when text changes
    and implement onPopulateAccessibilityEvent to let accessibility services
    know the text has changed (and what it changed to)
  • implement onInitializeAccessibilityNodeInfo which is called by
    accessibility services to detect what is on screen; this gives us a chance
    to explain that this TerminalView is a multiline text container and not
    some random eye candy that may be skipped over
  • no longer set content description, since we now set actual content by
    way of AccessibilityNodeInfo and AccessibilityEvent; replaced with
    all of the above
  • added custom accessibility actions for copy (screen), paste, and
    open termux menu (the More... menu). Accessibility Actions are something
    an Accessibility Service may query and present in a way that makes sense
    to the user. In the case of TalkBack, it has an Actions "reading mode"
  • added new strings to strings.xml and added 3 new IDs to ids.xml to
    support the custom accessibility actions.
  • refactor method out from onTouchEvent, branch BUTTON_TERTIARY,
    to use it for the accessibility paste action
  • in getText(), check mEmulator exists, otherwise return blank; TalkBack might get there first

Limitations

The Copy Screen action is a compromise since TerminalView doesn't report
on the cursor. I'm still going through TalkBack's code trying to understand what
it actually expects for things to work right. There are actions which are hard coded
to the class being android.widget.TextEdit (source1,
source2;
but I'm still trying to understand what's going on there and in the braille code;
also, that's for TalkBack 14, there are many versions of TalkBack in use out there).
In any case, the correct way to track the cursor between the terminal emulator's
scroll back and the accessibility view of the text is a whole separate opinionated discussion,
which could be left for later...

Something ought to be done about the side panel, since you rather have to
know it's there. But since the More... menu is now easily discoverable,
someone with TalkBack might stumble onto the Help panel and read that there's
a secret side panel on the left. That wouldn't be much different to the way
I myself discovered that sessions side panel on the left :-)

It would be worth adding landmarks for the terminalview, button bar and
side panel, but that's a different PR, since it probably needs to be done
in the Activity, not here.

There's also something iffy about the fact that AccessibilityManager.isEnabled()
is only checked at start up. Maybe it should be documented somewhere that
accessibility is checked only at startup for performance reasons. For people
who turn TalkBack on after launching the app, it might appear broken, even
though it really isn't. I didn't want to touch that behaviour since it was
something requested in the review of !344. I am merely stating my dissent.

To add to mAccessibilityEnabled: the a11y related method overrides are
simply not called if there's no active accessibility service, hence no checks needed.

Someone also mused about having the terminal speak out text as it comes in,
but that's also a different PR, since it rather requires interplay between
the TerminalEmulator code and TerminalView (providing accessibility), plus
setting up a virtual view that is live; and I assume people might want to
turn that off / control verbosity, which is a lot of work, so... not in this
PR.

It is theoretically possible to do partial text updates in onPopulateAccessibilityEvent,
(instead of blasting away the whole screen each time you type a character)
but I'm not convinced it actually works the way I hope, and it requires a
lot of work to get there (see previous paragraph). So... not in this PR.

But this PR makes interacting with the TerminalView itself much nicer
if you can't see what you're doing, and there's no reason to require people
to see what they're doing in this case.

@alzwded alzwded changed the title Improve accessibility of the Terminal View by making Reading Mode navigation more intuitive and adding accessibility actions for the context menu Improve accessibility of the Terminal View by making Reading Mode navigation more intuitive and add accessibility actions for the context menu Aug 15, 2024
@alzwded

This comment was marked as resolved.

@alzwded
Copy link
Author

alzwded commented Aug 18, 2024

Added a second patch which adds two Accessibility Actions to announce (= speak aloud) the cursor position, and the full line the cursor is on, respectively.

As of right now, there is nothing on the Android UI side to tell you this.

In the meantime, I'm continuing to explore more "intended" ways to represent a text area with a terminal style cursor, but I keep hitting roadblocks and dead ends. Hence, these two accessibility actions.

(This commit also reordered the actions and changed some resource strings and IDs added in the previous commit; I can shift the diff chunks around or squash the whole branch, just let me know what works best for reviewers).

@alzwded
Copy link
Author

alzwded commented Aug 18, 2024

Added a couple of new commits for extra keys (esc / end ctrl alt etc):

  • mark them as setTextEntryKey, so that they behave like text entry keys (= soft keyboard keys) on recent versions of Android + TalkBack, thus respecting the setting (double tap to activate vs activate on touch up)
  • announce state changes of special keys (e.g. "Control on", "Control off")

Improved reading mode navigation in `TerminalView`, allowing char/word/paragraph
navigation to work intuitively (e.g. *activate* -> *paragraphs* -> *swipe up*
will take you to the last line of output). Navigate by *Controls* or Touch
to get out of the Terminal View.

Added *Copy Screen*, *Paste* and *Termux Menu* accessibility actions on the
`TerminalView`, which *TalkBack* will now report.
There's a bit of a discrepancy where the "accessibility" cursor is, and where
the true terminal cursor is. So add a couple of actions to help figure out
where your text will go, or what it is you've just typed. Without this, there
is currently nothing on the Android UI side to tell you where it is.
…he character under cursor

In the abscence of something better, "Speak Cursor Position" doing its best to
read what is under the cursor in combination with "Speak Cursor Line" should
give you a reasonable idea where your input will go / what you want to delete.

If anyone brings up "TextEdit works, use that", be aware we're dealing with like 3-5
different cursors (the terminal's, selection handles, the "where was I reading"
accessibility cursor, which may have a begining and end).

If you need a line spelled out, then these two accessibility actions should help
you get your "what am I saying" accessibility cursor on the right paragraph and
ask it to spell out what's there.
This makes the additional soft keys behave like soft keyboard keys (if Android
and TalkBack are recent enough). There is a setting where text entry keys may
be activated on touch up, activated on double tap, or a mix of both.

This makes the soft arrow keys easier to use, since there's less tapping.
@alzwded
Copy link
Author

alzwded commented Aug 18, 2024

Rebased to latest master.

Added an extension to the second commit in the series (the one which adds "Speak Cursor Position" and "Speak Cursor Line") which also speaks out a unicode description of the character under cursor in order to properly explain "where your input will go", which otherwise can be rather unclear since the emulated terminal's cursor position and the various Android cursors and accessibility cursors aren't in sync, and there's nothing to sync them right now.

I had to add a getChar method on TerminalEmulator and TerminalBuffer, because getSelectedText was a bit overkill.

Overall changes up to this point, all applicable when an accessibility service is running:

  • signal availability of movement at granularity in TerminalView, which is now treated as a text control you stay in, so you don't accidently move out of it when moving by paragraph/word/character
  • Added accessibility commands for copy/paste/more... (these are a navigation mode, or they show up in e.g. the TalkBack menu; you don't see them anywhere else)
  • Added accessibility commands to speak out the cursor position and the line it's on to get a sense of where your input will go.
  • The extra keys behave like soft keyboard keys in TalkBack, since I assume the intention was to have them behave like keyboard keys? There's a marker that is set via AccessibilityNodeInfo.setTextEntryKey available in recent Android versions, which makes such buttons sensitive to a TalkBack setting having to do with typing preference on soft keyboards.
  • The special extra keys (ctrl, alt, etc) announce their state change. E.g., when hitting/activating CTRL, it will announce "control on" or "control on, locked" after it's done speaking everything else. Without this, there would be no indication of this global state change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant