Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[iOS] Full keyboard access scrolling #56606

Open
wants to merge 20 commits into
base: main
Choose a base branch
from

Conversation

LongCatIsLooong
Copy link
Contributor

@LongCatIsLooong LongCatIsLooong commented Nov 15, 2024

This PR adds basic FKA scrolling support: when the iOS focus (the focus state is maintained separately from the framework focus, see the previous PR) switches to an item in a scrollable container that is too close to the edge of the viewport, the container will scroll to make sure the next item is visible.

Previous PR for context: #55964

Screen.Recording.2024-11-18.at.1.48.19.PM.mov

Why the UIScrollView subclass in the focus hierarchy

The iOS focus system does not provide an API that allows apps to notify it of focus highlight changes. So if we were to keep using the transforms sent by the framework as-is and not introducing any UIViews in the focus hierarchy, the focus highlight will be positioned at the wrong location after scrolling (via FKA or via framework). That does not seem to be part of the public API and the focus system seems to only know how to properly highlight focusable UIViews.

Things that currently may not work

  1. Nested scroll views (have not tried to verify)

The UIScrollViews are always subviews of the FlutterView. If there are nested scrollables the focus system may not be able to properly determine the focus hierarchy (in theory the iOS focus system should never depend on UIView.parentView but I haven't tried to verify that).

  1. If the next item is too far below the bottom of the screen and there is a tab bar with focusable items, the focus will be transferred to tab bar instead of the next item in the list

Video demo (as you can see the scrolling is really finicky):

Screen.Recording.2024-11-18.at.2.09.09.PM.mov

I've tried doing the same thing using a UITableView with similar configurations but it seems to have the same problem. I'll try to dig a bit deeper into this and see if there's a workaround.

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
  • I listed at least one issue that this PR fixes in the description above.
  • I added new tests to check the change I am making or feature I am adding, or the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
  • I updated/added relevant documentation (doc comments with ///).
  • I signed the CLA.
  • All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@github-actions github-actions bot added platform-ios platform-fuchsia platform-web Code specifically for the web engine labels Nov 15, 2024
@@ -174,7 +176,9 @@ - (void)accessibilityBridgeDidFinishUpdate {
// contentOffset is 0.0, only the scroll down action is available.
self.scrollView.frame = self.accessibilityFrame;
self.scrollView.contentSize = [self contentSizeInternal];
[self.scrollView setContentOffset:[self contentOffsetInternal] animated:NO];
if (!self.scrollView.isDoingSystemScrolling) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This means if the scroll view is in a FKA scrolling animation, all framework updates will be disregarded.

This is not optimal, if the user wants to interrupt the scrolling animation via swipe gestures, the UIScrollView will be temporarily out of sync with the framework scroll offset, but once the animation is done, this UIScrollView will send one last offset update to the framework, so it's guaranteed that the framework semantics tree will need another update and the new tree will also update the accessibility bridge and the offsets maintained by the scroll view and the framework will become consistent again.

Copy link
Contributor Author

@LongCatIsLooong LongCatIsLooong Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tried removing this flag, and the offsets echoed back from the framework seem always to be one step behind and I am not sure exactly why:

It seems Shell::OnPlatformViewDispatchSemanticsAction and Shell::OnEngineUpdateSemantics both run tasks using RunNowOrPostTask. Now that the UI thread is the same as the platform thread, there should be no delay, I was expecting:

<<<FKA setContentOffset: {0, 13.333333333333334}
<<<FKA setContentOffset: {0, 13.333333333333334}
>>>framework setContentOffset: {0, 13.333333333333343}
<<<FKA setContentOffset: {0, 15}
<<<FKA setContentOffset: {0, 15}
>>>framework setContentOffset: {0, 15}

but instead it's always delayed:

<<<FKA setContentOffset: {0, 13.333333333333334}
<<<FKA setContentOffset: {0, 13.333333333333334}
// ... framework sending back the previous outdated value
<<<FKA setContentOffset: {0, 15}
<<<FKA setContentOffset: {0, 15}
>>>framework setContentOffset: {0, 13.333333333333334}
full log
flutter: ▄▄▄▄▄▄▄▄ Frame 19          1m 36s 583.332ms ▄▄▄▄▄▄▄▄
<<<FKA setContentOffset: {0, 0.66666666666666663}
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 0.7), SchedulerPhase.idle
<<<FKA setContentOffset: {0, 3}
flutter: ▄▄▄▄▄▄▄▄ Frame 20          1m 38s 933.334ms ▄▄▄▄▄▄▄▄
<<<FKA setContentOffset: {0, 3}
>>>framework setContentOffset: {0, 0.66666666666666663}
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 3.0), SchedulerPhase.idle
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 3.0), SchedulerPhase.idle
<<<FKA setContentOffset: {0, 5}
flutter: ▄▄▄▄▄▄▄▄ Frame 21          1m 38s 949.999ms ▄▄▄▄▄▄▄▄
<<<FKA setContentOffset: {0, 5}
>>>framework setContentOffset: {0, 3}
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 5.0), SchedulerPhase.idle
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 5.0), SchedulerPhase.idle
<<<FKA setContentOffset: {0, 6.666666666666667}
flutter: ▄▄▄▄▄▄▄▄ Frame 22          1m 38s 966.665ms ▄▄▄▄▄▄▄▄
<<<FKA setContentOffset: {0, 6.666666666666667}
>>>framework setContentOffset: {0, 5}
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 6.7), SchedulerPhase.idle
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 6.7), SchedulerPhase.idle
<<<FKA setContentOffset: {0, 8.3333333333333339}
flutter: ▄▄▄▄▄▄▄▄ Frame 23          1m 38s 983.332ms ▄▄▄▄▄▄▄▄
<<<FKA setContentOffset: {0, 8.3333333333333339}
>>>framework setContentOffset: {0, 6.666666666666667}
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 8.3), SchedulerPhase.idle
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 8.3), SchedulerPhase.idle
<<<FKA setContentOffset: {0, 9.6666666666666661}
flutter: ▄▄▄▄▄▄▄▄ Frame 24          1m 38s 999.999ms ▄▄▄▄▄▄▄▄
<<<FKA setContentOffset: {0, 9.6666666666666661}
>>>framework setContentOffset: {0, 8.3333333333333339}
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 9.7), SchedulerPhase.idle
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 9.7), SchedulerPhase.idle
<<<FKA setContentOffset: {0, 11}
flutter: ▄▄▄▄▄▄▄▄ Frame 25           1m 39s 16.665ms ▄▄▄▄▄▄▄▄
<<<FKA setContentOffset: {0, 11}
>>>framework setContentOffset: {0, 9.6666666666666661}
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 11.0), SchedulerPhase.idle
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 11.0), SchedulerPhase.idle
<<<FKA setContentOffset: {0, 12.333333333333334}
flutter: ▄▄▄▄▄▄▄▄ Frame 26           1m 39s 33.333ms ▄▄▄▄▄▄▄▄
<<<FKA setContentOffset: {0, 12.333333333333334}
>>>framework setContentOffset: {0, 11}
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 12.3), SchedulerPhase.idle
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 12.3), SchedulerPhase.idle
<<<FKA setContentOffset: {0, 13.333333333333334}
flutter: ▄▄▄▄▄▄▄▄ Frame 27           1m 39s 49.999ms ▄▄▄▄▄▄▄▄
<<<FKA setContentOffset: {0, 13.333333333333334}
>>>framework setContentOffset: {0, 12.333333333333334}
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 13.3), SchedulerPhase.idle
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 13.3), SchedulerPhase.idle
<<<FKA setContentOffset: {0, 14.333333333333334}
flutter: ▄▄▄▄▄▄▄▄ Frame 28           1m 39s 66.666ms ▄▄▄▄▄▄▄▄
<<<FKA setContentOffset: {0, 14.333333333333334}
>>>framework setContentOffset: {0, 13.333333333333334}
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 14.3), SchedulerPhase.idle
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 14.3), SchedulerPhase.idle
<<<FKA setContentOffset: {0, 15.333333333333334}
flutter: ▄▄▄▄▄▄▄▄ Frame 29           1m 39s 83.333ms ▄▄▄▄▄▄▄▄
<<<FKA setContentOffset: {0, 15.333333333333334}
>>>framework setContentOffset: {0, 14.333333333333334}
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 15.3), SchedulerPhase.idle
flutter: _RenderScrollSemantics#0a6f4 relayoutBoundary=up1 => Offset(0.0, 15.3), SchedulerPhase.idle
<<<FKA setContentOffset: {0, 16}
flutter: ▄▄▄▄▄▄▄▄ Frame 30 

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, i think this makes sense since FKA set content will change the scrollcontroller, but it will have to wait for one frame to get the layout and semantics update back

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a comment covering this here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it looks like I was looking at an older checkout, and it was changed here. The event ordering becomes the way I expected it to be if we run DispatchSemanticsAction synchronously like before. Jonah said RunNowOrPostTask semantics action dispatch is fine we'll just need to post an empty task afterwards to flush the dart futures. I think if we do that instead we could eliminate this flag.

That seems to be the better option because we don't need the flag and the scroll view will be responsive to user gestures during the FKA scroll animation, but I think I'll make the attempt a separate PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am curious to see if it will work, but I think even if the action is send to framework synchronously, it may still take one frame for flutter to rebuild layout and flush semantics back

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am curious to see if it will work, but I think even if the action is send to framework synchronously, it may still take one frame for flutter to rebuild layout and flush semantics back

That's fine I think. The problem with DispatchSemanticsAction being async is that we are essentially maintaining two sources of truth so it becomes a distributed system where the communication channels only guarantee FIFO ordering. If the UIScrollView here and the framework semantics tree having different opinion about what the content offset should be, the update takes time to travel and the inconsistency can be observed.

If the communication is synchronous then the inconsistent state won't be observable as the updates would arrive instantaneously.

Copy link
Contributor

@reidbaker reidbaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

android code LGTM

/// This flag is set by the `FlutterSemanticsScrollView` itself, typically in
/// one of the `UIScrollViewDelegate` methods.
///
/// When this flag is true, the Flutter framework should typically cease
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have code from the dart side to enforce this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like the update will be dropped. I will probably say that in the here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flag is only read by the engine code and the framework side does not know about this. Yeah the wording is confusing I'll change it to "When this flag is true, the SemanticsObject ignores all incoming framework updates"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(updated the comment)

@@ -48,6 +55,9 @@ - (void)didUpdateFocusInContext:(UIFocusUpdateContext*)context
}

- (id<UIFocusEnvironment>)parentFocusEnvironment {
if ([self.parent isKindOfClass:[FlutterScrollableSemanticsObject class]]) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't this need to be recursively up?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we recursively looks up in frame calculation

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should return the parent focus container, or nil if none. Since [FlutterScrollableSemanticsObject parentFocusEnvironment] returns its parent, and FlutterSemanticsScrollView is a UIScrollView, this looks right to me.

Copy link
Contributor Author

@LongCatIsLooong LongCatIsLooong Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by recursively up? Returning the root node? This method is supposed to return the parent object in the focus hierarchy not the root (this method is part of the UIFocusEnvironment protocol and the iOS focus engine will use this method to traverse the focus hierarchy).

@@ -87,16 +145,99 @@ - (CGRect)frame {
//
// This method is only supposed to return items within the given
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a note that I have run into weird bug before because i returned child something outside of the rect of accessibilityContainer. I am not sure if it applies here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I can try adding culling here but I'll have to test it separately (and it seems to be working fine w/o the culling).

// to a Float64List in dart.
// The size of a CGFloat is architecture-dependent and it's typically the word
// size. The last iOS with 32-bit support was iOS 10.
static_assert(sizeof(CGPoint) == sizeof(CGFloat) * 2);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we make it a soft-crash or short circuit if this condition does not hold

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a compile-time check. We typically deliver the engine as a prebuilt binary so it's not going to a concern for most users / devs. This assert is mostly for code readers to see the assumptions we made about CGFloat here.

Copy link
Member

@cbracken cbracken Nov 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels pretty naughty. Even just for the sake of readability, I'd be tempted to create a double[], copy the values in, then encode and ship that. The extra CPU work is negligible and the one line of code will save you five lines of comment. Though I'd be tempted to still have a one line comment stating that you're encoding a Float64List for Dart.

@@ -174,7 +176,9 @@ - (void)accessibilityBridgeDidFinishUpdate {
// contentOffset is 0.0, only the scroll down action is available.
self.scrollView.frame = self.accessibilityFrame;
self.scrollView.contentSize = [self contentSizeInternal];
[self.scrollView setContentOffset:[self contentOffsetInternal] animated:NO];
if (!self.scrollView.isDoingSystemScrolling) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, i think this makes sense since FKA set content will change the scrollcontroller, but it will have to wait for one frame to get the layout and semantics update back

@@ -86,6 +87,17 @@ class SemanticsAction {
/// scrollable.
static const SemanticsAction scrollDown = SemanticsAction._(_kScrollDownIndex, 'scrollDown');

/// A request to scroll the scrollable container to a given scroll offset.
///
/// The payload of this [SemanticsAction] is a flutter-stanard-encoded
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// The payload of this [SemanticsAction] is a flutter-stanard-encoded
/// The payload of this [SemanticsAction] is a flutter-standard-encoded

Copy link
Member

@cbracken cbracken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it!

  • LGTM + codeowners approval for embedder.h change.
  • Some comments and nits on the iOS side. Shout when you want a second pass!

@@ -48,6 +55,9 @@ - (void)didUpdateFocusInContext:(UIFocusUpdateContext*)context
}

- (id<UIFocusEnvironment>)parentFocusEnvironment {
if ([self.parent isKindOfClass:[FlutterScrollableSemanticsObject class]]) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should return the parent focus container, or nil if none. Since [FlutterScrollableSemanticsObject parentFocusEnvironment] returns its parent, and FlutterSemanticsScrollView is a UIScrollView, this looks right to me.

// to a Float64List in dart.
// The size of a CGFloat is architecture-dependent and it's typically the word
// size. The last iOS with 32-bit support was iOS 10.
static_assert(sizeof(CGPoint) == sizeof(CGFloat) * 2);
Copy link
Member

@cbracken cbracken Nov 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels pretty naughty. Even just for the sake of readability, I'd be tempted to create a double[], copy the values in, then encode and ship that. The extra CPU work is negligible and the one line of code will save you five lines of comment. Though I'd be tempted to still have a one line comment stating that you're encoding a Float64List for Dart.

@@ -174,7 +176,9 @@ - (void)accessibilityBridgeDidFinishUpdate {
// contentOffset is 0.0, only the scroll down action is available.
self.scrollView.frame = self.accessibilityFrame;
self.scrollView.contentSize = [self contentSizeInternal];
[self.scrollView setContentOffset:[self contentOffsetInternal] animated:NO];
if (!self.scrollView.isDoingSystemScrolling) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a comment covering this here.

[reversedItems addObject:((FlutterScrollableSemanticsObject*)child).scrollView];
} else {
[reversedItems addObject:child];
}
Copy link
Member

@cbracken cbracken Nov 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider extracting something a static like this for readability:

static id<UIFocusItem>* GetFocusItemForNode(SemanticsObject* node) {
  if ([node isKindOfClass:[FlutterScrollableSemanticsObject class]]) {
    return ((FlutterScrollableSemanticsObject*)node).scrollView;
  }
  return node;
}

That simplifies this code just doing one thing (reversing the order of the focus nodes), rather than two at the same time (reversing the order + determining the focus node).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made this an instance method to avoid isKindOfClass:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even better!

@@ -17,6 +18,11 @@
const float kFloatCompareEpsilon = 0.001;

@interface SemanticsObject (UIFocusSystem) <UIFocusItem, UIFocusItemContainer>
@property(nonatomic) FlutterSemanticsScrollView* scrollView;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're re-declaring this property in three places now.

  • On SemanticsObject in SemanticsObject.mm for internal use in the class.
  • On the category on FlutterScrollableSemanticsObject, which is a subclass of SemanticsObject.
  • Here, for testing.

I think it's safe for us to push the property declaration into SemanticsObject.h and remove it in the other places.

@LongCatIsLooong
Copy link
Contributor Author

Alright the checks are all green (except the one that flutter/flutter#159108 should fix) and I think I've addressed all comments. Let me know if I missed anything. Ready for re-review.

@LongCatIsLooong
Copy link
Contributor Author

Oops sorry didn't meant to request re-review from the Android codeowners since I didn't make any changes.

Copy link
Member

@cbracken cbracken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM stamp from a Japanese personal seal

  • lgtm/codeowners approval for embedder.h change
  • lgtm for iOS changes

Copy link
Contributor

@chunhtai chunhtai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, for dart and c++ semantics code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants