Apple Pencil should not display menu cursor on hover

[frenzibyte]

Expectation is that hovering over an element in game with an Apple Pencil will show the element hovered without a menu cursor displayed over it.

Backlogged until SDL implementation is ready, can be prioritised with a local fork if wanted.

Tasks

Preview Give feedback

1. [SDL3] Pen - IOS pen detecting as mouse libsdl-org/SDL#9911
   +0
   
2. Add a way to distinguish mouse from tablet pen inputs osu-framework#6488
   area:input size/L +2
   
3. Configure GlobalCursorDisplay to hide cursor when onscreen pen input is detected

[Susko3]

I guess this would apply to all pens/tablets where the pen is directly over the display. This is true on iPadOS if it doesn't support external graphics tablets. Android phones with pencils also fall in this category, but not if it's using an external graphics tablet. Same for Windows Ink, a tablet may or may not have a display. (All of these use the same SDL pen API.)

It's difficult to tell whether a pen event is direct or indirect, we can provide a sane default, but we should have a setting to hide or show the cursor. We can use the existing setting for touch, just rename it to "Show gameplay cursor during pen or touch input".

For the framework implementation, I would have this implementation:

public enum PenDeviceType
{
    Unknown,
    Direct,
    Indirect
}

public interface IPenInput : IInput
{
    PenDeviceType DeviceType { get; }
}

// or "MousePositionAbsoluteInputFromPen"
public class PenPositionAbsoluteInput : MousePositionAbsoluteInput, IPenInput
{
    public PenDeviceType DeviceType { get; init; }
}
// same for all other mouse events
// we would use these pen inputs instead of mouse inputs for pen handling

This way, osu! can check IPenInput.DeviceType and decide whether or not to show the cursor the same way it currently checks ISourcedFromTouch.

[frenzibyte]

It's difficult to tell whether a pen event is direct or indirect, we can provide a sane default, but we should have a setting to hide or show the cursor. We can use the existing setting for touch, just rename it to "Show gameplay cursor during pen or touch input".

I would hope we can detect "direct" pen events directly. In the case of iOS, I don't see any support for external graphic tablets in SDL, and even if they do, the "Apple Pencil" one can be explicitly identified with some SDL changes for that. Most likely however, we'll be able to identify between "Apple Pencil" and external graphic tablets based on whether the input is coming from SDL's pen input events or OTD's events (whenever they support it). Something to think about later, not relevant now.

For Android, I'm not sure how the API goes there but I want to imagine it's going to be more or less similar to the above.

The setting you mentioned will be renamed as such probably, and it will still be default to false.

This way, osu! can check IPenInput.DeviceType and decide whether or not to show the cursor the same way it currently checks ISourcedFromTouch.

This is more or less what I have in mind, but, depending on how correct we want this to be, there should also be a full input event flow for pen inputs (i.e. PenState, PenMotionEvent, etc.), and the conversion to mouse input will be done similar to how touch works right now. This is required for input to be translated correctly when entering a PassThroughInputManager.

[Susko3]

I think the current touch-to-mouse conversion is flawed, there's too many hacks and workaround to make it work. I would rather keep the status quo of "pen events are just mouse events".

This is required for input to be translated correctly when entering a PassThroughInputManager.

We can hack in support for pen inputs if needed:

diff --git a/osu.Framework/Input/PassThroughInputManager.cs b/osu.Framework/Input/PassThroughInputManager.cs
index 82cce921e..99e5a8bd7 100644
--- a/osu.Framework/Input/PassThroughInputManager.cs
+++ b/osu.Framework/Input/PassThroughInputManager.cs
@@ -99,6 +99,8 @@ protected override bool Handle(UIEvent e)
             if (e is MouseEvent && e.CurrentState.Mouse.LastSource is ISourcedFromTouch)
                 return false;
 
+            var penInput = e.CurrentState.Mouse.LastSource as IPenInput;
+
             switch (e)
             {
                 case MouseDownEvent mouseDown:
@@ -109,6 +111,11 @@ protected override bool Handle(UIEvent e)
                     new MouseButtonInput(mouseUp.Button, false).Apply(CurrentState, this);
                     break;
 
+                case MouseMoveEvent penMove when penInput != null:
+                    if (penMove.ScreenSpaceMousePosition != CurrentState.Mouse.Position)
+                        new PenPositionAbsoluteInput { Position = penMove.ScreenSpaceMousePosition, DeviceType = penInput.DeviceType }.Apply(CurrentState, this);
+                    break;
+
                 case MouseMoveEvent mouseMove:
                     if (mouseMove.ScreenSpaceMousePosition != CurrentState.Mouse.Position)
                         new MousePositionAbsoluteInput { Position = mouseMove.ScreenSpaceMousePosition }.Apply(CurrentState, this);

[frenzibyte]

I'll need proper reasoning for that. To me that looks like making things hacky for the sake of being hacky. I can already imagine OsuTouchInputMapper would benefit from receiving pen input motion as UIEvents to allow support for a playstyle of hovering with a pen and tapping on the corners.

[frenzibyte]

@Susko3 Want to take care of the osu!-side changes to get this done?

[Susko3]

I'm still unsure what settings should be presented to the user, and what their defaults should be. The default is important to get right, and the current TabletPenDeviceType is not available when the default is set.

Touch input is always direct, but we still allow the user to decide if they want to show the cursor with OsuSetting.GameplayCursorDuringTouch. So TabletPenDeviceType cannot override any setting, and is, therefore, completely ignored.

I guess we just add OsuSetting.GameplayCursorDuringPenInput, default to false on mobile (for Apple Pencil and Samsung S-pen users), and true on desktop (as most people have an indirect tablet, without a display).

I want the setting for pen input to be separate from touch, as they should have different defaults on desktop.

[frenzibyte]

The goal here ultimately is to hide the cursor on menu and gameplay when the pen input is identified to be "direct", otherwise show the cursor even if it's unknown on whether it's
direct or indirect.

I can agree with making the above behaviour with direct pen input user configurable, but I cannot agree with making it a separate setting IMO. That just overwhelms the user with too much settings presented to their face.

I would say just use the existing setting and rename it to "Show gameplay cursor during touch / on-screen pen input" or something.

[Susko3]

That setting makes perfect sense on iOS, but the new name is confusing for every other platform, as other platforms always report Unknown or Indirect input.

I'm fine with "Show gameplay cursor during touch / on-screen pen input" but only if that's displayed on platforms that support differentiating on-screen pen input (iOS). I would keep it "Show gameplay cursor during touch input" on all others.

Do we just hardcode iOS as supporting on-screen pen input, or have something smart like:

public interface IPenHandler
{
    /// <summary>
    /// Whether this <see cref="InputHandler"/> reports <see cref="TabletPenDeviceType.Direct"/> <see cref="ISourcedFromPen"/> input.
    /// </summary>
    bool ReportsDirectInput { get; }
}

And change the displayed string in settings:

diff --git a/osu.Game/Overlays/Settings/Sections/Gameplay/InputSettings.cs b/osu.Game/Overlays/Settings/Sections/Gameplay/InputSettings.cs
index c245a1a9ea..08306ec74d 100644
--- a/osu.Game/Overlays/Settings/Sections/Gameplay/InputSettings.cs
+++ b/osu.Game/Overlays/Settings/Sections/Gameplay/InputSettings.cs
@@ -15,8 +19,10 @@ public partial class InputSettings : SettingsSubsection
         protected override LocalisableString Header => GameplaySettingsStrings.InputHeader;
 
         [BackgroundDependencyLoader]
-        private void load(OsuConfigManager config)
+        private void load(OsuConfigManager config, GameHost host)
         {
+            bool supportsDirectPenInput = host.AvailableInputHandlers.OfType<IPenHandler>().Any(h => h.ReportsDirectInput);
+
             Children = new Drawable[]
             {
                 new SettingsSlider<float, SizeSlider<float>>
@@ -32,7 +38,7 @@ private void load(OsuConfigManager config)
                 },
                 new SettingsCheckbox
                 {
-                    LabelText = SkinSettingsStrings.GameplayCursorDuringTouch,
+                    LabelText = supportsDirectPenInput ? SkinSettingsStrings.GameplayCursorDuringTouchOrPen : SkinSettingsStrings.GameplayCursorDuringTouch,
                     Keywords = new[] { @"touchscreen" },
                     Current = config.GetBindable<bool>(OsuSetting.GameplayCursorDuringTouch)
                 },

[frenzibyte]

That setting makes perfect sense on iOS, but the new name is confusing for every other platform, as other platforms always report Unknown or Indirect input.

I'm fine with "Show gameplay cursor during touch / on-screen pen input" but only if that's displayed on platforms that support differentiating on-screen pen input (iOS). I would keep it "Show gameplay cursor during touch input" on all others.

Why though? If it's because due to lack of support then that's on us to fix, no? Changing the label based on platform / detection still feels like over-complication to me.

[Susko3]

If it's because due to lack of support then that's on us to fix, no?

We can't easily fix it: ppy/osu-framework#6488 (comment).

Changing the label based on platform / detection still feels like over-complication to me.

It's complicated, but the simple solution will confuse users that want the option to work. Or maybe users won't care. We can go with your solution and see if we get complaints.