[interest invokers] Touch inputs

[lukewarlow]

We've discussed this before but I don't think there's a dedicated issue to talk about what could constitute interest on a touch device (e.g. phone, tablet, certain laptops).

We should also discuss what potential issues there are with this and potential work arounds.

[lukewarlow]

So far I think we've been talking about a long press as a touch input interest mechanism.

Pros:

• This isn't a complete new behaviour, touch devices already use long press for some secondary actions (e.g. text selection, copy, paste, and showing tooltips in the case of Android)

Potential issues to overcome:

• Potentially overloaded because it's already used for other things
• Might interfere with text selection
• Might not be possible to trigger for users with certain conditions
• Might need to handle focus differently on touch so it doesn't count as a trigger
• Hard to polyfil due to lack of "long press" event.

[lukewarlow]

Potentially overloaded because it's already used for other things

Might interfere with text selection

Potential solutions are to encourage Devs to add user-select: none to interesttarget buttons (perhaps apply automatically but less keen on this)

For elements that already trigger a context menu on long press we can add this trigger action to that context menu just like copy link is already an option.

Might not be possible to trigger for users with certain conditions

Hopefully those users will be able to use TalkBack and that will be able to trigger this. Needs confirming.

Might need to handle focus differently on touch so it doesn't count as a trigger

Unfortunate but not impossible.

Hard to polyfil due to lack of "long press" event.

This will just have to be accepted, but perhaps we should add this to the platform too.

[mfreed7]

It might be controversial to say this, but I think it might be necessary to leave some parts of the concept of "interest" to user agents to define. For example, in a Vision Pro like platform, with eye-tracking and a distinctly non-keyboard, non-mouse user interface, it likely needs to be up to the platform to decide when a user is "interested" in an element. I don't think we should (or need to) write down exactly how this works. In contrast, keyboard users and mouse users have fairly standard ways to indicate "interest", namely focusing or hovering the element. These, I feel, should be standardized so they behave in the same way across browsers and platforms. Somewhere in the middle is touch. Perhaps touch is standardized enough that we can specify it, but perhaps not? What's important, to me at least, is that the standard says, normatively, that the user must have a way to "indicate interest" in an element, no matter their input modality.

Stepping back off the soap-box, per your questions about touch:

Potentially overloaded because it's already used for other things

In chromium at least, I could see just adding an item to the existing context menu.

Might interfere with text selection

I agree this might be an issue, but I would see it as an implementation issue maybe? It must be possible for the user to do both, somehow.

Might not be possible to trigger for users with certain conditions

I might need more context on this one. But again, the UA should provide an accessible way to indicate interest.

Might need to handle focus differently on touch so it doesn't count as a trigger

Agree completely.

Hard to polyfill due to lack of "long press" event.

Agree. But it's not a requirement that new features be polyfillable. Definitely nice to have though.

[css-meeting-bot]

The Open UI Community Group just discussed [interest invokers] Touch inputs.

The full IRC log of that discussion

<jarhar> Luke: we've discussed before the idea of interest invokers is that theyre somewhat agnostic to the input method. if its mouse you have hover, keyboard is focus, eye tracking you get whatever
<jarhar> Luke: one of the input modalities is touch screens, we've discussed various ideas how that should work
<jarhar> Luke: this is really to get the ball rolling on approaches to it, what we envisage seeing
<jarhar> Luke: is this even something that we want to do? it might be that on touch screens that its up to the ua to decide how it works
<jarhar> Luke: the two proposed resolutions i want are - do we think that intreest invokers should always be able to be used?
<jarhar> Luke: if youre on a touch screen, does the ua have to provide some mechanism? we could say its optional, and up to the ua
<jarhar> Luke: for hover invokers we could require it
<gregwhitworth> q+
<jarhar> Luke: or, as mason pointed out, the ua must provide this in some mechanism depending on the input
<jarhar> Luke: or we could spec some kind of how this should actually work
<jarhar> Luke: personally i would like to require it for mouse keyboard and touch
<jarhar> Luke: i think touch might be difficult to spec because it depends on platform specifics
<jarhar> Luke: long press is a thing on android to show tooltips, if you have an icon button. i dont know if this is a generic thing, but you long press and get a description
<jarhar> Luke: pro of that is that it at least matches one platform, but there are issues to overcome, it coudl be overloaded or mess with text selection
<jarhar> Luke: buttons in some uas dont even have text selection enabled, links you can text select from a link very well because it loads the context menu
<jarhar> Luke: some users might not be able to do long press so we need to figure out at story as well
<jarhar> Luke: we could hook it into talkback somehow
<jarhar> Luke: or if its a thing in ats
<jarhar> Luke: we would have to handle focus differently, on mobile the prototype is a bit weird
<jarhar> Luke: and the other bit thats not too big a deal is that its hard to polyfill because theres no longpress event
<jarhar> Luke: you would have to impelentnt that yourself
<jarhar> q?
<keithamus> q+
<jarhar> gregwhitworth: i missed the last discussion. the interest invokers statement, what is the actual job to be done? i conceptually understand it with the long press
<jarhar> gregwhitworth: im making an assumption that its going to occur. i guess that is an interest in the popup
<jarhar> gregwhitworth: are you able to give me a one sentence heres the job its trying to fulfill
<jarhar> gregwhitworth: how long its held down to invoke a popover
<jarhar> keithamus: i think there are two strong use cases for the application of interest invokers
<jarhar> keithamus: one is tooltips, and another is page previews, like wikipedia or github
<jarhar> keithamus: on any input devices, the idea is that it should be an at a glance discovery of the element youre showing interest on
<jarhar> keithamus: users avatar on twitter or facebook or whatever,
<jarhar> gregwhitworth: that gives me the use case
<jarhar> gregwhitworth: i guess my - i agree that if you go down this road it should be a must. you've called out the problem that the devils in the details, how do we get everyone to agree
<jarhar> gregwhitworth: im curious if we're not able to - pointerevents is very generic, is doesn't define behaviors on certain elements
<jarhar> gregwhitworth: if that was the resolution i would +1 it, but the devils in the details
<gregwhitworth> ack gregwhitworth
<gregwhitworth> ack keithamus
<jarhar> keithamus: my general feelings on this is that its potentially hard or impossible to spec
<jarhar> keithamus: i do think its easier with mouse and keyboard
<jarhar> keithamus: then it gets very difficult when we talk about touch screens because there are many applications and they treat ui differently
<sanketj_> q+
<jarhar> keithamus: keyboard and mouse are standardized, mice have 2 buttons
<jarhar> keithamus: keyboard universally have a tab key
<jarhar> keithamus: as luke points out, long press on android is different from ios
<gregwhitworth> q+
<jarhar> keithamus: i cant imagine us being able to spec or suggest something that would fucntion idiomatically that apple would be satisfied with
<jarhar> keithamus: i dont know how much leverage we would have to add some verbage in the spec that says you gotta do this
<jarhar> keithamus: yeah sure they gotta do everything
<jarhar> keithamus: tail call recursions in the ecmascript spec but 2/3 browsers dont do it
<jarhar> keithamus: we can say it, but the rubber has to meet the road somewhere
<jarhar> keithamus: not that we shouldn't do it, but whats the tractable part of this
<gregwhitworth> ack sanketj_
<jarhar> sanketj_: i want to +1 what others have said. theres a lot of behaviors that are different on different platforms when it comes to touch. i dont think the ecosystem is going to get to a point where theyre consistent
<jarhar> sanketj_: ipads have this feature called tap to hover, when you tap on something you get a hover preview
<jarhar> sanketj_: bringint that to other touch devices has been problematic
<keithamus> q+
<jarhar> sanketj_: i dont think theres going to be a standardized way to say that this kind of touch input should do this everywhere
<jarhar> sanketj_: maybe theres no other way than to say its up to the ua, or we dont say anything, which feels worse
<gregwhitworth> ack gregwhitworth
<gregwhitworth> ack keithamus
<jarhar> keithamus: theres other things to think about to spark some imagination, tvs have a very different mechanism for focus and stuff
<jarhar> keithamus: usually you have a dpad, and i dont think we want to have focus follow interest in that example because the only way to navigate is with focus, and it exposes more than a tv user would want
<jarhar> keithamus: and of course eye tracking - you cant have interest land on wherever the user is looking at
<jarhar> keithamus: so theres a whole world of considerations there. is it a delay, a visual indicator that youre about to expose that your einterested in this element
<jarhar> keithamus: theyre in the camp of this - its like mason, its an unsolved problem. if it was solved then it would be consistent, but we dont even have consistency within one as sanket points out
<jarhar> Luke: question 1: do we want to want to require touch devices to have some mechanism to show interest?
<jarhar> Luke: while that may seem slihgtly unimportant, the key thing is that im building an interface and im using interesst target, can i use it knowing that it wont show on all platforms?
<zcorpan> q+
<keithamus> q+
<jarhar> Luke: could it be on mobile devices like wikipedias preview it doesn't show and thats ok?
<jarhar> Luke: or do we want a mechanism on those platforms
<jarhar> Luke: it could be like mason points out that we dont require it for keyboard or mouse, up to the ua to decide what they want to do
<jarhar> Luke: they should say that you should provide interst nomatter the modality
<bkardell_> "you should no matter the modality" makes sense
<jarhar> Luke: i dont want to end up in a situation where its only implemented for mouse and keybard
<jarhar> Luke: i want to see long pressing on a button do something
<jarhar> Luke: but at the same time i dont want to put too much emphasis on uas to do stuff and delay the whole thing
<gregwhitworth> ack zcorpan
<jarhar> zcorpan: one possible user interface for touch devices would be to render another button next to the thing and you would touch that to get the popover
<jarhar> zcorpan: maybe that could be an opt in
<jarhar> zcorpan: if the author feels that its important and needs to be availble everywhere
<scotto8> q+
<jarhar> zcorpan: please make it available and show another button
<jarhar> zcorpan: with mouse and keyboard, its more discoverable because you hover or focus before click/enter
<jarhar> zcorpan: with touch you just touch and theres no opportunity for discovering
<jarhar> zcorpan: but if you have a questionmark button next to it then you have something to tap
<gregwhitworth> ack keithamus
<jarhar> keithamus: i think that sounds good, i would want that to be a pseudo element so its configurable for design system
<jarhar> keithamus: i wonder if we could map it to existing heuristics
<jarhar> keithamus: whenever focus visible is applied, then it should or must show an invoker. maybe that covers some use cases?
<gregwhitworth> ack scotto
<jarhar> scotto8: i was going to +1 simons idea
<gregwhitworth> s/scotto8/scotto
<jarhar> scotto8: dont make it an opt in, because if someone sees a bunch of these buttons show up everywhere maybe they'll rethink their life choices, and then they design it in a way where they dont have all these things showing up on touch
<jarhar> scotto8: i think it makes sense to have that be the way that it does represent itself
<jarhar> scotto8: if someones not happy with how that renders, then they can change their ui and make it something more consistent across devices
<jarhar> Luke: alternative to that is we dont do it auitomatically at all and they can implement it themselves
<jarhar> Luke: we have css media queries for input modalities, so you could hide your interest invoker and use a real invoker instead - or prevent it from doing what you want
<jarhar> Luke: good ideas
<jarhar> scotto8: to coutner that, if there was also a pseudo element then someone could use css media queries and then add their own and then best of both worlds
<jarhar> scotto8: i just worry about not doing it by default, gives people false belief that thjey dont have to do anything
<jarhar> scotto8: when working with developers, html allows you to do this, that doesnt mean you should
<jarhar> gregwhitworth: we are at time, we will have to pick this up again
<gregwhitworth> Zakim, end meeting

[flackr]

Hard to polyfill due to lack of "long press" event.

Agree. But it's not a requirement that new features be polyfillable. Definitely nice to have though.

On Chrome / Firefox, contextmenu is a long press event. You can check that the event's pointerType is touch to filter out right clicks.

[flackr]

Might interfere with text selection

I agree this might be an issue, but I would see it as an implementation issue maybe? It must be possible for the user to do both, somehow.

And drag and drop, and contextmenu of course. You can have all of these things defined and they already get in the way of each other, this is one more thing but seems fine.

Some platforms have a shorter timeout plus haptic feedback for one action and a longer long press timeout for a different action. E.g. The homescreen on iOS picks up the icon for dragging before showing the menu. This is something we could explore to disambiguate.

[css-meeting-bot]

The Open UI Community Group just discussed [interest invokers] Touch inputs, and agreed to the following:

• RESOLVED: the spec should say that any user "must" be able to show interest via some input modality.

The full IRC log of that discussion

<hdv> Luke: we didn't get to a resolution on this last time
<masonf> q+
<flackr> q+
<hdv> Luke: will we require invokers to always be invokable, try our best to make it work for all modalities, or do we accept it's not necessarily possible in all modalities?
<hdv> Luke: at least on Android there's buttons you can long press to show tooltips, not sure if there's similar mechanisms on other platforms like iOS
<hdv> Luke: one issue around this is that interest should naturally be before invoking… if you have the long press, that's naturally more of an invokation than showing interestg
<hdv> s/interestg/interest
<keithamus> q+
<hdv> Luke: there's also the issue that longpress on some elements already does stuff, eg on links. So can we spec this, would UAs actually be able to implement that?
<hdv> Luke: or would people get a separate toggle?
<hdv> Luke: personally feel we'd rather have something buitl in if anything
<hdv> ack mason
<hdv> masonf: my opinion is that for keyboard and for mouse, the standard should dictate how it works
<hdv> masonf: the spec should also say for other input modalities, like touch and others, there should 'be a way' to show interest
<hdv> masonf: that way being up to the UA, because of the many differences between platforms today
<hdv> masonf: what's most important is that all users can do the thing
<hdv> masonf: and probably should be specced that interest is before invoke
<masonf> ack flackr
<keithamus> q-
<hdv> flackr: I agree, we should have a way to access these with touch. these features should work with the standard modality of the device, on some devices maybe you have something like gaze… but on some devices you might only have touch and that should be supported
<hdv> flackr: there are lots of ways people have talked about this… there are a lot of overloaded actions
<hdv> flackr: there's menu actions, drag and drop… many ways this has been done in native UIs
<hdv> flackr: could see this working if interest is nondestructive, you could probably do thisif the finger lingered long enough without scroll
<masonf> q?
<hdv> flackr: there are a lot of details around how this interacts with other featueres
<hdv> s/featueres/features
<hdv> flackr: another issue is disembiguating between different actions, eg you could have drag and drop and context menu at the same time that doesn't conflict
<hdv> Luke: as in HTML drag and drop? does that support touch devices?
<hdv> flackr: yes on iOS, there's work underway for Android too
<hdv> masonf: how standardised is this?
<hdv> flackr: that's literally the homescreen behaviour on iOS and Android… but on the web it's not standardised
<hdv> masonf: my fear is how it is supported
<hdv> flackr: even if we don't spec it you would still get a contextmenu on longpress
<hdv> flackr: iOS supports this and I think Android does in some contexts
<hdv> masonf: I don't know if there is a standard for contextmenu, I think there isn't detail in the spec on long press vs hard press
<masonf> q?
<hdv> flackr: for 3d press could look at maximum pressure or something like that
<hdv> Luke: we could resolve that we should make it a MUST, for any given device/set of modalities, at least one of them should be available to show interest?
<masonf> Proposed resolution: the spec should say that UAs "must" support showing interest via any input modality.
<hdv> Luke: any input modality seems ambiguous
<hdv> dbaron: kind of like 'any user must be able to'?
<masonf> Proposed resolution: the spec should say that any user "must" be able to show interest via some input modality.
<Luke> +1
<keithamus> +1
<hdv> +1
<flackr> +1
<masonf> RESOLVED: the spec should say that any user "must" be able to show interest via some input modality.
<dbaron> +1
<brecht_dr> +1