Control window that is under cursor

[jdpreston30]

Hi, do you think it is possible we could create a feature that allows you to simply control the window that is under the cursor instead of having to cycle through focus of windows by clicking on them? this is especially useful, say, when you are running two different apps with different rules on two separate screens.

The old app I used to use (X gestures) had this feature, but the app as a whole didnt work well so i changed to MacGestures

[username0x0a]

I think it would be possible to detect which application is actually topmost visible at the point where the gesture starts 🤔 at least that's my idea how that possibly worked there, so it could be considered. But no estimates at this point, tho. ⏳

[jdpreston30]

Hey! i got in touch with Brian who created X gestures about this and this is what he said. Thoughts?:

Actually that feature never worked especially well in xGestures. But I can share with you the overall approach for how to implement it.

The first thing you'll want to check out is the macOS Accessibility API:
https://developer.apple.com/documentation/applicationservices/axuielement_h

In particular, the function AXUIElementCopyElementAtPosition will let you figure out what UI element exists at a position on the screen. You can then determine which application it comes from.

Be careful, though. The Accessibility API is often buggy and confusing. Apps need a special permission to use it which can only be granted in the Security & Privacy preference pane. Some apps don't play nicely with Accessibility queries. Various functions like AXUIElementCopyElementAtPosition can take a long time to finish executing (since it requires communication with the target app behind the scenes) and these functions can only safely be called from an app's main thread. So it can be difficult to avoid causing your app to lock up when using it.

The part xGestures didn't get right is that it would use the above function to determine which app's gestures should apply, but it would still apply them even if the window the user was gesturing over isn't active. For some types of gesture actions, most notably anything that involves typing keystrokes, you'll need to activate the window and wait for it to become active before sending it any keystrokes. AXObserverAddNotification may help with that.

It's tricky getting this stuff right, but can be magical for the user when it does, so best of luck!

• Brian

[username0x0a]

Thanks for the details! 👍

Yea, I was expecting Accessibility framework to help with it, as pretty much the whole functionality of MacGesture relies on it as well; I'd say the permission required by MacGesture will be sufficient to use those calls (some APIs do require some really special capabilities included in entitlements, only approvable by Apple alone).

Querying the element is a cherry on top of the cake, yet hearing about those issues in use scare me a bit 😅 as it introduces some not very nice asynchronisation not done very well in these old APIs. We'll see how it goes. 🤞