Any way to query the value of "Label" property of AXUIElemnt?

[avegetablechicken]

macOS tends to write more and more useful values as "Label" property instead of "Title" or "Value" properties of UIElements. I found no methods to query the value of "Label" property either by AppleScript or by hs.axuielement. Anybody knows way to workaround it?

[latenitefilms]

You should be able to access every AX element of an object. Use Xcode's Accessibility Inspector to see what values you have access to.

CommandPost has a bunch of AX helper utilities that might be helpful?

https://github.com/CommandPost/CommandPost/blob/develop/src/extensions/cp/ui/axutils.lua

[avegetablechicken]

@latenitefilms It seems that all properties accessible in Accessibility Inspector can be accessed by hs.axuielement except "Label" :(

[latenitefilms]

Are you able to give an example of a built-in macOS application that uses this label so I can try and reproduce?

Can you take a screenshot of what you're seeing in Accessibility Inspector, or better yet, UI Browser (with "Technical Terminology" turned on)?

Thoughts @asmagill ?

[asmagill]

"AXLabelUIElements" and "AXLabelValue" are defined for URLs (i.e. disks, files, and network connections), but are not considered standard for other objects. That said, an application can define any attributes it wants to, so its entirely possible that its a valid property for some applications.

I agree that we need to know more -- what applications are being queried, screen shots of what UI Browser is showing, what does hs.inspect(hs.axuielement.applicationElement(hs.application("app-name")):attributeNames()) (or equivalent), etc.

[avegetablechicken]

I can see the "Label" properties of a button in the display panel of Control Center in the window of Accessibility Inspector:

@latenitefilms @asmagill

[asmagill]

Apparently Apple is using Title in more places than the docs suggest; in any case, I ran the following:

ap = hs.application("Control Center")
apx = hs.axuielement.applicationElement(ap)

apx:elementSearch(function(m, r, c)
    if m == "completed" then
        if c == 1 then
            r[1]:doAXPress()
            t = hs.timer.doAfter(2, function()
                f = apx:elementSearch(function(m2, r2, c2)
                    if m2 == "completed" then
                        r[1]:doAXPress() -- close control center
                        hs.console.clearConsole()
                        print(hs.inspect(r2))
                    end
                end, { objectOnly = false })
                t = nil
            end)
        else
            print("criteria not specific enough: expected 1 result, found " .. tostring(c))
        end
    -- else search not complete
    end
end, hs.axuielement.searchCriteriaFunction({ attribute = "AXIdentifier", value = "com.apple.menuextra.controlcenter" }))

The results output to the console show that an awful lot of element attributes are returning error = "Requested value does not exist". One reason this error might be returned is if they (Apple) has implemented some sort of security that limits access to these specific attributes -- you'll notice that the required ones all have values (e.g. AXRole, AXSubrole, etc.) -- to only Apple signed applications (which is why XCode/UI Browser can see them).

Not sure that there is anything we can do about this.

[asmagill]

Oh... Label not Title... misremembered. Same answer, though... a lot of error = "Requested value does not exist".

[latenitefilms]

"Requested value does not exist" wouldn't be that unexpected would it? I feel like most AX values aren't actually populated?

[latenitefilms]

[latenitefilms]

I'm not sure where Accessibility Inspector actually gets the Label - I'm GUESSING it's probably actually using AXValue?

[asmagill]

"Requested value does not exist" wouldn't be that unexpected would it? I feel like most AX values aren't actually populated?

Yes and no... the code snippit I gave decides what attributes to query based on the results from hs.axuielement:attributeNames() which returns a table of attribute names that the application claims to provide. There is no actual check by Apple or the OS that the app actually does provide those attributes, and the return value for the attribute names is determined by the programmers, so... it could be that they copy-pasted and forgot/didn't intend to include that property, or it could be that the application performed some of its own internal logic and decided not to provide an answer at this time.

[asmagill]

If you run the snippit I gave above, it will dump the Control Center applications properties to the console, and AXLabel does appear for some items in the returned output; it just returns an error when the property is actually queried (by Hammerspoon at least)