Add ability to get full bounds of an element

[MrGVSV]

Problem

The bounds attribute only returns the visible bounds of an element. There should be a way to either get the visible bounds and the full bounds of an element.

It also seems like this is what's returned by driver.getElementRect() as well.

Solution

Add an attribute that allows users to get the full bounds. Ideally, this would be bounds and visibleBounds. If compatability/breakage is a concern, though, it could also be bounds and fullBounds.

Use Case

My own use-case for this is to check whether a particular edge of an element is fully scrolled to when only part of the element is currently displayed.

This is straightforward on iOS as I can just use the element's frame to calculate the additional amount needed to be scrolled. However, on Android, attempting to calculate the offscreen edge using its bounds will result in the calculation incorrectly indicating it is fully scrolled to (as the height becomes element's top edge to where it's cut off by the scroll view).

Specs

appium: 2.0.0-beta.40
appium-uiautomator2-driver: 2.2.0

[mykola-mokhnach]

Have you tried simpleBoundsCalculation setting ?

[MrGVSV]

Have you tried simpleBoundsCalculation setting ?

Unfortunately, it's still returning only the visible frame.

[MrGVSV]

I found this chunk of code, which I think might be related?

appium-uiautomator2-server/app/src/main/java/io/appium/uiautomator2/core/AxNodeInfoHelper.java

Lines 235 to 280 in e3ce25d

	/**
	* Returns the node's bounds clipped to the size of the display, limited by the MAX_DEPTH
	* The implementation is borrowed from `getVisibleBounds` method of `UiObject2` class
	*
	* @return Empty rect if node is null, else a Rect containing visible bounds
	*/
	@SuppressLint("CheckResult")
	private static Rect getBounds(@Nullable AccessibilityNodeInfo node, Rect displayRect, int depth) {
	Rect ret = new Rect();
	if (node == null) {
	return ret;
	}
	// Get the object bounds in screen coordinates
	node.getBoundsInScreen(ret);
	// Trim any portion of the bounds that are not on the screen
	ret.intersect(displayRect);
	// Trim any portion of the bounds that are outside the window
	Rect window = new Rect();
	if (node.getWindow() != null) {
	node.getWindow().getBoundsInScreen(window);
	ret.intersect(window);
	}
	// Find the visible bounds of our first scrollable ancestor
	int currentDepth = depth;
	Set<AccessibilityNodeInfo> ancestors = new HashSet<>();
	AccessibilityNodeInfo ancestor = node.getParent();
	// An erroneous situation is possible where node parent equals to the node itself
	while (++currentDepth < MAX_DEPTH && ancestor != null && !ancestors.contains(ancestor)) {
	// If this ancestor is scrollable
	if (ancestor.isScrollable()) {
	// Trim any portion of the bounds that are hidden by the non-visible portion of our
	// ancestor
	Rect ancestorRect = getBounds(ancestor, displayRect, currentDepth);
	ret.intersect(ancestorRect);
	return ret;
	}
	ancestors.add(ancestor);
	ancestor = ancestor.getParent();
	}
	return ret;
	}

[mykola-mokhnach]

Yes it is related. As far as I can see https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo only provides a single method for fetching element's bounds: https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo#getBoundsInScreen(android.graphics.Rect)

[MrGVSV]

Yes it is related. As far as I can see https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo only provides a single method for fetching element's bounds: https://developer.android.com/reference/android/view/accessibility/AccessibilityNodeInfo#getBoundsInScreen(android.graphics.Rect)

Does this return the full bounds? If it does, can it be used directly for getting the full bounds of the element and then use the custom getBounds method above for getting the visible bounds?

[mykola-mokhnach]

It returns exactly what's described in the documentation. The above setting makes the server to return the raw result of the getBoundsInScreen method:

appium-uiautomator2-server/app/src/main/java/io/appium/uiautomator2/core/AxNodeInfoHelper.java

Line 213 in 2abe64d

node.getBoundsInScreen(rect);

[MrGVSV]

Ah okay I see now. Yeah after looking at the documentation and running a few tests, it seems it does not account for any parts of an element that are offscreen. An element halfway off the bottom edge of the screen results in half the height being returned.

I don't suppose there's a way to get the full bounds apart from the A11yNodeInfo? 😅

[MrGVSV]

I suppose if we had access to the UiObject, we could get this information, right? At least according to these docs (though, I'm not sure if this just returns the same on-screen bounds or not).

We could maybe extract that method here, but that might be too inconsistent/complex:

appium-uiautomator2-server/app/src/main/java/io/appium/uiautomator2/core/AxNodeInfoExtractor.java

Lines 56 to 66 in 2abe64d

	private static AccessibilityNodeInfo extractAxNodeInfo(Object object) {
	if (object instanceof UiObject2) {
	return (AccessibilityNodeInfo) invoke(getMethod(UiObject2.class,
	"getAccessibilityNodeInfo"), object);
	} else if (object instanceof UiObject) {
	return (AccessibilityNodeInfo) invoke(getMethod(UiObject.class,
	"findAccessibilityNodeInfo", long.class), object, 0L);
	}
	throw new IllegalArgumentException(String.format("Unknown object type '%s'",
	object == null ? null : object.getClass().getName()));
	}

[mykola-mokhnach]

You could try to experiment locally and see what it returns. Maybe there is still something of use. This is the actual code of getBounds and getVisibleBounds methods from UIAutomator framework:

    /**
     * Returns the view's <code>bounds</code> property. See {@link #getVisibleBounds()}
     *
     * @return Rect
     * @throws UiObjectNotFoundException
     * @since API Level 16
     */
    public Rect getBounds() throws UiObjectNotFoundException {
        Tracer.trace();
        AccessibilityNodeInfo node = findAccessibilityNodeInfo(mConfig.getWaitForSelectorTimeout());
        if(node == null) {
            throw new UiObjectNotFoundException(mUiSelector.toString());
        }
        Rect nodeRect = new Rect();
        node.getBoundsInScreen(nodeRect);

        return nodeRect;
    }

    /**
     * Finds the visible bounds of a partially visible UI element
     *
     * @param node
     * @return null if node is null, else a Rect containing visible bounds
     */
    private Rect getVisibleBounds(AccessibilityNodeInfo node) {
        if (node == null) {
            return null;
        }

        // targeted node's bounds
        int w = getDevice().getDisplayWidth();
        int h = getDevice().getDisplayHeight();
        Rect nodeRect = AccessibilityNodeInfoHelper.getVisibleBoundsInScreen(node, w, h);

        // is the targeted node within a scrollable container?
        AccessibilityNodeInfo scrollableParentNode = getScrollableParent(node);
        if(scrollableParentNode == null) {
            // nothing to adjust for so return the node's Rect as is
            return nodeRect;
        }

        // Scrollable parent's visible bounds
        Rect parentRect = AccessibilityNodeInfoHelper
                .getVisibleBoundsInScreen(scrollableParentNode, w, h);
        // adjust for partial clipping of targeted by parent node if required
        nodeRect.intersect(parentRect);
        return nodeRect;
    }

[jlipps]

it seems it does not account for any parts of an element that are offscreen

this is a fact about working with android screens that I've noticed, anything not on the screen doesn't exist.