Exploring iPhone UI with Organismo. An XCUITest Tree Inspector.

To watch the composition of an App’s UI can be fascinating. The structure to construct what it seems just a flat surface with some squares and text can be surprising even to the programmer of the App.

This article shows how to visualise the composition of iOS UI on any App, Springboard and other System Apps included.

Exploring the internals of iOS UI is possible in several ways. We can divide them in two categories : Intrusive and Non Intrusive to Apps.

Intrusive to Apps entails the injection of custom code in the process of every App that would profit from all the iOS API to extract information about the App’s UI and transfer that information to a visualiser, Organismo.

In the Non Intrusive approach there is no injection of custom code, the Apps and the iOS remain unmodified. The current existing technology to obtain information about Apps UI is Apple’s XCUITest testing framework. This approach allows to inquire the App in front at any time, included the Springboard and System Apps. This fits the goals of this article, therefore we will take the Non Intrusive XCUITest way.

XCUITest and WDA Runner

XCUITest is Apple’s framework to build UI Automation tests in native Objective-C/Swift. This framework arrived in iOS 9 to to overtake the unfortunate UIAutomation framework, based on JavaScript coding.

XCUITest provides an API to query UI elements and perform gestures on them. The interesting part of it is that the tests are executed in a separate application , created and installed when the tests run. This secondary application called the “Runner”, starts and remains in background, executing one by one all existing tests. The “Runner” is using the XCUITest framework to access the iOS serverices and get access to the front App UI.

This in itself looks like a promising feature but we still need a way to get some process on the Device to make queries from Organismo and get responses. Here is where the WebDriverAgent of Facebook/Appium comes to help.

In a smart leap, Facebook WDA is based on an XCUITest Runner App with only one test function, which instead of making a UI test, as it is meant to, it opens a REST Server at port 8100 on the device, ready to respond to requests. This smart move doesn’t end here, the REST server is a Selenium/Appium compliant WebDriver with a well documented set of queries and actions on UI elements.

Now let’s see how do we get WDA up and running on a iOS device ready to be used by Organismo.

Running the WDA

Clone the WDA Repository from Appium. It’s a fork of Facebook original project with some interesting added commands.

Follow this guide to start the WDA. If you are a programmer to run it from Xcode might be the most straightforward.

Connecting Organismo to a Device

If you are new to Organismo please read the Get Started.

Once you have Organismo installed you can read the guide to connect to a Device.

Exploring the UI in Organismo

If everything went correctly you should be seeing the device on the 3D Scene. To get the UI tree of the application in front, select “UI Tree” on the right hand tabs and click on Refresh. Gathering the full screen information may take up to 15 secs, please be patient. On success, the UI Tree of elements will be displayed.

In order to get the App’s UI information, Organismo makes an HTTP GET call to WDA Server in 8100 port with the following URL: localhost:8100/source?format=json. If you want to know more about the WebDriver API you can read it in the W3C page or in Appium documentation.

The name of the elements are simplified for easier reading. The original names follow the “XCUIElementType” prefix convention. This is how XCUITest names the elements to programmers, XCUIElementTypeButton etc. There are about 80 of them.

Notice the ammount of “Other” elements. These are generally UIViews that do not have a proper UI Control class or definition. Notice as well that many elements are listed with the “not visible” icon. The visibility flag is controversial in XCUITest and WDA, it requires expensive heuristic calculations to define if an element is visible.

Scrolling through the tree will highlight the element on the device in the scene. You may even highlight elements outside the screen, giving you a nice perception of the existance of the elements in the memory and geometry space. Clicks and right clicks on the scene and UI tree will offer some interesting actions.

The Unreliability of XCUITest

If the time to get the UI Tree is too long, you can blame on the element’s visibility calculations. Visibility is not a property of XCUITest XCUIElements, it seems not to be considered necessary according to Apple. Or maybe it’s a tricky concept to respond to.

Appium does make use of the notion and it is an important one, therefore they are obliged to define Visibility. An element is considered visible if its bounds are inside the screen, or intersects enough to be hittable. The problem is that the element bounds that XCUITest provides can’t be relied upon, often gives bounds {0,0,0,0} for elements that have visible children. You may browse the tree in Organismo and see it happening often. The heuristics of WDA are here. Not straightforward. They are obliged to traverse the children before deciding the visibility of an item. Appium WDA is using YYCache to gain sime cycles in this recursive tasks.

Another serious problem of XCUITest is that it can build a UI Tree with an incorrect order of the elements. Elements in front layer can be represented in earlier leafs of the tree, an viceversa. This for instance, will cause mistakes in finding elements by coordinate.

It’s exciting anyway to see an UI exposed this way, explore it and enjoy all iOS Apps. You can learn a lot about UIKit.

Thanks !

If you liked it, you can Star the Github repository and this article. :)