Get a Thrill Exploring Apps with Organismo-iOS-Driver
In the article Exploring iPhone UI with Organismo I showed how fascinating can be to explore mobile Apps using Organismo-Desktop and the WebDriverAgent (WDA). However if you want to have a real trip exploring your App’s UI then you have to use Organismo-iOS-Driver.
The WDA is an smart leap from the XCUITest framework to provide a tool to automate tests and explore not only your applications but any application included system applications. The WDA is an special App that communicates with the testmanager service to retrieve information and execute actions of the application currently running in front. This approach is non intrusive to Apps, i.e. Apps do not need to have any special property or code nor compilation requests, they are 100% the Apps you will send to the App Store, and that is of course the ideal principle for testing. Unfortunatelly the limitations are big. WDA is limited to what XCUITest allows, even using the private API as it heavily does. All the capabilities provided by XCUITest to explore and control Apps are, so to speak, at an “amateur” level. And the good or bad news, depending on what side you are, is that it will always remain limited in capabilities in order to avoid explotation.
Technically there is another option to control and explore Apps. This would be the injection of custom code in the App, therefore an intrusive solution with its drawbacks and advantages. One advantage is that we could do pretty much anything we want, specially with a reflective language like ObjectiveC/Swift, where introspection and intercession are at our disposal. Organismo-iOS-Driver is such code.
Organismo-iOS-Driver Framework
Organismo-iOS-Driver is an open source framework, written in Objective-C. When loaded into the memory space of an App it can communicate with Organismo-Desktop and do anything we please in the App. For instance we can build UI exploration on steroids.
Probably the most important consideration when building Organismo-iOS-Driver is that it is a non invited guest to the App. Its main duty is to duly deliver its services and be transparent in the party. It must be extremely careful with communcations, resources access and symbol name collision avoidance. Every class, function and globals names in the library and in the third party libraries it uses, must be unique so that will never collide with any symbol of any application.
Organismo-iOS-Driver creates an HTTP server in the device localhost in port 5567, ready to respond to web socket connections at /main and /secondary endpoints. The Organismo-Desktop application has those two channels to request actions, for instance “getScreenshot” and “getTree”. The reason to have a second channel is to have a dedicated channel for continuous “getScreenshot” requests, which will provide a live update of the device screen in Organismo’s 3D device.
A limitation of Organismo-iOS-Driver is that if the App is in background it will not be able to get the screenshots therefore it will loose the video on the Organismo 3D viewer.
There are three ways to inject Organismo-iOS-Driver framework into an App :
- Adding the framework to your App’s Xcode project.
Organismo-iOS-Driver is like any other framework, you can add it to the list of linked frameworks and add a build phase to copy it to the Framework folder.
2. Injecting the framework into an existing “ipa/app” with bypass.
bypass is a cool OS X comamnd line tool that allows to inject Organismo-iOS-Driver into any .IPA or .APP you can get hold off. You don’t need the source code of the APP. You just need to have a developer certificate and a mobile provision.
3. Using MobileSubstrate in jailbroken devices.
For your fun search for Mobile Substrate in the jailbroken world.
Delivering the UI for 3D Representation
Gathering the information of the UI tree is very straightforward. It consists on iterating every window and their subviews, and getting the information of every UIView.
// Iterate App's windows
for (UIWindow * window in [UIApplication sharedApplication].windows) ...// Iterate a windows subviews
for (UIView *subView in window.subviews) ...
Check ORGUIViewHierarchy.m for the code.
The tree information together with a screenshot we can get the same result we got with the WDA, i.e. the screenshot and the UI tree representation side by side. See the following picture:
But we can do something different now that we are within the App. We can turn the UI exploration in something really striking. If every UI element would provide the image it renders we could isolate every element and render them not only in their X, Y plane but along the Z axis as well.
Well this is a pretty cool idea and turns out that it is possible too. This is our first lucky step: UIView has a function to draw itself onto a graphic context and create a PNG or JPEG out of it.
- (BOOL)drawViewHierarchy:(CGRect)rect afterScreenUpdates:(BOOL)afterWe are not there yet, drawViewHierarchy draws the view and all its subviews, which is not what we want, we need the image without its subviews. One way to achieve this is described in the next three steps:
- Hide all subviews.
for (UIView *subview in hiddenViews) {
[subview setHidden:YES];
[view setNeedsDisplay];
}2. Draw the view onto a graphic ocntext with drawViewHierarchy.
3. Restore visibility of the subviews.
This trick works and does the magic. See the picture bellow. We have the information of every UI element and its visual representation, now we can build an amazing 3D UI exploration. Check ORGScreenshot.m to see the code that creates the image of a UIVIew.
Levels of Detail of the UI Tree
Although I have mentioned the gathering of the UITree information as straightforward, it might not be so. Running the UI tree with UIKit API gives us the maximum level of detail, which could be excessive. Let’s see some cases of degree of detail that Organismo allows to choose.
- Private and Public Classes
Besides the UIKit classes you are familiar with (e.g UIView, UIButton), there are hundreds of classes that are not available or documented in the API, those are the Apple’s private classes that the UIKit uses to build the UI. These private classes usually have an underscore as prefix. In Organismo you can choose if you want to see them.
- Visible & Hidden Elements
UIViews can sometimes be hidden for multiple reasons. For instance an App can have hidden views on a particular screen to turn them visible after a condition. Those hidden views have no visual respresentation but you may want to see them represented in the tree for very fine exploration.
- Control Subviews
UI controls can be very complex in their composition, e.g. a date picker has a very complex internal structure that generaly might not be of interest to see it expanded on its full detail. But sometimes you might, therefore you can choose your preference to see the inners of controls.
- Gestures and Segues
From the hundreds of UI Elements of a particular screen, only a few have user interaction response and launch some action. If you are interested in investigating only the responsive elements you can turn the visualization flag on.
- Planes Distance and Range
Accessing and viewing inbetween UI Elements is often necessary and can be achieved with the two sliders for changing the Z distance between planes and the range of the planes. See the layers distance control in the image bellow.
- The Keyboard
This articles has Thrill in its title and not in vane, explore the Keyboard. It is a good place to feel the power of the representation of UI in 3D. The model breaks the boundaries of the physical screen to meet the representation of the memory.
Thanks !
If you liked it, you can Star the Organismo-iOS-Driver Github repository and this article. :)
