3D UI for a Mobile Devices Lab
Exploring the possibilities of 3D GUI to operate mobile devices.
Why a 3D Metaphor for a GUI ?
3D user interfaces are rare, so much so that probably you haven’t seen or used any. Of course I’m not referring to 3D games, architecture, molecular chemistry, CAD/CAM or now more in fashion AR/VR. Those are 3D intrinsic, and usually all of them are a mixed mode of 3D for data representation and standard 2D GUI for control.
I refer to other areas or programs where 3D could be a metaphorical choice not an imposition. The representation of a desktop with 2D windows, tables and icons is a metaphorical choice. Another metaphor could be a 3D room with tables and cabinets.
If we accept 1963 Sutherland Sketchpad as the birth of 3D Computer Graphics plus GUI, that is almost 60 years of software development that haven’t brought any successful 3D GUI to our desktops.
Alan Kays’s 2D windows, tables and buttons, controlled by Engelbart’s mouse seems to have killed any attempt ever since. Would the fellows at PARC had some more time before Steve Jobs went to visit them, and had they already developed Croquet, would we be in a different situation ?
History tells to think twice before adding 3D to your program, there is a simple faster cheaper way in 2D.
Later in this story I will expose some history of 3D GUI failures and some mixed successes. But before I will show some implementation exercises of a 3D GUI for mobile devices in Organismo.
A Devices Lab in Organismo
Organismo is a 3D playground for mobile devices. It is based on 3D basically because I believe mobile is an area where the 3D metaphor can provide interesting advantages.
In particular I have expanded Organismo with a demo to explore the possibilities to visualise a lab of devices.
The first exercise is to represent a world view of labs, with their location and a visual cue that could represent some meaningful value, like traffic or usage. We can appreciate the power of 3D to quickly understand the situation of our labs.
A second exercise would be to explore the interior of a specific lab. Bellow we can see the racks in a lab that hold the physical devices, and their distribution in the floor. Every rack could be represented with visual indicators that could warn malfunctions, traffic etc.
The third exercise would be to explore possibilities to represent available devices. The possibilities here are many, the geometry of the distribution, sorting and grouping possibilities by different properties, etc.
The results of the experimental simulations are, as if always happens with 3D, allow me to say it, spectacular. This is the allure of 3D that has attracted so many attempts through the short history of GUI, but we have already said, none has reached mainstream.
If you like to play with these simulation you can get them here:
If you are interested in 3D GUIs I have built a list of some of the most interesting projects. The list is large specially if we count the extraordinary projects developed in close doors around research centers. I will focus on some that actually reached to real users and had some success.
Beware 3D user interaction is a broad subject that in no way I’m covering it here. If you are interested you may start pulling the thread from Doug A. Bowman.
FSN File System Navigator. 1993.
File System Navigator was an experimental application to view a file system in 3D, made by SGI for IRIX systems.
Even though it was never developed to a fully functional file manager, it gained some fame after appearing in the movie Jurassic Park in 1993
FSN lays out the directories in a hierarchy with each directory represented by a pedestal. The height of the pedestal is proportional to the size of the files in the directory. The directories are connected by wires, on which it is possible to travel. On top of each directory are boxes representing individual files. The height of the box represents the size of the file, while the color represents the age.
FSN is intended as an investigation in information landscape navigation. Although the file system was chosen as a system to navigate through, similar paradigms could be applied to other trees and graphs, such as call graphs.
Gopher VR. 1995
In 1995, the Gopher developers at the University of Minnesota released GopherVR. GopherVR is an enhanced Internet Gopher client that includes a 3D visualization tool for viewing resource collections as 3D scenes.
GopherVR is still alive today thanks to some enthusiasts :
I strongly suggest the paper “A Preliminary Design for a 3-D Spatial User Interface for Internet Gopher” to get the very first thoughts of the developers os GopherVR about the advantages and shortcomings of Gopher VR and 3D GUIs in general.
Project Looking Glass. 2003
This is one of those jewels in history of computing that influenced so much the GUIs that would follow, specially in Linux. The following text is from Wikipedia, it is awesome.
Looking Glass was first developed by Hideya Kawahara, a Sun programmer who wrote it in his spare time on a small Linux laptop. After demonstrating an early version to Sun executives, he was assigned to it full-time with a dedicated team and open sourced the project. It was first demonstrated publicly by Jonathan Schwartz at LinuxWorld Expo 2003 in San Francisco, and since then has gathered momentum in development. After unveiling the prototype, Steve Jobs called Schwartz’s office and told him that Apple would sue Sun if they moved forward to commercialize it.
Jobs claimed that the project is infringing Apple’s IP. Regardless of the threat, Sun determined that the project was not a priority and decided not to put more resource to develop it further into product quality. The project thus had been continuing in an experimental mode. However, Sun’s financial situation became tighter and the management determined they had higher priority projects on their hands. The project has been inactive (practically dead) since late 2006.
Xgl, AIGLX, Compiz. 2006–2008
Xgl is an obsolete display server implementation supporting the X Window System protocol designed to take advantage of modern graphics cards via their OpenGL drivers, layered on top of OpenGL. AIGLX a more powerful replacement for Xgl. I would highlight Xgl, AIGLX for being pioneers in the use of OpenGL and hardware acceleration to create GUIs. Although by themselves they do only provide the tools for 3D graphics, not a 3D GUI desktop.
Compiz is a compositing window manager for the X Window System, using 3D graphics hardware to create fast compositing desktop effects for window management. Again a great platform to create 3D GUIs, but not a 3D GUI.
Croquet. 2007
Alan Kay and his fellows created at Xerox PARC the GUI that we still use today. It was more an invention than an innovation, using his own terminology. Later he became very disappointed for the lack of innovation of GUIs and embarked with other fellows to write a GUI for operative systems. In 3D of course. The project is called Croquet to create and deliver collaborative multi-user online applications.
Implemented in Squeak Smalltalk, Croquet supports communication, collaboration, resource sharing, and synchronous computation among multiple users.
BumpTop Desktop. 2009
BumpTop diserves a special place in this list for is audacity and quality. A pure 3D GUI (no 2D concessions) was a skeuomorphic desktop environment app that simulates the normal behavior and physical properties of a real-world desk and enhances it with automatic tools to organize its contents. It is aimed at stylus interaction, making it more suitable for tablet computers and handheld PCs. It was created at the University of Toronto as Anand Agarawala’s master’s thesis.
In April 2010, it was announced that Bumptop had been acquired by Google. It was unknown what Google had planned for the software, though there were speculations about plans to revamp it for a new Android based tablet UI. Shortly after its acquisition, Bumptop announced plans to remove the software completely from its website, only giving “End of Life” support to those who bought the Pro version.
Oh well .. again.
Recent Examples
Uber AVS
Uber’s Autonomous Vehicle Visualisation is one good example we can see nowadays in some articles, with a mixed mode GUI (3D and 2D). Although as I mentioned in the introduction, in AVS the 3D choice is intrinsic, almost obliged. It’s a 3D representation for a 3D world.
AVS is a new standard for describing and visualizing autonomous vehicle perception, motion, and planning data, offering a powerful web-based toolkit to build applications for exploring, interacting and, most critically, making important development decisions with that data.
As a stand-alone, standardized visualization layer, AVS frees developers from having to build custom visualization software for their autonomous vehicles. With AVS abstracting visualization, developers can focus on core autonomy capabilities for drive systems, remote assistance, mapping, and simulation.
NupSys
Finally I’d like to mention NupSys and their “nuVIZ 3d Visualization Data Center Infrastructure” as one the best successful 3D GUI implementations for a real product I could find on the internet.
They could have perfectly well go for a 2D approach and probably no customer would have asked why wasn’t in 3D. Still they decided to take a harder road for a better product. I’m not a user nor related to NupSys, I don’t know the product as a user, but certainly it looks great and as a 3D believer I appreciate their effort.
3D-Visualization (physically and virtually) for data center infrastructure management (DCIM), uniquely capable of visualizing virtual machines, virtual network topology, physical rackspace, physical device details (including its modules, line-cards, etc.) power utilization and heat-map, data center walk through. nuVIZ™ use cases include data center planning and inventory drastically reducing time. In its next release nuVIZ™ is expected to provide dynamic DCIM visualization, including adds, moves and changes.
Conclusion
After all the 3D GUI R&D, we are left only with the 2D 1/2 of the OS X Time Machine.
I found with joy that the same question I open this article with, is the one the GopherVR asks in their paper Why a 3D-Spatial Interface ?
Their answers are very interesting. “Providing a 3-D spatial interface is, in part, just providing a concrete embodiment of language we already use. ..it provides a flexibile conceptual framework with which many variables can be integrated .. for viewing collections of documents; People understand a lot about space.there is a rich vein of knowledge about how the design of 3-D spaces that is ready to be tapped. Disiciplines such as architecture, landscape design, and urban design have a sophisticated undertanding of the issues that arise in spatial design …that may be applied to the design of virtual environments.”
No less interesting are the drawbacks they mention. “The Lost-in-Space Problem, The Grouping Problem, The Browsing Problem”. They were in 1993 and experimenting. Now in 2020 after so many attempts we should be able to draw some conclusions.
In my opinion there is always a way to represent anything in 2D, therefore in a more cost effective way. Development in 3D is considerably more complex than 2D, and often is not as appropriate as the 2D option.
A mixed mode user interaction, this is, a combination of 3D and 2D would be the best way to create better UI/UX. Every feature on the program could be represented in the most suitable manner.
In reality though, it all comes down to costs. All the 3D user interaction try outs have not proven more that “coolness” and “spectacular looks”. No 3D user interaction feature has proven to be an smashing success that we can’t live without it. Therefore it is difficult to imagine Apple or Microsoft to increase their costs to provide a GUI API for 3D.
Only the 3D intrinsic fields of AR/VR need to keep on developing user interaction techniques and APIs. And maybe from there, some spill outs could come to our desktops.
Thanks !
Feel free to experiment with Organismo.
