UX of ARKit Projects - Part 1 (Bad UX)

With the release of iOS 11 and ARKit, and with our own coming ARKit titles, we decided we should take a look at how multiple apps are onboarding their users into the AR environment.  Some do it well.  Others - not so much.  We've been working in AR for more than 5 years, so we know many of the challenges involved in showing users/players how to interact with AR space.  Our goal is to help shed light on how the best app shops are bringing their users up to speed on familiarity with working in AR, and hopefully help shortcut the process for new devs working in AR for the first (or a second) time.

Requirements

Sometimes your content is going to be AR-driven, where the AR scene is essential for the content to be meaningful.  Other times, the AR is just an additional display layer, and your content could otherwise be displayed on a traditional screen with no loss in experience.  We believe the best experiences are the former, where the AR is spatially aware and is a functional driver of additional content value. In order to really benefit from this transformational quality that AR can bring to content, good AR experiences should provide the player/user with easy-to-understand instructions on how to interact with the AR scene and the content.  We believe the following types of user directions are important:

DIRECTING THE AR SCENE

Bear with us and think about movie production for just a moment - each scene has lighting, mic, camera angle requirements, which are controlled by the director. In traditional mobile apps, the developer is the director and the “scene” is basically the device's screen; you know the exact pixel dimensions, layout, etc. of that scene.  But AR developers can't necessarily control where their experiences happen, either in terms of physical location or in terms of interactions on the device’s screen:

  • Your players will interact with your content in different places, like an outdoor park, an indoor office, the inside of your car, the outside of your car, etc. - all of these "scenes" have different weather/lighting/GPS issues involved.  

  • Because the device screen acts much more like a monitor on a camera lens - allowing your AR content to scale in depth and left/right panning as the player moves the device through the scene - you can’t control how much of the device screen is actually dedicated to the AR content versus real world objects.

It’s exceptionally hard, if not logistically impossible, for a developer to control for all arrangements of a scene that their players will encounter.  So often, you have to encourage (and assist) your players to become their own directors, to ensure that they're getting the full benefit of the AR scene.  Developers have to train their players to be more engaged in the scene - to use the space, to use the camera angles, to minimize lighting or tracking distractions, to understand spatial audio cues, etc.  Give players some simple breadcrumbed instructions on this spatiality, but also provide more optional details for those interested in controlling more.  

PHYSICALITY

The player should be encouraged to physically explore the ways that the AR content is situated into their real physical environment. But you should always remember that spatiality requires 10x more physical activity than traditional apps.  Physical fatigue is a real UX issue that should be designed around - don’t inadvertently put too many interactions above waist height unless your specific intent is to exhaust your player’s arms or physique.  Forcing physical movement as part of an interaction or gameplay - while interesting and admirable to get people off their keisters -  means that your universe of willing participants will shrink. And all of this physicality is still mediated through the device’s screen; whether that’s through a 4” porthole via a phone’s screen or 10” on a tablet, it’s still a fractional window onto the real world.  And the surface of this window is your interaction point for your content - not via gestures out in the space you’re focused on.

RIGHT TIMING

AR requires permissions to use many of the device's sensors.  These are not granted by default in iOS and each permission for each app is explicitly called by the app and granted/not granted by the user.  These permissions should be presented at the right time during the user sessions, so as to maximize opting in to the permissions that are necessary to run AR experiences.  Don’t require camera or notification permissions without giving the player some context for why these are important.  If your Scene Directions are set up appropriately, it should be easy to get 100% opt-in for your permission requirements.

AR Content Type Drives UX Quality

There are over 300 ARKit apps currently available on the App Store.  For this ARUX analysis, we downloaded and tested more than 90 of the top titles.  We've grouped them into 14 categories, ranging from the physical (furniture) to the active (sports) to the esoteric (storytelling) (full list can be seen here).  We found that generally, the more directly there was a physical tie to the AR content (i.e. furniture), the better the initial setup and experience with the AR scene.  Then, generally, as the AR content moved from the physical space towards the more conceptual, the directions on how to interact with the AR space became worse and worse.  This varied depending on the publisher’s familiarity with AR of course, but as a general rule, as the AR content was less and less tied to a physical analog, the quality of the instructions provided to the app’s users diminished. This implies an intuitive disconnect between bringing these content types into an AR layer - if the developers, who are taking a deep dive on their AR content and understand it in more nuanced ways than others, can’t even relate how-to instructions properly, it is unlikely that users will intuitively understand how to best experience or utilize that AR layer.  Listen, we don’t need AR just for AR’s sake.  We need experiences that are qualitatively changed by presenting content in new contextual or spatial ways.  We need new ways of engaging with our devices and thinking about our spatial surroundings.  Pulling content off of a 2D plane (paper, screens, etc.) and into our native 3D world means that our education, utilities, decision-making, and entertainment are in the process of a transformational shift, and even these beginning ARKit experiences need to embrace that shift, as opposed to just repackaging traditional content and slapping it into space.  Part and parcel of that new spatial awareness is bringing your users into alignment with how to experience it.  

The fact that the better introductions and how-to cases are often presented by developers of experiences tied to a more physical analog also provides some insights about the UX of AR.  We all know how to sit on a chair.  We know where a “chair” fits into our world - we know how to physically move it, we know what qualities we like about certain types, we know how chair design impacts its surroundings, we can recognize even a dim outline of one in twilight, and we understand the physics of how a chair sits on the floor or ground and whether it might topple over with us in it.  So it should come as no surprise that furniture visualization tools are the easiest early experiences to understand in the ARKit environment.  ARKit’s floor-plane driven experience starts with gravity and the floor as the primary building blocks of the experience.  The further the experience moves away from that grounding metaphor - from embedded gestalt principles of comprehending the physical world - the harder it will be to present an understandable user experience, both in terms of content consumption as well as in term of simply instructing your users how to experience it.

Poor UX in Current ARKit Apps

We would prefer a world where no apps appear in this category.  And we apologize for singling out those experiences that we don't think meet our own (unreasonably high?) expectations, but without examples, how will the community learn?  So please don't look at inclusion of these apps as us pointing the finger at these devs to shame - rather, these are simply examples of what we wouldn't do or wouldn't have done, if we were in their shoes.  And because of the publisher’s size or prominence in their respective industries, A) criticism here isn’t going to affect their bottom line, and B) these companies should have known -and spent- better, and C) despite their industry presence/brand, these experiences shouldn’t be used as examples by smaller shops, regardless of how popular they may (or may not) be with the public.  Here, then, is a small collection of apps that have attempted to bring their users up to speed on how to work with AR, but for some reason (stated below), they miss the mark.

CHALK by VUFORIA

Honestly, we were very excited to see how Chalk would present AR to its users.  After all, Chalk comes from AR powerhouse Vuforia, which until ARKit came out, was THE dominant AR library in use across the market.  (Mea Culpa: Chalk is not an ARKit app, but was released at the same time, and should have provided good example on how to use AR.)  We played with the early Chalk prototypes at AWE 2017 and the app is getting a lot of good PR exposure.  However, the user experience is... to put it mildly... shitty.  The onboarding session is comprised solely of forms and terms and agreements and even an email verification, and after you slog your way through those unexplained steps, you can't even see or experience the app's content or functionality unless you've connected with someone else through the app.  What a waste of our time! From there, you're stuck and by this time, we've forgotten what Chalk is supposed to do for us.  So technically we can't even say the AR experience is good or bad - it's not there, so this is more of a traditional UX failure than a clear AR UX failure, but the AR is absent, and that's the whole point.  FORM -> OUT OF APP ACTION -> PERMISSION -> PERMISSION -> PERMISSION -> SPAM YOUR CONTACTS -> NO AR is just a terrible sequence.

PORSCHE AR IMAGINE

We were similarly interested to see what a luxury monied brand would bring to the UX of AR. Unfortunately, this seems to be one of those examples of bad timing in the steps of the onboarding sequence and is one of a myriad of examples in the ARKit category where camera permissions are the very first interaction with the app, without any explanatory context.  Many of these apps just assume that users will understand that camera permissions are required for AR to work, and just because this is an AR-first app, then those permissions will be granted.  In our opinion, users need to be given some context about the role of the camera in the AR experience.

The AR-specific instructions at the end of this sequence are what we consider to be rudimentary-to-basic: some context about the spatiality of the scene, but no appreciation for the scene sizing (it's going to place a life-sized vehicle in your space but there’s no warning) and little instruction on improving the experience.  Overall, PERMISSIONS -> TERMS -> BASIC INSTRUCTIONS -> UNEXPECTED SCENE SIZING is a subpar UX.

HOLO by 8i

 

45 long seconds to reach the AR content...

 

Holo is a camera app published by volumetric video company 8i.  They’re a venture funded B2B software company and this consumer-aimed app is an attempt to build awareness and use of holographic video.  The artwork design is great, but the UI and UX are really subpar.  To start, getting from install to even the beginning of AR content takes over 40 seconds of really useless interactions.  There’s an age requirement, permission screens, and “latest” details - but very, very little in the way of AR instructions, and the little that IS there occurs in the very beginning, so by the time you reach the scene, you’ve forgotten the 1-2 instructions they provided - not that they were terribly good to begin with. Context is king with AR!  Panning around doesn’t reveal any actions to the user, either - no indication that any scene detection is happening at all, and there are no onscreen prompts for how to build your AR environment.  It appears that the experience must be targeted at the Snapchat demographic - a userbase that normally tries to figure out functions without any onscreen prompting - because only when you mash the screen trying to make anything happen does the plane detection indicator appear.  AR requires more prompting than simple, traditional mobile content, so going the Snapchat directionless route is really not yet a viable option for AR experiences in our opinion.  And then there’s the AR content… ahem.

MEASUREMENT APPS

ARKit is still in its public infancy.  So, just like the early App Store, many of the apps on the market are copycats of one idea or another.  The biggest instance of this copycatting is with measurement apps.  In our spreadsheet tracker, we identified at least 39 measurement apps and for the most part, these apps worked very similarly, with little instruction or guidance on how best to effectuate the measuring scene.  We found 2 positive outliers in this group (which we will identify in our subsequent post on Good ARKit UX).  But for the others, despite having such a clear tie to the physical environment (it seems like “measuring” should conceptually go hand-in-hand with furniture visualization experiences, and so should be almost as adept at helping users understand their AR space), these measurement apps did nothing to help their users understand their environment and how to best interact with the AR measurements.  Rather, this segment of apps just seems to be the low-hanging fruit of ARKit experiences, and maybe the entrypoint for dev teams to expand their experimentation with AR experiences.  These apps feel like the flashlight apps of 2010.

Conclusion

Let me reiterate that this post is not about calling out Porsche, 8i, and Vuforia - this is about helping AR developers to recognize bad user experience, despite the source.  Steering AR content through an ideal experience is hard to get right, and an important skill for any AR dev team to build, including ours, is being able to set the stage for proper content interaction.  We’ve never gotten the experience 100% right, but over the past 5+ years of AR development, we’ve certainly tried a huge number of ways to introduce people to what AR is and can do.  Now that ARKit is probably the world’s most successful AR tech stack, many more people are going to use and play with AR, both for its functionalities as well as its transformative properties for content.  Help them - don’t assume they’ll know how to operate this new digital-spatial world.  Focus on clearly defined interactions and experiences that are not physically demanding.  Ask for permissions at the right place and time so that users understand why they’re granting you access to their camera and other capabilities.  And help people get to the AR as soon - and easily - as possible.  Again - help them.

3D Spatial Aiming Technologies

We've been playing around this week with our aiming solution.

IMG_0621.jpg

THE PROBLEM:

How to create an aiming system that effectively uses spatial relationships and 6DOF for positioning.

OUR GAMEPLAY ELEMENTS:

GAME STYLE

When we developed this game title, we decided against a strictly arcade stylistic gameplay.  Instead, we really like what AR can bring to the table - spatial relationships and a more direct tie to our real world.  To that end, we opted for a hyper-real style - can we make the gameplay look like something you might see out in the universe?  Can we build a gameplay mechanic that the Seed civilizations might have actually implemented, if the technology in fact existed?  If we had to defend a planetary system using energized meteors, how would we "realistically" impart those controls onto the rocks and best display that interaction to the system engineer?  So "homing meteors" or too much on-screen UI dependencies were all mechanisms that we avoided. 

VOLUMETRIC SPACE

We're creating a game that allows the player to fully roam their spatial environment and interact with the game elements.  Unlike many ARKit projects that focus solely on a flat plane game environment, PROJECT XII uses the full volume of your space to position the gamepieces and interactions.  We stitch your home planet into your living room - elevated approximately 3 feet off the floor.  A colorful nebula lives on your ceiling.  During the next minutes of the first chapter, enemy ships are inbound towards the planet from multiple approach vectors & angles.  The primary defense mechanism the player uses is firing meteors from the nebula overhead to intersect with the enemy ships at the right place and time in space. This is not a game best played while sitting on the couch.  In fact, in order to fire upon the enemy ships, you have to select them, which requires a certain physical proximity to the ship (in terms of where it is placed inside your living room).  We couldn't opt for a simple point and shoot.

SPATIAL GRID

We've constructed a gravity well around that planet and an accompanying grid system that runs parallel to the floor plane. This grid systems serves the dual purpose of helping to visually establish the attack angle of the incoming enemy ships, but it also provides a positional backdrop against which the player can judge the inbound speeds of the attacking enemies.  We're still working out the gravitational effects of the gameplay, but the gravity well in the grid was important for us to help the player get a sense of the physics at play here.

RELATIVISTIC ASTROPHYSICS

At Heavy Projects, we're unabashed science nerds and love the idea of working real physics into our experiences.  We're rationalists, and if we can bring general relativity, string theory, or other advanced science explorations into a popular medium, we're going to do it, dammit. We're using Unity to render our scenes, and Unity's engine has exemplary physics engines built in.  So we know (A) the IRL size of the planet in our scene (we assume it's Earth-sized in Chapter 1), (B) the size of the planet in Scene Units (0.4u), and (C) the speed of light relative to a body at rest (299,792 km/sec) - so a meteor travelling at the speed of light (an impossibility) would move approximately 23 planetary diameters per second in our scene.  Pushing a meteor to 0.1c or 0.2c gets us to 2.3 or 4.6 planetary diameters per second - a good gameplay mechanic for what we're doing.  Additionally, we take into account the increase in relativistic mass of the meteor (the increase in energized momentum required to push the meteor faster and faster, as explained here: https://arxiv.org/pdf/hep-ph/0602037.pdf) as an energy drain on the nebula reflective of the transformational requirements of E=mc2 (i.e. the energy drain on the nebula is not linear but is instead logarithmic towards infinity).  In our game, you'll be able to see what 10% the speed of light looks like, relative to a planet in space.  Fun!

OUR SOLUTION PROCESS

For those of you who have played with TiltBrush in a VR environment, we initially conceived of an aiming tool that the player could "dab" in space in front of the approaching enemy. The spatial paint dab would indicate a generalized area that the meteor might intersect with, with some margin of error randomization applied as to the final exact intersection point.  This proved to be incredibly difficult to play with because the meteor would of course keep moving past and through that dab spot - it wouldn't explode like anti-aircraft ammunition (it could have, but that would have stretched credulity and would have a required a more "smart" energy technology inside the nebula - I mean, come on, let's be real here).  And in all honesty, it was simply difficult to play with any solution that imparted a true 6DOF UX.  Instead, we opted to fall back on the grid system than initially contemplated.  We could use the grid system to eliminate one of the dimensional properties (here, it's the Y-coordinate Height) by (1) for each selected inbound enemy ship, fixing the grid to its approach angle keeping the planet as the fulcrum point, and (2) fixing our aiming crosshairs to the grid.  Then the player is only required to focus on 3 dimensions - X & Z coordinates + Time - in order to match up the meteor's arrival with the enemy ship's.  

We then added a 2D radar system, to help the player find the inbounds.  We kept the planet as the center of the radar, since that is also the central fulcrum of the pivoting grid system.  It works well enough for now - we'll improve the radar function in sub-releases. But trust us, at a certain wave of this Chapter - you're going to appreciate any help you can get!

So we now have a new aiming system that is (A) volumetric, (B) grid driven, (C) incorporates real physics, and (D) approaches what a hyper-real system might require, if this were all possible.  Good luck playing!