Using spatial experience ladder to define participant perception level
A spectrum to reach full immersion
For the last seven decades, interactions with computing systems have always funneled through rectangular windows of different sizes, overlooking a sea of mostly 2D data. Whenever we wanted to reach out and interact with that data, we would do so from behind a glass and through indirect interfaces such as a mouse or keyboard, or even by touching the glass. Direct and actual touching of the presented data was not possible until Spatial Computing started to become mainstream in the last 5 years or so.
Spatial Experiences (SX) exit flatland and extend beyond traditional rectangular frames, to enter 3D realms where many human senses are engaged simultaneously in processing digital illusions.
They engage the full biological and psychological spectrums and thus require broader ways of thinking of the context-of-use and deeper empathy with people.
The Climb by Crytek studio is a great example of this spatial characteristic of a video game created for Virtual Reality (VR). In order for a player to climb a cliff or even a skyscraper, they have to stretch arms to reach for boulders all around, to make grips to hold onto edges, and sometimes to swing their whole body and jump in order to move across gaps.
With these physical motion requirements, coupled with the realistic surrounding scenery settings, and the convincing ambient sounds, the game guarantees your heart to skip a beat if you failed to hold tight and fall. A high immersion level designed by stimulating haptics, motor, proprioception, vestibular, audial, and visual sensory systems, all in synchrony and in 360 degrees without clicking a button or tapping a screen.
Many psychological and physiological factors are in play when designing a Spatial Experience (SX) like that of The Climb. They define how an experience aesthetically looks, how it feels to interact with its different parts, how it connects people, and how it allows them to physically move. The goal is to design something that immerses people’s full perception of the synthetic world and places them as active participants instead of passive users. They live the story, not just listen to or see it unfolds, and participate in co-authoring its narrative.
Spatial Experience (SX) Design = look + feel + connect + move
In part 1 of this series of articles, I introduced the concepts of Perception Design (PD), Spatial Experience (SX), and Participant Experience (PX) to help form a new mindset that spatial designers should adopt when creating experiences for 3D worlds: Extended Reality (XR), 3D Web, or the Metaverse. I highlighted that people inhabit and interact with these worlds using their physical, emotional, sensorial, social, and behavioral systems the same way they would do with the real one. Their opinions are shaped through their perception and several external parameters including people, places, ideas, objects, and inspiration. My argument was that we need to design for perception in order to create plausible, spatial experiences in these worlds.
In this article I will continue my endeavor of building that lateral design mindset for SX by introducing the concept of Spatial Experience Ladder. This is another framework that helps designers categorize the type of experience they want their participants to perceive. Not all spatial experiences are as intimate and emotional as The Climb, nor should they be. It is important thus to be able to distinguish between the possible experience levels and know how to target them.
2. The Spatial Experience Ladder
The more we target perception and the senses in spatial designs, the more intuitive experiences we create that require less training to use, and the higher we climb up what I call the Spatial Experience Ladder. It is a scaling system used to describe the degree of immersion participants go through in a spatial experience and comprises of four levels: Interactive, Emotional, Immersive, and Presence. Each has certain requirements for the parameters of Perception Design (motion, emotion, senses, social, and cognition) and builds on top of the previous level. The higher the level is, the more people in that experience turn into participants than users.
2.1. Interactive Level
An Interactive experience on the Spatial Experience Ladder can be considered as a utility used for short bursts of interactions to do a specific task. People at this level are “users” of that tool rather than participants in its experience. They interact with 3D objects or environments through a screen. They lack the real sense of being present inside them. Their perception of connection and control is granted mainly from the outside-in, whereby they overlook objects from behind that screen and manipulate them through some indirect interfaces like menus.
Web-based 360 degrees hotel virtual tours and online 3D car configurators are two examples of spatial experiences on the Interactive level. They are “interactive” not “immersive” as oftentimes mistakenly claimed because they only expose richer information about a product or a service from different angles with higher visual fidelity — though from a participant’s fixed point-of-view. They do not bloom emotional, social, or sensory connections around them. Additionally, they mostly offer one-way conversation between the user and the product with no engaging narratives to elevate the personalization of the experience.
Designing for perception at this level is relatively straight forward and is something we have been doing for several decades already. It requires minimal physical motion (e.g. hand movement or gestures), low cognitive load (through familiar interactions like dragging, tapping, swiping, etc.), and it is often limited to the sense of vision (some ambient audio may be present too). Carefully curated visual looks can stir good inspiration at times but their impact on perception is relatively low.
The experience result at this level is the lowest in terms of you actually “being in” the synthetic world. On the other hand, it is the simplest to build and is ready for scale due to its lower technical barriers and the more familiar interactions it mandates.
2.2. Emotional Level
This level evokes the feelings of wonderment, joy, fear, community, etc. through carefully knitted stories. It is similar to the Interactive level whereby people engage with and manipulate 3D objects and spaces via low motor and cognitive processing from the outside in. The difference however is that they are accompanied through their journeys with story narratives that influence deeper perceptual beliefs.
Good and interactive storytelling has the power to immerse the senses and even make us feel we are living inside the story. Adding a social element makes the emotional attachments to that story even stronger. When emotions are triggered, we tend to connect more with the message and retain memories of the experience for longer. According to Dr Talmi Deborah, an emotional neuroscientist, “memory for emotional events is typically more vivid and accurate than memory for neutral ones”.
As well as better memory retention, we tend to become more involved as users in emotional experiences — a step closer to becoming participants in them.
Designing Emotional experiences requires a larger focus on the psychological and social aspects of human behaviors than Interactive experiences do, and more emphasis on storytelling to inspire the imagination. Moving emotions is top priority here, which can be achieved through mixing clever copywriting with special audio-visual effects. Hence, this level of experience relies heavily on the creative capacity (visual designs, 3D content and avatars, ambient environments, spatial audio, crafted storylines, direction etc.) more than on the technical one.
Moving emotions is top priority for experience at the Emotional level.
The 360 virtual tour by EDF energy company is a powerful demonstration of this Emotional experience level whereby one learns about how nuclear reactors work by actually shrinking down to an atomic size, journeying its different compartments, and freely looking around and explore. There is a calming presenter voice from above narrating to participants as they are progressing in the story to the bottom of the reactor.
You may not remember all the scientific details after the experience. However, the picture of the goliath engineering and the grandiose precision that EDF applies will never leave your mind after living the story of the explosive neutrons from inside the reactor core.
2.3. Immersive Level
Immersion is the degree and range at which one’s sensorium channels are substituted and engaged with simulated cues: eyes receive artificial light, ears artificial sounds, fingers artificial touch etc. It requires designing for several sensory aspects such as visual field of view (FOV), stereoscopy, depth and perspective, person-to-environment relative size, embodiment, binaural audio, spatial orientation, realistic lighting and shadows, proprioception, and many others.
The more of those aspects are addressed in a virtual environment design, the higher the immersion level is according to how Prof Mel Slater, a pioneer VR and HCI researcher, defines it:
“Immersion is a description of a technology, and describes the extent to which the computer displays are capable of delivering an inclusive, extensive, surrounding and vivid illusion of reality to the senses of a human participant”.
Control and agency in this Immersive level are critical topics. They “have to” be inside-out. Participants would directly interact and impact virtual elements from within, whether augmented onto the real world or fully enveloped by a virtual one, using natural gestures and input methods (i.e. hands). Having accurate embodiment for those interactions, expressions and behaviors helps preserve the sensorial illusion.
For example, leaning forward to pick up a virtual wrench in VR should react in correspondence to the physical body motion: leaning to come closer, and stretching arm to pick up.
Even picking a virtual cube with your real hand in AR and tossing it in the air should behave as you would expect to (see Microsoft’s Touching Holograms interaction concept for the Hololens in part 1 of this articles series).
With higher immersion comes higher cognitive processing to perceive it, even if subconsciously. This makes designing for human perception more demanding at this experience level than the two before.
Because Immersion aims to substitute real sensory cues, it is inherently more of a technical topic than a creative one, unlike the Emotional level. This makes the dependency on external inspiration and story narratives lower, and on social interactions and emotional stimulation more moderate. Wonderment, surprise, imagination, and other psychological states become more of expectations to reach the next experience level, Presence, as I shall explain next.
One note worth mentioning here is that we come across the term “immersion” or describe the mental state of being “immersed” very often as part of our everyday language. Reading a good book can disconnect us from the physical surroundings and immerse us in its story. Watching a well-directed movie can stir our emotions and leave us in tears. Video games can deeply hook players in virtual environments and social plays. Immersion in the context of Perception Design however applies more on spatial synthetic experiences where participants play direct part in their environments, not just control them from outside such as in video games.
2.3.1. AR “now” is not immersive — sorry!
According to this technical definition of Immersion, and the the required sensory modalities to target it, I would argue that AR as it stands today cannot be categorized as an Immersive experience! I know many will disagree with my personal bold statement, but the reason I say this is because state-of-the-art AR headsets unanimously produce low visual quality (fractional FOV, degraded imagery, loss of brightness, rainbow effect, etc.), reducing thus the accommodation of the Senses parameter. Additionally, not all AR headsets provide direct and natural ways of interaction (some require using your mobile phones, others a controller). So, two of the required parameters have been diminished.
Similarly, mobile AR does not offer a first-person view of the world but that of the mobile-camera view. We look through an obtrusive window and interactions happen indirectly by touching its glass. This reduces the impact on the Senses, Motion, and Emotions parameters of Perception design.
AR as it stands today cannot be categorized as an Immersive technology — yet. It is either Emotional or Interactive.
These technical limitations of AR unfortunately do not offer convincing sensory substitutions, making current AR experiences not Immersive — yet. They will improve for sure as “headware” advances. For now, I would categories them as Emotional since they facilitate interactivity and they weave in wow and inspiration to participants. But when no stories are involved in the experience, like placing a virtual kitchen mixer on a real table, AR becomes more of a utility and drops to the Interactive level in my opinion.
2.4. Presence Level
Presence is the degree at which the brain suspends any disbelief of the sensory substitutions and accepts them as being real. It is a psychological state after the senses have been effectively immersed. Immersion is the precursor to Presence.
Compelling, inspiring stories and natural interactions are what differentiate Presence from Immersion. They absorb the participant’s imagination, emotions, and feelings, and make them believe they are living the simulated experience. Whilst Immersion stands for what technology can deliver to influence perception, Presence is the psychological consequence for it.
Presence can be felt in siloed experiences or communal ones, as long the narratives are engaging, and interactions in the virtual worlds accurately simulate the real ones.
Immersion is the precursor to Presence.
PETA, an animal rights NGO, developed “EYE to EYE” to change the way they do their communication. It is a first-of-its-kind VR experience that follows a photorealistic rabbit to its final fate. It weaves together VR, live acting, full face and body tracking of an actor, highest level of 3D artistic details, custom binaural sounds, thematic environments, natural conversation with the rabbit, and an emotional tragic story. An experience that takes you inside the slaughterhouse and shakes your heart, making you feel present there and empathize with the rabbit.
Although physical movement of the participant is minimal, and it is mainly a single user experience, other elements such as realistically behaving rabbit (driven by a real-time actor behind the scenes) compensates for those parameters due to its novelty and true-to-life conversation simulation.
3. Final remarks and what’s next
It is important to note that the Spatial Experience Ladder is not meant for measuring immersion but rather categorizing experiences against it. There are many subjective and objective academic approaches that have been proposed and evaluated in the last 20–30 years to measure immersion in VR which could be borrowed for SX (for example here and here). I have even developed one myself that leveraged Design Thinking techniques to measure if an XR experience is hyper or actual.
The aim of the Spatial Experience Ladder is to guide spatial designers on the expected parameters to evoke in order to reach the intended experience level. It is more of a checklist to consider during the design process than a measurement tool.
In the next article, I will discuss how we can learn from other disciplines to inspire and inform our spatial design decisions. Lots we can borrow from the films industry, videos games, advertising, theatre and architecture. We do not need to start from ground 0.
Also, in future articles I will cover the 8 Heuristics of Spatial Experience Design I am working on with my team. They expand on what techniques to use to accommodate each of the Perception Design parameters and how to apply them to reach the target experience level. I cannot deny the fact that this would most likely remain a subjective exercise given the broad spectrum of external influences involved in spatial experiences, the diverse backgrounds of design practitioners creating them, and the many internal human aspects touched. That is the nature of design, right? It will be one interesting point to observe as Perception Design evolves.
To conclude I leave you with this quote by James Gibson, one of the most prominent contributors to the field of Visual Perception, as he describes the state of human perception when engaging with traditional 2D media back in 1980’s, which exist today and being reshaped by Spatial Computing:
“We modern, civilized, indoors adults are so accustomed to looking at a page or a picture, or through a window, that we often lose the feeling of being surrounded by the environment, our sense of the ambient array of light… We live boxed up lives.”
References:
Alger, M. (2015) VR Interface Design Pre-Visualisation Methods.
Kalawsky, R. S. (2000). The Validity of Presence as a Reliable Human Performance Metric in Immersive Environments. 3rd Annual International Workshop on Presence, (pp. 1–16). Delft.
Mestre, D., Fuchs, P., Berthoz, A., and Vercher, JL. (2006) Immersion and Presence. The treaty of virtual reality. Paris: Ecole des Mines de Paris, 309–38.
Papagiannis, H. (2017). Augmented Human (1st ed.). O’Reilly.
Slater, M., and Wilbur, S. (1997) A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments. Presence: Teleoperators and Virtual Environments, 6(6) 603–616, MIT Press.
Slater, M. (1999). Measuring presence: A response to the Witmer and Singer presence questionnaire. Presence: Teleoperators and Virtual Environments, 8 (5), 560–565.
Talmi, D. (2013) Enhanced emotional memory: Cognitive and neural mechanisms. Current Directions in Psychological Science 22 (6), 430–436.
Feel free to reach out to Fadi on LinkedIn or Twitter @FadiChehimi
Copyright © 2021 Accenture. All rights reserved.
