An exploration in collective interfaces
Designing for shared experiences: from personal computing to computing environments
written by IXD Research Collective
Pierluigi Dalla Rosa, Vaidehi Vartak, Nitin Sauran, Mayur Dhande, Nikhila Isukapally, Norris Hung, Jack Love & Isra Safawi
Humans are collaborative in almost every aspect of life. The work of researchers, doctors, intellectual workers, artists, construction workers and probably your work too; a lot is done collaboratively, sitting around a table, gathering on the sides of a whiteboard or joining hands to create an artifact, big or small, that needs more than one pair of hands.
We are living in a world of three dimensions, yet some of us spend a lot of our time interacting with two-dimensional UI. Our bodies are so expressive — jumping, stretching, running, actuating, fiddling, and yet we scroll on our phones with one finger. Bret Victor spoke about a future of interaction design that isn’t ‘dull’ or ‘full of permanent numbness.’
As a group of interaction designers based in San Francisco, we are researching the intersection of our collaborative nature and the digital world, trying to answer some of the following questions:
Beyond collaborative tools, what can interactive systems do when we are co-present in the same space? What is work when we are working on the same materials and objects?
The history of computing has been marked by being ‘personal’ for the last 40 years, even though the inception of digital interfaces was marked by highly collaborative intentions. As time progressed, the landscape saw a shift towards personal computing crystallizing into the predominant mode of interaction.
The precursor to collaborative real-time text editing was already showcased in 1968 with Doug Engelbart’s classic Mother of All Demos.
Fast forward to 1991, Loren and Rachel Carpenter conducted a large social experiment at SIGGRAPH ’91 by projecting a game of pong in an auditorium and observed the emergent behavior of hundreds of people collaborating to play the game together. This experiment took the principles of collaboration demonstrated in Engelbart’s work a step further. People were met in the environment of the theater, rather than in front of a personal computer. The experience was mediated by a new type of input device for general interaction, a camera; the power of image processing and computer vision could generate an output based on what it saw. The experience took the participant in a new interaction dimension, where behavior is emerging and the paradigms of interaction are more flexible and open.
In the past, researchers have explored components and software designed for supporting individual and group work co-situated, like in the example of Roomware by the research group of Carsten Magerkurth where they imagine a DynaWall, InteracTable, CommChairs, ConnecTable, and a protocol called Passage to transfer transporting virtual information using physical objects.
State of the Art
The landscape of collective computing has witnessed notable developments and projects, each aimed at enhancing collaborative work facilitated by computation. These endeavors encompass diverse domains, emphasizing the potential of collaboration through digital interfaces and systems.
In the realm of video game consoles, it’s common to see multiple controllers and split screens. This has been a successful way of experiencing the digital world in a shared space, yet it is limited and constrained. The video game industry has innovated in the domain of multiplayer games with the advent of VR headsets. Companies like SandboxVR or ZeroLatencyVR offer full-body, low-latency, immersive virtual reality experiences for groups of people. In these experiences, a group enters a shared virtual environment, and each person is located within the same space both virtually and physically. Users interact with their full bodies, moving, jumping, or using an input device like a weapon. These experiences make use of a mirror world, where most of the stimuli for the participants come from the digital environment. A fully occluded headset, immersive and spatial sound, and a haptic vest are employed.
While this is an interesting use of technology that could harness a lot of future potential if applied beyond video gaming — such as in real-world simulation, problem-solving, or training — there is an inherent disconnect between the replacement of all our senses and the negation of physicality and presence within our spaces.
In the world of interactive installations and interactive spaces a few companies have pushed the boundaries of what is possible in creating digitally mediated experiences that are rooted in the physical environment. The studios Tellart and Rockwell Labs have been pioneers in creating digitally mediated experiences in physical spaces with multi-user interfaces.
The image above showcases an interactive experience that anticipates a future where learning becomes more experiential. Instead of traditional lectures, readings, or media consumption, individuals can engage in dynamic simulations. In this specific sand table, scholars learn about the rain cycle through an orography simulation in a topography constructed by sand inside a sandbox that can be intuitively moved and shaped. Digital information, represented by elevations, colors, and precipitation, is projected onto the sand. This interface facilitates simultaneous interaction by multiple users, encouraging wonder and dialogue.
Another innovative venture in the realm of interactive learning is exemplified by Reactables, a tangible interface developed for collaborative music creation. While differing in focus from the aforementioned sand table, Reactables similarly employs a multi-user interface to promote shared engagement. Users manipulate physical objects on a tabletop interface, triggering electronic sound generation and visual feedback. This hands-on approach not only enhances collaborative learning experiences but also aligns with the broader trend of leveraging dynamic simulations in physical spaces for educational and creative purposes.
Taken a step even further, the project DynamicLand, created by the designer and technologist Bret Victor, operates as a sort of operating system for physical space, utilizing overhead cameras and projectors as the input and output of the dynamic system. The computer vision component in DynamicLand identifies custom patterns associated with specific behaviors, which can be programmed by anyone within the space. This innovative approach empowers individuals within the space to contribute to the ongoing evolution of a dynamic system, blurring the lines between traditional computing and the physical world. The result is an immersive and participatory experience where the boundaries of digital and tangible interactions are redefined in a shared physical space.
These experiments are anticipatory of a future where work can be at room scale, mix analog and digital interfaces and truly make the best of the richness, tactile and intuitiveness of the physical world, with the potential of the digital world, from simulation, emerging patterns and behavior or making sense of big data.
Prototyping Collective Interfaces
During our exploration, we set up two experiences to help immerse ourselves in collective computing and reflect on how these experiences need to be designed.
The first exercise was a mechanical exercise — we took two game controllers and hooked them up to an image gallery. The game controller we used is floor based, and has an elastic wire that connects to one’s hand. The hand’s motion is tracked in three-dimensional space thanks to the wire that can sense forward, lateral, and vertical movement.
With two of these game controllers, it allowed two people to connect their hands to each individual controller, and their lateral hand motion scrolled the image gallery left and right. If the two people worked concurrently, they would scroll the images in the same direction, but if they moved their hands in opposite directions, they would get to a standstill.
We observed very quickly how the interaction with the machine wasn’t sufficient for an effective navigation; Not surprisingly, a layer of verbal communication emerged, and body expression that hinted at the desired direction.
This is a feature, to bring back human-to-human communication. The computer is not the intermediary of this conversation, the computer doesn’t feel like another presence in the room, but rather part of the environment.
The inputs and outputs were appropriate for the task, they accommodated one user or two, the system did not create a hierarchy where one user would have more control — it didn’t need to do that.
The second experiment was a situated vignette, where we imagined having a conversation supported by an information/computing system that could aid us without interrupting a social exchange.
The setting is around a table where an interface is projected and touch enabled. A microphone was part of the system as well, picking up audio in the background, and based on the context of the conversation, offering relevant suggestions to quickly execute search, image lookups, and map browsing.
The people having a conversation were not interrupted by the machine, with minimal effort, they were able to access information that was relevant in that moment.
Conclusion & Principles for Collective Interfaces
Collective computing explores how technology can facilitate collaborative work in shared spaces, aiming to enhance both familiar and emerging forms of collaboration through innovative interfaces and operating systems.
Our objective is to cultivate a harmonious environment where technology serves to augment human capabilities without dominating the central focus. The physical space, shaped by our embodiment and the qualities of objects within it, sets the norms for our interactions.
To ground our explorations in collective computing, we’ve established a few guiding principles:
- No Hierarchy
In this setup, everyone’s input is equally valued, fostering open collaboration and decision-making without hierarchical structures. Ownership transitions from individuals to the collective. - Task-lessness
For a lack of better term, by task-lessness we mean using computers in an unstructured manner. Technology should support human collaboration without imposing a task for the whole system. For instance, the system should not enter a global mode, but allow each individual choice of action. This approach grants individuals control over what they want to pursue, promoting interaction between people rather than with the controls of the machine. This opens up to foster curiosity and play rather than rigid instructions. - Whole-Body Interaction:
Going beyond limited, finger-based interactions, incorporating natural, whole-body interaction enhances the intuitive nature of collaboration, allowing individuals to express themselves more naturally.
The challenge is to create a new computational paradigm where computation inhabits the collaborative space rather than existing within a confined individual box. Our ongoing exploration aims to integrate computing seamlessly into the shared space, aligning with the social dynamics that communities have embraced for thousands of years. Technology, in this context, should play a supportive role in fostering innovation and, perhaps, aid us in reconnecting with the essence of our humanity.
