Project 'Ghost Dance' is where VR meets dance – and it's like real-life Black Mirror (in a good way)

Researchers Rui and Cecilia and their team used motion capture to explore the possibility of avatar dance duets.

For millennia, dancing has been a profoundly human experience. People move in time to the beat, using instinct to guide them through the music. But what if your dance partner wasn’t made of flesh and blood? Instead of having a real person in front of you, you can now step into the metaverse and have someone waiting for you there.

A consortium led by Lusófona University set out to explore this with their research project, Ghost Dance. The goal was to use Movella's motion capture tech to record choreography, and then monitor the differences between dancing with a virtual partner and a human one. Directors Rui Antunes and Cecilia de Lima’s backgrounds in biomechanics and dance made for a highly technical project that explored previously untouched areas (you can learn to animate a character with mocap here).

“Cecilia and I have different specialties, but a shared interest in how technology can shape our day-to-day lives,” says Rui. “Ghost Dance questions the very foundations of dance, and enabled us to see how the art form could develop to accommodate digital formats in the future.”

Covid lockdowns and digital interaction

The Ghost Dance project stems back to Covid-19. “The pandemic emphasized the fact that we are increasingly communicating through technology,” explains Cecilia. “It raised a question for Rui and me: how would this translate to more physical activities like dancing?”

Human connection is undeniably important to the way we dance. Moving alongside another person requires synchronicity, intimacy, and familiarity: it’s the only way we know. Bringing technology into the fold could change everything. “Small nuances like body language, breathing, and the sound of footsteps are crucial in dance,” Cecilia continues. “So we wanted to see how dancers could perform if those details were removed and their partner replaced with a virtual counterpart.”

To assess this difference, Cecilia’s dance students needed to perform a series of choreographed duets with both a human and an avatar.

Making a dancing avatar

There were two stages in creating a virtual dance partner. The first was to record the dancers’ choreography to collect data for the avatars. “For this, we used motion capture,” says Rui. “We used an Xsens inertial suit for data collection because we could see what was happening on-screen in real time, unlike the more lengthy optical motion capture process.”

The markerless suits allowed the dancers to be in their usual location, Escola Superior de Dança’s dance studios, instead of a traditional mocap volume stage. “It was hugely convenient for those working on the project, and meant that we could get high-quality data right here on campus,” Rui continues. “The dancers also felt comfortable with the suits on, as if they were wearing normal clothes, so they could work as naturally as possible with no disruption.”

This data formed the basis for the digital dance partners. The second stage was watching and dancing with them using a VR headset, which is where they truly came to life. “It was an odd feeling for the dancers,” says Cecilia. “Not only because they aren’t used to using technology in their performances, but also because the avatars weren’t tangible beings, so they could pass straight through their bodies.”

As professionally trained dancers, performing with an expressionless, uncanny humanoid was a unique experience. Despite these fundamental differences, the project proved that this type of human-technology duet is indeed possible.

From here, the team wanted to make the avatars perform even better. Rui and Cecilia looked to improve the virtual dancing experience by allowing the characters to improve on their own from the provided motion capture data. “We choreographed minute-long sequences using eight efforts from the Laban movement, a common type of analysis in dancing,” she continues. “This is so the machines can categorize and make the movements independently.” Because of the scale of the machine learning project, the team hopes to continue this into the future.

Improving virtual dance

The research from the Ghost Dance project only propelled the researchers at Lusófona University to go further with their studies, looking to secure more funding for specific attributes of the virtual dance partner, such as assessing the differences in eye contact, facial expressions, and realism.

“The project confirmed that dancing with a virtual partner could, in the future, be a viable option,” says Rui. “To go even further, the avatars might soon learn and move for themselves in line with their human counterparts, thanks to an accurate baseline of motion capture data.”

Find out more about Movella here.

Mocap has been used to good effect recently – see how this artist used the tech to recreate the Severance titles his living room.