What is Spatial Computing? And Are We There Yet?

April 3, 2024

Maybe you’ve seen the video. A man behind the steering wheel of a moving car. He’s got VR goggles on. And he’s moving his hands as if he’s typing on a keyboard that isn’t there.

Except the keyboard is there – to the guy in the goggles, at least. These sorts of virtual keyboards are the work of spatial computing. It’s a term that’s closely related to augmented reality and the metaverse. Here, IEEE members Todd Richmond and Juan Galindo discuss spatial computing: what it is, what it’s used for and where it’s headed.

“When we use spatial computing, we can engage with computers directly in our surroundings, manipulating virtual objects and information within our physical space,” Galindo said. “This approach requires computers to adapt and understand the human world, rather than the other way around. By doing this, spatial computing aims to seamlessly integrate computers into our lives as natural partners for our daily tasks and activities.”

What is the future of spatial computing, and how could it change our interaction with the digital and physical worlds?

Richmond: To understand the future, it is useful to consider the past. Spatial computing is an evolution of traditional computing, which depends on a screen display. The early mainframes with dumb terminals dictated a very limited set of use cases. As laptop computers began to appear, the ideal of taking digital capabilities anywhere became mainstream. AR and VR were attempts to break free of traditional screen experiences. Experiential design and the user experience for spatial computing don’t have a lot of obvious right answers. It will take time to sort out how we blend the virtual and physical worlds in a way that doesn’t lead to cognitive overload and attention tunneling

What fields are most likely to use spatial computing?

Richmond: If the user experience challenges are solved, spatial computing will be used across academic disciplines and business verticals. Mobile computing is a good analogy – when the iPhone debuted, people asked the same question. Now, 16 years later, it is hard to imagine life without a mobile device. While there were user experience challenges in moving content and experiences to smaller screens, it still was a screen. The core issue with spatial is that the screen goes away – and with it a lot of the design tricks we’ve evolved over the centuries.

What are the key technologies and devices that enable spatial computing?

Richmond: Sensors, network and edge computing are three major technologies. Multi-sensor platforms are critical for gathering information about the local environment – cameras to capture the surroundings as well as movements of the user and others, accelerometers to determine motion, and biometric capabilities to capture heart rate, eye movements, and eventually, the ability to understand user thought and intent. We’re not there yet.

What are the key challenges of spatial computing?

Galindo: The primary challenges are related to ergonomics. Some people feel tired using spatial computing devices, and they will say that the device doesn’t fit well. Another challenge is related to reliability and connectivity. Finally, there are some safety and privacy concerns, as digital identities may be targets for hackers.

What are the big research questions in this field?

Richmond: As spatial computing becomes the norm, our design of the user experiences needs to hold dear the concept that the user is a human – and that ethical design cannot be a nice-to-have but rather is essential for a sustainable and equitable future. This is part of why we have been talking about standing up a new field dedicated to this concept – EI/EX, which stands for equitable interfaces, ethical experiences. These considerations are not optional if we want to have a future that is inclusive and gets us closer to the goal of having technology level the playing field.

What’s Next: How will artificial intelligence, next generation communications, and the metaverse work together? This article from IEEE Xplore provides potential research ideas to guide the work of developers