Virtual Reality Beyond Sight and Sound

October 4, 2022

When people put on a virtual reality (VR) headset, they’re transported.

But the VR experience is limited to sight and sound. And that can be limiting, especially for people with disabilities.

IEEE Member Anderson Maciel is working to remove those limits by making VR even more immersive. The key? Adding more senses – a sense of touch, and of smell.

“There’s no reason for a person with an amputated limb or a visual impairment to experience more barriers in VR than in the physical world,” Maciel said.

And in doing so, VR could also present new opportunities to teach people who do not have disabilities what it is like to live with one.

His research includes the development of a vibrotactile head-mounted display (HMD). Essentially, it is a VR headset that, instead of providing visual information to the user, is equipped with a set of vibrating motors arranged around the head. The motors vibrate, and that stimuli can be used to direct users when walking down the street.

“The idea was to offer the possibility to communicate directions to users without the need to see signs or gestures and leaving the auditory channel free for talking and environmental noise. The vibration motors can be placed in different configurations. We provided an efficient one, able to indicate a direction around the head and an elevation.”

Q: What interested you in applying the tools of virtual reality to help people with disabilities?

Accessibility and inclusiveness are always a goal in human-computer interaction. With VR, there are two sides. One the one hand, living in the body of another allows us to experience what it is like to have a missing limb, or to have a visual impairment. That’s a powerful and unique tool to engender empathy.

The work we do is not just about permanent blindness. It could be applied to situations in which a person might not be able to use their sense of vision, for example of a firefighter in a smoky building, or a miner in a dark cave. They are unable to see and must rely on other senses and knowledge to navigate and accomplish their mission.

Q: Do you think the blind or visually impaired might be able to fully participate in the metaverse?

The world is becoming more and more available for blind individuals, but there is a long way yet for total accessibility. Fortunately, technology is helping a lot. But another important aspect is that people in general need to be educated to understand the differences and difficulties of any impairment. I think VR and the metaverse will help on both sides, especially because it is easier to implement sensory substitution in the virtual world than in the real one.

Q: What have haptics taught you about human perception, about how we feel and sense?

Computer hapticss have been studied for decades. We’ve learned that hairy and hairless skin feel vibrations differently, even when hair is removed, that the perceptual resolution of skin varies from the body midline. Perceptual resolution of skin is heterogeneous in terms of vertical and horizontal variation, and that stretching the skin in specific areas changes the weight perception and the sense of direction.

The goal of VR is to provide full immersion, i.e. block all five senses and replace them with computer-generated stimuli.

In multimodal interaction with sighted individuals, we’ve used sound and vibrotactile notifications to point the eyes. We’ve noticed that the more senses we add to the experience, the more people are able to feel them.

Q: What are some of the ways you can use haptics to convey information?

Vibration patterns applied on different body locations can convey iconic information, such as what direction a person should go in, or the presence of obstacles. It can also convey more abstract information, such as yes or no. We’ve investigated whether people are capable of articulating haptic words or sentences, for example, “give me the object to your right”.

Q: VR is primarily an audio-visual medium. What other sensory experiences are on the horizon in VR?

Displays that convey smells or temperature have been available for some time, and they are making immersive environments richer.

I’d say olfactory systems have been successful in improving presence and we may see some applications in the near future.

Peltier-type thermoelectric generators are small enough and cheap enough to be integrated in wearables. I have tried some experiences where I could recognize some virtual objects were colder and some were hotter. But the actuators are slow. Users need to wait several seconds before the temperature change is perceived. So, the hardware technology is not perfected for VR yet.

Q: How might the hardware you are using now be used in the metaverse?

The metaverse is a huge opportunity for anyone developing interactive hardware and software for user experience, and this includes haptics. We believe that the fact that the haptic actuators are easily integrated into HMDs and controllers means that they have an important role on how people experience the metaverse. The possibilities are unlimited.

Learn more: IEEE members are on the front line of a rapidly developing metaverse, dealing with a wide range of questions that includes ethical and accessible design of virtual environments and the interoperability of VR equipment. If you’d like to join a global community of experts discussing these issues, check out the IEEE Standards Association’s Metaverse Congress.