Professor Vikram Kapila Explains the Integration of Mixed Reality/Augmented Reality/Virtual Reality into Various Technologies

November 22, 2017

IEEE Senior Member Vikram Kapila, Professor, Department of Mechanical and Aerospace Engineering Director, Mechatronics, Controls, and Robotics Laboratory New York University, Tandon School of Engineering

IEEE Transmitter: What is your area of MR/AR/VR expertise?

Professor Kapila: In the area of mixed, augmented and virtual reality (MR/AR/VR), my lab has expertise in conceiving, developing, implementing and assessing marker and image-based approaches to render highly natural and intuitive human-machine interaction mechanisms. We have integrated the MR/AR/VR technologies with mechatronics, robotics, computer vision, and control engineering to develop an array of novel applications in several domains, for example:

Engineering laboratory education, intuitive and natural human-robot interaction, mechatronics and robotic solutions for disabilities (stroke rehabilitation, senior living, etc.).

IEEE Transmitter: Why is MR/AR/VR important? How is it benefitting society today, and how will it benefit society in the future?

Professor Kapila: As evidenced from the examples below, the MR/AR/VR technologies are important because they allow the creation of immersive environments that promote user engagement and motivation and often include intuitive mechanisms to interact with objects in a synthetic or physical world. An additional benefit is that these MR/AR/VR experiences can be rendered through pervasive mobile devices that people already own. The examples below illustrate the important elements and benefits of these technologies.

In engineering laboratory education, the MR/AR/VR technologies can offer learners multiple linked representations, a technique that is known to effectively account for different learning styles and preferences of learners. Moreover, these technologies can bridge the gap between traditional (brick-and-mortar) and non-traditional (online) laboratory education.

For human-robot interaction, the MR/AR/VR technologies can produce highly intuitive and natural interfaces that do not require operator training. Moreover, with the use of mobile-based MR/AR/VR technologies, it is possible to eliminate research-grade, laboratory hardware as well as reliance on structured laboratory environment, thereby allowing effortless deployment of robotic platforms in real-world, unprepared environments.

In stroke rehabilitation, deployment of the MR/AR/VR technologies can render a rehabilitation environment that maintains patients’ engagement, sustains their motivation, and offers goal-driven activities, all of which can promote the concept of telerehabilitation.

Various academic and industry research teams have recently explored compelling applications of the MR/AR/VR technologies in 3D-design, rendering, and assessment for engineering and architecture fields; product assembly and disassembly; and machinery, plant, and facility inspection and maintenance; among others.

In the future, we expect increasing convergence of MR/AR/VR with technologies such as robotics, computer vision, machine learning, additive manufacturing, etc., to deeply impact fields such as education, workforce training, human-robot interaction, remote exploration and leisure (sports, entertainment, and gaming) as well as diverse industries such as physical and online retail (e.g. fashion, apparel, furniture, appliances, etc.); construction, remodeling, and maintenance; real estate marketing; and healthcare; among others.

IEEE Transmitter: How, specifically, are you are working with MR/AR/VR? What makes your work special?

Professor Kapila: Our work is unique in that we have relied on pervasive consumer-grade mobile devices to render our MR/AR/VR environments. Moreover, we have demonstrated that the use of mobile devices can additionally help to eliminate reliance on research-grade sensing and computing hardware and can render flexible and portable solutions that can be deployed outside of the structured laboratory environments.

IEEE Transmitter: How can MR/AR/VR benefit the world’s next leaders in research and teaching, medicine and exploration (e.g. space, underwater, archaeology)?

Professor Kapila: The MR/AR/VR technologies can benefit the world’s next leaders by allowing their frictionless integration in applications that employ convergent-technologies, e.g., robotics, computer vision, machine learning, additive manufacturing, among others. Similar to open-source hardware and software platforms such as Arduino, Raspberry Pi, ROS, OpenCV, etc., development, dissemination, and deployment of corresponding open-source hardware and software for MR/AR/VR can enable researchers from disparate domains to explore these technologies.

Compelling use-cases will further showcase broad utility of these technologies to potential users. These technologies can of course facilitate highly intuitive, natural, and immersive telerobotics which is highly relevant to remote exploration in space, underwater, and archaeology; remote observation of land and sea creatures without human invasion of their natural habitat; as well as in surgery. MR/AR/VR have myriad applications in education, for example, laboratory education, anatomy, astronomy, engineering graphics, spatial relations, etc. Moreover, in a manufacturing environment, maintenance workers can benefit from the MR/AR/VR technologies during inspection and repair operations.

IEEE Transmitter: Where do you see MR/AR/VR heading the next 5-10 years?

Professor Kapila: An increasing convergence of MR/AR/VR with technologies such as robotics, computer vision, machine learning, additive manufacturing, etc., will have a broad impact in varied fields. For example, in education these technologies will yield novel hands-on learning environments as well as immersive learning platforms as alternatives to static, printed, or online textbooks. Moreover, these technologies will enable learners unable to participate in field trips to undertake virtual walk-throughs of physical spaces (e.g., a museum, zoo, planetarium, etc.). The MR/AR/VR technologies will create novel means to deliver inspection and repair instructions to technicians. With innovations from companies such as Lowes and the impending arrival of Amazon in retail sector, the MR/AR/VR technologies will have significant implications for the retail sector. In the coming decade, the MR/AR/VR technologies will facilitate collaborative experiences in engineering and architectural design, teaching and learning, and gaming, among other applications.

IEEE Transmitter: Please add anything else you deem important regarding AR/VR technology

Professor Kapila: It is great to be living in, working, and experiencing this moment of technological convergence that is poised to shape all manner of human experience, from everyday living, education, work, leisure, etc.