In early May last year, Microsoft announced a new feature for the augmented reality headset, HoloLens 2. Prior to the upgrade, the headset had issues with mobile environments confusing the headset’s sensors.
The issue at hand saw the augmented reality headset’s sensors conflict with each other due to tracking movement through a camera and tracking through sensors. When used inside a car, the headset detected motion but saw a static environment around it causing “confusion”.
The new enhancement was developed in conjunction with Volkswagen and is designed to present drivers with virtual objects inside and outside the vehicle as well as a virtual map on the car’s dashboard. HoloLens 2 can also display navigation arrows at key intersections and alert drivers of upcoming pedestrians crossing in front of the vehicle.
We spoke with Marc Pollefeys, Director, Microsoft Mixed Reality and AI Lab Zurich, to learn more about this new technology.
Just Auto (JA): How did the partnership with Volkswagen come about?
Marc Pollefeys (MP): In 2015, Volkswagen embarked on a research project using autonomous vehicles and augmented reality to teach driving on a racetrack.
Volkswagen hoped to use HoloLens for the project, but found that when the device was placed in a moving vehicle, its sensors lost tracking and the holograms it normally displayed disappeared.
The Volkswagen researchers reached out to Microsoft for help and got in touch with myself, Microsoft’s Chief Science Officer and an expert in 3D computer vision and machine learning.
The Volkswagen team traveled to Microsoft headquarters in Redmond, Washington to show me around and team up with their work with HoloLens and their vision for augmented reality technology. The two teams began collaborating around 2018 to develop mobile platform functionality for HoloLens 2.
When used in a vehicle, how does augmented reality work?
To solve the problem of sensor tracking loss, my team developed an algorithm that models the gaps between sensors and allows HoloLens to continue tracking.
The new “mobile platform” mode for HoloLens 2 overcomes a major limitation of mixed reality headsets and creates the potential for technology to be used in new ways – by training drivers to handle harsh road conditions.
Could you explain the virtual card technology, which is presented on the dashboard of the car, and what it includes.
This is a demo app exploring what future navigation systems might look like that leverages the capability of the mobile platform. Note that it is not necessarily representative of future technology plans or currently intended for wide use.
What are the main benefits of this technology for vehicle occupants?
In the future, augmented reality tools like Mobile Platform Mode will enable seamless and connected mobility experiences including receiving information, assisting with navigation and entertainment as people travel throughout the day.
Are there any plans for future technology development/partnerships with other companies that you are able to discuss?
The technology is already attracting interest from maritime businesses and organizations, which are using HoloLens to remotely connect maritime workers with mechanical experts via Microsoft Dynamics 365 Remote Assist.
Microsoft plans to further refine the mobile platform mode for use in elevators, trains, cars, and other mobile environments. The technology can be used in many ways to connect the interior of vehicles to the outside world, such as helping truck or bus drivers navigate narrow streets, identifying points of interest along driving or providing passenger entertainment.
#improved #augmented #reality #headset #Microsoft