I worked on an augmented reality application called Remote Assist that enables factory floor technicians and engineers to get remote assistance on complex machinery via 3D annotations. Our clients were reporting issues with annotations being placed inaccurately in their environment. My team and I created a multifaceted solution mixing an algorithmic change in how AR Core worked with a new input model that allowed the placement experience to be more clear.
While the HoloLens version of Remote Assist had been out for 2 years already, the mobile version had just been released before I joined the team. Because of this, we had a lot of tech and design debt address. After gathering the most pressing customer feedback, we found the top issue was in the accuracy of the annotations. came to these three problem areas:
I tackled all three facets of the problem with their own unique solutions:
I designed a new patented placement experience that:
I also created a series of error messaging that would give more precise feedback when tracking wasn’t working. Instead of giving them a generic error message, we’d tell them specifically what they should do to correct it.
Lastly, I worked with my SWE and design prototyper to understand how the algorithm was detecting planes to place annotations onto and created our own solution that generated mini planes out of features points which ended up being more accurate.
I iterated through many versions exploring different placement experiences, different rotation interactions, and visual indicators of the grid and what the camera was picking up on.
This project taught me that when you bring multiple disciplines into the brainstorming process, you can create more thorough solutions.
Next steps for this project would have been iterating on each feature based on customer feedback
