Magic Leap App: “heAR”
2019
— Magic Leap App
— Unity, Magic Leap One,IBM Watson SDK, Lumin SDK, gcloud, flask
— Collaborator: Richard, James, Mustafa, Devanshi
— My role: Magic Leap+Unity set up and integration, Unity, wireframing, user flow definition, visuals.
Events
— 2019 MIT Media Lab Reality Virtually Hackathon, Winner of the Mobility & Communication Prize
During my final semester of undergrad, I participated in the 2019 Reality Virtually Hackathon at the MIT Media Lab. The goal was to explore how immersive technologies like AR and VR could be applied to real-world problems.
I teamed up with four other participants who shared an interest in communication and augmented reality. We quickly aligned around a core question: How might AR help improve face-to-face communication for people with hearing impairments?
Concept & Inspiration
To ground our approach, we started by discussing common challenges in face-to-face communication, especially for elders or individuals with hearing loss. We scoped our idea toward supporting speech-to-text and sign language-to-text translation in real-time using AR headsets like the Magic Leap One.
The interface was inspired by MMORPG games, where players often communicate via floating text bubbles above their characters. We reimagined this idea in an AR context:
-
The user wears a Magic Leap headset.
-
When someone speaks, the user can point at them using a controller.
-
A live speech-to-text bubble appears next to the speaker's head.
We also borrowed from in-game message logs to help users track ongoing conversations, a particularly useful feature for those who need more time to process auditory or visual information.
A screenshot from World of Warcraft
Core Features
-
Speech-to-text: Real-time transcription using IBM Watson
-
ASL-to-text: Camera-based sign language recognition
-
Message log: Scrollable history of conversations
We called the prototype heAR.
Development Process
We built the app using the Magic Leap SDK for Unity, alongside the Lumin SDK for spatial mapping and object placement.
-
Speech Recognition: Integrated IBM Watson’s Unity SDK
-
ASL Translation: Trained a VGG neural network using a Kaggle dataset, deployed via Flask on Google Cloud
-
AR UI: Custom 3D speech bubbles and UI assets were created using After Effects, Blender, and C4D
-
Hardware: Magic Leap One, with raycasting for identifying and targeting speakers
One of the biggest challenges was dealing with the limitations of the Magic Leap platform—it lacked some basic features we expected (like image recognition), so we had to pivot quickly and find workarounds. Despite the constraints, the process was extremely rewarding.
Listing out development milestones and user flow.ASL -> Text Implementation.
Outcome
After just two and a half days, we delivered a working prototype with real-time speech and sign language translation, AR UI integration, and cloud-hosted neural processing. While we didn’t have time to fully polish the interface or test UX flows, the core system worked.
We were honored to win the Mobility & Communication Prize and were featured by Magic Leap for our work.