Bubble.io
Bubble.io is an augmented reality app that assists individuals suffering from hearing loss
Project Overview
Bubble.io is an award winning augmented reality app designed for the Seattle 2018 VR/AR hackathon.
This augmented reality app aims to assist individuals suffering from hearing loss by providing speech bubbles in the user’s field of view so that the user is to remain connected to the conversation and the group.
I was the lead UX designer on this project.
Skills: Research, persona creation, user-flow creation, prototyping, visual design, empathy-building demo for judges
Timeline: 48 hours
Tools: Pen & paper, Sketch, ear plugs, Magic-leap immersive headset, Unity game engine
Seattle VR/AR hackathon
The Seattle VR/AR hackathon is a weekend event where individuals form teams around compelling project ideas to build a product from start to finish. A panel of judges grades each demo on the final day and awards a set of prizes.
The Seattle VR/AR hackathon focuses on immersive technologies such as Virtual Reality, Augmented Reality and 3d User Interfaces.
Research | The Impact of Hearing Loss is Serious
Data shows that individuals who experience hearing loss late in life were twice as likely to experience depression.
20% of Americans report some form of hearing loss
50% of these individuals lost their hearing late in life
Individuals with hearing loss are twice as likely to have depression as those with normal hearing
This data was coupled with anecdotal stories that people at the hackathon shared about elderly family members at social gatherings. Below i’ve shared some responses I gathered when I posed the question, “Can you describe your experience with hearing disabled family members in social settings?”
“I remember Grandpa often sitting on the sidelines at parties because he couldn’t hear what was going on.”
“I felt frustrated when talking to my Great-Aunt at events because she would have so much trouble understanding me.”
“Sometimes it seemed like my Grandma would be in her own little world at parties, humming to herself and not paying attention to what was going on.”
Problem and Goal Definition
Problem: People who lose hearing late in life feel disconnected in group settings.
Goal: Leverage augmented reality technology to increase understanding, participation, and connection by adding speech bubbles into a user’s field of view.
Product and story sketching
Now that the problem was uncovered, I began a sketch and ideation session.
The best UI would be zero UI — the immersive headset adds speech bubbles to the user’s field of view automatically
We decided relatively early to focus on telling a user’s story rather than building a working prototype. The story that we envisioned: a family member having trouble tracking a dinnertime conversation.
The family member with hearing loss cannot hear what the other two individuals are saying. Once they put the headset on, speech bubbles appear like close captioning. This allows the family member to interact at the table and provide insight into exactly what ingredient is missing from great-grandma’s chili.
Demo videos
First-person perspective
Demonstration of the speech bubbles inserted into the user’s field of view. Frame-rate jitters are not reflective of the Magic Leap One’s capabilities — we simply had issues compiling our demo to run in production mode.
Third-person perspective
To simulate hearing loss, we had testers insert ear plugs and walk through a scene with us. You can see the speech bubbles on the laptop screen as Philip is stepping through the scripted dialogue.
Result
The Bubble.io team was honored with an award for Outstanding Novelty!
The story that we told to the testers and judges resonated.
We simulated a hearing loss environment to show how disconnecting it feels, then showed how speech transcription in the field of view can help someone re-connect with the group.
Having the judges insert ear plugs during the demo was key in fostering empathy. By ‘walking in the shoes’ of our target users, the judges were more equipped to understand the need for the bubble.io solution.
Next steps
This project is a proof of concept. Further research is needed to assess the needs of individuals with profound hearing loss in group settings, and the efficacy of existing solutions.
Integrate speech to text capability, with voice identification
From facial recognition data, use articulation of mouth to link speaker’s face with voice ID
Develop working dynamic speech bubbles in accordance with existing closed captioning UX & accessibility standards
Team Bubble.io:
Nicole Eskandari (UX research, UX design, deck design)
Ian Page-Echols (initial pitch, UX design, Unity development)
Philip Kobernik (story development, UX design)