Accessible AR Glasses
Academic Project
Smart glasses that enhances understanding in group conversations for people with hard of hearing, even in loud environments.
Feedback
"I truly enjoyed working with you on this. You took it so far in just a few weeks."
- Ryan, the co-design participant
Skills
Co-design
User Research
Prototyping
Role & Team
Product Designer
Team of 4
Timeline
3 months
We partnered with Ryan, who is hard-of-hearing, to address his specific challenges. While our solution is designed for him, we understand one user's experience does not represent the entire community.
Context
Our Method: Co-Design
We used co-design to:
Share the power in a collaborative relationship
Honor the expertise of people with disabilities
Ensure the outcome meets their needs
Accessible Research Practices
As a lead interviewer, I conducted a literature review of 6 papers and implemented accommodations like:
Microphone to improve accuracy in zoom captioning
Round table so the team can all be lipread conveniently
Slideshow of questions as a written aid
Interview: Understanding the Problem Areas
I led the semi-structured interview to understand Ryan's history of hearing loss, test our assumptions, and learn his pain points in a few areas.
Safety
Pain Point:
He needs to be aware of his surroundings for ambulances while driving
Social
Pain Point:
He doesn't want to interrupt conversations because of his disability.
Recreation
Pain Point:
He's limited in social events depending on how loud the environment is.
Brainstorming to Scope
We ideated for 5 minutes to address the 3 problem areas listed above.
My 2nd idea was inspired by our caption set-up. Ryan wished he had it for his company meetings.
Our user ideated about showing the direction of the audio source in a group conversation.
As a group, we scoped it to the most intriguing ideas that were related to social interactions.
Interview Findings
Through affinity diagramming, we learned that:
Hearing loss negatively impacted social interactions:
Decreased engagement, confidence, and opportunities accessible to him
Using hearing aids has a positive impact on social interactions:
Increased engagement and confidence when he could hear and understand more.
Hearing aids still have gaps in ability, leading to missed conversations when:
People are too far away, background noise is too loud, or high-pitched voices.
Problem
During the interview, we learned that Ryan was missing conversations because:
Group conversations went too quickly for him to lipread
People talked over each other, which is impossible to lipread
Background noise would be too loud to hear even with aids
Low-fi Sketches
As a group, we explored 16 ideas for a variety of devices. We then narrowed it down to 2 ideas.
1. Bluetooth Microphone with Captioning App
Setting: Office meeting when the speaker is too far away to hear
Features:
-
Petal will light up in the direction of the audio
-
Cue the speaker with requests like "slow down"
2. Smart Glasses with Request Features
Setting: Restaurant with high background noise
Features:
-
Highlight who is speaking
-
Request accommodations for restaurant beforehand
The story shows how it would be set up and how speakers might respond to "slow down" requests.
The story shows the flow for accommodations and the highlighted speaker with captions below.
Mid-fi
We created physical prototypes and mid-fi screens to gather accurate feedback for our physical ideas. We adjusted from our low-fi sketches based on Ryan's feedback over email.
1. Clip-On Microphone and Captioning App
Discreet and Comfortable Accommodations
-
Smaller and lighter mic
-
In-app requests that activate LED indicators on phone stand
-
Read captions comfortably with the phone stand
Invite Others to Improve Captioning
-
Attendees can edit captions and topics to improve its accuracy
-
Questions can be asked and answered without interrupting the flows
Midfi Insights
Ryan had experience with movie captioning glasses, which impacted his preference for smart glasses.
Looking at a phone distracts from lipreading, which requires focus on the face.
He only needs to peek at captions occasionally to understand the general context.
A phone can be distracting, awkward, and disrespectful in social settings
Ryan wanted to appear engaged and didn't want to interrupt the group too much.
Bringing extra items was unpractical and a hassle
Wearable tech was unintrusive without obvious signs it's an assistive technology.
The first idea of a microphone with a captioning app didn't align with Ryan's needs because:
Ryan felt like it was seamless how the captions were always in his field of vision. He didn't need to look away or down from the movie to view captions.
2. Smart Glasses
Highlight Speakers and Focus Captions
People speaking will light up in different colors and you can choose which captions to read
We acted out a conversation to emulate the experience of a smart glasses that highlight faces
We moved the captions around to show him the options and asked him how many lines of captions he preferred
Flexibility with Captions
The positions and size of captions will follow the speaker and can be adjusted.
After deciding on smart glasses, we took turns going back and forth creating or expanding on ideas for a minute.
Co-Design Activity: Hot Potato
Ryan iterating with Yifan about NFC idea to onboard people into a text group.
I'm ideating that accommodation notifications could have a unique vibration pattern
From this session, we narrowed it down to smart glasses with an NFC feature to make it convenient without needing to download another app.
Solution
Onboarding Flow
I created this flow chart to determine how we could easily onboard attendees in person. With our NFC brainstorming session, we were inspired to create a Bluetooth room that adds everyone to a text group chat and send accommodation notifications.
I added the share group code step to prevent random strangers from joining and disrupting the online room.
1. Connect to Ryan's Group
2. Enter pin to join room
3. Start tutorial
4. Learn about notifications
5. Thank you note from Ryan
6. Onboarded and connected!
I designed the accommodation request features and unique vibration patterns so people could check the notification without lifting up their phones over time.
Feature Design: Request Accomodations
Users are walked through the different kinds and why it's important to follow during the onboarding process. They can play vibrations to familiarize themselves with it.
The slow-down notification is 4 slow vibrations that mimic a steady and fading heartbeat.
Create a private room and onboard attendees via Bluetooth.
Minimal explanations are needed with the tutorial.
Easy Onboarding
Ryan's App View
Attendee's View
Captions follow your vision so you don't need to look away from lipreading.
Remember who is speaking with custom color-coding to captions.
Know Who Is Speaking
Subtly alert specific attendees to keep conversations going.
Alerts with brief vibrations and messages allow people to stay informed at a glance.
Request Accommodations Discreetly
Attendee's View
Ryan's App View
Pair with your hearing aids and adjust it to your settings for the best experience
Captioning can change in size and position to avoid blocking important parts of your vision.
Customize to Your Needs
Feedback
“This would be helpful and easy to use! I truly enjoyed working with each of you on this project. You guys took it so far in just a few weeks."
- Ryan, our participant
“I would love to have this request accommodation feature in the classroom as a teacher. It would be so helpful to see and use in in-person settings."
- Tyler Fox, UW Professor of Interaction Design
If I had more time, I would test it with other users and measure its effectiveness with these metrics:
Confidence in Understanding Group Conversations
Participants will rate their confidence pre- and post-product, gauging real-world applicability and prompting consideration of edge cases.
System Usability Score (SUS)
The average score would give insight into its simplicity and ease of use for both users and attendees. I would use it for comparisons between different designs and versions.
Impact: Success Metrics
Explore repeated usage
We prioritized first-time attendee flows, briefly addressing saved color settings for speakers who may join again. I would create flowcharts and test with repeated guests to identify areas for efficiency.
Test with other kinds of settings
Our primary use case was at a loud restaurant. Further testing for other scenarios would bring considerations like if people are blocking the view of other people at the bar counter.
Next Steps
Always test your assumptions and listen to lived experiences.
Literature reviews fall short of capturing nuanced disability experiences. Our participant's expertise prevented bias and misunderstandings, guiding tailored support to his needs.
Co-creation can be easily incorporated and leads to effective solutions.
Co-design with Ryan gave us firsthand insight that led to innovative ideas we hadn't considered. Even 5-minute ideation activities with him offered clear direction for product refinement.
Lessons Learned
Solution
We confirmed Ryan's priorities in group conversations and formed our design requirements:
Identify who is speaking in the group
1
Enable captioning as needed in a group setting
2
Requesting accommodations in a comfortable manner
3