Novel bionic glasses help the blind ‘see’ through sound

When The Voice coaches all turned their chairs during Julee-Anne Bell’s 2021 audition, the blind, classically trained singer had no idea that Jessica Mauboy, had swivelled first. “I should probably let you know that I actually can’t see, I can’t see you at all,” explained Bell, 48, who has been without sight since birth. “There’s four of us,” Guy Sebastian enthused. “If there had been a fifth chair, that would have turned around too,” Keith Urban added, before Guy Sebastian jumped up to duet with Bell, singing Climb Every Mountain, from the Sound of Music.  

Bell relied on her cane and help from a crew member to walk on to the Channel Seven stage for her performance. The process would have been far easier had she also been able to use a pair of novel bionic glasses that she has been helping to trial. The ground-breaking ‘smart’ glasses are the brainchild of ARIA, an Australian vision technology innovation company. Their development recently received NSW Health support via a commercialisation training course. They also enjoyed the opportunity to showcase their innovation at the 2023 Joint Economic Meeting, where over 65 leaders from the NSW and Guangdong health and life sciences sectors, came together to explore opportunities for collaboration.

Mimicking the echolocation of bats and whales  

ARIA glasses employ Augmented Sensory Reality, which means they use sound to tell the wearer what the glasses ‘see’ via small cameras in the device. “This is similar to how bats or whales use echolocation as their primary spatial sense,” says Robert Yearsley, Co-founder and Chief Executive Officer of ARIA. “The glasses provide a person living with vision disability with a simplified vision equivalent (or analog) in sound,” Yearsley further explains. “It doesn’t matter what has caused a person’s blindness – as long as they have hearing, the glasses can help them.”  

In Bell’s case, the smart glasses would have helped her better navigate the unfamiliar backstage and onstage environment of a television soundstage. After her audition, the experienced music teacher, who performed with Queensland Opera for seven years, joined Guy Sebastian’s team until she was eliminated by the series winner in the ‘Cut’ round. “Regardless, I loved every moment of my time on the Voice and I made some friends I will be in touch with for life,” says Bell, who is also Managing Director of the not-for-profit organisation, World Access for the Blind, Australia. 

‘Seeing’ through sensory augmentation

Vision impairment is a growing challenge throughout the world, with 338 million people currently affected and projections that this figure will rise to more than half a billion people by the year 2050. In Australia, 690,000 people are vision impaired, with 70,000 people impacted by blindness. To address this growing disability, ARIA (which stands for Augmented Reality in Audio), has developed a non-invasive device that requires no surgery or implantation and can be worn like ordinary glasses. “The glasses use a new form of sensory augmentation that combines elements of visual perception and spatial awareness,” says Yearsley. “They also use procioception which is the ability to sense movement, action and objects nearby and around you,” Yearsley adds. “Using this combination of sensory elements, the glasses aim to complement and enhance a blind person’s primary senses of hearing and touch, rather than trying to restore vision.” 

 Each device contains three tiny video cameras, and an advanced Artificial Intelligence (AI), system that uses sound to convey what and where things are in relation to the user. “The speakers, which are located in the arms of the glasses, transmit an intuitive illusion of the location of specific objects which is updated in real time, relative to the user’s body and head position,” Yearsley explains. “This results in a revolutionary level of spatial foreknowledge for people who are blind.”  
 
Information about nearby objects and the surrounding environment is presented to the wearer in two ways. The glasses use ‘spearcons’ where a voice provides verbal cues that state the name of the object, stating words such as ‘shoes’, ‘computer and ‘stool’ when the camera on the glasses recognises those objects nearby. The glasses also use ‘earcons’, where a sound drawn from a large soundbank is used to represent the object – for example – a meow for a cat or the buzz of a vibrating phone. These verbal cues and sounds are loud enough to be heard by the wearer but cannot be heard by those standing near them. A person wearing the glasses can choose to only use the verbal cues or the sound cues or a combination of both. In the future, it is hoped they will also be able to personalise their soundbank as well. 

‘Hearing’ objects in space 

The high or low volume of the sound cues from the glasses indicates proximity to each object. This helps the person with vision impairment identify objects such as a desk, chair, wine glass, backpack or dumbbell and reach out to touch it with a good degree of accuracy.  In an early 2022 trial, Bell was asked to identify a bottle, book, bowl and cup sitting on a table. “I’ve never had light perception or been able to see anything, so ‘hearing’ objects and identifying them in front of me without touching them, was a very strange sensation,” she says. “I had to learn to move my head in a scanning pattern to kind of ‘look’ at things, but once I got the hang of that, I could quickly and correctly identify the difference between the objects and where they were, which felt amazing and exciting.” 

Enabling this differentiation between objects is a huge game changer for blind people, according to Bell, who adds that the glasses “Are incredibly helpful, whether you’re locating a tea cup on a cluttered kitchen bench or distinguishing an apple from a pear while out grocery shopping.” 

Supporting independence for people with vision impairment

When moving outside of their homes, 90 percent of people who are blind rely on others to assist them. “When you’re blind, if an object isn’t making sound, or you’re not touching it, you can’t really sense it,” says Marx Melencio, who is ARIA’s machine vision engineer and rapid prototyping expert and was the first blind person to test the glasses in 2020. Melencio 43, became blind after a random gun attack in 2003. For years, when moving around, he relied on help from his devoted wife Cherry, and information from his watch, laptop, phone and cane. So, he is very excited that the glasses provide a similar range of information in one single device. “When I wear them, suddenly everything ‘sings’ – parked cars, furniture, doors, my keys or my wallet,” Melencio explains. “Using the glasses allows me to do everything more independently – such as find a bottle of water, locate my phone and make my way to chairs and tables.” 

 On his first three different outdoor road-tests Melencio wore the smart specs to walk his dog, go to the park and venture out to nearby shops to buy something to eat. “It was amazing to feel so much independence,” he recalls of using the bionic glasses to ‘look’ around and perceive the layout of the space, objects and people in the environment outside. “The glasses allow me to walk down the crowded and cluttered sidewalks, where I live, steering around people and objects such as motorbikes on the path,” he says. “In a social situation, I can enter a room and the glasses give me a good idea of how many people are there. If I attend a meeting, they help me know where to look, even if no-one has spoken.” 

Bell echoes his enthusiasm. “With the glasses on, I don’t have to ask about the layout of the room because the sounds tell me where the doors and windows are and where people are gathering,” she says. “Even in a very busy place like Market St in Sydney’s Chinatown, they help me easily discern a food court from a coffee shop.”

Future innovations

ARIA’s breakthrough bionic glasses have been developed in collaboration with the Sydney University of Technology, University of Sydney, Blind Citizens Australia and World Access for the Blind Australia, with support from the Australian Federal Government and through the NSW Health Commercialisation Training Program. Co-design input has also been invited from people who are vision impaired, including Daniel Kish, the inventor of ‘flash-sonar’, a form of human echo-location that uses tongue-clicking to judge the location and size of objects from the sound that bounces back. These collaborations have enabled ongoing and amazing new capabilities to be added to the glasses, tailored to the experience of living with vision disability. They have also contributed to ARIA being named Australian Technology Company of the Year in the 2023 Australian Technologies Competition. 

One proposed future development of this smart vision device is the inclusion of features to track the wearer’s heart rate and breath. “If these markers then indicate a person is feeling mental overload, the glasses will reduce the amount of information they receive, to more simple cues,” Yearsley says. Down the track, the ARIA engineering team hope to also utilise AI to help blind users interpret nonverbal cues that sighted people take for granted. This means that a person with vision impairment would know when someone they are talking to is changing their facial expression, nodding their head, making hand gestures and making direct eye contact.  

Light and fashionable 

Lightweight and black, ARIA’s novel glasses have been designed to look like fashion sunglasses. “This is very important,” says Bell. “Just because we can’t see other people, it doesn’t mean that people who are blind are not aware that other people can see us. When you have a disability, you can sometimes stand out a little bit more, so the last thing you want is to be wearing something that makes you look like you just stepped off the Starship Enterprise on Star Trek. If you can wear some glasses that look like a cool pair of sunglasses, that’s a real bonus.”