Meet Zelda, an Auslan-speaking virtual assistant for Deaf people, and her developers

 

Meet Zelda, an Auslan-speaking virtual assistant for Deaf people, and her developers


A cylindrical device sits next to a tablet computer on a bench.
Voice-controlled technology is on the rise, so where does this leave Deaf people?()

When we were making updates to our house a year or so ago, my family decided to go a bit techy with it. 

The lights can now be controlled from a switch like normal, but also through an app which connects to a voice-activated smart speaker.

It's fun — but it turns out it's more than just a fun party trick. I use it all the time.

"What's the weather going to be like tomorrow?"

"Turn on the hall lights." 

"What's 350 degrees Fahrenheit in Celsius?" (I ask this one so often I really should know the answer by now. For reference, it's about 175C.)

The voice assistant has become something I take for granted, along with something else I take for granted — being able to hear.

So much of our digital world relies on these voice-activated assistants. And that reliance is growing at an astonishing rate. 

It's actually changing so fast it's hard to come up with good numbers, but one industry study in 2018 said there were 1.35 million smart speakers in Australian households, and a 2021 article said 26 per cent or 5.6 million Australians owned a smart speaker, up from 17 per cent in 2020. 

You can see why they're gaining traction — sometimes you just can't get to your phone to tap out a question with your thumbs.

So with a shift towards more voice-activated devices, where does this leave people in the Deaf community?

'Hey Siri, do you speak Auslan?'

The Auslan Communication Technology Pipeline is aiming to create a virtual assistant named Zelda that recognises and speaks Auslan.

Julie Lyons, a Deaf woman from Brisbane who's a research assistant on the project, sees it as a way of giving people independence. 


"Deaf people don't want to become co-dependent on hearing people," Julie says. 

"A lot of Deaf participants that we interviewed were really keen to buy this type of product and that's because they want to become more independent.

"I think it's very important for Deaf people to access Auslan because Auslan is our native language."

A smiling woman makes the Auslan sign for Deaf pride.
Jessica Korte, who leads the Auslan Communication Technology Pipeline, makes the Auslan sign for "I love you" or deaf pride.c at the University of Queensland.

Dr Korte is a hearing woman, but her research specialises in something called participatory design — creating technology with, not just for, people from groups that are traditionally marginalised.

The first step in the project was lots of interviews with Deaf folks "to make sure we're designing a technology they actually want to use in a way that they actually want to use it".

It's not the first time people have tried to come up with a technology to help better connect hearing and Deaf people. But often those attempts have been driven by assumptions about what Deaf people want, says Adele Greedy-Vogel, another research assistant on the project who is a hearing child of Deaf adults.

"A lot of technology in this space … is so that hearing people can understand Deaf people," Adele says.

"There's always an emphasis on hearing people being able to understand and having access to all the information.

"We really need to change that framing because it's Deaf people that need the access, not hearing people."

How does it work?

If you've got a device like a Google Home or an Apple HomePod, you probably have a good idea of how this would work, but instead of an audio interface, it's a visual one.

A person can sign to it with a question or command and have the device understand and come back with a response. For example, "350 degrees Fahrenheit is 175 degrees Celsius" 

An animated woman makes the Auslan sign for "name". 
The avatar makes the Auslan sign for "name". 

But in their co-design process, the Deaf people the team interviewed added a few other features they'd like to see, Dr Korte says.

"Doorbells are a big one," she says.

"But also, Deaf folks have said they'd be interested if it could do things like give them notifications if there were particularly interesting sounds going on in the house or in the neighbourhood that they're not hearing for themselves.

"Things like dogs barking, car alarm going off, or babies crying in the house."

Julie says, at the moment, many Deaf people rely on neighbours to keep them informed.

"With a product like this, they would be able to ask someone in their home … and it would be able to sign back," she says.

"That's equal access for Deaf people. Genuine equal access."

How to build a virtual assistant 

So, what does it actually take to build a tool like this? Siri, Alexa and the other big players in the voice assistant space have the full weight of digital empires like Apple and Amazon behind them.

Can a scrappy team from a university have the grunt to produce something that will work? And is the Australian market big enough to make Zelda commercially viable?

Julie Lyons in mocap suit 2
Research assistant Julie Lyons wears a motion-capture suit to "teach" Auslan to the artificial intelligence system behind the avatar

These questions remain to be answered Dr Korte says, but the team is forging ahead anyway. 

"What we need from an AI perspective is a lot of data, because that's how these machine learning systems work. You need lots of people providing lots of data so that the computer systems can learn all the variations in the way people will sign the same set of commands," she says. 

"We also then need high-powered computers in order to train the machine learning systems, because you're essentially throwing the data at it and letting the computer figure out what are the patterns that are important

Post a Comment

أحدث أقدم