AI Can't Do Sign Language...Yet (Part 2)

Vlog | Destiny Yarbro | August 12, 2026 | 14 min watch

The MOMENT I published publish on my last AI video, new thoughts popped into my head regarding AI and sign language. This video presents 6 more reasons AI will struggle with sign languages.

0:20 What Most Tech Articles Get Wrong About AI and Sign Language

Most tech articles you see say something like "AI knows sign language!" or "AI ring or AI gloves are capable of understanding its user's sign language!" or "robot capable of signing!" But if you read the article more in depth, you realize it's talking about fingerspelling not actual signing, not actual sign language, just a glove that understands fingerspelling.

But maybe you're thinking, "Isn't fingerspelling sign language though?" Yes, BUT...

It's like if you were to speak English only with letters, spelling each word. "H O W A R E D O I N G [question mark]" Is that English? Is that speaking English? No! That's just letters.

Same idea, fingerspelling is not equal to sign language. But a lot of people think "I know how to fingerspell that means I know how to sign." ...

And a lot of the people who are building these AI technologies, the rings, the gloves, they don't know sign! They might be able to fingerspell a bit. I've watched them, they can barely fingerspell. They don't know sign, they don't know how to fingerspell really yet they're the ones trying to teach AI how to sign...hmmm.

1:49 Information Communicated Outside of the Signs Themselves

Now suppose these rings and gloves do figure out how to sign, let's just suppose. As I mentioned in the previous video, understand that ASL signs, handshapes, movements all have meaning. But there are so many more ways to show meaning in sign.

For example, facial expressions. You can be signing along but if you change one facial expression, the meaning completely changes.

Remember back in the day when we would text without emojis? You know the faces with all of the facial expressions? What happened? Misunderstandings right and left because you had to try to discern, you didn't know if what they texted was funny, a joke, or serious, or sarcastic. It was really hard to tell.

That! That's what AI is doing when it's only focused on the signs without facial expressions and all these other parameters. A ring or a glove is not going to include that information.

3:08 Nuances That Change Meaning in ASL (5 Parameters)

For example, facial expressions. You can be signing along but if you change one facial expression, the meaning completely changes.

Remember back in the day when we would text without emojis? You know the faces with all of the facial expressions? What happened? Misunderstandings right and left because you had to try to discern, you didn't know if what they texted was funny, a joke, or serious, or sarcastic. It was really hard to tell.

That! That's what AI is doing when it's only focused on the signs without facial expressions and all these other parameters. A ring or a glove is not going to include that information.

For example, this sign with two facial expressions or this sign with two facial expressions have the same handshape, the same sign, the same movement, but different meanings.

Facial expressions is just one of many tiny nuances that can completely change meaning. Others are:

Second, the weight of the sign. Let me give you an example of what that is.

I can sign "hard" like this as in "Wow, that homework was hard!" Compare that with signing it like this. The signs themselves are the same but I'm putting a different weight into the second to show "I'm so sorry, wow, you are going through so much."

A funny example, I remember back in the day, I was chatting with a woman, a Deaf mom and as we're chatting her kid kept coming up again and again saying, "Hey Mom, Mom, hey mom". Her kid was Deaf too. "Mom, hey, mom mom mom." The two of us tried to keep chatting but it was a struggle so the mom turned to her kid and told him to stop. She signed "finish" meaning, "Stop interrupting."

Anyways, we continue with our conversation but the kid just keeps coming back, "hey mom, mom, mom" and each time the mom would sign "finish" bigger and bigger with increasing frustration. "Finish. Finish!!"

Finally, she had had enough and I watched this mom turn to her kid and sign "finish" like this. Hardly a movement. What did the kid do? He shut up immediately and slunk off. So the mom had been signing bigger and bigger but it was finally when she was at the end of her rope that she signed it like this, just the tiniest movement, but the weight behind the sign communicated so much more. Like "You better stop or else..."

Third, speed is another nuance, whether a sign is slow or fast. For example, this sign can mean "sometimes" or it can mean "every now and then" or it can mean "rarely". These are all the same sign but with different nuances depending on the size of the sign.

Fourth, and this is a fun one. Palm orientation changes meaning. For example: (ready for this?) Good. Bad. Good, the palm orientation is facing up. Bad, the palm turns to face out, the palm orientation faces down. These signs are opposites just like the signs for "I know" and "I don't know." It's just a palm orientation difference.

Fifth, location, where it's signed on the body. For example: this signed by the head means "terrible" while this signed at the shoulder means "Chile". Even if the first four are signed the same, if the fifth one is different, the meaning is completely different.

So maybe AI could come to understand the 2nd nuance and the 5th nuance but remember these are all happening at the same time.

They're all influencing each other. That is something that AI is going to struggle to capture all that visual information at the same time.

These 5 points lead into another thing that I think AI just won't understand for many many years.

5:13 Variations: Sign Locations

Remember the 5th nuance? Location and where a sign is located on the body?

Maybe you're thinking, well, that will be easy for AI to quickly pick up. And you're right, AI could learn that this sign here at my chin means cute while this away from my body means "haha," this means "water" and this means "weather". Over time it can figure out where the signs are on the body and in relation to the head.

Like "girl" is signed at the chin. The letter A is signed out here. "Adam" is signed here at the head. So yes, it could figure out where a sign is located, mapping it out. "The face is here so girl is signed here, that makes sense, and Adam is signed up here and A is out here." It could figure those out.

BUT. ONE. BIG. PROBLEM.

Do signs' locations stay in the same place? Nope. They move all around, they change locations depending on accent, style, age of the signer, when someone picked up sign language.

So maybe you sign "I don't know" up here at the forehead, but I sign it here by the cheek bone. Maybe, like in a car if you're driving, I would sign it out here way in front of me so you can see it in your peripheral vision while you're driving.

Or maybe I'll sign it like this because I'm whispering.

8:06 Variations: Whisper Signing

Wait, are you thinking, "How do you whisper in sign?"

Yes, you can whisper is sign! How? Really small in front of your chest so you can block your hands so no one can see what you're signing.

Typically you sign a box about this big. If you want to have a private conversation, however, this box shrinks down so that it's just about this big in front of your chest.

So this is where the head is and this is where the waist is relatively. So "I don't know" which is normally signed up her becomes "I don't know" down here.

8:44 Variations: Baby Signing

AND if you see a Deaf baby signing. It means their parents have been signing with them and they'll try to copy their parents' signing and they'll even try to fingerspell, moving their little fingers. Now can a 8-month-old baby or a 3-year-old toddler actually fingerspell?

Of course not! They can't just like hearing babies can't spell. So what are the babies actually doing when they fingerspell? They're copying the shape of the fingerspelled word. For example, fingerspelling the name "William" would maybe be this:

Maybe they've been able to spot a letter here and there but really they're copying the shape. The same thing happens with signs. For example, the sign for "bathroom" becomes this or this because they can't form a T handshape, a baby can't sign that, a toddler can't sign that.

AI would struggle to understand a kid's signing, it just couldn't do it.

9:58 Other Physical Variations That Will Impact AI’s Ability to Understand Signing

But another physical adaptation to signs is if a signer is cold, they're going to struggle to fingerspell or if they're drunk, their signs are going to be all sloshy.

I remember some time ago, I was chatting with a woman and for some reason I was struggling a little with understanding her signing, I kept misunderstanding her a bit. It wasn't until later that I realized that she was missing a finger, her ring finger.

So I was understanding enough but it was a little tricky. That, I'm sure, is like hearing who speak with a lisp or a stutter. It can be hard to understand, right.

A signer might have Cerebral Palsy or Arthritis and struggle to produce the signs.

But maybe you're thinking, we have this in the hearing world too though...

That's true but I think it's important to understand that 40% of Deaf people have other disabilities. Sometimes that disability may have caused them to lose their hearing. So there is a significantly higher number of disabled signers which means a wider variety of signers.

11:09 One-Handed Signing

And another adaptation that's very common is if someone signs one-handed.

For example, if someone is chatting on the phone, they're signing one-handed. This is super common. AI would watch the signing and say "What on earth is that?" The signs, shapes, movement all change because you're signing one-handed.

So all of these examples show the very normal variances that occur in sign language that may cause AI to struggle.

11:42 Every Signer Has a Different Accent

Before we finish this video, I want to add a thought.

In my last AI video, I had 4 points and I want to expand a bit on two.

First, I talked about accents. I mentioned that each person's signing looks very different because their language backgrounds are different which means that you may have two neighbors.

They live in the same area, they live next to each other, but their signing is completely different. One neighbor's parents signed while the other grew up with parents who didn't sign but spoke only. Their signing is going to look very different.

Imagine if every time you met an English speaker, they had a different accent. Each person had not just a slight difference, but a different strong accent.

12:41 ASL Data Limitations

Second, in my last video, I mentioned how sign language training data for AI is limited. A comparison came to mind that I wanted to show you.

AI required a vast amount of data collected to be able to understand English, right? But that data is limited for ASL why?

Most videos online (and I've searched extensively!) most videos are either baby sign videos or ASL 101 videos teaching basic signs. Like my new YouTube channel here.

Anyway. How is AI going to learn sign using only these two types of videos?

It's like if AI learned English only using the homework of 5-year-old children or learning from non-native English speakers or only learning English through English 101 or English As a Second Language classes.

Ummm...could AI really learn true English that way? No, obviously.

It's a funny example but it means that AI needs more fluent signers online, it needs videos about more complex topics, it needs more in-depth teaching videos not just surface-level ASL 101 videos.

My Videos