gabe
< writing

Hearing Is Not Understanding

2026-10-24 / 14 min

Most voice AI today never actually hears you. What happens is your speech gets turned into text, and the model reads the text.

So what?

everything about how you said it gets thrown away. Some(but not all) paralinguistic factors that get thrown away are:

Over the past week or so I've been researching audio models and messing around with them. First, I'm going to map out audio AI and the different types of models, and then I go into an experiment where I test if a modern 'fused' model can actually hear to the level that humans do.

How audio works

This isn't mandatory to understand, but it helps lay the groundwork for how these models work. Kinda like understanding how the brain works and applying that to Neural Networks.

So, How do humans hear? At a high level, we have cochlea in our ears that take vibrations and pressure(what sound is) and translate them into a continuous electrical nerve signal for our brain to process with a frequency decomposition.

As sound travels through the air, the pressure dips up and down. The rate of these dips is the frequency. The higher the individual peak, the louder the sound.

Building upon this,

You can store one second of 16 kHz audio as a 16,000-dimension vector, then turn it into a spectrogram, which is the representation of sound in an image format.

A spectrogram of speech

Now, let's get into the important part because who cares about humans in this AI filled world?

How AI models hear

There are three ways to build a voice AI model today.

Cascade: speech to text to speech

The easiest to comprehend and most common setup is a chain of three separate steps:

  1. Speech-to-text (also called ASR, automatic speech recognition) turns your voice into a transcript.
  2. A text LLM reads the transcript and writes a reply.
  3. Text-to-speech (TTS) reads the reply out loud.

This is easy to build and easy to debug, because you can read every step. The catch is the middle step only ever sees words. Tone, sarcasm, and hesitation are gone before the "brain" gets involved. It also has to wait for you to finish talking before it can respond, which is why these systems can feel slow or talk over you.

Some companies in this field are

2. Native audio, turn-based

For this, one model takes the audio itself as input, with no transcript in between. So it could hear tone. But it still works in turns, like a walkie-talkie: you talk, it listens, then it talks. This is the kind of model I mostly tested.

Some players here are

Generally these models are more modern and are trained on pure audio

3. Fused, full-duplex

"Full-duplex" means both sides can talk at the same time, like a phone call instead of a walkie-talkie. These models listen and speak at once, all the time. That lets them do the small things humans do: say "mm-hm" while you're still talking, handle you cutting them off, and jump in at the right moment. The tradeoff is that they're harder to give instructions to, because there's no clean text prompt in the middle. Another limitation of these models is a small context window so they can't reason much.

Some players here are

Now that you have a basic understanding of different models, lets see how they perform in a high stakes environment of doing sales calls!

The experiment

I compared a turn-based model that accepts audio and follows instructions with a full-duplex model that listens and talks at once. To see if the dull-duplex model is able to detect tone and emotion and act on it in a sales call environment

Setup

First attempt

My first version priced a $100 clock. Even then, the pattern showed up.

When Gemini isn't told to factor in tone or mood:

Here's Gemini's reasoning on one happy clip(take note of the discrepency!):

"The customer sounds frustrated and impatient. Offering the full $100 list price to an angry customer carries a high risk of rejection, so offering a substantial discount down to $60 maximizes the expected value by significantly increasing the likelihood of a quick, frictionless purchase.

'I understand you're looking for a new alarm clock. Since you seem like you just want to get this done quickly, I can offer you our top model, usually $100, for a discounted price of $60 today.'"

A happy customer, heard as frustrated, offered a 40% discount.

Without being told, the model ignores tone

In the not told condition, prices barely moved across tones: 70% of audio runs chose exactly $35, slightly above list. The model's stated reasons describe a "direct, no-nonsense" customer, not an angry one.

When told, prices follow what the model thinks it heard

Gemini 4 Argon's price by actor and tone. Colored dots are runs where the prompt says emotion matters; the black bar is their average. Hollow gray dots are the same clip when the prompt doesn't mention emotion. Look at the highlighted happy row: same tone, same sentence, $35.20 from one actor and $23.60 from the other. Typed words with no audio stay at $30, so the shifts come from the audio.

For actor 1006 the told prompt mostly works as intended: angry $20.90, happy $35.20. For actor 1069 every tone is discounted, including happy ($23.60), because the model hears that clip as angry.

Look closer and something else shows up. Told to weigh tone, Gemini discounts almost every voice by $7 to $14, whatever the emotion. Neutral and sad get the same treatment as angry. Only one of the twelve clips, actor 1006's happy take, gets a markup. Typed words barely move at all (about $2). This could be a prompting issue because I mention happiness and anger.

One way to measure this is like classifier-free guidance in image models: run once with the instruction and once without, and the difference is how much the instruction moved the output.

The model's own reasons show the split. On 1006 happy ($38): "The customer speaks with a bright, pleasant, and upbeat tone of voice, indicating she is in a good mood and likely willing to pay above the standard $30 price." On 1069 happy ($24): "The customer sounds quite stern, abrupt, and slightly irritated or impatient."

The told prompt also invites made-up tone. On typed words, with no audio at all, Gemini sometimes said "the customer's tone is completely neutral," a tone it can't hear from text. It priced at $30 anyway, so no harm here, but it's worth watching.

Hearing is weak, and depends on who's talking

Asked directly, Gemini named the intended emotion in 21 of 60 trials. Random guessing would get about 9. And it was far better on one voice than the other:

What Gemini 4 Argon answered in 5 trials per clip, colored by the emotion it chose, with the intended tone on the left and whether human listeners recognized the clip on the right. All 60 transcripts were word-perfect, so the audio arrived intact. The model heard the words fine. It missed the feeling.

This explains the price chart. The only clip that got a markup (1006 happy) is the only happy clip Gemini heard as happy. 1069's happy take was heard as "angry" 4 of 5 times, so it got discounted like the upset tones. And since people recognized 5 of 6 clips for both actors, 1069's acting isn't the main problem.

The mistakes aren't random. Loud, tense clips become "angry" (including 1069's happy and fearful takes), and quiet clips become "neutral" or "sad". The model seems to track arousal (how energetic the voice is) but not valence (whether the feeling is good or bad).

Other models make the same kind of mistake. On this clip, most people heard angry. A small classifier trained for emotion agreed. Both audio LLMs called it neutral:

Do other models do any better?

Next I ran the same pricing task on ten more setups: other audio models, two streaming voice models (GPT Realtime and Gemini Live), a small model running on my laptop (Ultravox), and a cascade where Whisper writes a transcript and Gemini 4 Argon only reads the text.

How to read the chart: each row is one model. Each colored dot is the average price it charged for one tone. The dashed line is the $30 list price.

The number on the far right is the one-line answer: once the prompt says emotion matters, how much more did a happy customer pay than an angry one? It should be positive.

Average price per tone, 20 runs per dot (two actors, ten runs each; 16 for GPT Realtime). Left: the prompt never mentions emotion. Right: the prompt says happy customers pay more and upset ones less. The black bar is the same sentence typed, with no audio. Rows are sorted by the happy minus angry gap.

See the exact prompts

Every model got the same system prompt. The only difference between the two panels is the middle paragraph.

Left panel, emotion not mentioned:

You are a sales rep at a home goods store, talking with a customer in person.
Our alarm clocks usually sell for $30, and each one costs the store $12.
You may charge more or less than $30. Customers have paid anywhere from $15 to $50.
Pick the price that maximizes expected profit: (price minus $12) times the chance they buy.

Right panel, emotion in the prompt (the added lines are the ones starting "How the customer sounds"):

You are a sales rep at a home goods store, talking with a customer in person.
Our alarm clocks usually sell for $30, and each one costs the store $12.
You may charge more or less than $30. Customers have paid anywhere from $15 to $50.
How the customer sounds matters a lot. If they sound happy, excited, or in a good mood,
you can charge more than $30. If they sound angry, upset, or sad, they usually pay less.
Listen to their tone of voice, not just their words.
Pick the price that maximizes expected profit: (price minus $12) times the chance they buy.

The customer's turn. The user message is "The customer walks up to you and says:" followed by the audio clip. For the typed-words control, the sentence is written out instead: "I would like a new alarm clock."

How the price comes back. Text models are asked for JSON:

Respond with JSON only:
{"spoken_reply": "what you say to the customer", "price": <number in dollars>,
 "purchase_probability": <0 to 1>, "reasoning": "why you chose that price"}

Models that answer out loud (GPT Realtime, Gemini Live) are asked instead to "speak your reply to the customer out loud, in one or two sentences" and to always say the exact words "the price is" before the number, so it can be read back from the transcript.

Flip from the left panel to the right one and watch what moves:

See how it works with this video

So the pattern from the Gemini deep dive holds across the field. Getting a model to act on tone is easy (one sentence in the prompt). Getting it to act on the right tone is not, and only one model of eleven did it convincingly.

What this suggests

Next, I want to:

end.