Hearing Like Humans? Sound Symbolism and Perceptual Alignment in Speech Language Models

2026-07-11

AI Summary

This study investigates whether speech language models (SLMs) possess the human-like capacity for sound symbolism—the perceptual association between speech sounds and sensory attributes such as roundedness or sharpness. Departing from prior text- or image-based approaches, this work presents the first systematic evaluation of SLMs’ sound symbolic judgments using authentic human speech recordings across auditory, cross-modal, and visual dimensions, benchmarked against human behavioral data. Through acoustic feature analysis—particularly spectral tilt—and controlled visual experiments, the study reveals that SLMs significantly diverge from human perception in auditory judgments, fail to capture the key acoustic cues underlying sound symbolism, and cannot reliably map sounds to corresponding shapes, thereby exposing critical limitations in their speech representations.

Abstract

Sound symbolism, the human tendency to map speech sounds to perceptual qualities such as roundness or sharpness, arises primarily from the acoustics of speech rather than spelling. Whether Speech Language Models (SLMs) share this tendency remains open, as prior evaluations rely on text or images rather than real speech. We study it using genuine human speech recordings, comparing model judgments against human data across the auditory, crossmodal, and visual components of the effect. We find that SLMs' auditory judgments align poorly with human perception and miss the acoustic cues, such as spectral tilt, that drive human intuitions, and open-weight models cannot reliably link a heard sound to its corresponding shape. With a visual-only control ruling out shape perception, the weakness localizes to how speech is represented, suggesting that perceptual alignment depends not on stronger vision but on speech representations that capture the cues humans hear.

Problem

• sound symbolism
• speech language models
• perceptual alignment
• acoustic cues
• human perception

Innovation

• sound symbolism
• speech language models
• perceptual alignment
• acoustic cues
• spectral tilt