EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Point-light Talkers: Multisensory Enhancement of Speech Tracking by Co-speech Movement Kinematics

Jacob P Momsen, Seana Coulson iD

Journal of Cognitive Neuroscience · 2025

DOI: 10.1162/jocn.a.62 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Cognitive Neuroscience, 2025. Verfasst von Jacob P Momsen und 1 weiteren Autor:innen. Beteiligte Einrichtung: San Diego State University and UC San Diego. Hauptthemen laut MEDLINE: Gestures, Speech Perception, Speech, Visual Perception. Die Arbeit nennt 61 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

While multisensory super-additivity has been demonstrated in the context of visual articulation, it is unclear whether speech and co-speech gestures are similarly subject to super-additive integration. The current study investigates multisensory integration of speech and bodily gestures, testing whether biological motion signatures of co-speech gestures enhance cortical tracking of the speech envelope. We recorded EEG from 20 healthy adults as they watched a series of audiovisual (AV) discourse clips from four conditions: AV congruent clips with co-speech gestures that were naturally aligned with speech, AV incongruent clips in which gestures were not aligned with the speech, audio-only clips in which speech was delivered in isolation, and video-only clips presenting the gesture content with no accompanying speech. As we hypothesize that the kinematics of co-speech gestures are sufficient to drive gestural enhancement of speech tracking, our clips employed minimalistic “point-light” depictions of a speaker's movements: point-light talkers. Using neural decoder models to predict the amplitude of the speech envelope from EEG elicited in all four conditions, we compared speech reconstruction performance between multisensory (AV congruent) and additive models, that is, those representing the summed neural response across the two unisensory conditions. We found significant improvement in decoder scores for models trained on AV congruent trials relative to both audio-only and additive models. Forward models of brain activity indicated signatures of multisensory integration 140–160 msec following changes to the speech envelope. These results provide novel evidence for a multisensory enhancement effect of co-speech gesture kinematics on continuous speech tracking.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

GesturesSpeech PerceptionSpeechVisual PerceptionHumansMaleFemaleAdultBiomechanical PhenomenaYoung AdultElectroencephalographyPhotic StimulationMovement

Autor:innen und Institutionen

  • San Diego State University and UC San Diego.
  • Yale School of Medicine.
  • Kavli Institute for Brain & Mind.

ORCID-Profile: Seana Coulson

Zitieren

APA

Momsen, J. P., & Coulson, S. (2025). Point-light Talkers: Multisensory Enhancement of Speech Tracking by Co-speech Movement Kinematics. Journal of Cognitive Neuroscience. https://doi.org/10.1162/jocn.a.62

Vancouver

Momsen JP, Coulson S. Point-light Talkers: Multisensory Enhancement of Speech Tracking by Co-speech Movement Kinematics. Journal of Cognitive Neuroscience. 2025. doi:10.1162/jocn.a.62

BibTeX
@article{Momsen2025,
  author = {Momsen, Jacob P and Coulson, Seana},
  title = {Point-light Talkers: Multisensory Enhancement of Speech Tracking by Co-speech Movement Kinematics},
  journal = {Journal of Cognitive Neuroscience},
  year = {2025},
  doi = {10.1162/jocn.a.62}
}
Kostenlos starten →