EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language

Xizi Deng iD, Elise McClay iD, Erin Jastrzebski, Yue Wang, H Henny Yeung

PLOS ONE · 2024

DOI: 10.1371/journal.pone.0304150 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS ONE, 2024. Verfasst von Xizi Deng und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Linguistics, Simon Fraser University, Burnaby BC, Canada. Hauptthemen laut MEDLINE: Language, Speech Perception, Phonetics. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: China Scholarship Council, Natural Sciences and Engineering Research Council of Canada, Social Sciences and Humanities Research Council of Canada.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

When comprehending speech, listeners can use information encoded in visual cues from a face to enhance auditory speech comprehension. For example, prior work has shown that the mouth movements reflect articulatory features of speech segments and durational information, while pitch and speech amplitude are primarily cued by eyebrow and head movements. Little is known about how the visual perception of segmental and prosodic speech information is influenced by linguistic experience. Using eye-tracking, we studied how perceivers’ visual scanning of different regions on a talking face predicts accuracy in a task targeting both segmental versus prosodic information, and also asked how this was influenced by language familiarity. Twenty-four native English perceivers heard two audio sentences in either English or Mandarin (an unfamiliar, non-native language), which sometimes differed in segmental or prosodic information (or both). Perceivers then saw a silent video of a talking face, and judged whether that video matched either the first or second audio sentence (or whether both sentences were the same). First, increased looking to the mouth predicted correct responses only for non-native language trials. Second, the start of a successful search for speech information in the mouth area was significantly delayed in non-native versus native trials, but just when there were only prosodic differences in the auditory sentences, and not when there were segmental differences. Third, (in correct trials) the saccade amplitude in native language trials was significantly greater than in non-native trials, indicating more intensely focused fixations in the latter. Taken together, these results suggest that mouth-looking was generally more evident when processing a non-native versus native language in all analyses, but fascinatingly, when measuring perceivers’ latency to fixate the mouth, this language effect was largest in trials where only prosodic information was useful for the task.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

LanguageSpeech PerceptionPhoneticsHumansFemaleMaleAdultYoung AdultFaceVisual PerceptionEye MovementsSpeechEye-Tracking Technology

Autor:innen und Institutionen

  • Department of Linguistics, Simon Fraser University, Burnaby BC, Canada.

ORCID-Profile: Xizi Deng, Elise McClay

Förderung

  • China Scholarship Council
  • Natural Sciences and Engineering Research Council of Canada
  • Social Sciences and Humanities Research Council of Canada

Zitieren

APA

Deng, X., McClay, E., Jastrzebski, E., Wang, Y., & Yeung, H. H. (2024). Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language. PLOS ONE. https://doi.org/10.1371/journal.pone.0304150

Vancouver

Deng X, McClay E, Jastrzebski E, Wang Y, Yeung HH. Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language. PLOS ONE. 2024. doi:10.1371/journal.pone.0304150

BibTeX
@article{Deng2024,
  author = {Deng, Xizi and McClay, Elise and Jastrzebski, Erin and Wang, Yue and Yeung, H Henny},
  title = {Visual scanning patterns of a talking face when evaluating phonetic information in a native and non-native language},
  journal = {PLOS ONE},
  year = {2024},
  doi = {10.1371/journal.pone.0304150}
}
Kostenlos starten →