EUVIMEDEUVIMED entdecken
Crossref

Fine-tuning strategies for large-scale face-conditioned text-to-speech

Umi Okamoto, Sei Ueno, Akinobu Lee

The Journal of the Acoustical Society of America · 01.10.2025 · 158 (4_Supplement) · A138-A138

DOI: 10.1121/10.0040379 ↗

Kurz gefasst

Fachartikel in der Zeitschrift The Journal of the Acoustical Society of America, erschienen am 01.10.2025. Verfasst von Umi Okamoto und 2 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Speech-conditioned text-to-speech (TTS) models, which generate speech from text and reference speech, have been successful due to large-scale datasets and parameter size. Similarly, scaling up face-conditioned TTS models, which generate speech from text and facial images, is expected to enhance both speech quality and speaker similarity. However, the effectiveness of such models is hindered by the scarcity and low quality of audio-visual data, which often contain noise in either the speech or the images. Previous work has leveraged a large-scale speech-conditioned TTS model as a pre-trained model for fine-tuning with an audio-visual dataset. This approach compensates for the limited quantity and quality of such data, while achieving speaker similarity comparable to that of speech-conditioned models. While fine-tuning the TTS model improves speaker similarity, it degrades intelligibility, which is likely influenced by noise in the audio-visual data or by the modality gap between the conditioning inputs used during training and inference. To address this issue, we investigate fine-tuning strategies for face-conditioned TTS by randomly mixing audio-only and audio-visual data during fine-tuning, and by evaluating whether to use the audio-only or the audio-visual dataset during each training stage. In addition, we conduct comparative experiments to evaluate the effectiveness of these strategies.

Zitieren

APA

Okamoto, U., Ueno, S., & Lee, A. (2025). Fine-tuning strategies for large-scale face-conditioned text-to-speech. The Journal of the Acoustical Society of America, 158(4_Supplement), A138-A138. https://doi.org/10.1121/10.0040379

Vancouver

Okamoto U, Ueno S, Lee A. Fine-tuning strategies for large-scale face-conditioned text-to-speech. The Journal of the Acoustical Society of America. 2025;158(4_Supplement):A138-A138. doi:10.1121/10.0040379

BibTeX
@article{Okamoto2025,
  author = {Okamoto, Umi and Ueno, Sei and Lee, Akinobu},
  title = {Fine-tuning strategies for large-scale face-conditioned text-to-speech},
  journal = {The Journal of the Acoustical Society of America},
  year = {2025},
  volume = {158},
  number = {4_Supplement},
  pages = {A138--A138},
  doi = {10.1121/10.0040379}
}
Kostenlos starten →