Intelligent Speech Interaction System with Real-Time Speech Modification and Recognition
Programmnaya Ingeneria · 11.11.2026 · 17 (11) · 613-625
DOI: 10.17587/prin.17.613-625 ↗Kurz gefasst
Fachartikel in der Zeitschrift Programmnaya Ingeneria, erschienen am 11.11.2026. Verfasst von Vladimir A. Nechaev und 1 weiteren Autor:innen.
Abstract (Originalsprache)
The paper presents an intelligent speech interaction system designed for real-time voice communication scenarios, primarily in call centers. When conventional systems are used, both conversational efficiency and the quality of downstream automatic processing deteriorate due to multilingual and mixed-language speech, foreign accents, background noise, and domain-specific terminology. The proposed approach integrates, within a unified architecture, a multilingual domain-oriented automatic speech recognition (ASR) system and a speech modification system. Parallel streams are employed — alongside the direct (unprocessed) audio signal, a modified stream is generated, which can be used both to improve intelligibility for the interlocutor and to increase recognition robustness. At the same time, both versions of the speech data are preserved for subsequent analysis and continuous service quality improvement. The study formalizes a correction-level indicator that reflects the depth of speech modification — from accent conversion to pronunciation correction and full speech-to-speech translation. It is shown that increasing the correction level requires a longer context window for the corresponding speech transformation and generation models and, consequently, leads to higher end-to-end real-time latency. This makes it possible to justify the choice of the speech modification strategy under the constraints of interactive dialogue, such as the user's proficiency in the intermediary language and the maximum permissible real-time delays. A practical threshold of approximately 30o ms is emphasized, beyond which latency becomes perceptible to users and may cause discomfort during communication. Experiments were conducted on real call center data comprising Russian speech and mixed Russian-English speech that includes foreign accents, background noise, and domain-specific terminology. The results show that combining a multilingual, domain-oriented ASR system with a speech modification system provides the best recognition quality for both Russian-only and mixed Russian-English data. For practical deployment, the proposed intelligent speech interaction system was implemented for GPU-based inference. The average recognition latency under concurrent load was 90-100 ms, while the total latency, including the accent conversion module, reached up to 350 ms. At the same time, the real-time delay between the client and the operator in the modified channel remains within 250 ms, confirming the practical applicability of the proposed approach.
Zitieren
APA
Nechaev, V. A., & Kosyakov, S. V. (2026). Intelligent Speech Interaction System with Real-Time Speech Modification and Recognition. Programmnaya Ingeneria, 17(11), 613-625. https://doi.org/10.17587/prin.17.613-625
Vancouver
Nechaev VA, Kosyakov SV. Intelligent Speech Interaction System with Real-Time Speech Modification and Recognition. Programmnaya Ingeneria. 2026;17(11):613-625. doi:10.17587/prin.17.613-625
BibTeX
@article{Nechaev2026,
author = {Nechaev, Vladimir A. and Kosyakov, S. V.},
title = {Intelligent Speech Interaction System with Real-Time Speech Modification and Recognition},
journal = {Programmnaya Ingeneria},
year = {2026},
volume = {17},
number = {11},
pages = {613--625},
doi = {10.17587/prin.17.613-625}
}
EUVIMED