EUVIMEDEUVIMED entdecken
Crossref

EVALUATING SIMILARITY MEASURES FOR MALAY NOISY TEXT NORMALIZATION: PERFORMANCE AND THRESHOLD ANALYSIS

Azilawati Azizan, Nurkhairizan Khairudin, Muhammad Fitri Shazwan Fadzely, Nursyahidah Alias, Norshuhani Zamin, Norlina Mohd Sabri

Malaysian Journal of Computer Science · 11.08.2025 · 38 (Special)

DOI: 10.22452/mjcs.vol38spc.1 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Malaysian Journal of Computer Science, online erschienen am 11.08.2025. Verfasst von Azilawati Azizan und 5 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Keywords: Noisy Text; Text Normalization; Malay Noisy Text; Similarity Measure; Threshold. Abstract Noisy text normalization is a critical preprocessing step in natural language processing (NLP), particularly for user-generated content (UGC) that contains a lot of slang, abbreviations, and typographical errors. This extended study investigates the performance of multiple similarity measures in normalizing Malay noisy text, addressing gaps in prior study that predominantly relied on rule-based approaches and single similarity measures. By systematically evaluating token-based, edit-based, and sequence-based similarity measures across various thresholds, this study provides a comprehensive analysis of their effectiveness and computational efficiency. The methodology comprises a two-phase experiment: an initial phase to identify optimal thresholds using a small dataset and a second phase that generalizes findings on a larger dataset. Key findings reveal that edit-based measures, such as Levenshtein Distance and Damerau-Levenshtein, consistently outperform other measures at lower thresholds, achieving normalization success rates exceeding 83%. Ratcliff/Obershelp emerged as the most effective sequence-based measure, while token-based measures like Jaccard and Cosine demonstrated limited performance. The study also highlights the critical role of threshold in balancing normalization accuracy and flexibility. Additionally, computational time analysis underscores the trade-offs between accuracy and efficiency across similarity categories. These findings pave the way for more robust and adaptable text normalization strategies, particularly for Malay language studies.

Zitieren

APA

Azizan, A., Khairudin, N., Fadzely, M. F. S., Alias, N., Zamin, N., & Sabri, N. M. (2025). EVALUATING SIMILARITY MEASURES FOR MALAY NOISY TEXT NORMALIZATION: PERFORMANCE AND THRESHOLD ANALYSIS. Malaysian Journal of Computer Science, 38(Special). https://doi.org/10.22452/mjcs.vol38spc.1

Vancouver

Azizan A, Khairudin N, Fadzely MFS, Alias N, Zamin N, Sabri NM. EVALUATING SIMILARITY MEASURES FOR MALAY NOISY TEXT NORMALIZATION: PERFORMANCE AND THRESHOLD ANALYSIS. Malaysian Journal of Computer Science. 2025;38(Special). doi:10.22452/mjcs.vol38spc.1

BibTeX
@article{Azizan2025,
  author = {Azizan, Azilawati and Khairudin, Nurkhairizan and Fadzely, Muhammad Fitri Shazwan and Alias, Nursyahidah and Zamin, Norshuhani and Sabri, Norlina Mohd},
  title = {EVALUATING SIMILARITY MEASURES FOR MALAY NOISY TEXT NORMALIZATION: PERFORMANCE AND THRESHOLD ANALYSIS},
  journal = {Malaysian Journal of Computer Science},
  year = {2025},
  volume = {38},
  number = {Special},
  doi = {10.22452/mjcs.vol38spc.1}
}
Kostenlos starten →