BanglaMix: a dataset of token-level annotated Bengali-English code-mixed social media comments with Bengali transliteration
Data in Brief · 2026-08 · 69 · 113317
DOI: 10.1016/j.dib.2026.113317 ↗Kurz gefasst
Fachartikel in der Zeitschrift Data in Brief, 2026-08. Verfasst von Sabbir Mahmud Afridi und 2 weiteren Autor:innen. Beteiligte Einrichtung: University of Dhaka, Dhaka 1000, Bangladesh. Die Arbeit nennt 14 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Government of the People's Republic of Bangladesh ICT Division.
Abstract (Originalsprache)
This article describes BanglaMix, a dataset of 12,576 Bengali-English code-mixed social media comments collected from a publicly available video-sharing platform (YouTube) using its public Data API. Each record pairs the original (raw) comment text with a lightly cleaned version, a Bengali-script (Unicode) transliteration of that text, and a word-level, token-by-token language annotation using five labels (BAN, ENG, ROM_BAN, OTHER, UNK). Each comment is additionally characterised by three comment-level attributes: script type (latin, bengali, mixed, or other), code-mix type (an eight-category label describing which languages and scripts co-occur in the comment, e.g., monolingual_BAN, ENG+ROM_BAN, BAN+ENG), and a boolean flag indicating whether the comment is likely to be code-mixed. All annotations, including the Bengali transliterations and the token-level language labels, were produced by a single primary annotator (the author); an inter-annotator reliability study in which a second bilingual annotator independently re-labelled a random 500-comment sub-sample found substantial agreement at both the token level (Cohen's kappa = 0.74) and comment level (Cohen's kappa = 0.67). Comments were retained as anonymous text records identified only by an alphanumeric comment identifier; no usernames, channel names, or other personal identifiers were stored, and records that had retained a public @-mention of another commenter were removed before release. Across the dataset, 57.3% of comments are written primarily in Latin script, 28.4% in Bengali script, and 13.6% in mixed script; 36.8% of comments are flagged as likely code-mixed. At the token level, of 120,866 labelled tokens, 44.7% are tagged as Bengali (BAN), 23.5% as English (ENG), and 22.1% as Romanized Bengali (ROM_BAN). Comments are short on average (median 6 words; mean 9.61 words per comment) and exhibit no duplicate source texts. The dataset is intended for reuse in word-level language identification, Bengali back-transliteration, and other Bengali-English code-mixed natural language processing tasks, for which large, token-annotated, naturalistic social media resources remain scarce.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- University of Dhaka, Dhaka 1000, Bangladesh.
Förderung
- Government of the People's Republic of Bangladesh ICT Division
Zitieren
APA
Afridi, S. M., Partha, A. C., & Khan, M. M. (2026). BanglaMix: a dataset of token-level annotated Bengali-English code-mixed social media comments with Bengali transliteration. Data in Brief, 69, 113317. https://doi.org/10.1016/j.dib.2026.113317
Vancouver
Afridi SM, Partha AC, Khan MM. BanglaMix: a dataset of token-level annotated Bengali-English code-mixed social media comments with Bengali transliteration. Data in Brief. 2026;69:113317. doi:10.1016/j.dib.2026.113317
BibTeX
@article{Afridi2026,
author = {Afridi, Sabbir Mahmud and Partha, Abir Chakraborty and Khan, Md Mosaddek},
title = {BanglaMix: a dataset of token-level annotated Bengali-English code-mixed social media comments with Bengali transliteration},
journal = {Data in Brief},
year = {2026},
volume = {69},
pages = {113317},
doi = {10.1016/j.dib.2026.113317}
}
EUVIMED