Automated Dataset Preparation and Management from Raw Data Using Context-Aware Chunking and LLMs
International Journal of Innovative Science and Research Technology · 20.04.2026 · 1191
DOI: 10.38124/ijisrt/26apr952 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Innovative Science and Research Technology, online erschienen am 20.04.2026. Verfasst von S. Saraswathi und 3 weiteren Autor:innen. Die Arbeit nennt 10 Literaturangaben.
Abstract (Originalsprache)
The increasing adoption of Large Language Models (LLMs) has created a critical need for high-quality, structured training data derived from real-world, unstructured sources such as documents and web content. However, existing research primarily focuses on model architectures or isolated data processing stages, lacking a unified solution for end-to-end dataset preparation. This project proposes a reusable, production-ready Dataset Preparation Pipeline that automates the ingestion, cleaning, chunking, task-specific data generation, and quality evaluation of raw data for multiple LLM applications including question answering, summarization, and classification. The system integrates automated quality assessment to filter low-quality or biased samples and exports curated datasets in standard formats for seamless downstream use. By adopting a data-centric approach, the proposed pipeline bridges the gap between raw real-world data and reliable LLM training datasets, enabling scalable, continuous, and efficient dataset generation for modern language model development.
Zitieren
APA
Saraswathi, S., M., V., S., V., & M., T. (2026). Automated Dataset Preparation and Management from Raw Data Using Context-Aware Chunking and LLMs. International Journal of Innovative Science and Research Technology, 1191. https://doi.org/10.38124/ijisrt/26apr952
Vancouver
Saraswathi S, M. V, S. V, M. T. Automated Dataset Preparation and Management from Raw Data Using Context-Aware Chunking and LLMs. International Journal of Innovative Science and Research Technology. 2026:1191. doi:10.38124/ijisrt/26apr952
BibTeX
@article{Saraswathi2026,
author = {Saraswathi, S. and M., Vignesh and S., Vijayalakshmi and M., Tholkapian},
title = {Automated Dataset Preparation and Management from Raw Data Using Context-Aware Chunking and LLMs},
journal = {International Journal of Innovative Science and Research Technology},
year = {2026},
pages = {1191},
doi = {10.38124/ijisrt/26apr952}
}
EUVIMED