EUVIMEDEUVIMED entdecken
Crossref

Automated Dataset Preparation and Management from Raw Data Using Context-Aware Chunking and LLMs

S. Saraswathi, Vignesh M., Vijayalakshmi S., Tholkapian M.

International Journal of Innovative Science and Research Technology · 20.04.2026 · 1191

DOI: 10.38124/ijisrt/26apr952 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Innovative Science and Research Technology, online erschienen am 20.04.2026. Verfasst von S. Saraswathi und 3 weiteren Autor:innen. Die Arbeit nennt 10 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The increasing adoption of Large Language Models (LLMs) has created a critical need for high-quality, structured training data derived from real-world, unstructured sources such as documents and web content. However, existing research primarily focuses on model architectures or isolated data processing stages, lacking a unified solution for end-to-end dataset preparation. This project proposes a reusable, production-ready Dataset Preparation Pipeline that automates the ingestion, cleaning, chunking, task-specific data generation, and quality evaluation of raw data for multiple LLM applications including question answering, summarization, and classification. The system integrates automated quality assessment to filter low-quality or biased samples and exports curated datasets in standard formats for seamless downstream use. By adopting a data-centric approach, the proposed pipeline bridges the gap between raw real-world data and reliable LLM training datasets, enabling scalable, continuous, and efficient dataset generation for modern language model development.

Zitieren

APA

Saraswathi, S., M., V., S., V., & M., T. (2026). Automated Dataset Preparation and Management from Raw Data Using Context-Aware Chunking and LLMs. International Journal of Innovative Science and Research Technology, 1191. https://doi.org/10.38124/ijisrt/26apr952

Vancouver

Saraswathi S, M. V, S. V, M. T. Automated Dataset Preparation and Management from Raw Data Using Context-Aware Chunking and LLMs. International Journal of Innovative Science and Research Technology. 2026:1191. doi:10.38124/ijisrt/26apr952

BibTeX
@article{Saraswathi2026,
  author = {Saraswathi, S. and M., Vignesh and S., Vijayalakshmi and M., Tholkapian},
  title = {Automated Dataset Preparation and Management from Raw Data Using Context-Aware Chunking and LLMs},
  journal = {International Journal of Innovative Science and Research Technology},
  year = {2026},
  pages = {1191},
  doi = {10.38124/ijisrt/26apr952}
}
Kostenlos starten →