EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Efficient data preprocessing for extractive question answering models

Sivakumar S., Meenakshi S. P.

Boletim da Sociedade Paranaense de Matemática · 01.11.2025 · 43 (4)

DOI: 10.5269/bspm.78619 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Boletim da Sociedade Paranaense de Matemática, online erschienen am 01.11.2025. Verfasst von Sivakumar S. und 1 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 17 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Building a domain-specific dataset for extractive question answering requires addressing challenges posed by complex and unstructured textual sources. Official documents from the Indian Lok Sabha are typically lengthy, semi-structured, and often contain noise such as interruptions, repetitions, and inconsistente formatting. To overcome these challenges, we designed a systematic pipeline for text extraction and preprocessing, resulting in a clean and well-structured dataset suitable for training high-performance QA models. The pipeline includes handling diverse PDF formats, removing unwanted symbols and metadata, identifying ministries, and extracting precise question–answer pairs using regular expressions. Additional steps such as context segmentation, token alignment, and answer start indexing were applied to ensure compatibility with modern QA architectures. A BERT-based model was fine-tuned on the processed dataset, and experimental results confirmed that effective preprocessing significantly enhances performance. Our ablation study showed consistent improvements across different settings, while benchmark comparisons with SQuAD v1.1 and Natural Questions demonstrated that the Lok Sabha dataset performs competitively, achieving Exact Match (EM) and F1 scores of 74 % and 81 %, respectively. Furthermore, bias analysis based on answer lengths validated the robustness of the dataset. Overall, this study demonstrates that parliamentary data, when carefully processed, can serve as a reliable resource for developing domain-adapted QA systems in the fields of public policy and governance.graphs.

Zitieren

APA

S., S., & P., M. S. (2025). Efficient data preprocessing for extractive question answering models. Boletim da Sociedade Paranaense de Matemática, 43(4). https://doi.org/10.5269/bspm.78619

Vancouver

S. S, P. MS. Efficient data preprocessing for extractive question answering models. Boletim da Sociedade Paranaense de Matemática. 2025;43(4). doi:10.5269/bspm.78619

BibTeX
@article{S2025,
  author = {S., Sivakumar and P., Meenakshi S.},
  title = {Efficient data preprocessing for extractive question answering models},
  journal = {Boletim da Sociedade Paranaense de Matemática},
  year = {2025},
  volume = {43},
  number = {4},
  doi = {10.5269/bspm.78619}
}
Kostenlos starten →