EUVIMEDEUVIMED entdecken
ValidierungsstudieOpen Access · CC BY 4.0

Human‐supervised LLM triage of pig butchering complaints: A validation study of multi‐model identifier extraction

Sanghyeob Ko iD, Jumi Kim, Jin Young Jang, Gibum Kim iD

Journal of Forensic Sciences · 2026-11 · 71 (6) · 2747-2759

DOI: 10.1111/1556-4029.70411 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift Journal of Forensic Sciences, online erschienen am 22.07.2026. Verfasst von Sanghyeob Ko und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Forensic Science, Sungkyunkwan University, Seoul, Republic of Korea. Hauptthemen laut MEDLINE: Large Language Models, Fraud, Triage. Die Arbeit nennt 19 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Korean National Police Agency.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Digital forensic investigations increasingly process unstructured cryptocurrency‐fraud complaints at intake while preserving analyst oversight before evidentiary or downstream investigative use. This validation study evaluates whether human‐supervised multi‐model LLM extraction can recover triage‐relevant identifiers from California DFPI pig butchering complaints (2022–2024, N = 273). We introduce a four‐tier, information‐based forensic taxonomy that classifies cases by the presence of traceable identifiers—cryptocurrency wallet addresses, fraud‐related platform URLs, and reported loss amounts—rather than by narrative length. Three commercial LLMs (Claude Sonnet 4.5, GPT‐5.2, Gemini 3 Pro) were evaluated against adjudicated ground truth annotated by two independent raters (Cohen's κ = 0.969–0.988). The taxonomy classified 96.7% of cases (264/273) as actionable, with 9.2% containing wallet addresses that support direct blockchain tracing. LLM extraction achieved F1 = 1.000 for wallet addresses, F1 = 0.911–0.918 for URLs, and F1 = 0.865–0.904 for loss amounts; lexical regex and pre‐trained spaCy NER baselines achieved F1 ≤0.381 for loss extraction, where context‐dependent interpretation is required. The pipeline routed 82 of the 264 actionable cases (31.1%) to prioritized human review based on cross‐model URL or loss‐amount discrepancies, functioning as a quality‐control mechanism rather than an output‐aggregation ensemble. Pipeline‐stage extraction completed in 74 min at approximately $15 USD; this efficiency is reported separately from the human‐review queue that the framework mandates by design. Because commercial API processing involves data‐sovereignty and CJIS‐equivalent compliance considerations, we position the protocol as triage support within analyst‐mediated workflows. We identify local deployment and pre‐transmission anonymization as priorities for future operational validation.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsFraudTriageAnimalsHumansSwineForensic Sciences

Stichwörter der Autor:innen

blockchain forensicscryptocurrency frauddigital forensic triagehuman‐supervised triageidentifier extractionlarge language modelsmulti‐model validationpig butchering scam

Autor:innen und Institutionen

  • Department of Forensic Science, Sungkyunkwan University, Seoul, Republic of Korea.
  • Korean Police Investigation Academy, Asan, Republic of Korea.
  • Daejeon Provincial Police Agency, Daejeon, Republic of Korea.

ORCID-Profile: Sanghyeob Ko, Gibum Kim

Förderung

  • Korean National Police Agency

Zitieren

APA

Ko, S., Kim, J., Jang, J. Y., & Kim, G. (2026). Human‐supervised LLM triage of pig butchering complaints: A validation study of multi‐model identifier extraction. Journal of Forensic Sciences, 71(6), 2747-2759. https://doi.org/10.1111/1556-4029.70411

Vancouver

Ko S, Kim J, Jang JY, Kim G. Human‐supervised LLM triage of pig butchering complaints: A validation study of multi‐model identifier extraction. Journal of Forensic Sciences. 2026;71(6):2747-2759. doi:10.1111/1556-4029.70411

BibTeX
@article{Ko2026,
  author = {Ko, Sanghyeob and Kim, Jumi and Jang, Jin Young and Kim, Gibum},
  title = {Human‐supervised LLM triage of pig butchering complaints: A validation study of multi‐model identifier extraction},
  journal = {Journal of Forensic Sciences},
  year = {2026},
  volume = {71},
  number = {6},
  pages = {2747--2759},
  doi = {10.1111/1556-4029.70411}
}
Kostenlos starten →