Evaluating the Reliability of LLMs in OSINT Investigations: A Friend or Foe?
European Conference on Cyber Warfare and Security · 15.06.2026 · 25 (1) · 72-79
DOI: 10.34190/eccws.25.1.4818 ↗Kurz gefasst
Fachartikel in der Zeitschrift European Conference on Cyber Warfare and Security, online erschienen am 15.06.2026. Verfasst von Errol Baloyi und 6 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
This study investigates the use of Large Language Models (LLMs), including GPT, Claude 3, Gemini, Meta, DeepSeek, Qwen 2.5, Mistral Large, and Grok, in Open-Source Intelligence (OSINT) investigations, focusing on their capabilities, limitations, and practical implications. Using a controlled fictional organization, the CtrlZ Society, to simulate a plausible online footprint, the LLMs were provided with a synthetic dataset purportedly linked to the organization. The dataset included social media posts, a Reddit thread, a Pastebin document, a GitHub repository, a blog post, a Telegram broadcast, a WHOIS record, and a news article. Based on this dataset, the models were evaluated on accuracy, timeline reconstruction, account attribution, evidence traceability, susceptibility to hallucination, and handling of ambiguity and incomplete information. Results revealed substantial variation among models: Claude 3, GPT, and Qwen 2.5 demonstrated strong analytical performance and reliable synthesis of investigative outputs, while Gemini and DeepSeek exhibited weaker capabilities. Some models, including Meta were also prone to forced narrative construction when prompted adversarially, highlighting risks of misinterpretation or overreach. Despite these limitations, all LLMs provided valuable support for structuring and summarising complex data, demonstrating their potential as efficiency multipliers in OSINT workflows. Based on these findings, the study provides recommendations for practitioners, including rigorous human oversight, multi-model validation, adherence to verification protocols, and careful evaluation of outputs to mitigate risks and maximise the reliability of LLM-assisted investigations.
Zitieren
APA
Baloyi, E., Siphambili, N., Ntshangase, N., Letshwenyo, M., Makharamedzha, F., Mmbodi, R., & Hlongwane, N. (2026). Evaluating the Reliability of LLMs in OSINT Investigations: A Friend or Foe?. European Conference on Cyber Warfare and Security, 25(1), 72-79. https://doi.org/10.34190/eccws.25.1.4818
Vancouver
Baloyi E, Siphambili N, Ntshangase N, Letshwenyo M, Makharamedzha F, Mmbodi R, et al. Evaluating the Reliability of LLMs in OSINT Investigations: A Friend or Foe?. European Conference on Cyber Warfare and Security. 2026;25(1):72-79. doi:10.34190/eccws.25.1.4818
BibTeX
@article{Baloyi2026,
author = {Baloyi, Errol and Siphambili, Nokuthaba and Ntshangase, Ntomfuthi and Letshwenyo, Mpho and Makharamedzha, Fhatuwani and Mmbodi, Rendani and Hlongwane, Ndabezinhle},
title = {Evaluating the Reliability of LLMs in OSINT Investigations: A Friend or Foe?},
journal = {European Conference on Cyber Warfare and Security},
year = {2026},
volume = {25},
number = {1},
pages = {72--79},
doi = {10.34190/eccws.25.1.4818}
}
EUVIMED