EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Evaluating the Reliability of LLMs in OSINT Investigations: A Friend or Foe?

Errol Baloyi, Nokuthaba Siphambili, Ntomfuthi Ntshangase, Mpho Letshwenyo, Fhatuwani Makharamedzha, Rendani Mmbodi, Ndabezinhle Hlongwane

European Conference on Cyber Warfare and Security · 15.06.2026 · 25 (1) · 72-79

DOI: 10.34190/eccws.25.1.4818 ↗

Kurz gefasst

Fachartikel in der Zeitschrift European Conference on Cyber Warfare and Security, online erschienen am 15.06.2026. Verfasst von Errol Baloyi und 6 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This study investigates the use of Large Language Models (LLMs), including GPT, Claude 3, Gemini, Meta, DeepSeek, Qwen 2.5, Mistral Large, and Grok, in Open-Source Intelligence (OSINT) investigations, focusing on their capabilities, limitations, and practical implications. Using a controlled fictional organization, the CtrlZ Society, to simulate a plausible online footprint, the LLMs were provided with a synthetic dataset purportedly linked to the organization. The dataset included social media posts, a Reddit thread, a Pastebin document, a GitHub repository, a blog post, a Telegram broadcast, a WHOIS record, and a news article. Based on this dataset, the models were evaluated on accuracy, timeline reconstruction, account attribution, evidence traceability, susceptibility to hallucination, and handling of ambiguity and incomplete information. Results revealed substantial variation among models: Claude 3, GPT, and Qwen 2.5 demonstrated strong analytical performance and reliable synthesis of investigative outputs, while Gemini and DeepSeek exhibited weaker capabilities. Some models, including Meta were also prone to forced narrative construction when prompted adversarially, highlighting risks of misinterpretation or overreach. Despite these limitations, all LLMs provided valuable support for structuring and summarising complex data, demonstrating their potential as efficiency multipliers in OSINT workflows. Based on these findings, the study provides recommendations for practitioners, including rigorous human oversight, multi-model validation, adherence to verification protocols, and careful evaluation of outputs to mitigate risks and maximise the reliability of LLM-assisted investigations.

Zitieren

APA

Baloyi, E., Siphambili, N., Ntshangase, N., Letshwenyo, M., Makharamedzha, F., Mmbodi, R., & Hlongwane, N. (2026). Evaluating the Reliability of LLMs in OSINT Investigations: A Friend or Foe?. European Conference on Cyber Warfare and Security, 25(1), 72-79. https://doi.org/10.34190/eccws.25.1.4818

Vancouver

Baloyi E, Siphambili N, Ntshangase N, Letshwenyo M, Makharamedzha F, Mmbodi R, et al. Evaluating the Reliability of LLMs in OSINT Investigations: A Friend or Foe?. European Conference on Cyber Warfare and Security. 2026;25(1):72-79. doi:10.34190/eccws.25.1.4818

BibTeX
@article{Baloyi2026,
  author = {Baloyi, Errol and Siphambili, Nokuthaba and Ntshangase, Ntomfuthi and Letshwenyo, Mpho and Makharamedzha, Fhatuwani and Mmbodi, Rendani and Hlongwane, Ndabezinhle},
  title = {Evaluating the Reliability of LLMs in OSINT Investigations: A Friend or Foe?},
  journal = {European Conference on Cyber Warfare and Security},
  year = {2026},
  volume = {25},
  number = {1},
  pages = {72--79},
  doi = {10.34190/eccws.25.1.4818}
}
Kostenlos starten →