EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Integrating Vision and Olfaction via Multi-Modal LLM for Robotic Odor Source Localization

Sunzid Hassan iD, Lingxiao Wang iD, Khan Raqib Mahmud

Sensors · 2024

DOI: 10.3390/s24247875 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, 2024. Verfasst von Sunzid Hassan und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science, Louisiana Tech University, 201 Mayfield Ave, Ruston, LA 71272, USA. Hauptthemen laut MEDLINE: Odorants, Robotics, Algorithms, Smell. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 74 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Louisiana Board of Regents.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Odor source localization (OSL) technology allows autonomous agents like mobile robots to localize a target odor source in an unknown environment. This is achieved by an OSL navigation algorithm that processes an agent’s sensor readings to calculate action commands to guide the robot to locate the odor source. Compared to traditional ‘olfaction-only’ OSL algorithms, our proposed OSL algorithm integrates vision and olfaction sensor modalities to localize odor sources even if olfaction sensing is disrupted by non-unidirectional airflow or vision sensing is impaired by environmental complexities. The algorithm leverages the zero-shot multi-modal reasoning capabilities of large language models (LLMs), negating the requirement of manual knowledge encoding or custom-trained supervised learning models. A key feature of the proposed algorithm is the ‘High-level Reasoning’ module, which encodes the olfaction and vision sensor data into a multi-modal prompt and instructs the LLM to employ a hierarchical reasoning process to select an appropriate high-level navigation behavior. Subsequently, the ‘Low-level Action’ module translates the selected high-level navigation behavior into low-level action commands that can be executed by the mobile robot. To validate our algorithm, we implemented it on a mobile robot in a real-world environment with non-unidirectional airflow environments and obstacles to mimic a complex, practical search environment. We compared the performance of our proposed algorithm to single-sensory-modality-based ‘olfaction-only’ and ‘vision-only’ navigation algorithms, and a supervised learning-based ‘vision and olfaction fusion’ (Fusion) navigation algorithm. The experimental results show that the proposed LLM-based algorithm outperformed the other algorithms in terms of success rates and average search times in both unidirectional and non-unidirectional airflow environments.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

OdorantsRoboticsAlgorithmsSmellHumans

Stichwörter der Autor:innen

large language models (LLMs)multi-modal roboticsodor source localizationrobot operating system (ROS)

Autor:innen und Institutionen

  • Department of Computer Science, Louisiana Tech University, 201 Mayfield Ave, Ruston, LA 71272, USA.
  • Department of Electrical Engineering, Louisiana Tech University, 201 Mayfield Ave, Ruston, LA 71272, USA.

ORCID-Profile: Sunzid Hassan, Lingxiao Wang

Förderung

  • Louisiana Board of Regents

Zitieren

APA

Hassan, S., Wang, L., & Mahmud, K. R. (2024). Integrating Vision and Olfaction via Multi-Modal LLM for Robotic Odor Source Localization. Sensors. https://doi.org/10.3390/s24247875

Vancouver

Hassan S, Wang L, Mahmud KR. Integrating Vision and Olfaction via Multi-Modal LLM for Robotic Odor Source Localization. Sensors. 2024. doi:10.3390/s24247875

BibTeX
@article{Hassan2024,
  author = {Hassan, Sunzid and Wang, Lingxiao and Mahmud, Khan Raqib},
  title = {Integrating Vision and Olfaction via Multi-Modal LLM for Robotic Odor Source Localization},
  journal = {Sensors},
  year = {2024},
  doi = {10.3390/s24247875}
}
Kostenlos starten →