EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Extended Reality Telemedicine Collaboration System Using Patient Avatar Based on 3D Body Pose Estimation

Matko Šarić, Mladen Russo iD, Luka Kraljević iD, Davor Meter

Sensors · 20.12.2023 · 24 (1) · 27

DOI: 10.3390/s24010027 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 20.12.2023. Verfasst von Matko Šarić und 3 weiteren Autor:innen. Beteiligte Einrichtung: Faculty of Electrical Engineering, Mechanical Engineering and Naval Architecture, University of Split, Ruđera Boškovića 32, 21000 Split, Croatia. Hauptthemen laut MEDLINE: Running, Telemedicine. Die Arbeit wurde bisher 21-mal zitiert (Crossref) und nennt 21 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Croatian Government and the European Union through the European Regional Development Fund - the Competitiveness and Cohesion Operational Programme, Virtual Telemedicine Assistance.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent advances in extended reality (XR) technology have opened the possibility of significantly improving telemedicine systems. This is primarily achieved by transferring 3D information about patient state, which is utilized to create more immersive experiences on VR/AR headsets. In this paper, we propose an XR-based telemedicine collaboration system in which the patient is represented as a 3D avatar in an XR space shared by local and remote clinicians. The proposed system consists of an AR client application running on Microsoft HoloLens 2 used by a local clinician, a VR client application running on the HTC vive Pro used by a remote clinician, and a backend part running on the server. The patient is captured by a camera on the AR side, and the 3D body pose estimation is performed on frames from this camera stream to form a 3D patient avatar. Additionally, the AR and VR sides can interact with the patient avatar via virtual hands, and annotations can be performed on a 3D model. The main contribution of our work is the use of 3D body pose estimation for the creation of a 3D patient avatar. In this way, 3D body reconstruction using depth cameras is avoided, which reduces system complexity and hardware and network resources. Another contribution is the novel architecture of the proposed system, where audio and video streaming are realized using WebRTC protocol. The performance evaluation showed that the proposed system ensures high frame rates for both AR and VR client applications, while the processing latency remains at an acceptable level.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

RunningTelemedicineHumansAvatarHandPatients

Stichwörter der Autor:innen

3D body pose estimationavatarextended realitytelemedicine system

Autor:innen und Institutionen

  • Faculty of Electrical Engineering, Mechanical Engineering and Naval Architecture, University of Split, Ruđera Boškovića 32, 21000 Split, Croatia.

ORCID-Profile: Mladen Russo, Luka Kraljević

Förderung

  • Croatian Government and the European Union through the European Regional Development Fund - the Competitiveness and Cohesion Operational Programme
  • Virtual Telemedicine Assistance

Zitieren

APA

Šarić, M., Russo, M., Kraljević, L., & Meter, D. (2023). Extended Reality Telemedicine Collaboration System Using Patient Avatar Based on 3D Body Pose Estimation. Sensors, 24(1), 27. https://doi.org/10.3390/s24010027

Vancouver

Šarić M, Russo M, Kraljević L, Meter D. Extended Reality Telemedicine Collaboration System Using Patient Avatar Based on 3D Body Pose Estimation. Sensors. 2023;24(1):27. doi:10.3390/s24010027

BibTeX
@article{ari2023,
  author = {Šarić, Matko and Russo, Mladen and Kraljević, Luka and Meter, Davor},
  title = {Extended Reality Telemedicine Collaboration System Using Patient Avatar Based on 3D Body Pose Estimation},
  journal = {Sensors},
  year = {2023},
  volume = {24},
  number = {1},
  pages = {27},
  doi = {10.3390/s24010027}
}
Kostenlos starten →