Designing AI-Ready Healthcare Infrastructure: Scalable Feature Generation Using PySpark, Delta Lake, and Databricks
International Journal of Scientific Research in Computer Science, Engineering and Information Technology · 25.09.2025 · 11 (5) · 592-601
DOI: 10.32628/cseit2612422 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Scientific Research in Computer Science, Engineering and Information Technology, online erschienen am 25.09.2025. Verfasst von Naga RaviTeja Tadiboina und 1 weiteren Autor:innen. Die Arbeit nennt 33 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
The increase in generation of healthcare data due to Electronic Health Records (EHRs), wearable devices, medical imaging systems, and Internet of Medical Things (IoMT) devices has been problematic in regard to data management, scalability, and real-time analysis. The emergence of new data sources, such as Electronic Health Records (EHRs), wearable devices, medical imaging systems and Internet of Medical Things (IoMT) devices has brought about significant challenges in data management, scalability, and real-time analysis in the healthcare industry. Challenges in the healthcare sector involve effective management of huge volumes of varied data, thus impacting the potential of AI applications in the sector. Efficient data management, especially in managing huge volumes of varied data, is a key challenge in the healthcare industry, thus the limitations of AI applications in the same. In this case, the researchers have proposed a healthcare infrastructure that is ready for AI applications through PySpark, Delta Lake, and Databricks that will enable scalability, distributed computing, and secure data management in healthcare. The architecture allows for batch and streaming data processing, maintains data consistency, is fault-tolerant, and complies with healthcare security requirements. The experiments were conducted using typical values of the datasets to evaluate the proposed framework with Traditional ETL, Batch Processing, AWS Data Pipeline and Apache Spark on performance metrics such as Processing Time, Throughput, Feature Generation accuracy, Latency and Resources Utilization. The results show that the proposed framework is scalable, can process the data quickly, has a low latency, high throughput, and better feature quality, which are suitable for healthcare AI-based analytics and intelligent clinical decision-support systems. The proposed infrastructure would provide a solid foundation for developing future secure, scalable, and efficient healthcare data platforms.
Zitieren
APA
Tadiboina, N. R., & Tadiboina, S. G. (2025). Designing AI-Ready Healthcare Infrastructure: Scalable Feature Generation Using PySpark, Delta Lake, and Databricks. International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 11(5), 592-601. https://doi.org/10.32628/cseit2612422
Vancouver
Tadiboina NR, Tadiboina SG. Designing AI-Ready Healthcare Infrastructure: Scalable Feature Generation Using PySpark, Delta Lake, and Databricks. International Journal of Scientific Research in Computer Science, Engineering and Information Technology. 2025;11(5):592-601. doi:10.32628/cseit2612422
BibTeX
@article{Tadiboina2025,
author = {Tadiboina, Naga RaviTeja and Tadiboina, Sai Gowtham},
title = {Designing AI-Ready Healthcare Infrastructure: Scalable Feature Generation Using PySpark, Delta Lake, and Databricks},
journal = {International Journal of Scientific Research in Computer Science, Engineering and Information Technology},
year = {2025},
volume = {11},
number = {5},
pages = {592--601},
doi = {10.32628/cseit2612422}
}
EUVIMED