EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Accurate Segmentation of Overlapping Cervical Cells Using an Optimized Deep Learning Framework for Cytology Screening

Amal A Alzu'bi, Mohammad Khatatbeh, Wan Azani Mustafa iD, Norhayati Mohd Zain iD, Hiam Alquran iD, Alia Al-Mohtaseb, Khaled Z Alawneh iD, Mohammad Fawaeer iD, Bara'a Fawaeer iD, Shatha Salameh, Ahmad Alhussain

Diagnostics · 17.07.2026 · 16 (14) · 2240

DOI: 10.3390/diagnostics16142240 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Diagnostics, online erschienen am 17.07.2026. Verfasst von Amal A Alzu'bi und 10 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Information Systems, Jordan University of Science and Technology, P.O. Box 3030, Irbid 22110, Jordan. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: KPJ Healthcare University, Jordan University of Science and Technology, King Abdullah University Hospital.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Cervical cancer remains one of the leading causes of cancer-related morbidity among women worldwide. The Papanicolaou (Pap) smear is widely used for early detection; however, its manual interpretation is time-consuming, requires substantial expertise, and is often affected by inter-observer variability, particularly in cases with dense and overlapping cells.

Methods

This study developed an optimized deep learning framework for cervical cell instance segmentation, specifically targeting the separation of overlapping cells in Pap smear images. The proposed framework was based on Mask R-CNN with a ResNet-50 backbone and Feature Pyramid Network. A public development dataset of 460 cervical smear images was used for model development and internal evaluation, while an independent 210-image dataset collected from King Abdullah University Hospital was reserved as an external held-out clinical assessment set. To improve small-cell detection and mask refinement, the Region Proposal Network was adapted using biologically informed anchor scales (16, 32, 64, 128, 256), followed by soft calibration-based post-processing.

Results

The proposed framework achieved an AP50 of 89.90% and a Mask IoU of 90.44%. Small-cell performance, measured using APs under the COCO small-object convention, reached 22.80%, while APm and APl reached 68.45% and 81.42%, respectively. Clinical cell-count evaluation on the independent KAUH held-out set was performed using the mean count of five physicians as the reference standard and achieved an overall clinical detection accuracy of 93.00%.

Conclusions

The optimized Mask R-CNN framework improved the detection and separation of overlapping cervical cells in Pap smear images and may serve as a supportive tool for cytopathology workflows. The results suggest that biologically informed anchor optimization and soft calibration can improve cell-level instance segmentation, particularly for small and overlapping cells. Further validation on larger multi-center datasets remains necessary before routine clinical deployment.

Schlagwörter

Stichwörter der Autor:innen

cervical cancercomputer-aided screeninginstance segmentationmask R-CNNmedical image analysisoverlapping cellspap smearsmall-cell detection

Autor:innen und Institutionen

  • Department of Computer Information Systems, Jordan University of Science and Technology, P.O. Box 3030, Irbid 22110, Jordan.
  • Advanced Computing (AdvCOMP), Centre of Excellence, Universiti Malaysia Perlis (UniMAP), Pauh Putra Campus, Arau 02600, Perlis, Malaysia.
  • Medical Imaging Department, School of Integrative Medicine & Life Sciences, KPJ Healthcare University, Nilai 71800, Negeri Sembilan, Malaysia.
  • Department of Biomedical Systems and Informatics Engineering, Yarmouk University, P.O. Box 566, Irbid 21163, Jordan.
  • Department of Pathology, Jordan University of Science and Technology, P.O. Box 3030, Irbid 22110, Jordan.
  • Department of Radiology and Nuclear Medicine, Jordan University of Science and Technology, P.O. Box 3030, Irbid 22110, Jordan.

ORCID-Profile: Wan Azani Mustafa, Norhayati Mohd Zain, Hiam Alquran, Khaled Z Alawneh, Mohammad Fawaeer, Bara'a Fawaeer

Förderung

  • KPJ Healthcare University
  • Jordan University of Science and Technology
  • King Abdullah University Hospital

Zitieren

APA

Alzu'bi, A. A., Khatatbeh, M., Mustafa, W. A., Zain, N. M., Alquran, H., Al-Mohtaseb, A., Alawneh, K. Z., Fawaeer, M., Fawaeer, B., Salameh, S., & Alhussain, A. (2026). Accurate Segmentation of Overlapping Cervical Cells Using an Optimized Deep Learning Framework for Cytology Screening. Diagnostics, 16(14), 2240. https://doi.org/10.3390/diagnostics16142240

Vancouver

Alzu'bi AA, Khatatbeh M, Mustafa WA, Zain NM, Alquran H, Al-Mohtaseb A, et al. Accurate Segmentation of Overlapping Cervical Cells Using an Optimized Deep Learning Framework for Cytology Screening. Diagnostics. 2026;16(14):2240. doi:10.3390/diagnostics16142240

BibTeX
@article{Alzubi2026,
  author = {Alzu'bi, Amal A and Khatatbeh, Mohammad and Mustafa, Wan Azani and Zain, Norhayati Mohd and Alquran, Hiam and Al-Mohtaseb, Alia and Alawneh, Khaled Z and Fawaeer, Mohammad and Fawaeer, Bara'a and Salameh, Shatha and Alhussain, Ahmad},
  title = {Accurate Segmentation of Overlapping Cervical Cells Using an Optimized Deep Learning Framework for Cytology Screening},
  journal = {Diagnostics},
  year = {2026},
  volume = {16},
  number = {14},
  pages = {2240},
  doi = {10.3390/diagnostics16142240}
}
Kostenlos starten →