EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A Unified Deep Learning Framework for Instance Segmentation Across Diverse Cytological Stains

Luís Otávio Santos, Rodrigo de Paula E Silva Ribeiro, Bibiana Quatrin Tiellet, Laura Maria Effting Silva, Fabiana Botelho de Miranda Onofre, Alexandre Sherlley Casimiro Onofre, Aldo von Wangenheim

Cytopathology · 2026-09 · 37 (5) · 493-502

DOI: 10.1111/cyt.70087 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Cytopathology, online erschienen am 11.05.2026. Verfasst von Luís Otávio Santos und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Sciences, Federal University of Santa Catarina, Florianopolis, Brazil. Hauptthemen laut MEDLINE: Deep Learning, Staining and Labeling, Image Processing, Computer-Assisted. Die Arbeit nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Coordenação de Aperfeiçoamento de Pessoal de Nível Superior.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Assess whether a single instance‐segmentation model can operate robustly across multiple cytological stains, avoiding stain‐specific pipelines without sacrificing accuracy.

Methods

We consolidated three expert‐annotated datasets (Papanicolaou, Feulgen, AgNOR), standardised their formats and trained Mask R‐CNN, Mask2Former, YOLO11n and YOLO12n under two regimes: per‐stain and unified multi‐stain. We reported AP 50 and AP 75 on validation/test to decouple detection tolerance from boundary precision, using identical training and evaluation protocols across models.

Results

Feulgen specialist models yielded the highest scores in our benchmark, with YOLO11n reaching AP 75 0.566 and AP 50 0.643. Mask2Former led in geometric precision (AP 75 ) in complex scenarios (AgNOR and Papanicolaou), achieving the best performance on the combined dataset. Crucially, unified training matched or exceeded stain‐specific models, maintaining detection (AP 50 ) while improving boundary definition (AP 75 ).

Conclusions

A single multi‐stain model is feasible for cytology instance segmentation, preserving detection and improving boundary accuracy in diverse stains. Transformer‐based Mask2Former showed the best AP 75 on the combined dataset, supporting a unified multi‐stain, scalable path to AI‐assisted cytopathology and practical screening workflows. Inside This Month's Cytopathology Can a single computer vision model handle the diversity of cytological stains found in clinical routine? This study validates a unified deep learning approach that effectively segments cells across Papanicolaou, Feulgen and AgNOR protocols. Results suggest that laboratories can deploy simplified, stain‐invariant screening tools without compromising diagnostic precision, overcoming a major barrier in digital pathology interoperability.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Deep LearningStaining and LabelingImage Processing, Computer-AssistedHumansPapanicolaou TestDetection AlgorithmsConvolutional Neural NetworksFemale

Stichwörter der Autor:innen

computer‐assistedcytopathologydeep learningimage processinginstance segmentationwhole slide imaging

Autor:innen und Institutionen

  • Department of Computer Sciences, Federal University of Santa Catarina, Florianopolis, Brazil.
  • Department of Clinical Analysis, Federal University of Santa Catarina, Florianopolis, Brazil.
  • National Institute for Digital Convergence (INCoD), Federal University of Santa Catarina, Florianopolis, Brazil.

Förderung

  • Coordenação de Aperfeiçoamento de Pessoal de Nível Superior

Zitieren

APA

Santos, L. O., Ribeiro, R. P. E. S., Tiellet, B. Q., Silva, L. M. E., de Miranda Onofre, F. B., Onofre, A. S. C., & von Wangenheim, A. (2026). A Unified Deep Learning Framework for Instance Segmentation Across Diverse Cytological Stains. Cytopathology, 37(5), 493-502. https://doi.org/10.1111/cyt.70087

Vancouver

Santos LO, Ribeiro RPES, Tiellet BQ, Silva LME, de Miranda Onofre FB, Onofre ASC, et al. A Unified Deep Learning Framework for Instance Segmentation Across Diverse Cytological Stains. Cytopathology. 2026;37(5):493-502. doi:10.1111/cyt.70087

BibTeX
@article{Santos2026,
  author = {Santos, Luís Otávio and Ribeiro, Rodrigo de Paula E Silva and Tiellet, Bibiana Quatrin and Silva, Laura Maria Effting and de Miranda Onofre, Fabiana Botelho and Onofre, Alexandre Sherlley Casimiro and von Wangenheim, Aldo},
  title = {A Unified Deep Learning Framework for Instance Segmentation Across Diverse Cytological Stains},
  journal = {Cytopathology},
  year = {2026},
  volume = {37},
  number = {5},
  pages = {493--502},
  doi = {10.1111/cyt.70087}
}
Kostenlos starten →