A Unified Deep Learning Framework for Instance Segmentation Across Diverse Cytological Stains
Cytopathology · 2026-09 · 37 (5) · 493-502
DOI: 10.1111/cyt.70087 ↗Kurz gefasst
Fachartikel in der Zeitschrift Cytopathology, online erschienen am 11.05.2026. Verfasst von Luís Otávio Santos und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Sciences, Federal University of Santa Catarina, Florianopolis, Brazil. Hauptthemen laut MEDLINE: Deep Learning, Staining and Labeling, Image Processing, Computer-Assisted. Die Arbeit nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Coordenação de Aperfeiçoamento de Pessoal de Nível Superior.
Abstract (Originalsprache)
Objective
Assess whether a single instance‐segmentation model can operate robustly across multiple cytological stains, avoiding stain‐specific pipelines without sacrificing accuracy.
Methods
We consolidated three expert‐annotated datasets (Papanicolaou, Feulgen, AgNOR), standardised their formats and trained Mask R‐CNN, Mask2Former, YOLO11n and YOLO12n under two regimes: per‐stain and unified multi‐stain. We reported AP 50 and AP 75 on validation/test to decouple detection tolerance from boundary precision, using identical training and evaluation protocols across models.
Results
Feulgen specialist models yielded the highest scores in our benchmark, with YOLO11n reaching AP 75 0.566 and AP 50 0.643. Mask2Former led in geometric precision (AP 75 ) in complex scenarios (AgNOR and Papanicolaou), achieving the best performance on the combined dataset. Crucially, unified training matched or exceeded stain‐specific models, maintaining detection (AP 50 ) while improving boundary definition (AP 75 ).
Conclusions
A single multi‐stain model is feasible for cytology instance segmentation, preserving detection and improving boundary accuracy in diverse stains. Transformer‐based Mask2Former showed the best AP 75 on the combined dataset, supporting a unified multi‐stain, scalable path to AI‐assisted cytopathology and practical screening workflows. Inside This Month's Cytopathology Can a single computer vision model handle the diversity of cytological stains found in clinical routine? This study validates a unified deep learning approach that effectively segments cells across Papanicolaou, Feulgen and AgNOR protocols. Results suggest that laboratories can deploy simplified, stain‐invariant screening tools without compromising diagnostic precision, overcoming a major barrier in digital pathology interoperability.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Sciences, Federal University of Santa Catarina, Florianopolis, Brazil.
- Department of Clinical Analysis, Federal University of Santa Catarina, Florianopolis, Brazil.
- National Institute for Digital Convergence (INCoD), Federal University of Santa Catarina, Florianopolis, Brazil.
Förderung
- Coordenação de Aperfeiçoamento de Pessoal de Nível Superior
Zitieren
APA
Santos, L. O., Ribeiro, R. P. E. S., Tiellet, B. Q., Silva, L. M. E., de Miranda Onofre, F. B., Onofre, A. S. C., & von Wangenheim, A. (2026). A Unified Deep Learning Framework for Instance Segmentation Across Diverse Cytological Stains. Cytopathology, 37(5), 493-502. https://doi.org/10.1111/cyt.70087
Vancouver
Santos LO, Ribeiro RPES, Tiellet BQ, Silva LME, de Miranda Onofre FB, Onofre ASC, et al. A Unified Deep Learning Framework for Instance Segmentation Across Diverse Cytological Stains. Cytopathology. 2026;37(5):493-502. doi:10.1111/cyt.70087
BibTeX
@article{Santos2026,
author = {Santos, Luís Otávio and Ribeiro, Rodrigo de Paula E Silva and Tiellet, Bibiana Quatrin and Silva, Laura Maria Effting and de Miranda Onofre, Fabiana Botelho and Onofre, Alexandre Sherlley Casimiro and von Wangenheim, Aldo},
title = {A Unified Deep Learning Framework for Instance Segmentation Across Diverse Cytological Stains},
journal = {Cytopathology},
year = {2026},
volume = {37},
number = {5},
pages = {493--502},
doi = {10.1111/cyt.70087}
}
EUVIMED