Enhanced Precision of Fluorescence In Situ Hybridization (FISH) Analysis Using Neural Network-Based Nuclear Segmentation for Digital Microscopy Samples
Sensors · 2026
DOI: 10.3390/s26030873 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, 2026. Verfasst von Annamaria Csizmadia und 9 weiteren Autor:innen. Beteiligte Einrichtung: Doctoral School of Pathological Sciences, Semmelweis University, H-1085 Budapest, Hungary. Hauptthemen laut MEDLINE: In Situ Hybridization, Fluorescence, Cell Nucleus, Neural Networks, Computer. Die Arbeit nennt 38 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Government, NKFI, National Research and Development Council of Hungary.
Abstract (Originalsprache)
Introduction
Accurate nuclear segmentation is essential for the reliable diagnostic interpretation of fluorescence in situ hybridization (FISH) results. However, automated 2D digital algorithms often fail in samples with dense or overlapping nuclei, such as lymphomas, due to the loss of spatial depth information. Here, we tested if AI-based 3D nuclear segmentation can improve the accuracy, reproducibility, and diagnostic reliability of FISH reading in critical situations. Materials and Methods: Formalin-fixed follicular lymphoma sections were FISH-labeled for BCL2 gene rearrangements and digitally scanned in multilayer Z-stacks. The analytic performance in nuclear segmentation of the adaptive thresholding-based FISHQuant, and the freely accessible AI-based NucleAIzer, StarDist, and Cellpose algorithms, were compared to the eye control-based traditional FISH testing, primarily focusing on nuclear segmentation.
Results
We revealed that the Cellpose algorithm showed limited sensitivity to low-intensity signals and the adaptive thresholding 2D segmentation, and FISHQuant struggled to resolve densely packed nuclei, occasionally underestimating their counts. In contrast, 3D segmentation across focal planes significantly improved the nuclear separation and signal localization. AI-driven 3D models, especially NucleAIzer and StarDist, showed improved precision, lower variance (VP/VS ≈ 0.96), and improved gene spot correlation (r > 0.82) across multiple focal planes. The similar average number of gene spots per cell nuclei in the AI-based analyses as the eye control counting, despite the elevated number of cell nuclei found with AI, validated the AI nuclear segmentation results.
Conclusions
Inaccurate segmentation limits automated diagnostic FISH signal evaluation. Deep learning 3D approaches, particularly NucleAIzer and StarDist, may overcome thresholding and 2D constraints and improve the consistency of nuclear detection, resulting in better classification of pathogenetic gene aberrations with automated workflows in digital pathology.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Doctoral School of Pathological Sciences, Semmelweis University, H-1085 Budapest, Hungary.
- 3DHISTECH Ltd., H-1141 Budapest, Hungary.
- Department of Internal Medicine and Oncology, Semmelweis University, H-1083 Budapest, Hungary.
- Doctoral School of Applied Informatics and Applied Mathematics BioTech, Obuda University, H-1034 Budapest, Hungary.
- Synthetic and Systems Biology Unit, Biological Research Center, H-6726 Szeged, Hungary.
- Department of Pathology and Experimental Cancer Research, Semmelweis University, Üllői út 26, H-1085 Budapest, Hungary.
ORCID-Profile: Marianna Dimitrova Kucarov, Krisztian Koos, Robert Paulik, Laszlo Krenacs, Gergo Papp, Tibor Krenacs
Förderung
- Government, NKFI
- National Research and Development Council of Hungary
Zitieren
APA
Csizmadia, A., Molnar, B., Kucarov, M. D., Koos, K., Paulik, R., Kapczar, D., Krenacs, L., Csernus, B., Papp, G., & Krenacs, T. (2026). Enhanced Precision of Fluorescence In Situ Hybridization (FISH) Analysis Using Neural Network-Based Nuclear Segmentation for Digital Microscopy Samples. Sensors. https://doi.org/10.3390/s26030873
Vancouver
Csizmadia A, Molnar B, Kucarov MD, Koos K, Paulik R, Kapczar D, et al. Enhanced Precision of Fluorescence In Situ Hybridization (FISH) Analysis Using Neural Network-Based Nuclear Segmentation for Digital Microscopy Samples. Sensors. 2026. doi:10.3390/s26030873
BibTeX
@article{Csizmadia2026,
author = {Csizmadia, Annamaria and Molnar, Bela and Kucarov, Marianna Dimitrova and Koos, Krisztian and Paulik, Robert and Kapczar, Dora and Krenacs, Laszlo and Csernus, Balazs and Papp, Gergo and Krenacs, Tibor},
title = {Enhanced Precision of Fluorescence In Situ Hybridization (FISH) Analysis Using Neural Network-Based Nuclear Segmentation for Digital Microscopy Samples},
journal = {Sensors},
year = {2026},
doi = {10.3390/s26030873}
}
EUVIMED