HybridHiT-UNet: Multi-Scale Temporal U-Net with Hierarchical Shot-Aware Transformers for Video Summarization
Machine Learning and Knowledge Extraction · 2026-05-20 · 8 (5) · 135
DOI: 10.3390/make8050135 ↗At a glance
Journal article in Machine Learning and Knowledge Extraction, published online on 2026-05-20. Written by Saadman Sakib and 3 co-authors. Cited 1 times so far (Crossref); 37 references. The full text is freely available (open access, CC BY 4.0).
Abstract (original language)
Video summarization aims to produce a short yet informative summary of a long video while reducing the amount of redundancy. Most transformer-based methods are single-temporal scale or are unconcerned with shot-level structure, limiting temporal coherence and cross-dataset generalization. To fill these gaps, we present HybridHiT-UNet, a supervised framework that combines three complementary parts: a pretrained Vision Transformer encoder to provide spatially rich frame representations, a multi-scale 1D Temporal U-Net backbone to provide hierarchical temporal modeling of frame representations, and a shot-aware hierarchical transformer scoring head to provide inter-shot context to importance prediction. Frame-level scores are summed into shot-level utilities and optimized with a knapsack selection on a fixed-length budget, and a weighted focal loss is used to address extreme class imbalance. Wide experiments using four benchmarks (SumMe, TVSum, OVP, and YouTube) under canonical, augmented, and transfer protocols demonstrate that HybridHiT-UNet achieves F1-scores of 65.8% on SumMe and 79.92% on TVSum, which is higher than the existing methods, which still achieve diversity scores of 64.98% and 48.68%, respectively. A systematic study further demonstrates that a 20% summary budget would yield a consistently superior coverage-diversity trade-off than the traditional 15% one, which provides useful evidence-based advice on the selection of summary length.
Cite
APA
Sakib, S., Mahmud, T., Andersson, K., & Deb, K. (2026). HybridHiT-UNet: Multi-Scale Temporal U-Net with Hierarchical Shot-Aware Transformers for Video Summarization. Machine Learning and Knowledge Extraction, 8(5), 135. https://doi.org/10.3390/make8050135
Vancouver
Sakib S, Mahmud T, Andersson K, Deb K. HybridHiT-UNet: Multi-Scale Temporal U-Net with Hierarchical Shot-Aware Transformers for Video Summarization. Machine Learning and Knowledge Extraction. 2026;8(5):135. doi:10.3390/make8050135
BibTeX
@article{Sakib2026,
author = {Sakib, Saadman and Mahmud, Tanjim and Andersson, Karl and Deb, Kaushik},
title = {HybridHiT-UNet: Multi-Scale Temporal U-Net with Hierarchical Shot-Aware Transformers for Video Summarization},
journal = {Machine Learning and Knowledge Extraction},
year = {2026},
volume = {8},
number = {5},
pages = {135},
doi = {10.3390/make8050135}
}
EUVIMED