EUVIMEDEUVIMED entdecken
Crossref

InvPT++: Inverted Pyramid Multi-Task Transformer for Visual Scene Understanding

Hanrong Ye, Dan Xu

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-12 · 46 (12) · 7493-7508

DOI: 10.1109/tpami.2024.3397031 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-12. Verfasst von Hanrong Ye und 1 weiteren Autor:innen. Die Arbeit wurde bisher 29-mal zitiert (Crossref) und nennt 76 Literaturangaben. Gefördert von: Early Career Scheme of the Research Grants Council.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multi-task scene understanding aims to design models that can simultaneously predict several scene understanding tasks with one versatile model. Previous studies typically process multi-task features in a more local way, and thus cannot effectively learn spatially global and cross-task interactions, which hampers the models' ability to fully leverage the consistency of various tasks in multi-task learning. To tackle this problem, we propose an Inverted Pyramid multi-task Transformer, capable of modeling cross-task interaction among spatial features of different tasks in a global context. Specifically, we first utilize a transformer encoder to capture task-generic features for all tasks. And then, we design a transformer decoder to establish spatial and cross-task interaction globally, and a novel UP-Transformer block is devised to increase the resolutions of multi-task features gradually and establish cross-task interaction at different scales. Furthermore, two types of Cross-Scale Self-Attention modules, i.e., Fusion Attention and Selective Attention, are proposed to efficiently facilitate cross-task interaction across different feature scales. An Encoder Feature Aggregation strategy is further introduced to better model multi-scale information in the decoder. Comprehensive experiments on several 2D/3D multi-task benchmarks clearly demonstrate our proposal's effectiveness, establishing significant state-of-the-art performances.

Förderung

  • Early Career Scheme of the Research Grants Council

Zitieren

APA

Ye, H., & Xu, D. (2024). InvPT++: Inverted Pyramid Multi-Task Transformer for Visual Scene Understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12), 7493-7508. https://doi.org/10.1109/tpami.2024.3397031

Vancouver

Ye H, Xu D. InvPT++: Inverted Pyramid Multi-Task Transformer for Visual Scene Understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(12):7493-7508. doi:10.1109/tpami.2024.3397031

BibTeX
@article{Ye2024,
  author = {Ye, Hanrong and Xu, Dan},
  title = {InvPT++: Inverted Pyramid Multi-Task Transformer for Visual Scene Understanding},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
  volume = {46},
  number = {12},
  pages = {7493--7508},
  doi = {10.1109/tpami.2024.3397031}
}
Kostenlos starten →