EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Adversarially Robust Vision-Language Alignment via Stability-Aware Task Vector Merging

Maren Schaefer

Journal of Innovative Research and Technology · 10.05.2026 · 3 (1) · 140-150

DOI: 10.66857/jirt.v3i1.34 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Innovative Research and Technology, online erschienen am 10.05.2026. Verfasst von Maren Schaefer. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Vision-language models have demonstrated remarkable capabilities in aligning complex visual and textual representations, thereby enabling significant advancements in multimodal learning tasks such as image captioning, cross-modal retrieval, and visual question answering. However, these models remain highly susceptible to adversarial perturbations, wherein microscopic, carefully crafted noise applied to input images or text can severely degrade alignment and lead to catastrophic misclassifications. Current adversarial defense mechanisms often compromise standard performance or require prohibitively expensive adversarial training regimes. To address this critical vulnerability, this paper introduces a novel framework for achieving adversarially robust vision-language alignment through stability-aware task vector merging. By conceptualizing the parameter differences between pre-trained and fine-tuned models as task vectors, we perform weight-space interpolations that integrate robust features without catastrophic forgetting. We propose a stability-aware merging mechanism that evaluates the sensitivity of individual parameter dimensions using gradient variance metrics, subsequently down-weighting unstable components that are highly susceptible to adversarial shifts. Comprehensive theoretical analysis and extensive empirical evaluations across multiple multimodal benchmarks demonstrate that our proposed approach significantly enhances robustness against gradient-based attacks while maintaining, and in some cases improving, clean standard accuracy. The results suggest that stability-aware task vector merging offers a highly scalable and effective alternative to traditional adversarial training in large-scale vision-language systems.

Zitieren

APA

Schaefer, M. (2026). Adversarially Robust Vision-Language Alignment via Stability-Aware Task Vector Merging. Journal of Innovative Research and Technology, 3(1), 140-150. https://doi.org/10.66857/jirt.v3i1.34

Vancouver

Schaefer M. Adversarially Robust Vision-Language Alignment via Stability-Aware Task Vector Merging. Journal of Innovative Research and Technology. 2026;3(1):140-150. doi:10.66857/jirt.v3i1.34

BibTeX
@article{Schaefer2026,
  author = {Schaefer, Maren},
  title = {Adversarially Robust Vision-Language Alignment via Stability-Aware Task Vector Merging},
  journal = {Journal of Innovative Research and Technology},
  year = {2026},
  volume = {3},
  number = {1},
  pages = {140--150},
  doi = {10.66857/jirt.v3i1.34}
}
Kostenlos starten →