EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

On Isotropy of Multimodal Embeddings

Kirill Tyshchuk, Polina Karpikova, Andrew Spiridonov, Anastasiia Prutianova, Anton Razzhigaev, Alexander Panchenko

Information · 10.07.2023 · 14 (7) · 392

DOI: 10.3390/info14070392 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Information, online erschienen am 10.07.2023. Verfasst von Kirill Tyshchuk und 5 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 17 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Ministry of Science and Higher Education.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Embeddings, i.e., vector representations of objects, such as texts, images, or graphs, play a key role in deep learning methodologies nowadays. Prior research has shown the importance of analyzing the isotropy of textual embeddings for transformer-based text encoders, such as the BERT model. Anisotropic word embeddings do not use the entire space, instead concentrating on a narrow cone in such a pretrained vector space, negatively affecting the performance of applications, such as textual semantic similarity. Transforming a vector space to optimize isotropy has been shown to be beneficial for improving performance in text processing tasks. This paper is the first comprehensive investigation of the distribution of multimodal embeddings using the example of OpenAI’s CLIP pretrained model. We aimed to deepen the understanding of the embedding space of multimodal embeddings, which has previously been unexplored in this respect, and study the impact on various end tasks. Our initial efforts were focused on measuring the alignment of image and text embedding distributions, with an emphasis on their isotropic properties. In addition, we evaluated several gradient-free approaches to enhance these properties, establishing their efficiency in improving the isotropy/alignment of the embeddings and, in certain cases, the zero-shot classification accuracy. Significantly, our analysis revealed that both CLIP and BERT models yielded embeddings situated within a cone immediately after initialization and preceding training. However, they were mostly isotropic in the local sense. We further extended our investigation to the structure of multilingual CLIP text embeddings, confirming that the observed characteristics were language-independent. By computing the few-shot classification accuracy and point-cloud metrics, we provide evidence of a strong correlation among multilingual embeddings. Embeddings transformation using the methods described in this article makes it easier to visualize embeddings. At the same time, multiple experiments that we conducted showed that, in regard to the transformed embeddings, the downstream tasks performance does not drop substantially (and sometimes is even improved). This means that one could obtain an easily visualizable embedding space, without substantially losing the quality of downstream tasks.

Förderung

  • Ministry of Science and Higher Education

Zitieren

APA

Tyshchuk, K., Karpikova, P., Spiridonov, A., Prutianova, A., Razzhigaev, A., & Panchenko, A. (2023). On Isotropy of Multimodal Embeddings. Information, 14(7), 392. https://doi.org/10.3390/info14070392

Vancouver

Tyshchuk K, Karpikova P, Spiridonov A, Prutianova A, Razzhigaev A, Panchenko A. On Isotropy of Multimodal Embeddings. Information. 2023;14(7):392. doi:10.3390/info14070392

BibTeX
@article{Tyshchuk2023,
  author = {Tyshchuk, Kirill and Karpikova, Polina and Spiridonov, Andrew and Prutianova, Anastasiia and Razzhigaev, Anton and Panchenko, Alexander},
  title = {On Isotropy of Multimodal Embeddings},
  journal = {Information},
  year = {2023},
  volume = {14},
  number = {7},
  pages = {392},
  doi = {10.3390/info14070392}
}
Kostenlos starten →