Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformers

2026-01-28 · Yiran Huang, Karsten Roth, Quentin Bouniot, Wenjia Xu, Zeynep Akata arxiv

Transformer-based multimodal large language models often exhibit in-context learning (ICL) abilities. Motivated by this phenomenon, we ask: how do transformers learn to associate information across modalities from in-context examples? We investigate this question through controlled experiments on small transformers trained on synthetic classification tasks, enabling precise manipulation of data statistics and model architecture. We begin by revisiting core principles of unimodal ICL in modern transformers. While several prior findings replicate, we find that Rotary Position Embeddings (RoPE) increases the data complexity threshold for ICL. Extending to the multimodal setting reveals a fundamental learning asymmetry: when pretrained on high-diversity data from a primary modality, surprisingly low data complexity in the secondary modality suffices for multimodal ICL to emerge. Mechanistic analysis shows that both settings rely on an induction-style mechanism that copies labels from matching in-context exemplars; multimodal training refines and extends these circuits across modalities. Our findings provide a mechanistic foundation for understanding multimodal ICL in modern transformers and introduce a controlled testbed for future investigation. Code is available at: https://github.com/YiranHuangIrene/multimodal-icl

📄 PDF Abstract BibTeX arXiv:2601.20796

Code (0)

등록된 구현이 없습니다.

Instruction Anchor: Dissecting the Mechanistic Dynamics of Modality Arbitration

2026-02-03 · Yu Zhang, Mufan Xu, Xuefeng Bai, Kehai Chen 외 arxiv

Modality following is the ability to selectively leverage multimodal contexts based on user instructions. It is fundamental to the safety and reliability of multimodal large language models (MLLMs) in real-world deployme…

Context-specific Credibility-aware Multimodal Fusion with Conditional Probabilistic Circuits

2026-03-27 · Pranuthi Tenali, Sahil Sidheekh, Saurabh Mathur, Erik Blasch 외 arxiv

Multimodal fusion requires integrating information from multiple sources that may conflict depending on context. Existing fusion approaches typically rely on static assumptions about source reliability, limiting their ab…

Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion

2023-06-15 · Ishaan Singh Rawal, Alexander Matyasko, Shantanu Jaiswal, Basura Fernando 외

While VideoQA Transformer models demonstrate competitive performance on standard benchmarks, the reasons behind their success are not fully understood. Do these models capture the rich multimodal structures and dynamics …

Benchmarking counterfactual

Rethinking Explainability in the Era of Multimodal AI

2025-06-16 · Chirag Agarwal

While multimodal AI systems (models jointly trained on heterogeneous data types such as text, time series, graphs, and images) have become ubiquitous and achieved remarkable performance across high-stakes applications, t…

On the Complementarity of Quantum and Classical Features: Adaptive Hybrid Quantum-Classical Feature Fusion for Breast Cancer Classification

2026-04-24 · Yasmin Rodrigues Sobrinho, João Renato Ribeiro Manesco, João Paulo Papa arxiv

The integration of quantum machine learning with classical deep learning offers promising avenues for medical image analysis by mapping data into high-dimensional Hilbert spaces. However, effectively unifying these disti…

Quantum Machine Learning Cancer Classification