Multi-modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models

2023-12-14 · Liqi He, Zuchao Li, Xiantao Cai, Ping Wang

Chain-of-thought (CoT) reasoning has exhibited impressive performance in language models for solving complex tasks and answering questions. However, many real-world questions require multi-modal information, such as text and images. Previous research on multi-modal CoT has primarily focused on extracting fixed image features from off-the-shelf vision models and then fusing them with text using attention mechanisms. This approach has limitations because these vision models were not designed for complex reasoning tasks and do not align well with language thoughts. To overcome this limitation, we introduce a novel approach for multi-modal CoT reasoning that utilizes latent space learning via diffusion processes to generate effective image features that align with language thoughts. Our method fuses image features and text representations at a deep level and improves the complex reasoning ability of multi-modal CoT. We demonstrate the efficacy of our proposed method on multi-modal ScienceQA and machine translation benchmarks, achieving state-of-the-art performance on ScienceQA. Overall, our approach offers a more robust and effective solution for multi-modal reasoning in language models, enhancing their ability to tackle complex real-world problems.

📄 PDF Abstract BibTeX arXiv:2312.08762

Code (0)

등록된 구현이 없습니다.

Tasks

Machine Translation

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

Multimodal Chain of Continuous Thought for Latent-Space Reasoning in Vision-Language Models

2025-08-18 · Tan-Hanh Pham, Chris Ngo arxiv

Many reasoning techniques for large multimodal models adapt language model approaches, such as Chain-of-Thought (CoT) prompting, which express reasoning as word sequences. While effective for text, these methods are subo…

Multimodal Reasoning

Learning Modal-Mixed Chain-of-Thought Reasoning with Latent Embeddings

2026-01-31 · Yifei Shao, Kun Zhou, Ziming Xu, Mohammad Atif Quamar 외 arxiv

We study how to extend chain-of-thought (CoT) beyond language to better handle multimodal reasoning. While CoT helps LLMs and VLMs articulate intermediate steps, its text-only form often fails on vision-intensive problem…

Reinforcement Learning Multimodal Reasoning

CoLT: Teaching Multi-Modal Models to Think with Chain of Latent Thoughts

2026-06-30 · Lianyu Hu, Shengqian Qin, Zeqin Liao, Qing Guo 외 arxiv

Chain-of-thought (CoT) reasoning has enabled multi-modal large language models (MLLMs) to tackle complex visual reasoning tasks by generating explicit intermediate reasoning steps in natural language. However, this text-…

Visual Reasoning

ReGuLaR: Variational Latent Reasoning Guided by Rendered Chain-of-Thought

2026-01-30 · Fanmeng Wang, Haotian Liu, Guojiang Zhao, Hongteng Xu 외 arxiv

While Chain-of-Thought (CoT) significantly enhances the performance of Large Language Models (LLMs), explicit reasoning chains introduce substantial computational redundancy. Recent latent reasoning methods attempt to mi…

Computational Efficiency

Parallel Test-Time Scaling for Latent Reasoning Models

2025-10-09 · Runyang You, Yongqi Li, Meng Liu, Wenjie Wang 외 arxiv

Parallel test-time scaling (TTS) is a pivotal approach for enhancing large language models (LLMs), typically by sampling multiple token-based chains-of-thought in parallel and aggregating outcomes through voting or searc…