How2: A Large-scale Dataset for Multimodal Language Understanding
In this paper, we introduce How2, a multimodal collection of instructional videos with English subtitles and crowdsourced Portuguese translations. We also present integrated sequence-to-sequence baselines for machine translation, automatic speech recognition, spoken language translation, and multimodal summarization. By making available data and code for several multimodal natural language tasks, we hope to stimulate more research on these and similar challenges, to obtain a deeper understanding of multimodality in language processing.
Code (2)
Tasks
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Machine Translationspeech-recognitionSpeech RecognitionTranslationSimilar Papers 제목 키워드 기반
OCC-MLLM:Empowering Multimodal Large Language Model For the Understanding of Occluded Objects
There is a gap in the understanding of occluded objects in existing large-scale visual language multi-modal models. Current state-of-the-art multimodal models fail to provide satisfactory results in describing occluded o…
Language ModelingLanguage ModellingLarge Language ModelMultimodal Large Language ModelInternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
This paper introduces InternVid, a large-scale video-centric multimodal dataset that enables learning powerful and transferable video-text representations for multimodal understanding and generation. The InternVid datase…
Action RecognitionContrastive LearningRepresentation LearningRetrieval+5MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning
With the rapid development of large language models (LLMs) and their integration into large multimodal models (LMMs), there has been impressive progress in zero-shot completion of user-oriented vision-language tasks. How…
Chart UnderstandingMM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments
Operating rooms (ORs) are complex, high-stakes environments requiring precise understanding of interactions among medical staff, tools, and equipment for enhancing surgical assistance, situational awareness, and patient …
2D Panoptic SegmentationGraph GenerationLanguage ModelingLanguage Modelling+2Understanding Chinese Video and Language via Contrastive Multimodal Pre-Training
The pre-trained neural models have recently achieved impressive performances in understanding multimodal content. However, it is still very challenging to pre-train neural models for video and language understanding, esp…
Contrastive LearningLanguage ModelingLanguage ModellingMasked Language Modeling+1