Cross-stitched Multi-modal Encoders
In this paper, we propose a novel architecture for multi-modal speech and text input. We combine pretrained speech and text encoders using multi-headed cross-modal attention and jointly fine-tune on the target problem. The resultant architecture can be used for continuous token-level classification or utterance-level prediction acting on simultaneous text and speech. The resultant encoder efficiently captures both acoustic-prosodic and lexical information. We compare the benefits of multi-headed attention-based fusion for multi-modal utterance-level classification against a simple concatenation of pre-pooled, modality-specific representations. Our model architecture is compact, resource efficient, and can be trained on a single consumer GPU card.
Code (0)
등록된 구현이 없습니다.
Tasks
ClassificationGPUSimilar Papers 제목 키워드 기반
Cross-stitched Multi-modal Encoders
In this paper, we propose a novel architecture for multi-modal speech and text input. We combine pretrained speech and text encoders using multi-headed cross-modal attention and jointly fine-tune on the target problem. T…
GPUInterpreting the linear structure of vision-language model embedding spaces
Vision-language models encode images and text in a joint space, minimizing the distance between corresponding image and text pairs. How are language and images organized in this joint space, and how do the models encode …
Language ModelingLanguage ModellingModification Takes Courage: Seamless Image Stitching via Reference-Driven Inpainting
Current image stitching methods often produce noticeable seams in challenging scenarios such as uneven hue and large parallax. To tackle this problem, we propose the Reference-Driven Inpainting Stitcher (RDIStitcher), wh…
Image StitchingESC: Evolutionary Stitched Camera Calibration in the Wild
This work introduces a novel end-to-end approach for estimating extrinsic parameters of cameras in multi-camera setups on real-life sports fields. We identify the source of significant calibration errors in multi-camera …
Camera CalibrationImage SegmentationSemantic SegmentationImage Quality Assessment for Omnidirectional Cross-reference Stitching
Along with the development of virtual reality (VR), omnidirectional images play an important role in producing multimedia content with immersive experience. However, despite various existing approaches for omnidirectiona…
Image Quality AssessmentImage Stitching