i-Code Studio: A Configurable and Composable Framework for Integrative AI
Artificial General Intelligence (AGI) requires comprehensive understanding and generation capabilities for a variety of tasks spanning different modalities and functionalities. Integrative AI is one important direction to approach AGI, through combining multiple models to tackle complex multimodal tasks. However, there is a lack of a flexible and composable platform to facilitate efficient and effective model composition and coordination. In this paper, we propose the i-Code Studio, a configurable and composable framework for Integrative AI. The i-Code Studio orchestrates multiple pre-trained models in a finetuning-free fashion to conduct complex multimodal tasks. Instead of simple model composition, the i-Code Studio provides an integrative, flexible, and composable setting for developers to quickly and easily compose cutting-edge services and technologies tailored to their specific requirements. The i-Code Studio achieves impressive results on a variety of zero-shot multimodal tasks, such as video-to-text retrieval, speech-to-speech translation, and visual question answering. We also demonstrate how to quickly build a multimodal agent based on the i-Code Studio that can communicate and personalize for users.
Code (0)
등록된 구현이 없습니다.
Tasks
Question AnsweringRetrievalSpeech-to-Speech TranslationText RetrievalVideo to Text RetrievalVisual Question AnsweringSimilar Papers 제목 키워드 기반
i-Code: An Integrative and Composable Multimodal Learning Framework
Human intelligence is multimodal; we integrate visual, linguistic, and acoustic signals to maintain a holistic worldview. Most current pretraining methods, however, are limited to one or two modalities. We present i-Code…
Contrastive LearningVideo UnderstandingMulti-Dimensional Reconfigurable, Physically Composable Hybrid Diffractive Optical Neural Network
Diffractive optical neural networks (DONNs), leveraging free-space light wave propagation for ultra-parallel, high-efficiency computing, have emerged as promising artificial intelligence (AI) accelerators. However, their…
FastIoT -- A framework and holistic approach for rapid development of IIoT systems
While lots of research has been conducted on the architecture of Industrial Internet of Things (IIoT) systems, concepts of structuring their development processes are missing. Therefore, we propose a holistic approach su…
Edge-computingMiscellaneousReconfigurable Computing Challenge: Transformer for Jet Tagging on Versal AI Engines
Transformer-based models achieve strong performance for jet tagging at the CERN LHC, but deploying them in low-latency, resource-constrained trigger systems is challenging. We present an initial implementation of a quant…
Jet TaggingGSCodec Studio: A Modular Framework for Gaussian Splat Compression
3D Gaussian Splatting and its extension to 4D dynamic scenes enable photorealistic, real-time rendering from real-world captures, positioning Gaussian Splats (GS) as a promising format for next-generation immersive media…
Benchmarking