Efficient and Effective Adaptation of Multimodal Foundation Models in Sequential Recommendation
Multimodal foundation models (MFMs) have revolutionized sequential recommender systems through advanced representation learning. While Parameter-efficient Fine-tuning (PEFT) is commonly used to adapt these models, studies often prioritize parameter efficiency, neglecting GPU memory and training speed. To address this, we introduced the IISAN framework, significantly enhancing efficiency. However, IISAN was limited to symmetrical MFMs and identical text and image encoders, preventing the use of state-of-the-art Large Language Models. To overcome this, we developed IISAN-Versa, a versatile plug-and-play architecture compatible with both symmetrical and asymmetrical MFMs. IISAN-Versa employs a Decoupled PEFT structure and utilizes both intra- and inter-modal adaptation. It effectively handles asymmetry through a simple yet effective combination of group layer-dropping and dimension transformation alignment. Our research demonstrates that IISAN-Versa effectively adapts large text encoders, and we further identify a scaling effect where larger encoders generally perform better. IISAN-Versa also demonstrates strong versatility in our defined multimodal scenarios, which include raw titles and captions generated from images and videos. Additionally, IISAN-Versa achieved state-of-the-art performance on the Microlens public benchmark. We will release our code and datasets to support future research.
Code (0)
등록된 구현이 없습니다.
Tasks
GPUparameter-efficient fine-tuningRecommendation SystemsRepresentation LearningSequential RecommendationSimilar Papers 제목 키워드 기반
CROSSAN: Towards Efficient and Effective Adaptation of Multiple Multimodal Foundation Models for Sequential Recommendation
Multimodal Foundation Models (MFMs) excel at representing diverse raw modalities (e.g., text, images, audio, videos, etc.). As recommender systems increasingly incorporate these modalities, leveraging MFMs to generate be…
parameter-efficient fine-tuningRecommendation SystemsSequential RecommendationIISAN: Efficiently Adapting Multimodal Representation for Sequential Recommendation with Decoupled PEFT
Multimodal foundation models are transformative in sequential recommender systems, leveraging powerful representation learning capabilities. While Parameter-efficient Fine-tuning (PEFT) is commonly used to adapt foundati…
GPUparameter-efficient fine-tuningRecommendation SystemsRepresentation Learning+1Multimodal Pre-training Framework for Sequential Recommendation via Contrastive Learning
Current multimodal sequential recommendation models are often unable to effectively explore and capture correlations among behavior sequences of users and items across different modalities, either neglecting correlations…
Contrastive LearningRecommendation SystemsRepresentation LearningSequential RecommendationTowards Efficient and Effective Adaptation of Large Language Models for Sequential Recommendation
In recent years, with large language models (LLMs) achieving state-of-the-art performance in context understanding, increasing efforts have been dedicated to developing LLM-enhanced sequential recommendation (SR) methods…
Sequential RecommendationA Framework for Elastic Adaptation of User Multiple Intents in Sequential Recommendation
Recently, substantial research has been conducted on sequential recommendation, with the objective of forecasting the subsequent item by leveraging a user's historical sequence of interacted items. Prior studies employ b…
Incremental LearningSequential Recommendation