Meta Multi-Task Learning for Sequence Modeling
Semantic composition functions have been playing a pivotal role in neural representation learning of text sequences. In spite of their success, most existing models suffer from the underfitting problem: they use the same shared compositional function on all the positions in the sequence, thereby lacking expressive power due to incapacity to capture the richness of compositionality. Besides, the composition functions of different tasks are independent and learned from scratch. In this paper, we propose a new sharing scheme of composition function across multiple tasks. Specifically, we use a shared meta-network to capture the meta-knowledge of semantic composition and generate the parameters of the task-specific semantic composition models. We conduct extensive experiments on two types of tasks, text classification and sequence tagging, which demonstrate the benefits of our approach. Besides, we show that the shared meta-knowledge learned by our proposed model can be regarded as off-the-shelf knowledge and easily transferred to new tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Multi-Task LearningRepresentation LearningSemantic Compositiontext-classificationText ClassificationSimilar Papers 제목 키워드 기반
Metagross: Meta Gated Recursive Controller Units for Sequence Modeling
This paper proposes Metagross (Meta Gated Recursive Controller), a new neural sequence modeling unit. Our proposed unit is characterized by recursive parameterization of its gating functions, i.e., gating mechanisms of M…
Code GenerationInductive BiasMachine TranslationMusic Modeling+2Recasting Continual Learning as Sequence Modeling
In this work, we aim to establish a strong connection between two significant bodies of machine learning research: continual learning and sequence modeling. That is, we propose to formulate continual learning as a sequen…
Continual LearningMERMAID: Metaphor Generation with Symbolism and Discriminative Decoding
Generating metaphors is a challenging task as it requires a proper understanding of abstract concepts, making connections between unrelated concepts, and deviating from the literal meaning. Based on a theoretically-groun…
Language ModelingLanguage ModellingMasked Language ModelingMeta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglement
A longstanding goal of artificial general intelligence is highly capable generalists that can learn from diverse experiences and generalize to unseen tasks. The language and vision communities have seen remarkable progre…
DisentanglementInductive BiasMuJoCoReinforcement Learning (RL)+2MERMAID: Metaphor Generation with Symbolism and Discriminative Decoding
Generating metaphors is a challenging task as it requires a proper understanding of abstract concepts, making connections between unrelated concepts, and deviating from the literal meaning. In this paper, we aim to gener…
Language ModelingLanguage ModellingMasked Language ModelingSentence