The curious case of developmental BERTology: On sparsity, transfer learning, generalization and the brain
In this essay, we explore a point of intersection between deep learning and neuroscience, through the lens of large language models, transfer learning and network compression. Just like perceptual and cognitive neurophysiology has inspired effective deep neural network architectures which in turn make a useful model for understanding the brain, here we explore how biological neural development might inspire efficient and robust optimization procedures which in turn serve as a useful model for the maturation and aging of the brain.
Code (0)
등록된 구현이 없습니다.
Tasks
Transfer LearningSimilar Papers 제목 키워드 기반
Maps Search Misspelling Detection Leveraging Domain-Augmented Contextual Representations
Building an independent misspelling detector and serve it before correction can bring multiple benefits to speller and other search components, which is particularly true for the most commonly deployed noisy-channel base…
Representation LearningMonolingual vs multilingual BERTology for Vietnamese extractive multi-document summarization
CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning
In open-ended environments, autonomous learning agents must set their own goals and build their own curriculum through an intrinsically motivated exploration. They may consider a large diversity of goals, aiming to disco…
DiversityEfficient ExplorationMulti-Goal Reinforcement Learningreinforcement-learning+2ViCGCN: Graph Convolutional Network with Contextualized Language Models for Social Media Mining in Vietnamese
Social media processing is a fundamental task in natural language processing with numerous applications. As Vietnamese social media and information science have grown rapidly, the necessity of information-based mining on…
Language Modellingtext-classificationText ClassificationVietnamese Social Media Text ProcessingBERTologyNavigator: Advanced Question Answering with BERT-based Semantics
The development and integration of knowledge graphs and language models has significance in artificial intelligence and natural language processing. In this study, we introduce the BERTologyNavigator -- a two-phased syst…
Knowledge GraphsNavigateQuestion AnsweringRelation+1