Shaping representations through communication: community size effect in artificial learning systems
Motivated by theories of language and communication that explain why communities with large numbers of speakers have, on average, simpler languages with more regularity, we cast the representation learning problem in terms of learning to communicate. Our starting point sees the traditional autoencoder setup as a single encoder with a fixed decoder partner that must learn to communicate. Generalizing from there, we introduce community-based autoencoders in which multiple encoders and decoders collectively learn representations by being randomly paired up on successive training iterations. We find that increasing community sizes reduce idiosyncrasies in the learned codes, resulting in representations that better encode concept categories and correlate with human feature norms.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderRepresentation LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Bioenergetic trophic trade-offs determine mass-dependent extinction thresholds across the Cenozoic
Body size drives the energetic demands of organisms, constraining trophic interactions between species and playing a significant role in shaping the feasibility of species' populations in a community. On macroevolutionar…
Improving the Ranging Performance of Random ISAC Signals Through Pulse Shaping Design
In this paper, we propose a novel pulse shaping design for single-carrier integrated sensing and communication (ISAC) transmission. Due to the communication information embedded in the ISAC signal, the resulting auto-cor…
Integrated sensing and communicationISACPredictive Coding for Boosting Deep Reinforcement Learning with Sparse Rewards
While recent progress in deep reinforcement learning has enabled robots to learn complex behaviors, tasks with long horizons and sparse rewards remain an ongoing challenge. In this work, we propose an effective reward sh…
Deep Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)End-to-end Optimization of Constellation Shaping for Wiener Phase Noise Channels with a Differentiable Blind Phase Search
As the demand for higher data throughput in coherent optical communication systems increases, we need to find ways to increase capacity in existing and future optical communication links. To address the demand for higher…
Resource-aware Computation-Communication Overlap for multi-GPU ML Workloads
The rapid growth of large-scale machine learning (ML) has made distributed training across multiple GPUs a fundamental component of modern ML systems. As model sizes and computational throughput continue to increase, com…