Optimization models of natural communication
A family of information theoretic models of communication was introduced more than a decade ago to explain the origins of Zipf's law for word frequencies. The family is a based on a combination of two information theoretic principles: maximization of mutual information between forms and meanings and minimization of form entropy. The family also sheds light on the origins of three other patterns: the principle of contrast, a related vocabulary learning bias and the meaning-frequency law. Here two important components of the family, namely the information theoretic principles and the energy function that combines them linearly, are reviewed from the perspective of psycholinguistics, language learning, information theory and synergetic linguistics. The minimization of this linear function is linked to the problem of compression of standard information theory and might be tuned by self-organization.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Quantized Frank-Wolfe: Faster Optimization, Lower Communication, and Projection Free
How can we efficiently mitigate the overhead of gradient communications in distributed optimization? This problem is at the heart of training scalable machine learning models and has been mainly studied in the unconstrai…
Distributed OptimizationQuantizationStochastic OptimizationDecentralized Riemannian natural gradient methods with Kronecker-product approximations
With a computationally efficient approximation of the second-order information, natural gradient methods have been successful in solving large-scale structured optimization problems. We study the natural gradient methods…
Communication-Efficient Federated Group Distributionally Robust Optimization
Federated learning faces challenges due to the heterogeneity in data volumes and distributions at different clients, which can compromise model generalization ability to various distributions. Existing approaches to addr…
Federated LearningDIAMOND: Taming Sample and Communication Complexities in Decentralized Bilevel Optimization
Decentralized bilevel optimization has received increasing attention recently due to its foundational role in many emerging multi-agent learning paradigms (e.g., multi-agent meta-learning and multi-agent reinforcement le…
Bilevel OptimizationMeta-LearningMulti-agent Reinforcement LearningThe Communication Complexity of Optimization
We consider the communication complexity of a number of distributed optimization problems. We start with the problem of solving a linear system. Suppose there is a coordinator together with $s$ servers $P_1, \ldots, P_s$…
Distributed Optimization