Stochastic Structured Prediction under Bandit Feedback
Stochastic structured prediction under bandit feedback follows a learning protocol where on each of a sequence of iterations, the learner receives an input, predicts an output structure, and receives partial feedback in form of a task loss evaluation of the predicted structure. We present applications of this learning scenario to convex and non-convex objectives for structured prediction and analyze them as stochastic first-order methods. We present an experimental evaluation on problems of natural language processing over exponential output spaces, and compare convergence speed across different objectives under the practical criterion of optimal task performance on development data and the optimization-theoretic criterion of minimal squared gradient norm. Best results under both criteria are obtained for a non-convex objective for pairwise preference learning under bandit feedback.
Code (1)
Tasks
PredictionStructured PredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Bandit Structured Prediction for Neural Sequence-to-Sequence Learning
Bandit structured prediction describes a stochastic optimization framework where learning is performed from partial feedback. This feedback is received in the form of a task loss evaluation to a predicted output structur…
Domain AdaptationMachine TranslationStochastic OptimizationStructured Prediction+1Structured Prediction via Learning to Search under Bandit Feedback
We present an algorithm for structured prediction under online bandit feedback. The learner repeatedly predicts a sequence of actions, generating a structured output. It then observes feedback for that output and no othe…
Active LearningDependency ParsingPredictionStructured PredictionCounterfactual Risk Minimization: Learning from Logged Bandit Feedback
We develop a learning principle and an efficient algorithm for batch learning from logged bandit feedback. This learning setting is ubiquitous in online systems (e.g., ad placement, web search, recommendation), where an …
counterfactualMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONBandit Structured Prediction for Learning from Partial Feedback in Statistical Machine Translation
We present an approach to structured prediction from bandit feedback, called Bandit Structured Prediction, where only the value of a task loss function at a single predicted point, instead of a correct structure, is obse…
Machine TranslationRerankingStructured PredictionTranslationAlmost Boltzmann Exploration
Boltzmann exploration is widely used in reinforcement learning to provide a trade-off between exploration and exploitation. Recently, in (Cesa-Bianchi et al., 2017) it has been shown that pure Boltzmann exploration does …
Multi-Armed BanditsReinforcement Learning