Adviser Networks: Learning What Question to Ask for Human-In-The-Loop Viewpoint Estimation
Humans have an unparalleled visual intelligence and can overcome visual ambiguities that machines currently cannot. Recent works have shown that incorporating guidance from humans during inference for monocular viewpoint-estimation can help overcome difficult cases in which the computer-alone would have otherwise failed. These hybrid intelligence approaches are hence gaining traction. However, deciding what question to ask the human at inference time remains an unknown for these problems. We address this question by formulating it as an Adviser Problem: can we learn a mapping from the input to a specific question to ask the human to maximize the expected positive impact to the overall task? We formulate a solution to the adviser problem for viewpoint estimation using a deep network where the question asks for the location of a keypoint in the input image. We show that by using the Adviser Network's recommendations, the model and the human outperforms the previous hybrid-intelligence state-of-the-art by 3.7%, and the computer-only state-of-the-art by 5.28% absolute.
Code (1)
Tasks
Viewpoint EstimationSimilar Papers 제목 키워드 기반
AI-in-the-loop: The future of biomedical visual analytics applications in the era of AI
AI is the workhorse of modern data analytics and omnipresent across many sectors. Large Language Models and multi-modal foundation models are today capable of generating code, charts, visualizations, etc. How will these …
Transferring Domain Knowledge with an Adviser in Continuous Tasks
Recent advances in Reinforcement Learning (RL) have surpassed human-level performance in many simulated environments. However, existing reinforcement learning techniques are incapable of explicitly incorporating already …
OpenAI Gymreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrieval
Unified multimodal retrieval aims to identify candidates that satisfy complex user intent expressed through heterogeneous inputs. Although Large Vision-Language Model (LVLM)-based retrievers are efficient and scalable, d…
Reinforcement LearningHumanwashing -- It Should Leave You Feeling Dirty
The phrase 'human in the loop' is increasingly used to imply a sense of safety in relation to AI decision systems. It shouldn't. There are contexts where it can be applied appropriately, but these are not in the deployed…
Financial Adviser Misconduct and Labor Market Penalties: Uncovering Racial Disparities in the Absence of Gender Gaps
Using a comprehensive matched employer-employee dataset for U.S. financial advisers from 2008 to 2018, we revisit established evidence on labor market penalties following financial misconduct. Prior studies report that f…