TroVE: Inducing Verifiable and Efficient Toolboxes for Solving Programmatic Tasks

2024-01-23 · Zhiruo Wang, Daniel Fried, Graham Neubig

Language models (LMs) can solve tasks such as answering questions about tables or images by writing programs. However, using primitive functions often leads to verbose and error-prone programs, and higher-level functions require expert design. To enable better solutions without human labor, we ask code LMs to curate reusable high-level functions, and use them to write solutions. We present TROVE, a training-free method of inducing a verifiable and efficient toolbox of functions, by generating via using, growing, and periodically trimming the toolbox. On 11 datasets from math, table question answering, and image reasoning tasks, TROVE consistently yields simpler solutions with higher accuracy than baselines using CODELLAMA and previous methods using GPT, while using 79-98% smaller toolboxes. TROVE further enables 31% faster and 13% more accurate human verification than baselines. With the same pipeline, it creates diverse functions for varied tasks and datasets, providing insights into their individual characteristics.

📄 PDF Abstract BibTeX arXiv:2401.12869

Code (1)

zorazrw/trove 공식 구현

Tasks

MathQuestion Answering

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Residual Connection 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Adam 설명 없음

Multi-Head Attention 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

A Compute-Matched Re-Evaluation of TroVE on MATH

2025-07-16 · Tobias Sesterhenn, Ian Berlot-Attwell, Janis Zenkner, Christian Bartelt arxiv

Reusing established theorems and formulas is central to mathematical problem solving, serving as essential building blocks for tackling increasingly complex challenges. Recent work, TroVE, argues that code-generating Lar…

TRoVe: Discovering Error-Inducing Static Feature Biases in Temporal Vision-Language Models

2025-11-30 · Maya Varma, Jean-Benoit Delbrouck, Sophie Ostmeier, Akshay Chaudhari 외 arxiv

Vision-language models (VLMs) have made great strides in addressing temporal understanding tasks, which involve characterizing visual changes across a sequence of images. However, recent works have suggested that when ma…

Programmatically Interpretable Reinforcement Learning

2018-04-06 · ICML 2018 7 · Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli 외

We present a reinforcement learning framework, called Programmatically Interpretable Reinforcement Learning (PIRL), that is designed to generate interpretable and verifiable agent policies. Unlike the popular Deep Reinfo…

Car Racing Deep Reinforcement Learning reinforcement-learning Reinforcement Learning+1

SymCode: A Neurosymbolic Approach to Mathematical Reasoning via Verifiable Code Generation

2025-10-29 · Sina Bagheri Nezhad, Yao Li, Ameeta Agrawal arxiv

Large Language Models (LLMs) often struggle with complex mathematical reasoning, where prose-based generation leads to unverified and arithmetically unsound solutions. Current prompting strategies like Chain of Thought s…

Mathematical Reasoning Logical Fallacies Code Generation

SKATE, a Scalable Tournament Eval: Weaker LLMs differentiate between stronger ones using verifiable challenges

2025-08-08 · Dewi S. W. Gould, Bruno Mlodozeniec, Samuel F. Brown arxiv

Evaluating the capabilities and risks of foundation models is paramount, yet current methods demand extensive domain expertise, hindering their scalability as these models rapidly evolve. We introduce SKATE: a novel eval…

Spatial Reasoning