$\texttt{ACCORD}$: Closing the Commonsense Measurability Gap
We present $\texttt{ACCORD}$, a framework and benchmark suite for disentangling the commonsense grounding and reasoning abilities of large language models (LLMs) through controlled, multi-hop counterfactuals. $\texttt{ACCORD}$ introduces formal elements to commonsense reasoning to explicitly control and quantify reasoning complexity beyond the typical 1 or 2 hops. Uniquely, $\texttt{ACCORD}$ can automatically generate benchmarks of arbitrary reasoning complexity, and so it scales with future LLM improvements. Benchmarking state-of-the-art LLMs -- including GPT-4o (2024-05-13), Llama-3-70B-Instruct, and Mixtral-8x22B-Instruct-v0.1 -- shows performance degrading to random chance with only moderate scaling, leaving substantial headroom for improvement. We release a leaderboard of the benchmark suite tested in this work, as well as code for automatically generating more complex benchmarks.
Code (1)
Tasks
BenchmarkingCommon Sense ReasoningCounterfactual ReasoningLarge Language ModelQuestion AnsweringSimilar Papers 제목 키워드 기반
Null Measurability at the Symmetrization Interface in VC Learning
Recent work revisiting measurability in the fundamental theorem of statistical learning imposes Borel measurability of ghost-gap suprema. We show that, at the one-sided ghost-gap interface actually used by the standard s…
Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuning
A recurring challenge in preference fine-tuning (PFT) is handling $\textit{intransitive}$ (i.e., cyclic) preferences. Intransitive preferences often stem from either $\textit{(i)}$ inconsistent rankings along a single ob…
Instruction FollowingMeasurability in the Fundamental Theorem of Statistical Learning
The Fundamental Theorem of Statistical Learning states that a hypothesis space is PAC learnable if and only if its VC dimension is finite. For the agnostic model of PAC learning, the literature so far presents proofs of …
Binary ClassificationPAC learningqPOTS: Efficient batch multiobjective Bayesian optimization via Pareto optimal Thompson sampling
Classical evolutionary approaches for multiobjective optimization are quite accurate but incur a lot of queries to the objectives; this can be prohibitive when objectives are expensive oracles. A sample-efficient approac…
Bayesian OptimizationComputational EfficiencyMultiobjective OptimizationThompson SamplingModel selection in hybrid quantum neural networks with applications to quantum transformer architectures
Quantum machine learning models generally lack principled design guidelines, often requiring full resource-intensive training across numerous choices of encodings, quantum circuit designs and initialization strategies to…
Quantum Machine Learning