Testing GPT-4 with Wolfram Alpha and Code Interpreter plug-ins on math and science problems

2023-08-10 · Ernest Davis, Scott Aaronson

This report describes a test of the large language model GPT-4 with the Wolfram Alpha and the Code Interpreter plug-ins on 105 original problems in science and math, at the high school and college levels, carried out in June-August 2023. Our tests suggest that the plug-ins significantly enhance GPT's ability to solve these problems. Having said that, there are still often "interface" failures; that is, GPT often has trouble formulating problems in a way that elicits useful answers from the plug-ins. Fixing these interface failures seems like a central challenge in making GPT a reliable tool for college-level calculation problems.

📄 PDF Abstract BibTeX arXiv:2308.05713

Code (0)

등록된 구현이 없습니다.

Tasks

Language ModelingLanguage ModellingLarge Language ModelMath

Methods 이 논문이 사용한 방법론

Refunds@Expedia|||How do I get a full refund from Expedia? “How do I get a full refund from Expedia? How do I get a full refund from Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Quick Help &…

Multi-Head Attention 설명 없음

Attention 설명 없음

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Cosine Annealing Cosine Annealing is a type of learning rate schedule that has the effect of starting with a large learning rate that is relatively rapidly decreased to a minimum value before…

Position-Wise Feed-Forward Layer 설명 없음

Testing GPT-4-o1-preview on math and science problems: A follow-up study

2024-10-11 · Ernest Davis

In August 2023, Scott Aaronson and I reported the results of testing GPT4 with the Wolfram Alpha and Code Interpreter plug-ins over a collection of 105 original high-school level and college-level science and math proble…

Math Spatial Reasoning

The Broad Optimality of Profile Maximum Likelihood

2019-06-10 · NeurIPS 2019 12 · Yi Hao, Alon Orlitsky

We study three fundamental statistical-learning problems: distribution estimation, property estimation, and property testing. We establish the profile maximum likelihood (PML) estimator as the first unified sample-optima…

CIBench: Evaluating Your LLMs with a Code Interpreter Plugin

2024-07-15 · Chuyu Zhang, Songyang Zhang, Yingfan Hu, Haowen Shen 외

While LLM-Based agents, which use external tools to solve complex problems, have made significant progress, benchmarking their ability is challenging, thereby hindering a clear understanding of their limitations. In this…

Benchmarking

Verifying Good Regulator Conditions for Hypergraph Observers: Natural Gradient Learning from Causal Invariance via Established Theorems

2026-03-10 · Max Zhuravlev arxiv

We verify that persistent observers in causally invariant hypergraph substrates satisfy the conditions of the Conant-Ashby Good Regulator Theorem. Building on Wolfram's hypergraph physics and Vanchurin's neural network c…

R1-Code-Interpreter: Training LLMs to Reason with Code via Supervised and Reinforcement Learning

2025-05-27 · Yongchao Chen, Yueying Liu, Junwei Zhou, Yilun Hao 외

Despite advances in reasoning and planning of R1-like models, Large Language Models (LLMs) still struggle with tasks requiring precise computation, symbolic manipulation, optimization, and algorithmic reasoning, in which…

Code Generation Reinforcement Learning (RL)