PuzzleGPT: Emulating Human Puzzle-Solving Ability for Time and Location Prediction
The task of predicting time and location from images is challenging and requires complex human-like puzzle-solving ability over different clues. In this work, we formalize this ability into core skills and implement them using different modules in an expert pipeline called PuzzleGPT. PuzzleGPT consists of a perceiver to identify visual clues, a reasoner to deduce prediction candidates, a combiner to combinatorially combine information from different clues, a web retriever to get external knowledge if the task can't be solved locally, and a noise filter for robustness. This results in a zero-shot, interpretable, and robust approach that records state-of-the-art performance on two datasets -- TARA and WikiTilo. PuzzleGPT outperforms large VLMs such as BLIP-2, InstructBLIP, LLaVA, and even GPT-4V, as well as automatically generated reasoning pipelines like VisProg, by at least 32% and 38%, respectively. It even rivals or surpasses finetuned models.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Solving Pictorial Jigsaw Puzzle by Stigmergy-inspired Internet-based Human Collective Intelligence
The pictorial jigsaw (PJ) puzzle is a well-known leisure game for humans. Usually, a PJ puzzle game is played by one or several human players face-to-face in the physical space. In this paper, we focus on how to solve PJ…
Explaining Puzzle Solutions in Natural Language: An Exploratory Study on 6x6 Sudoku
The success of Large Language Models (LLMs) in human-AI collaborative decision-making hinges on their ability to provide trustworthy, gradual, and tailored explanations. Solving complex puzzles, such as Sudoku, offers a …
Decision MakingDifficulty Rating of Sudoku Puzzles: An Overview and Evaluation
How can we predict the difficulty of a Sudoku puzzle? We give an overview of difficulty rating metrics and evaluate them on extensive dataset on human problem solving (more then 1700 Sudoku puzzles, hundreds of solvers).…
Using Small MUSes to Explain How to Solve Pen and Paper Puzzles
In this paper, we present Demystify, a general tool for creating human-interpretable step-by-step explanations of how to solve a wide range of pen and paper puzzles from a high-level logical description. Demystify is bas…
Solving Witness-type Triangle Puzzles Faster with an Automatically Learned Human-Explainable Predicate
Automatically solving puzzle instances in the game The Witness can guide players toward solutions and help puzzle designers generate better puzzles. In the latter case such an Artificial Intelligence puzzle solver can in…