Pop Quiz! Can a Large Language Model Help With Reverse Engineering?
Large language models (such as OpenAI's Codex) have demonstrated impressive zero-shot multi-task capabilities in the software domain, including code explanation. In this work, we examine if this ability can be used to help with reverse engineering. Specifically, we investigate prompting Codex to identify the purpose, capabilities, and important variable names or values from code, even when the code is produced through decompilation. Alongside an examination of the model's responses in answering open-ended questions, we devise a true/false quiz framework to characterize the performance of the language model. We present an extensive quantitative analysis of the measured performance of the language model on a set of program purpose identification and information extraction tasks: of the 136,260 questions we posed, it answered 72,754 correctly. A key takeaway is that while promising, LLMs are not yet ready for zero-shot reverse engineering.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingLarge Language ModelSimilar Papers 제목 키워드 기반
QuizRank: Picking Images by Quizzing VLMs
Images play a vital role in improving the readability and comprehension of Wikipedia articles by serving as `illustrative aids.' However, not all images are equally effective and not all Wikipedia editors are trained in …
Literature online quiz system project report.
Online Quiz System is a web-based quiz system for accessing students. It is a system by which students can sit in a quiz which need no pencil and paper. Nowadays, students take quiz manually .Lecturers need spend more ti…
QUENCH: Measuring the gap between Indic and Non-Indic Contextual General Reasoning in LLMs
The rise of large language models (LLMs) has created a need for advanced benchmarking systems beyond traditional setups. To this end, we introduce QUENCH, a novel text-based English Quizzing Benchmark manually curated an…
BenchmarkingCommon Sense ReasoningWorld KnowledgeQuiz Design Task: Helping Teachers Create Quizzes with Automated Question Generation
Question generation (QGen) models are often evaluated with standardized NLG metrics that are based on n-gram overlap. In this paper, we measure whether these metric improvements translate to gains in a practical setting,…
Question GenerationQuestion-GenerationReading ComprehensionSQUARES: A SQL Synthesizer Using Query Reverse Engineering
Nowadays, many data analysts are domain experts, but they lack programming skills. As a result, many of them can provide examples of data transformations but are unable to produce the desired query. Hence, there is an in…
Enumerative SearchProgram SynthesisSQL Synthesis