Leveraging LLMs to support co-evolution between definitions and instances of textual DSLs: A Systematic Evaluation
Software languages evolve over time for reasons such as feature additions. When grammars evolve, textual instances that originally conformed to them may become outdated. While model-driven engineering provides many techniques for co-evolving models with metamodel changes, these approaches are not designed for textual DSLs and may lose human-relevant information such as layout and comments. This study systematically evaluates the potential of large language models (LLMs) for co-evolving grammars and instances of textual DSLs. Using Claude Sonnet 4.5 and GPT-5.2 across ten case languages with ten runs each, we assess both correctness and preservation of human-oriented information. Results show strong performance on small-scale cases ($\geq$94% precision and recall for instances requiring fewer than 20 modified lines), but performance degraded with scale: Claude maintains 85% recall at 40 lines, while GPT fails on the largest instances. Response time increases substantially with instance size, and grammar evolution complexity and deletion granularity affect performance more than change type. These findings clarify when LLM-based co-evolution is effective and where current limitations remain.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Leveraging LLMs to support co-evolution between definitions and instances of textual DSLs
Software languages evolve over time for various reasons, such as the addition of new features. When the language's grammar definition evolves, textual instances that originally conformed to the grammar become outdated. F…
Bias Amplification in Language Model Evolution: An Iterated Learning Perspective
With the widespread adoption of Large Language Models (LLMs), the prevalence of iterative interactions among these models is anticipated to increase. Notably, recent advancements in multi-round self-improving methods all…
Language ModelingLanguage ModellingDe-jargonizing Science for Journalists with GPT-4: A Pilot Study
This study offers an initial evaluation of a human-in-the-loop system leveraging GPT-4 (a large language model or LLM), and Retrieval-Augmented Generation (RAG) to identify and define jargon terms in scientific abstracts…
Language ModelingLanguage ModellingLarge Language ModelRAG+2TEGEE: Task dEfinition Guided Expert Ensembling for Generalizable and Few-shot Learning
Large Language Models (LLMs) exhibit the ability to perform in-context learning (ICL), where they acquire new tasks directly from examples provided in demonstrations. This process is thought to operate through an implici…
Continual LearningDefinition ExtractionFew-Shot LearningIn-Context Learning+1An evolutionary approach to Function
Background: Understanding the distinction between function and role is vexing and difficult. While it appears to be useful, in practice this distinction is hard to apply, particularly within biology. Results: I take an…