Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation
Retrieval-Augmented Generation (RAG) systems are emerging as a key approach for grounding Large Language Models (LLMs) in external knowledge, addressing limitations in factual accuracy and contextual relevance. However, there is a lack of empirical studies that report on the development of RAG-based implementations grounded in real-world use cases, evaluated through general user involvement, and accompanied by systematic documentation of lessons learned. This paper presents five domain-specific RAG applications developed for real-world scenarios across governance, cybersecurity, agriculture, industrial research, and medical diagnostics. Each system incorporates multilingual OCR, semantic retrieval via vector embeddings, and domain-adapted LLMs, deployed through local servers or cloud APIs to meet distinct user needs. A web-based evaluation involving a total of 100 participants assessed the systems across six dimensions: (i) Ease of Use, (ii) Relevance, (iii) Transparency, (iv) Responsiveness, (v) Accuracy, and (vi) Likelihood of Recommendation. Based on user feedback and our development experience, we documented twelve key lessons learned, highlighting technical, operational, and ethical challenges affecting the reliability and usability of RAG systems in practice.
Code (0)
등록된 구현이 없습니다.
Tasks
Optical Character Recognition (OCR)RAGRetrievalRetrieval-augmented GenerationSemantic RetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applications
The aim of Reinforcement Learning (RL) in real-world applications is to create systems capable of making autonomous decisions by learning from their environment through trial and error. This paper emphasizes the importan…
Autonomous Drivingreinforcement-learningReinforcement LearningReinforcement Learning (RL)Prompt Less, Smile More: MTP with Semantic Engineering in Lieu of Prompt Engineering
AI-Integrated programming is emerging as a foundational paradigm for building intelligent systems with large language models (LLMs). Recent approaches such as Meaning Typed Programming (MTP) automate prompt generation by…
Prompt EngineeringData-driven Discovery of Cyber-Physical Systems
Cyber-physical systems (CPSs) embed software into the physical world. They appear in a wide range of applications such as smart grids, robotics, intelligent manufacture and medical monitoring. CPSs have proved resistant …
Large language models for partial differential equation workflows
Partial differential equations (PDEs) become actionable in science and engineering not as isolated formulae, but as executable workflows that connect modelling assumptions, governing equations, numerical solvers, diagnos…
Teaching Undergraduate Students to Think Like Real-World Systems Engineers: A Technology-Based Hybrid Learning Approach
A hybrid teaching approach that relied on combining Project Based Learning with Team Based Learning was trialled in an engineering module during the past five years. Our motivation was to expose students to real-world au…
Decision Making