An Assurance-Oriented Retrieval-Augmented Generation Framework for ISO 27001 and IT General Controls Audit Evaluation

Abstract

The increasing use of information systems in organizations has made information security and compliance auditing more important than ever. Auditors often need to review large volumes of policies, standards, and control documentation to identify relevant controls and supporting evidence. This process can be time-consuming and challenging, particularly when multiple frameworks are involved. In addition, although Large Language Models (LLMs) can assist in information retrieval, they may generate inaccurate or unsupported responses, which limits their reliability in audit environments. To address these challenges, this research proposes an assurance-oriented Retrieval-Augmented Generation (RAG) framework for ISO/IEC 27001 and Information Technology General Controls (ITGC) audit evaluation. The framework was developed to retrieve relevant information from compliance documents and provide responses based on retrieved evidence rather than relying solely on model knowledge. A hybrid retrieval approach combining semantic and keyword-based search techniques was implemented, and a proof-of-concept web application was developed to demonstrate the framework. The proposed framework was evaluated using both qualitative and quantitative approaches. Quantitative evaluation was performed using the RAGAS framework on twenty audit-related queries with GPT-4o Mini and Gemini 2.5 Flash. The findings showed that the framework was able to retrieve relevant information and generate evidence-based responses that support audit and compliance activities. This study is limited to ISO/IEC 27001 and ITGC documents. Future research may extend the framework to additional compliance standards, larger datasets, and real-world audit environments.