Document Type

Conference Proceeding

Publication Date

6-2025

Abstract

This exploratory paper builds on the EMMSAD 2024 paper “Could a Large Language Model Contribute Significantly to Requirements Analysis?” Eight versions of each of three LLM prompts (for system structure, analysis, and recommendations) were applied to three 3000+ word case studies. Those versions expressed different “treatments” including a control with no RAG augmentation, a version with RAG augmentation based on an analysis template used by MBA and EMBA students, and six other versions based on theoretical approaches such as activity theory, a BPM design space, work system principles, and so on. The LLM responses were somewhat reliable for summarizing system structure, less reliable for summarizing an analysis, and often generic and impractical for recommendations because the LLM did not understand contexts. This new paper adds three new capabilities: 1) RAG augmentation using a knowledge base consisting of “knowledge objects” built on work system theory, 2) application of that knowledge base using chain-of-thought reasoning, 3) inclusion of direct feedback from analysts during an analysis process in order to correct errors and to extend the prompt in new directions. Examples are used to illustrate results from applying those capabilities to 3 disparate case studies.

Comments

The final publication is available at Springer via The final publication is available at Springer via https://doi.org/10.1007/978-3-031-94569-4_2

Download

Share

COinS