Take Loads Off Your Developers: Automated User Story Generation using Large Language Model
Bibliographic record
Abstract
Software Maintenance and Evolution (SME) is moving fast with the assistance of artificial intelligence (AI), especially Large Language Models (LLM). Researchers have already started automating various activities of the SME workflow. Un-derstanding the requirements for maintenance and development work i.e. Requirements Engineering (RE) is a crucial phase that kicks off the SME workflow through multiple discussions on a proposed scope of work documented in different forms. The RE phase ends with a list of user stories for each unit task and usually created and tracked on a project management tool such as GitHub, Jira, AzurDev, etc. In this research, we collaborated with Bell Mobility to develop a tool “Geneus” (Generate UserSory) using GPT-4-turbo to automatically create user stories from software requirements documents. Requirements documents are usually long and contain complex information. Since LLMs typically suffer from hallucination when the input is too complex, this paper proposes a new prompting strategy, “Refine and Thought” (RaT), to mitigate that issue and improve the performance of the LLM in prompts with large and noisy contexts. Along with manual evaluation using RUST (Readability, Understandability, Specificity, Technical-aspects) survey questionnaire, automatic evaluation with BERTScore, and AlignScore evaluation metrics are used to evaluate the results of the “Geneus” tool. Results show that our method with RaT performs consistently better in most of the cases of interactions compared to the single-shot baseline method. However, the BERTScore and AlignScore test results are not consistent. In the median case, Geneus performs significantly better in all three interactions (requirements specifi-cation, user story details, and test case specifications) according to AlignScorebut it shows slightly low performance in requirements specifications according to BERTScore. Distilling RE documents requires significant time & effort from the senior members of the team through multiple meetings with stakeholders. We believe automating this process will certainly reduce additional loads off the software engineers and increase the ultimate productivity allowing them to utilize their time on other prioritized tasks.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.015 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.001 | 0.003 |
| Open science | 0.002 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.006 | 0.004 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".