AI-Based Detection of Hallucinated and Vulnerable Code Generated by LLMs

Shayma Mustafa Mohi-Aldeen

Abstract


The adoption of Large Language Models (LLMs) for code generation has grown, but there are two risks associated with the outputs of these models: code hallucination, which occurs when a model generates code for packages, modules, methods, or attributes that do not exist, and insecure code generation, which happens when a model produces insecure programming practices. This study introduces a lightweight, dependency-free desktop application named LLM Code Auditor to detect hallucinated software references and security vulnerabilities within the generated Python code by LLMs. It takes into account code written with LLM-based coding assistants, such as ChatGPT, Claude, and GitHub Copilot. A set of five representative Python code snippets were gathered for authentication, data retrieval operations, loading of configuration files, backup operations, and utility functions. The prompts were crafted to match realistic programming tasks and include references to API usage, dependency, and security-sensitive operations related to these application scenarios. A set of collected samples has been assessed with two complementary validation mechanisms. A first application of runtime introspection was used to check whether modules are imported or attributes are referenced in the target python environment. Second, a source code parser to find insecure constructs was used to create an AST (Abstract Syntax Tree) which was then passed to a rule analyzer that mapped the found vulnerabilities to Common Weakness Enumeration (CWE) identifiers and the severity level. The evaluation used quantitative measures such as the number and distribution of the vulnerabilities detected, the hallucination findings, CWE classification, level of confidence and a composite risk score, ranging from 0 to 100. The experimental findings were shown to be able to detect 14 vulnerability findings in 5 categories of CWE and 2 hallucination findings: One Nonexistent Package and One Unsupported Method. The mean risk score for the five samples was 54.4. The results show that runtime verification and explainable static analysis are a useful tool to investigate the reliability and security issues within LLM-generated Python code prior to deployment, and that they are complementary.

Keywords


AI code generation; code hallucination; large language models; secure coding; vulnerability detection

Full Text:

PDF

References


B. Radenkovic, S. Prokhorov, M. Radenkovic, and A. Labus, Application of Large Language Models in Software Development: Review of the Current State and Development Perspectives. 2025.

A. Patel, K. Z. Sultana, and B. Samanthula, A Comparative Analysis between AI Generated Code and Human Written Code: A Preliminary Study. 2024.

C. Gao et al., A Systematic Literature Review of Code Hallucinations in LLMs: Characterization, Mitigation Methods, Challenges, and Future Directions for Reliable AI. 2025.

Y. Lee, J. Song, D. Kim, J. Kim, M. Kim, and J. Nam, Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges. 2025.

M. Kharma, S. Choi, M. .Alkhanafseh, and D. Mohaisen, Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysis. 2025.

S. Ben Yaala and R. Bouallegue, “Vulnerability Detection in Large Language Models: Addressing Security Concerns,” J. Cybersecurity Priv., Vol. 5, No. 3, 2025.

F. Liu et al., “Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code,” IEEE Trans. Softw. Eng., Vol. 52, pp. 1037–1055, Jan. 2026.

M. Iavich, T. Kuchukhidze, and A. Lopata, “Static Analysis Techniques for Embedded, Cyber-Physical, and Electronic Software Systems: A Comprehensive Survey,” Electronics, Vol. 15, No. 5, 2026.

H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions,” in 2022 IEEE Symposium on Security and Privacy (SP), 2022, pp. 754–768.

N. Perry, M. Srivastava, D. Kumar, and D. Boneh, “Do Users Write more Insecure Code with AI Assistants ?,” No. 8, 2023.

G. Sandoval, H. Pearce, T. Nys, R. Karri, S. Garg, and B. Dolan-Gavitt, “Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants,” in 32nd USENIX Security Symposium (USENIX Security 23), 2023, pp. 2205–2222.

J. Spracklen, R. Wijewickrama, A. H. M. N. Sakib, A. Maiti, B. Viswanath, and M. Jadliwala, “We have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating {LLMs},” in 34th USENIX Security Symposium (USENIX Security 25), 2025, pp. 3687–3706.

A. Raj and S. Sahu, “Names can Hurt : Spotting Slopsquatting Risks caused by Package Name Hallucinations in Local Coding LLMs,” 2026.

S. G. Morkonda, M. Selim, and H. Assal, “Security of LLM-Generated Code : A Comparative Analysis .,” arXiv:2605.23091, 2025.

S. S. Aldosari and L. S. Aldawsari, “Securing LLM Code Generation: Leveraging Prompt Engineering to Mitigate Vulnerabilities Across Models and Languages,” SCI. Comput. Program., Vol. 251, p. 103446, 2026.

C. Negri-Ribalta, R. Geraud-Stewart, A. Sergeeva, and G. Lenzini, “A Systematic Literature Review on the Impact of AI Models on the Security of Code Generation,” Front. Big Data, Vol. Volume 7-2024, 2024.

Z. Li, D. Zou, S. Xu, X. Ou, and H. Jin, “VulDeePecker : A Deep Learning-based System for Vulnerability Detection,” arXiv:1801.01681, No. February, 2018.

Z. Li, D. Zou, S. Xu, H. Jin, Y. Zhu, and Z. Chen, “SySeVR: A Framework for using Deep Learning to Detect Software Vulnerabilities,” IEEE Trans. Dependable Secur. Comput., Vol. 19, No. 4, pp. 2244–2258, 2022.

Y. Zhou, S. Liu, J. Siow, X. Du, and Y. Liu, “Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks,” in Advances in Neural Information Processing Systems, 2019, Vol. 32.

M. Fu and C. Tantithamthavorn, “LineVul: A Transformer-based Line-Level Vulnerability Prediction,” in Proceedings - The 2022 Mining Software Repositories Conference, MSR 2022, 2022, pp. 608–620.

W. P. Konan, A. Da, and S. College, “WildCode Revisited : A Comprehensive Empirical Study on the Security of LLM-Generated Code,” arXiv:2512.04259, pp. 1–38, 2026.

V. Belozerov, P. J. Barclay, and A. Sami, “Secure Coding with AI – from Detection to Repair,” Empir. Softw. Eng., Vol. 31, No. 4, p. 93, 2026.

D. Cotroneo, C. Improta, and P. Liguori, “Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity,” in 2025 IEEE 36th International Symposium on Software Reliability Engineering (ISSRE), 2025, pp. 252–263.




DOI: https://doi.org/10.32520/stmsi.v15i11.7068

Article Metrics

Abstract view : 0 times
PDF - 0 times

Refbacks

  • There are currently no refbacks.


Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.