Researchers from the University of Sheffield recently conducted a study that shed light on the vulnerability of popular artificial intelligence(AI) applications, such as ChatGPT and others, to potential exploitation for crafting harmful Structured Query Language (SQL) commands. Their findings indicate the possibility of launching cyber attacks and compromising computer systems using these AI applications.
The study, co-led by Xutan Peng, a PhD student, and his team, targeted Text-to-SQL systems utilized for creating natural language interfaces to databases. Their investigation included applications like BAIDU-UNIT, ChatGPT, AI2SQL, AIHELPERBOT, Text2SQL, and ToolSKE.
Peng emphasized, “Many companies are unaware of these threats, and due to the complexity of chatbots, even within the community, there are aspects not fully understood.” Despite ChatGPT being a standalone system with minimal risks to its own service, the research revealed its susceptibility to producing malicious SQL code that could cause substantial harm to other services.
The vulnerabilities found within these AI applications opened doors for potential cyber threats, allowing the exploitation of systems, theft of sensitive information, manipulation of databases, and execution of Denial-of-Service attacks, rendering machines or networks inaccessible to users.
Peng highlighted an example where individuals, including professionals like nurses, employ AI models like ChatGPT for productivity purposes, inadvertently generating harmful SQL commands that could cause severe data mismanagement in scenarios like interacting with databases storing clinical records.
Additionally, the researchers identified a concerning issue during the training of Text-to-SQL models, where they could surreptitiously embed harmful code, resembling a Trojan Horse, within the models. This “invisible” code could potentially harm users who utilize these compromised systems.
Dr. Mark Stevenson, a senior lecturer at the University of Sheffield, stressed the complexity of large language models used in Text-to-SQL systems, acknowledging their potency but also their unpredictability. The research team shared their findings with companies like Baidu and OpenAI, leading to the resolution of these vulnerabilities in their AI applications.
The study, published in arXiv, emphasizes the need to recognize and address potential software security risks associated with Natural Language Processing (NLP) algorithms. Their findings underscore the importance of exploring methods to safeguard against such exploitations in the future.
Study Abstract:
The study conducted by the University of Sheffield revealed vulnerabilities in Text-to-SQL systems within several commercial applications, showcasing the potential exploitation of Natural Language Processing models to produce malicious code. This signifies a significant security threat that could result in data breaches and Denial of Service attacks, posing a serious risk to software security. The research aims to draw attention to these vulnerabilities within NLP algorithms and encourages further exploration into safeguarding strategies to mitigate these risks.
