The long-awaited AI revolution has finally arrived, with OpenAI’s ChatGPT leading the charge and ushering in a wave of generative AI that has transformed the tech landscape. However, alongside this progress, we are witnessing a significant shift in the threat landscape, as these advancements bring forth new risks and challenges.
AI-Powered Attacks: Enhancing Phishing and Fraud
In an alarming turn of events, attackers have harnessed the power of AI to bolster their malicious activities, particularly in the realms of phishing and fraud. The recent leak of Meta’s 65-billion parameter language model has set the stage for the emergence of more sophisticated phishing attacks. Prompt injection attacks have become a daily occurrence, posing a considerable threat to individuals and organizations alike.
Balancing Data Security with AI/ML-based Services
The widespread adoption of AI and machine learning-based services has inadvertently created a conundrum for security teams. Users often entrust business-sensitive data to these platforms, leaving security professionals scrambling to implement robust measures for controlling and protecting the use of such services. A notable example includes Samsung engineers inadvertently leaking proprietary code by incorporating it into ChatGPT for debugging purposes. Additionally, a survey by Fishbowl revealed that a significant 68% of employees using ChatGPT for work purposes chose not to disclose this fact to their superiors, further complicating data security.
The Growing Concern of AI Misuse
The misuse of AI is a mounting concern for consumers, businesses, and even governmental bodies. Recognizing the gravity of the situation, the White House has recently announced increased investments in AI research and forthcoming assessments and policies to address the issue. As the AI revolution progresses at breakneck speed, it has become imperative to address four major classes of issues that have emerged as a consequence.
Asymmetry in the attacker-defender dynamic
Attackers will likely adopt and engineer AI faster than defenders, giving them a clear advantage. They will be able to launch sophisticated attacks powered by AI/ML at an incredible scale at low cost.
Social engineering attacks will be first to benefit from synthetic text, voice and images. Many of these attacks that require some manual effort — like phishing attempts that impersonate IRS or real estate agents prompting victims to wire money — will become automated.
Attackers will be able to use these technologies to create better malicious code and launch new, more effective attacks at scale. For example, they will be able to rapidly generate polymorphic code for malware that evades detection from signature-based systems.
One of AI’s pioneers, Geoffrey Hinton, made the news recently as he told the New York Times he regrets what he helped build because “It is hard to see how you can prevent the bad actors from using it for bad things.”
ADVERTISEMENT
Security and AI: Further erosion of social trust
We’ve seen how quickly misinformation can spread thanks to social media. A University of Chicago Pearson Institute/AP-NORC Poll shows 91% of adults across the political spectrum believe misinformation is a problem and nearly half are worried they’ve spread it. Put a machine behind it, and social trust can erode cheaper and faster.
The current AI/ML systems based on large language models (LLMs) are inherently limited in their knowledge, and when they don’t know how to answer, they make things up. This is often referred to as “hallucinating,” an unintended consequence of this emerging technology. When we search for legitimate answers, a lack of accuracy is a huge problem.
This will betray human trust and create dramatic mistakes that have dramatic consequences. A mayor in Australia, for instance, says he may sue OpenAI for defamation after ChatGPT wrongly identified him as being jailed for bribery when he was actually the whistleblower in a case.
ADVERTISEMENT
New attacks
Over the next decade, we will see a new generation of attacks on AI/ML systems.
Attackers will influence the classifiers that systems use to bias models and control outputs. They’ll create malicious models that will be indistinguishable from the real models, which could cause real harm depending on how they’re used. Prompt injection attacks will become more common, too. Just a day after Microsoft introduced Bing Chat, a Stanford University student convinced the model to reveal its internal directives.
Attackers will kick off an arms race with adversarial ML tools that trick AI systems in various ways, poison the data they use or extract sensitive data from the model.
As more of our software code is generated by AI systems, attackers may be able to take advantage of inherent vulnerabilities that these systems inadvertently introduced to compromise applications at scale.
Externalities of scale
The costs of building and operating large-scale models will create monopolies and barriers to entry that will lead to externalities we may not be able to predict yet.
In the end, this will impact citizens and consumers in a negative way. Misinformation will become rampant, while social engineering attacks at scale will affect consumers who will have no means to protect themselves.
The federal government’s announcement that governance is forthcoming serves as a good start, but there’s so much ground to make up to get in front of this AI race.
AI and security: What comes next
The nonprofit Future of Life Institute published an open letter calling for a pause in AI innovation. It got plenty of press coverage, with the likes of Elon Musk joining the crowd of concerned parties, but hitting the pause button simply isn’t viable. Even Musk knows this — he has seemingly changed course and started his own AI company to compete.
It was always disingenuous to suggest innovation should be stifled. Attackers certainly won’t honor that request. We need more innovation and more action so that we can ensure that AI is used responsibly and ethically.
The silver lining is that this also creates opportunities for innovative approaches to security that use AI. We will see improvements in threat hunting and behavioral analytics, but these innovations will take time and need investment. Any new technology creates a paradigm shift, and things always get worse before they get better. We’ve gotten a taste of the dystopian possibilities when AI is used by the wrong people, but we must act now so that security professionals can develop strategies and react as large-scale issues arise.
Conclusion:
As AI continues its rapid advancement, it is crucial to recognize the evolving threat landscape and take proactive measures to safeguard against emerging security concerns. The impact of AI-powered attacks, the challenge of balancing data security with AI-based services, and the need to address AI misuse have become paramount considerations for individuals, organizations, and governments alike. By staying vigilant and proactive, we can navigate the AI revolution while minimizing potential risks and ensuring a secure technological future.
