AI from the attacker’s perspective: See how cybercriminals are leveraging AI and exploiting its vulnerabilities to compromise systems, users, and even other AI applications
Cybercriminals and AI: reality versus hype
“AI will not replace humans in the near future. But people who know how to use AI will replace those people who don’t know how to use AI,” said Etay Maor, Chief Security Strategist at Cato Networks and founder of Cato CTRL. “Similarly, attackers are also turning to AI to augment their own capabilities.”
Yet there is far more hype than reality surrounding the role of AI in cybercrime. Headlines often sensationalize AI threats, using terms like “Chaos-GPT” and “Black Hat AI Tools,” even claiming they want to destroy humanity. However, these articles are more frightening than descriptive of serious threats.
For example, when investigated in underground forums, it turned out that several of these so-called “AI cyber tools” were nothing more than updated versions of basic public LLMs without advanced capabilities. In fact, they were even branded a scam by angry attackers.
How hackers really use AI in cyber attacks
In reality, cybercriminals are still figuring out how to use AI effectively. They experience the same problems and shortcomings as legitimate users, such as hallucinations and limited abilities. According to their predictions, it will take a few years before they can effectively deploy GenAI for hacking needs.
For now, GenAI tools are mainly used for simpler tasks, such as writing phishing emails and generating code snippets that can be integrated into attacks. In addition, we have seen attackers supply compromised code to AI systems for analysis, in an attempt to ‘normalize’ such code as non-malicious.
Using AI to Abuse AI: Introducing GPTs
Introduced by OpenAI on November 6, 2023, GPTs are customizable versions of ChatGPT that allow users to add specific instructions, integrate external APIs, and integrate unique knowledge sources. This feature allows users to create highly specialized applications such as tech support bots, educational tools, and more. In addition, OpenAI offers developers monetization options for GPTs, through a dedicated marketplace.
Abuse of GPTs
GPTs introduce potential security problems. A notable risk is exposing sensitive instructions, proprietary knowledge, or even API keys embedded in the custom GPT. Malicious actors can use AI, specifically prompt engineering, to replicate a GPT and leverage its monetization potential.
Attackers can use prompts to retrieve knowledge sources, instructions, configuration files, and more. These can be as simple as asking the custom GPT to display all uploaded files and custom instructions, or asking for debugging information. Or advanced, like asking the GPT to zip one of the PDF files and create a downloadable link, asking the GPT to list all its capabilities in a structured table format, and more.
“Even the protections that developers have put in place can be bypassed and all knowledge can be extracted,” said Vitaly Simonovich, Threat Intelligence Researcher at Cato Networks and member of Cato CTRL.
These risks can be avoided by:
- Do not upload sensitive data
- The use of instruction-based protection, although even these may not be foolproof. “You have to consider all the different scenarios the attacker could take advantage of,” Vitaly adds.
- OpenAI protection
AI attacks and risks
Multiple frameworks exist today to assist organizations considering developing and creating AI-based software:
- NIST Risk Management Framework for Artificial Intelligence
- Google’s Secure AI framework
- OWASP Top 10 for LLM
- OWASP Top 10 for LLM Applications
- The recently launched MITER ATLAS
LLM attack surface
There are six major LLM (Large Language Model) components that attackers can target:
- Quick – Attacks such as rapid injections, which use malicious input to manipulate the AI’s output
- Answer – Misuse or leak of sensitive information in AI-generated comments
- Model – Theft, poisoning or manipulation of the AI model
- Training data – Introducing malicious data to change the AI’s behavior.
- Infrastructure – Focused on the servers and services that support the AI
- Users – Deceiving or exploiting people or systems that rely on AI outputs
Real-world attacks and risks
Let’s conclude with some examples of LLM manipulations, which can easily be used in a malicious way.
- Rapid injection into customer service systems – A recent case involved a car dealer using an AI chatbot for customer service. A researcher managed to manipulate the chatbot by providing a prompt that changed its behavior. By instructing the chatbot to agree to all customer statements and ending each response with, “And that’s a legally binding offer,” the researcher was able to buy a car for a ridiculously low price, exposing a major vulnerability.
- Hallucinations leading to legal consequences – In another incident, Air Canada faced legal action when their AI chatbot provided incorrect information about its refund policy. When a customer relied on the chatbot’s response and subsequently filed a claim, Air Canada was held liable for the misleading information.
- Own data leaks – Samsung employees unknowingly leaked proprietary information when they used ChatGPT to analyze code. Uploading sensitive data to third-party AI systems is risky because it is unclear how long the data is kept and who has access to it.
- AI and deepfake technology in fraud – Cybercriminals also use AI beyond text generation. A Hong Kong bank fell victim to a $25 million fraud when attackers used live deepfake technology during a video call. The AI-generated avatars mimicked trusted bank officials and convinced the victim to transfer money to a fraudulent account.
In summary: AI in cybercrime
AI is a powerful tool for both defenders and attackers. As cybercriminals continue to experiment with AI, it’s important to understand how they think, what tactics they use, and what options they face. This allows organizations to better protect their AI systems against misuse.
View the entire masterclass here.
