This blog written by: Jay Chen, Asher Davila Loranca, Brody Kutt, Haozhe Zhang, Yiheng An, Yu Fu, Qi Deng, Royce Lu
Technical editors: Nicole Nichols, Sam Kaplan, and Aryn Pedowitz
The rapid evolution of Generative AI (GenAI) brings about a double-edged sword. While revolutionizing industries with its transformative potential, it simultaneously presents significant security challenges. Cybercriminals are increasingly leveraging GenAI, exploiting its capabilities to enhance their malicious activities. This rise in GenAI-powered attacks enables adversaries to streamline and automate each stage of the Cyber Kill Chain, from reconnaissance and phishing to code generation and malware deployment. The result? Attacks are becoming more effective, harder to detect, and occurring at an alarming rate, scale, and sophistication.
Defenders must proactively combat this growing threat by turning the tables and utilizing GenAI to their own advantage. This includes leveraging GenAI to expand their security coverage, accelerate threat detection and response times, and develop more robust, proactive defense mechanisms. A key advantage for defenders lies in their access to high-quality, domain-specific data. By effectively collecting, processing, and feeding this data to GenAI models, organizations can empower their defenses with precise, context-aware decision-making capabilities.
The cybersecurity landscape will continue its rapid evolution as both attackers and defenders harness the power of GenAI. To stay ahead of the curve, organizations must prioritize continuous innovation and adaptation of their security strategies. Solutions like those from Palo Alto Networks, which are powered by Precision AI, offer a proactive approach, leveraging AI to anticipate, detect, and mitigate threats more efficiently, thus maintaining a robust security posture against the escalating threat of AI-driven cyber attacks.
The rollout of our Secure AI by Design product portfolio has begun. If you want to see how we can help secure AI applications, please see the Palo Alto Networks Can Help section below.
This blog concludes our GenAI Security Framework series, shifting our focus from defense to offense. While previous installments explored securing GenAI models, applications, and infrastructure, this piece delves into the adversarial potential of GenAI, examining how malicious actors can exploit this technology.
As a refresher, our series laid out a comprehensive GenAI security framework:
Building on this foundation, we now shift our focus to the attackers. In this blog, we will explore how GenAI can be weaponized to both accelerate traditional attacks and enable new forms of cyber threats. We will examine emerging attacks powered by GenAI and discuss how this technology can be abused to facilitate each stage of the Cyber Kill Chain.
Like any publicly available technology, GenAI can be harnessed for both positive and negative purposes. On the one hand, GenAI can effectively contextualize information, aiding in decision-making processes and generating beneficial content for society. On the other hand, it can also be exploited for malicious activities. GenAI has the potential to generate misinformation, craft phishing schemes, synthesize realistic malign images or videos, and even produce malicious code. Research has also shown that GenAI can be used to perform penetration testing, further demonstrating its capability to automate aspects of cyber offenses.
In mid-2023, WormGPT began surfacing on various cybercriminal forums. WormGPT is a chatbot service similar to ChatGPT but without safety restrictions. Its creator claims WormGPT was specially fine-tuned to assist users in generating malicious content, including malware, hate speech, and misinformation. Researchers have successfully used it to create convincing phishing texts for business email compromise (BEC). Following WormGPT, Evil-GPT, and Fraud-GPT emerged on the Dark Web, both claiming to be superior versions of WormGPT.
Microsoft and OpenAI revealed that multiple nation-state cybercriminal groups, including Forest Blizzard (Russia), Emerald Sleet (North Korea), Crimson Sandstorm (Iran), Charcoal Typhoon (China), and Salmon Typhoon (China), attempted to use OpenAI services for malicious purposes throughout 2023. These threat actors utilized LLMs for tasks such as understanding satellite technologies, crafting phishing content, and writing scripts.
In early 2024, CNN reported that a finance worker at a multinational firm was tricked into transferring $25 million to scammers who used a type of GenAI technology called deepfakes to impersonate the company’s CFO during a video call.
A plethora of academic research has also demonstrated the possibilities of using GenAI to carry out cyber attacks. Researchers from UIUC showed that LLM agents can autonomously identify and exploit vulnerabilities on a website without prior knowledge and human intervention. The same group of researchers developed a multi-agent framework with teams of LLM agents that successfully exploited zero-day and one-day vulnerabilities in real-world systems. Researchers from Nanyang Tech developed an LLM-powered penetration testing framework, PENTESTGPT, which successfully completed various Capture The Flag (CTF) challenges. Additionally, the CyberSecEval2 benchmark from Meta demonstrates that the most recent LLM models, such as GPT-4, which are granted access to Python and other code interpreters to execute code directly generated by the LLM in response to a prompt, are more effective at exploiting vulnerabilities.
The speed and scale of LLM-powered cyber attacks will continue to rise. Before we bolster our IT infrastructure to defend against more intense attacks, it is imperative to understand how threat actors could leverage GenAI to empower their arsenals. In the next section, we will outline how threat actors may utilize GenAI to facilitate cyber attacks.
The Cyber Kill Chain is a framework developed by Lockheed Martin to delineate the various stages of a cyber attack. These stages include Reconnaissance, Weaponization, Delivery, Exploitation, Installation, Command and Control (C2), and Actions on Objectives.
This framework aids organizations in understanding and mitigating cyber threats by breaking down the attack process into distinct, manageable phases. By mapping attacks to these stages, security teams can more effectively prevent, detect, and respond to potential breaches.
Given the rapidly advancing capabilities of GenAI and the real threats it has posed, we envision that GenAI could one day act as the “mastermind” that plans, directs, and executes a cyber attack. The subsections below dive into each stage of the attack chain to illustrate how the use of GenAI can facilitate each stage of the operation. As part of our analysis, we have not directly addressed the "Exploitation" stage, principally because successful exploitation is dependent on the execution of the planning and preparation phases which are extensively covered below.
Reconnaissance is the initial phase where the attacker gathers information about the target. It involves researching the organization to identify potential entry points, vulnerabilities, and valuable data. Common techniques used in this stage include scanning networks, harvesting employee names and emails, and collecting open-source intelligence. The goal is to understand the target's defenses and weaknesses to plan the attack.
GenAI's Impact: GenAI can automate and enhance reconnaissance through:
During the weaponization stage, attackers leverage the information gathered to create a tailored exploit. This could involve crafting a phishing email with a malicious attachment, building a specific malware payload, or exploiting a known software vulnerability.
GenAI's Impact: GenAI can make weaponization faster and more effective:
The delivery stage is where the attacker transmits the weaponized payload to the target. Common delivery methods include phishing emails, malicious websites, or infected USB drives. This phase represents the point at which the attacker enters the target’s environment.
GenAI's Impact: GenAI can make delivery methods more convincing and harder to detect:
In the installation phase, the attacker establishes a foothold in the target’s environment by installing malware. This often involves installing backdoors, creating new user accounts, or modifying system settings to maintain access. The malware is typically designed to remain undetected while providing ongoing control over the compromised system.
GenAI's Impact: GenAI can create stealthier and more persistent malware:
With persistence successfully accomplished at the installation stage, attackers need to establish a channel for remote communication with the compromised system. This allows them to send commands, exfiltrate data, and potentially move laterally within the network.
GenAI's Impact: GenAI can make command and control (C2) communication more resilient and harder to detect:
The final stage involves the attacker taking actions to achieve their ultimate goals, which could involve data theft, system destruction, or further network compromise. This phase is where the attacker extracts valuable information, disrupts operations, and conducts fraudulent transactions.
GenAI's Impact: GenAI can automate and optimize the attacker's objectives:
The emergence of GenAI in the cybersecurity landscape has ushered in a new era of potential threats, significantly lowering the entry barrier for cyber attacks and accelerating every stage of the cyber kill chain. From reconnaissance to command and control, GenAI's capabilities empower attackers to operate at unprecedented scales, potentially targeting thousands of victims simultaneously. As these AI-powered attacks grow more adaptive and sophisticated, evolving in tandem with GenAI's advancing capabilities, the cybersecurity community faces an urgent need to bolster its defenses.
GenAI technologies present substantial challenges to traditional cybersecurity measures. Attackers can leverage AI to rapidly adapt their tactics, evade detection through advanced obfuscation techniques, and exploit vulnerabilities across different environments. This adaptability makes conventional security approaches increasingly inadequate.
To counter this escalating threat landscape, it is crucial to employ AI for defensive purposes. The concept of "fighting AI with AI" has never been more pertinent, as organizations strive to outsmart adversaries with more intelligent and precise AI-driven solutions. By harnessing AI defensively, organizations can improve their ability to detect, prevent, and mitigate emerging threats in real time.
For instance, Palo Alto Networks' proprietary AI system, Precision AI, combines machine learning, deep learning, GenAI, and other AI techniques to predict and block attacks. It draws on extensive security datasets collected from cloud, endpoint, and network sources. Specifically, Palo Alto Networks operates a series of AI-based detection engines within its Advanced Threat Prevention (ATP) cloud to analyze traffic for advanced C2 and malware threats in real time, providing protection against zero-day threats. AI-powered ATP can stop 22% more unknown malware in real time than traditional methods. Additionally, the cloud-delivered WildFire malware analysis service leverages data and threat intelligence from the industry’s largest global community, applying advanced AI analysis to automatically identify unknown threats and halt attackers. Palo Alto Networks’ Advanced URL Filtering uses inline cloud-based deep learning detectors to assess suspicious web content in real time, protecting users from zero-day threats by detecting evasion techniques, targeted attacks, and new, unknown web-based threats.
The multifaceted AI-powered defense strategy represents a crucial step in staying ahead of the evolving threat landscape shaped by GenAI. By leveraging the power of AI, we can enhance our cybersecurity defenses and better protect against the sophisticated threats posed by AI-powered attackers.
The rollout of our Secure AI by Design product portfolio has begun.
We can help you solve the problem of protecting your GenAI infrastructure with AI Runtime Security that is available today. AI Runtime Security is an adaptive, purpose-built solution that discovers, protects, and defends all enterprise applications, models, and data from AI-specific and foundational network threats.
AI Access Security secures your company’s GenAI use and empowers your business to capitalize on its benefits without compromise.
Prisma® Cloud AI Security Posture Management (AI-SPM) protects and controls AI infrastructure, usage and data. It maximizes the transformative benefits of AI and LLMs without putting your organization at risk. It also gives you visibility and control over the three critical components of your AI security — the data you use for training or inference, the integrity of your AI models, and access to your deployed models.
These solutions will help enterprises navigate the complexities of Generative AI with confidence and security.
