Enhanced Security Measures in Place: To ensure a safer experience, we’ve implemented additional, temporary security measures for all users.
This blog written by Haozhe Zhang, Brody Kutt, Yiheng An, Yu Fu, Qi Deng, Royce Lu, and Scott Emo.
The rollout of our Secure AI by Design product portfolio has begun. If you want to see how we can help secure AI applications, please see the Palo Alto Networks Can Help section below.
Generative AI (GenAI) is redefining possibilities across diverse industries, including education, entertainment, marketing, legal, and healthcare. While these sectors already face varied security risks, GenAI introduces novel challenges and compounds existing complexities. With the right AI security portfolio, you should be able to secure AI by design rather than patching it afterwards.
To be able to secure AI Apps, we need to understand the infrastructure that underpins GenAI is different from other modern architectures in three key ways:
In this context, four essential elements of infrastructure security are critical for success: Access Management, Insecure Plugins, Supply Chain Attacks, and Model Denial-of-Service attacks. In the following segments, we will discuss each in their traditional context and highlight the unique needs of GenAI systems.
In modern system architectures, issues related to access management always take precedence as the concerns that security engineers want to address. This is because the aftermath of an access management failure can be unpredictable, and can depend on what components are included in the system architecture. When there is a database or filesystem, improper access management may lead to information leaks. When there are command dispatchers and task executors, improper access management may lead to malicious command execution. As we will discuss, these concerns also apply to GenAI system infrastructure.
Typical GenAI system infrastructure usually contains web services, storage (including filesystem, database, and cache), network services (like load balancers, proxies, and API gateways), and computational resources for processing and generating responses. These components cohesively work together to generate each image or word.
While data flows throughout the system, if access to any of these components is inadequately managed, it could potentially expose sensitive data or allow unauthorized users to manipulate the system's behavior. This can lead to severe security breaches or system misuse. For example, unauthorized access to the database could result in the leakage of confidential training data, while compromised network services might allow attackers to intercept or alter data in transit. Additionally, insufficient access controls on computational resources could enable malicious actors to deploy malware or backdoor, severely impacting the system's security and reliability.
Access management issues can be addressed or mitigated with appropriate security frameworks, like legacy access management principles or the implementation of a zero trust security framework. In GenAI systems, the security models and zero trust framework should be tailored to enforce strict verification for every access request, and to limit access to only what's necessary. This method enhances security by closely monitoring who gets in and what they can do, effectively reducing the risk of breaches.
In traditional access management frameworks, there are several legacy access control models that provide options for securing systems and data. These techniques still prove useful in GenAI applications. They have stood the test of time, though they are not a silver bullet. As AI system technology continues to progress, these more traditional access control models may need to give way to new frameworks like the zero trust model. However, they are still valuable mechanisms to regulate access control.
Zero Trust is a practical security framework that addresses access management challenges in GenAI systems. Its core philosophy assumes that no entity should be automatically trusted, regardless of whether it is inside or outside the network perimeter. This philosophy is especially relevant in GenAI environments, where the dynamic nature of AI operations and data flows requires a more adaptive and vigilant approach to security.
A properly implemented Zero Trust Architecture (ZTA) usually implements the below principles to ensure better access management:
Insecure plugins are another critical concern that must be addressed for the infrastructure security of GenAI systems. Some people have access to GenAI systems such as ChatGPT and Bard but have no idea how they could maximize the magnificent power of GenAI to benefit their lives. Plugins or extensions make this easy for them. Plugins are being created in increasing numbers and with an increasingly important role in the GenAI ecosystem. However, each plugin comes with varying levels of security. Additionally, the attack surface of GenAI systems continuously widens with each new plugin.
On one hand, plugins and extensions play an important role in GenAI ecosystems by extending GenAI's capabilities. For example, one of ChatGPT plugins released by the famous travel technology company, Expedia, uses LLMs to make travel plans in an innovative way. On the other hand, the integration of insecure plugins into GenAI systems introduces additional risks. The risks have a large variety of outcomes, ranging from information disclosure to remote code execution.
One example is related to a security flaw that previously existed in Google Workspace extension of Google Bard. Researchers managed to exploit the flaw and implement data exfiltration. In two blogs , the researchers demonstrated how the extension can be used to perform unauthorized actions. Specifically, they showed how the extension could read sensitive information from the user's system and send it to an untrusted location. This underscores the potential risks to users posed by such plug-ins, which can serve as vectors for unauthorized data access and transmission.
In another example, researchers from Salt Security conducted a series of investigations on ChatGPT's plugins and identified several vulnerabilities. These vulnerabilities include the potential for malicious plugin installation and account takeover. These plugins can act as masking vectors, potentially exposing user data or compromising user accounts.
In addition, researchers from Washington University in St. Louis and University of Washington shared some of their research on an evaluation framework for OpenAI's ChatGPT plugins. In their paper, they not only put forward a methodology with a framework that helps the security evaluation of LLM plugin ecosystems but also analyzed some attack surfaces with real-world test cases.
For plugin developers and maintainers:
For GenAI system maintainer:
GenAI systems usually feature complex components, libraries, and software dependencies. The intricate web of dependencies introduces a significant risk in the form of supply chain attacks.
Supply chain attacks on legacy infrastructures usually occur on familiar features like database management systems and network components. GenAI systems, while also often suffering from security flaws on those components, could also be compromised through additional, less familiar ways. Below are some common examples.
Detailed further in this blog, ChatGPT was once impacted by a security issue in a 3rd party library, leading to information disclosure vulnerability. In March 2023, an information disclosure vulnerability in ChatGPT was disclosed. This vulnerability allowed users to access other users' messages. This is known as Horizontal Privilege Escalation. This was because of an improper implementation logic in py-redis, a 3rd party library that works as a connector between the Python program and Redis servers.
Below are some helpful strategies to safeguard GenAI systems against supply chain security concerns:
Denial-of-Service attacks, occurring in different contexts or systems, are typically conducted through various methods of exploitation. However, the basic idea behind these methods is similar—to create or direct enough noise or signal to particular parts of a network to freeze or crash their operation, making certain resources unavailable to users. These resources can include websites, services, or applications.
In the context of GenAI systems, which include various AI models, Model Denial-of-Service refers to an attack method that, when interfering with large language models (LLMs), consumes such a high amount of resources that it reduces the performance or increases the cost of the model services.
Denial-of-Service can occur when context expansion happens and consumes excessive calculation or network resources.
Security researchers with Dropbox published a blog to discuss their research on OpenAI's GPT models. In their research, they conducted expensive repeat requests by crafting a prompt asking GPT to repeat certain phrases forever, and as a consequence, they observed abnormally long response times with a large amount of output. Thus, the researchers believe that denial-of-service is possible because it is easy to find short prompts that will generate a full context window of output.
Other than the DoS caused by excessive resource consumption, glitch tokens can also be utilized to conducted DoS when they are processed by GenAI models.
In this article, researchers discovered a set of anomalous tokens that trigger a previously undocumented failure mode in GPT-2 and GPT-3 models. These tokens, when used in prompts, lead to unusual and often bizarre completions, contradicting the models' intended functions. Based on their findings, we can assume that if these glitch tokens are inserted into the source of models and once the RAGs vector database is updated, the tokens will be added to the information store and may cause unexpected behaviors of LLMs.
Mitigation of Model Denial-of-Service is not easy due to the various unique exploitations that exist. Below are some effective ways in response to the model DoS attacks.
The rollout of our Secure AI by Design product portfolio has begun.
We can help you solve the problem of protecting your GenAI infrastructure with AI Runtime Security that is available today. AI Runtime Security is an adaptive, purpose-built solution that discovers, protects, and defends all enterprise applications, models, and data from AI-specific and foundational network threats.
AI Access Security secures your company’s GenAI use and empowers your business to capitalize on its benefits without compromise.
Prisma® Cloud AI Security Posture Management (AI-SPM) protects and controls AI infrastructure, usage and data. It maximizes the transformative benefits of AI and large language models without putting your organization at risk. It also gives you visibility and control over the three critical components of your AI security — the data you use for training or inference, the integrity of your AI models and access to your deployed models.
These solutions will help enterprises navigate the complexities of Generative AI with confidence and security.
