This blog written in collaboration with: Haozhe Zhang, Brody Kutt, Yiheng An, Yu Fu, Qi Deng, and Royce Lu.
Generative AI (GenAI) is redefining possibilities across diverse industries, including education, entertainment, marketing, legal, and healthcare. While these sectors already face varied security risks, the integration of GenAI introduces novel challenges and compounds existing complexities.
The infrastructure that underpins GenAI is different from other modern architectures in three key ways:
In this context, four essential elements of infrastructure security are critical: Access Management, Insecure Plugins, Supply Chain Attacks, and Model Denial-of-Service attacks. In the following segments, we will discuss each in their traditional context and highlight the unique needs of GenAI systems.
In modern system architectures, issues related to access management always take precedence as the concerns that security engineers want to address. This is because the aftermath of an access management failure can be unpredictable, and can depend on what components are included in the system architecture. When there is a database or filesystem, improper access management may lead to information leaks. When there are command dispatchers and task executors, improper access management may lead to malicious command execution. As we will discuss, these concerns also apply to GenAI system infrastructure.
Typical GenAI system infrastructure usually contains web services, storage (including filesystem, database, and cache), network services (like load balancers, proxies, and API gateways), and computational resources for processing and generating responses. These components cohesively work together to generate each image or word.
While data flows throughout the system, if access to any of these components is inadequately managed, it could potentially expose sensitive data or allow unauthorized users to manipulate the system's behavior. This can lead to severe security breaches or system misuse. For example, unauthorized access to the database could result in the leakage of confidential training data, while compromised network services might allow attackers to intercept or alter data in transit. Additionally, insufficient access controls on computational resources could enable malicious actors to deploy malware or backdoor, severely impacting the system's security and reliability.
Access management issues can be addressed or mitigated with appropriate security frameworks, like legacy access management principles or the implementation of a zero trust security framework. In GenAI systems, the security models and zero trust framework should be tailored to enforce strict verification for every access request, and to limit access to only what's necessary. This method enhances security by closely monitoring who gets in and what they can do, effectively reducing the risk of breaches.
In traditional access management frameworks, there are several legacy access control models that provide options for securing systems and data. These techniques still prove useful in GenAI applications. They have stood the test of time, though they are not a silver bullet. As AI system technology continues to progress, these more traditional access control models may need to give way to new frameworks like the zero trust model. However, they are still valuable mechanisms to regulate access control.
Zero Trust is a practical security framework that addresses access management challenges in GenAI systems. Its core philosophy assumes that no entity should be automatically trusted, regardless of whether it is inside or outside the network perimeter. This philosophy is especially relevant in GenAI environments, where the dynamic nature of AI operations and data flows requires a more adaptive and vigilant approach to security.
A properly implemented Zero Trust Architecture (ZTA) usually implements the below principles to ensure better access management:
Insecure plugins are another critical concern that must be addressed for the infrastructure security of GenAI systems. Some people have access to GenAI systems such as ChatGPT and Bard but have no idea how they could maximize the magnificent power of GenAI to benefit their lives. Plugins or extensions make this easy for them. Plugins are being created in increasing numbers and with an increasingly important role in the GenAI ecosystem. However, each plugin comes with varying levels of security. Additionally, the attack surface of GenAI systems continuously widens with each new plugin.
On one hand, plugins and extensions play an important role in GenAI ecosystems by extending GenAI's capabilities. For example, one of ChatGPT plugins released by the famous travel technology company, Expedia, uses LLMs to make travel plans in an innovative way. On the other hand, the integration of insecure plugins into GenAI systems introduces additional risks. The risks have a large variety of outcomes, ranging from information disclosure to remote code execution.
One example is related to a security flaw that previously existed in Google Workspace extension of Google Bard. Researchers managed to exploit the flaw and implement data exfiltration. In two blogs , the researchers demonstrated how the extension can be used to perform unauthorized actions. Specifically, they showed how the extension could read sensitive information from the user's system and send it to an untrusted location. This underscores the potential risks to users posed by such plug-ins, which can serve as vectors for unauthorized data access and transmission.
In another example, researchers from Salt Security conducted a series of investigations on ChatGPT's plugins and identified several vulnerabilities. These vulnerabilities include the potential for malicious plugin installation and account takeover. These plugins can act as masking vectors, potentially exposing user data or compromising user accounts.
In addition, researchers from Washington University in St. Louis and University of Washington shared some of their research on an evaluation framework for OpenAI's ChatGPT plugins. In their paper, they not only put forward a methodology with a framework that helps the security evaluation of LLM plugin ecosystems but also analyzed some attack surfaces with real-world test cases.
For plugin developers and maintainers:
For GenAI system maintainer:
GenAI systems usually feature complex components, libraries, and software dependencies. The intricate web of dependencies introduces a significant risk in the form of supply chain attacks.
Supply chain attacks on legacy infrastructures usually occur on familiar features like database management systems and network components. GenAI systems, while also often suffering from security flaws on those components, could also be compromised through additional, less familiar ways. Below are some common examples.
Detailed further in this blog, ChatGPT was once impacted by a security issue in a 3rd party library, leading to information disclosure vulnerability. In March 2023, an information disclosure vulnerability in ChatGPT was disclosed. This vulnerability allowed users to access other users' messages. This is known as Horizontal Privilege Escalation. This was because of an improper implementation logic in py-redis, a 3rd party library that works as a connector between the Python program and Redis servers.
Below are some helpful strategies to safeguard GenAI systems against supply chain security concerns:
Denial-of-Service attacks, occurring in different contexts or systems, are typically conducted through various methods of exploitation. However, the basic idea behind these methods is similar—to create or direct enough noise or signal to particular parts of a network to freeze or crash their operation, making certain resources unavailable to users. These resources can include websites, services, or applications.
In the context of GenAI systems, which include various AI models, Model Denial-of-Service refers to an attack method that, when interfering with large language models (LLMs), consumes such a high amount of resources that it reduces the performance or increases the cost of the model services.
Denial-of-Service can occur when context expansion happens and consumes excessive calculation or network resources.
Security researchers with Dropbox published a blog to discuss their research on OpenAI's GPT models. In their research, they conducted expensive repeat requests by crafting a prompt asking GPT to repeat certain phrases forever, and as a consequence, they observed abnormally long response times with a large amount of output. Thus, the researchers believe that denial-of-service is possible because it is easy to find short prompts that will generate a full context window of output.
Other than the DoS caused by excessive resource consumption, glitch tokens can also be utilized to conducted DoS when they are processed by GenAI models.
In this article, researchers discovered a set of anomalous tokens that trigger a previously undocumented failure mode in GPT-2 and GPT-3 models. These tokens, when used in prompts, lead to unusual and often bizarre completions, contradicting the models' intended functions. Based on their findings, we can assume that if these glitch tokens are inserted into the source of models and once the RAGs vector database is updated, the tokens will be added to the information store and may cause unexpected behaviors of LLMs.
Mitigation of Model Denial-of-Service is not easy due to the various unique exploitations that exist. Below are some effective ways in response to the model DoS attacks.
The security of GenAI systems' infrastructure might seem like a brand new frontier. However, it shares common ground with traditional infrastructure security concepts. This scenario is reminiscent of putting new wine into old bottles. The essence remains timeless, yet the context has shifted.
Palo Alto Networks closely keeps pace with these evolving trends and responds rapidly to new security challenges, and provides a series of products for protection towards different aspects of GenAI system infrastructure.
These solutions will help enterprises navigate the complexities of Generative AI with confidence and security.
