This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Community Hub.
Hello, everyone. I am writing this blog using a Generative AI (GenAI) assistant to boost my productivity. My assistant can access older documents I have written and rephrase them into blog material. While granting the assistant access to my documents significantly boosts my work efficiency, it raises important security questions. Can the assistant be used by an attacker to exfiltrate my documents? In this blog, we will show how to safely grant a GenAI based application access to sensitive or user data while lowering the risk of such data being leaked.
Introduction
If you are reading this, you are already involved in the development or operation of a Gen AI based application. As development of applications powered by these advanced AI tools surges, offering unprecedented capabilities in processing and generating human-like content, so does the rise of security and privacy concerns. Among those, the biggest risk is exploiting those tools for leaking sensitive data or performing unauthorized actions, putting the organization under business and legal risk. So, a critical aspect you must address in your application is the prevention of information leaks and unauthorized API access caused due to weaknesses in your Gen AI app.
This blog post delves into the best practices for securely architecting Gen AI-based applications, ensuring they operate within the bounds of authorized access and maintain the integrity and confidentiality of sensitive data.
Understanding the Risks
Gen AI applications inherently require access to diverse data sets to process requests and generate responses. This requirement spans general to highly sensitive data, contingent on the application's purpose and scope. Without careful architectural planning, these applications could inadvertently facilitate unauthorized access to confidential information or privileged operations.
The primary risks involve:
- Information Leaks: Unauthorized access to sensitive data through the exploitation of the application's features.
- Escalated Privileges: Unauthorized access elevation, enabling attackers or unauthorized users to perform actions beyond their standard permissions by assuming the Gen AI application identity.
Mitigating these risks necessitates a security-first mindset in the design and deployment of Gen AI-based applications.
Best Practice for Granting Permissions
Limit Application Permissions
Developers should operate under the assumption that any data or functionality accessible to the application can potentially be exploited by users through carefully crafted prompts. This includes reading fine-tunning data or grounding data and performing API invocations. Recognizing this, it is crucial to meticulously manage permissions and access controls around the Gen AI application, ensuring that only authorized actions are possible.
A fundamental design principle involves strictly limiting application permissions to data and APIs. Applications should not inherently access segregated data or execute sensitive operations. By constraining application capabilities, developers can markedly decrease the risk of unintended information disclosure or unauthorized activities. Instead of granting broad permission to applications, developers should utilize user identity for data access and operations.
Utilizing User Identity for Data Access and Operations
Access to sensitive data and the execution of privileged operations should always occur under the user's identity, not the application. This strategy ensures the application operates strictly within the user's authorization scope. By integrating existing authentication and authorization mechanisms, applications can securely access data and execute operations without increasing the attack surface.
Examples of Insecure Practices
Here are a few examples of careless practices that can lead to data breach:
- Placing sensitive data in training files is used for fine-tuning models, as such data could be later extracted through sophisticated prompts.
- Using the application identity to access segregated grounding data found in vector databases, APIs, files, or any other sources. Such practice should be limited to data that should be available to all application users, as users with access to the application can craft prompts to extract any such information.
- Granting application identity permissions to perform segregated operations, like reading or sending emails on behalf of users, reading, or writing to an HR database or modifying application configurations. Calling segregating API without verifying the user permission can lead to security or privacy incidents.
To mitigate risk, always implicitly verify the end user permissions when reading data or acting on behalf of a user. For instance, in scenarios that require data from a sensitive source, like user emails or an HR database, the application should employ the user’s identity for authorization, ensuring that users view data they are authorized to view.
Putting it into effect
In the diagram below we see an application which utilizes for accessing resources and performing operations. Users’ credentials are not checked on API calls or data access. This creates a security risk where users without permissions can, by sending the “right” prompt, perform API operation or get access to data which they should not be allowed for otherwise.
Let us remove those potential risks by explicitly validating user permission to APIs and data using OAuth. For this, my favorite approach is leveraging libraries like Semantic Kernel or LangChain. These libraries enable developers to define "tools" or "skills" as functions the Gen AI can opt to use for retrieving additional data or executing actions. Such tools can utilize OAuth to authenticate on behalf of the end-user, mitigating security risks while enabling applications to process user files intelligently. In the new design below, we remove sensitive data from finetune and static grounding. All sensitive data or segregated APIs are accessed by a LangChain / SemanticKernel tool which passes the OAuth token for explicit validation or users’ permissions.
Using Microsoft AI search for grounding
As an alternative, Microsoft provides an out of the box solution for user authorization when accessing grounding data by leveraging Microsoft AI search. You are invited to learn more at Using your data with Azure OpenAI securely - Azure OpenAI | Microsoft Learn.
Conclusion
The integration of Gen AIs into applications offers transformative potential, but it also introduces new challenges in ensuring the security and privacy of sensitive data. By adhering to the best practices outlined above, developers can architect Gen AI-based applications that not only leverage the power of AI but do so in a manner that prioritizes security.