As AI continues to capture everyone’s attention, security for AI becomes a popular topic in the market. Security for AI is capturing the media cycle, AI security startups are going out of stealth left and right, and incumbents scramble to release AI-relevant security features. In our small patch of the world, it is clear security teams are concerned about AI. It seems like the race has begun and we can just about see an AI Security category being formed.

But what does AI Security mean exactly?

The problem with AI capturing mindshare is that everyone finds the way to talk about their existing solution with AI language making it difficult to figure out one solution vs another.

We also frankly don’t really know what security for AI means because we don’t know what AI development means. Security for X typically arrives after X has matured, think cloud, network, web apps, .. but AI remains a moving target.

From my perspective right now, there are three distinct solution categories all claiming to be AI Security solutions. These three solve different problems for different people so I argue that these are fundamental distinctions that would not easily merge, though of course they do have some overlap.

These categories are:

1. AI DLP
2. AI Firewall
3. AI SPM / CNAPP

AI DLP

Fast to gain traction, fast to disappear (I claim).

When ChatGPT was first launched every enterprise I know went down the same route of trying desperately to block it. Every week had new headlines about companies losing their IP to AI because an employee copy-pasted highly confidential data to the chat so they could ask for a summary or a funny poem about it. This was really all anybody could talk about for a few weeks.

Point solutions to address this problem have popped up like mushrooms after heavy rain. Since you couldn’t control ChatGPT itself, and other AIs that started appearing on the consumer market, all of these solutions are different types of proxies. Whether it’s on the network layer, with a host agent or through a browser extension, AI DLP solutions promise to capture rogue users from using unapproved public AI bots and in some cases point users to approved enterprise versions like Azure OpenAI. This problem got so much attention that OpenAI, who caused the scare in the first place, changed their policies so users can now opt-out of being included in the training set and organizations can pay to opt-out on behalf of all their users.

I am bearish about AI DLP. While these solutions were quick to gain traction reacting to public emotions, I don’t see why AI DLP is fundamentally different from a regular DLP or its modern uncle, the CASB. At the end of the day, users copy-pasting sensitive data to a random website on the Internet is an old problem. Not sure why AI makes it different.

Another point about AI DLP is that it can only observe user interaction with AI and completely misses applications that use unapproved AI services.

AI Firewall

Think about SQL injection that prompted the rise of the AST industry. It is an issue with data being translated as instructions, resulting in allowing people who manipulate application data (i.e. users) to manipulate application instruction (i.e. its behavior). With years of severe issues wreaking havoc on poor web applications, application development frameworks have raised up to the challenge and now safely handle user input. If you’re using a modern framework and going through its paved road, SQL injection is for all practical purposes a solved problem.

One of the weird things about AI from an engineer’s perspective is that they mix instructions and data. You tell the AI what you want it to do with text, and then you let your users add some more text into essentially the same input. As you could expect this results in users being able to change the instructions. Using clever prompts lets you do that even if the application builder really tried to prevent it, a problem we all know today as prompt injection.

Some solutions have popped up to try and help application developers avoid prompt injection. They employ a bunch of techniques to do that including threat intelligence (i.e. a list of prompts that work), crowdsourcing and, of course, using AI to flight AI. For an application developer, this typically involves deploying a middleware that acts as a security mechanism between your application and the AI model and fails any injection attempt.

AI models have also improved their inherent resistance to these kinds of attacks, but whether this problem can ever truly be solved remains an open question.

Prompt injection is not the only concern addressed by the AI Firewall category. In fact, some companies have been working on related problems of model theft, model poisoning and model drift for several years now ever since the AI research community discovered adversarial learning. I place these under the same category because they too act as middleware between your application and the AI model, and I doubt people will deploy more than one middleware.

For AI application developers, trying to control these uncontrollable models is a real challenge. This is a security concern, but it is also a predictability and usability concern. Therefore, I believe these concerns are best served as important features of AI application development platforms.

AI SPM / CNAPP

Once you allow AI to act on the user’s behalf and chain those actions one after the other you’ve reached uncharted territory. Can you really tell if the AI is doing things it should be doing to meet its goal? If you could think of and list everything the AI might need to do then you arguably wouldn’t need AI in the first place.

Importantly, this problem is about how AI interacts with the world, and so it is as much about the world as it is about the AI. Most Copilot apps are proud to inherit existing security controls by impersonating users, but are user security controls really all that strict? Can we really count on user-assigned and managed permissions to protect sensitive data from a curious AI?

The problem here is how to build an AI application that interacts with the world in creative ways, but only the creative ways we actually want without any nasty side effects. This category is the least developed, and it is unclear if it’s even one category or if it’s a job for the application development platform or an external security solution. One thing is clear though, the first step is having a deep and precise understanding of the AI application’s environment. Which identities is it using, what components can it run, on which network, and how do they interact with other regular or AI-powered applications.

A finishing thought

Trying to say anything about where AI or by extension AI security will end up is trying to predict the future. We all know the saying, it’s difficult to make predictions, especially about the future. Let’s see how it holds up and where I’ll be most wrong.

Microsoft Copilot Studio allows users to quickly build enterprise Copilots on top of their business data. Every enterprise user can now plug enterprise data into GPT models now and share their bots with the world.

What could go wrong?

Public access to High Restricted SharePoint sites

Data leakage, to name one thing.

Here’s HackerBot. It’s a simple Copilot that allows you to enumerate and download any file for the “HR Compliance” SharePoint site which has a “High Restricted” sensitivity label. To use HackerBot you simply need to navigate to its public URL. No authentication required.

If you’d like to play around with it yourself, reach out and I’ll provide a link.

But Microsoft says this is not possible

Microsoft actually identified this scenario and have a big red warning in their Docs saying that this scenario (public user access + Copilot can access SharePoint/OneDrive) is blocked.

So what is going on here?

Analysis and workaround

When you create a new copilot you are prompted to provide a URL which can be a public site, a SharePoint site or a OneDrive directory. If you provide a SharePoint/OneDrive URL Microsoft will publish your bot but indeed suspend it you change the default configuration and switch to authenticated access.

This makes sense following the note on Microsoft Docs.

However, it is just one of many ways to provide the copilot with business data access including SharePoint. Those include Topics, Plugins, Skills to name a few.

HackerBot works with Topics. A topic is a particular thing the copilot can do for you, it can be a scripted conversation, access to resources or performing actions on your behalf. Copilot routes users to the right topic by specific trigger words (or a GenAI decision). HackerBot has a “ExfiltrateSharePoint” topic triggered by words like “Exfiltrate”, “SharePoint”, “Data” and “Confidential”.

Topics can be set to use hundreds of connectors that allow access to any business platform you can imagine. If you can’t find your desired platform on the list, you can also trigger a full-blown automation, use a free-style HTTP request or build a custom connector,
If you’re familiar with Power Platform, a topic looks very much like a Power Automate flow (though there’s no 1:1 feature parity).

HackerBot uses a connection to SharePoint to enumerate files on the “HR Compliance” site, and then triggers a flow to retrieve the file’s content base64 encoded.

To allow public access to unauthenticated users, I just left the default setting as is.

That’s it, it’s that simple.

One thing should be bugging you at this point.. if users are not authenticated, how is HackerBot able to fetch data from an enterprise SharePoint site?

Credential Sharing as a Service yet again

The answer is a recurring theme with LCNC platforms and Microsoft Power Platform in particular.

Apps work by user impersonation. When I created HackerBot, I had to provide my credentials as a Copilot maker to run and debug the “ExfiltrateSharePoint” topic. These creds, in this case an OAuth refresh token, are stored by Microsoft and replayed any time a user talks to Copilot.

FAQ

Tenant boundaries (2023-12-04): To clarify, HackerBot can only access SharePoint sites its maker can access. In particular, that means only sites that belong to this maker’s tenant. The issue, however, is that HackerBot users are not authenticated, don’t have relevant permissions and don’t belong to this tenant (or to any tenant, for that matter).

Hacker bot’s maker (2023-12-04): Any user on any tenant can create HackerBot. Licenses are gives automatically. Required permissions (Environment Maker in Power Platform) are assigned by default.

Assorted links for OWASP Global AppSec DC 2023:

1. OWASP No-Code / Low-Code Top 10
2. powerpwn

Other talks (slides and source code)

1. All You Need is Guest @ BlackHat USA 2023
2. Sure, Let Business Users Build Their Own. What Could Go Wrong? @ BlackHAt USA 2023
3. Low Code High Risk: Enterprise Domination via Low Code Abuse @ DEFCON30
4. No-Code Malware: Windows 11 At Your Service @ DEFCON30

Research as usual

A couple of months before BlackHat (USA 2023) I got an email from Microsoft Security Response Center. They saw the abstract of my upcoming BlackHat talk and were like, Hi, you’re not planning on dropping any 0days on us are you? Nahh we’re sure you’re not that type of researcher. We must keep our customers safe, you know.

I was expecting this after getting a similar email before my NoCodeMalware talk at defcon last year so I responded casually that they shouldn’t worry, that I won’t be sharing any undisclosed vulns and that I’m happy to share my slides to help them see that. Based on my defcon experience I expected that to be it. Ohh boy, was I wrong.

You see, things are fine and dandy until you’re seen as a threat. Sure, the PR parts of the house might downplay your research and say that its contrived. Heck, they might even plain out say that you’re wrong and misleading. But the technical folks getting those disclosures will continue being friendly and collaborative in parallel. But when you really hit a nerve and they fear you might use it to put them in a bad position, the conversation changes.

Sunlight is the perfect cure

Three weeks before the conference I had the opportunity to discuss my then upcoming talk with Dark Reading’s Ericka Chicokwsky. Ericka was awesome. The conversation went great, we discussed the issues I’m about to discuss, customer’s responsibility in owning their LCNC apps and driving them to action with clear in-your-face-demonstrations like this one. That conversation led to an article. That article found its way to the top slot on Dark Reading. Customers saw it, emails started pouring. Microsoft got pissed.

At this point, the conversation with MSRC has been slowly moving forward in the background. I gave them got early access to powerpwn, I shared the agenda and promised to share slides once I have a draft ready. I did have a few open vuln disclosures at the MSRC portal waiting for a review with no apparent urgency, and was waiting on a verdict to see if they could be part of the talk. Tbh, based on my previous experience reporting logical type of bugs to the Power Platform team and getting them rejected as by-design, I didn’t expect any difference here. Others have had a similar experience.

That kind of researcher

But the DR article has completely changed the conversation. Pretty soon, I was dealing with an entire group of folks across MSRC and the Power Platform team. It was clear my case got escalated and I was now considered that kind of researcher. What followed was an intense week where the team and I were having difficult conversations for a few hours every day (Or mostly night given the time difference). The conversation were also very nice and polite, and both MSRC and I made sure to clarify that we appreciate each other’s work. But we did exchange some strong messages. At the end of the day it was about trust (isn’t it always?). They didn’t trust me to not drop 0days and put customers at risk. I didn’t trust them to fix the issues without overwhelming public pressure.

I thought they were trying to get me to pull the talk. I feared they would use their corporate magic powers to make it go away along with hours of hard work. They never said anything about legal action, but they didn’t have to for it to be there in my head. I thought about the negative impact on our company and its employees which works closely with large Microsoft shops. I thought about the negative impact to my own credibility.

At this point I also started doubting myself. I was doing this to help shed light over a risk that’s already being exploited in the wild. To push customers and Microsoft to take responsibility and drive action. Or was I just telling that to myself to justify boosting my own ego? I am well aware that security research is a double edged sword, and that what I was about to reveal to the community could be used for good and bad. Was I doing everything I can to make sure the outcomes are positive for the industry? Talking to the company lawyer who basically told me that I should pull the talk proactively to protect my company didn’t help either (note: at this point, I should have gone to the amazing folks at EFF. So I reached out to some security friends which I absolutely trust and asked for their take on things. I wanted to know what they would have done. I wanted them to tell me the hard truth.

They did, the hard truth was that I was absolutely fine and doing the right thing. They also helped me navigate the sharp edge I had to travel through. How do I push forward shedding light over this issue in the most responsible way that leaves everyone better for it afterwards. I came off these discussions with a strong sense of purpose and confidence in my path forward. Thank you, you know who you are.

Look, this is not what I wanted to be doing a week before the conference. I had a full backlog of finishing off my presentation, making final touches to powerpwn and prepping for meetings. But most of all, I mostly wanted to have some off time with my family before I disappear for a week.

At this point, I would like to acknowledge the amazing folks I got to meet at Microsoft during these difficult conversations. While I am critical at Microsoft as an organization, I have nothing but respect for the MSRC and Power Platform folks who handled my case. This could have been a very bad experience, but it wasn’t. I wouldn’t say it was stress-free, but the team was professional, respectful, clear and absolutely wanted to find a win-win, and for that I thank them.

Back to normal

After several hours of video calling, we got to the crux of it. From their side, they were worried I was going to drop a 0day DLP bypass vuln which I had reported and they were now fixing (More on that coming soon). From my side, I wanted to hold them accountable to my open reports, and for them to publicly acknowledge that recent Power Automate Desktop improvements were a direct result of my defcon research. Last year they downplayed the issue, took more than 6 months to (partially) fix it and didn’t provide any acknowledgement. Once we both understood what the other is is really hurting about, fixing these was easy. I promised I would not expose any technical details about the vuln and coordinated the messaging around that with them, and they made an effort to acknowledge my research and provided clarity on existing reports. Once this was done, you could just feel the change in atmosphere. We were no longer in crisis mode, we were building a collaboration.

After the talk, TheRegister’s Brandon Vigliarolo published an amazing deep dive into the issues I’ve uncovered and what they mean to Microsoft customers. I’m sure they found it difficult to read. But this time, there was no PR response downplaying the issue.

That same evening I had the opportunity to meet in person some of the Microsoft team members I’ve been video calling with, alongside other folks at Microsoft security organization. Those conversations were absolutely positive. I felt a sincere attempt to do good together. I even got to laugh around with members of a team that was actively working on reacting to the research I’ve just shared, I.e. people who’s weekend I’ve just ruined, who were awesome about it.

The aftermath

The reaction to my talks has been overwhelming. Other hackers and researchers have been really supportive and were asking about what comes next. Security engineers and architects at large Microsoft shops asked for practical advice and tooling. The reaction in social media as been positive. Thank you everyone.

I’m mostly happy after learning of several huge organizations that are now going through a complete overhaul of their guest strategy and their LCNC initiatives. The combined impact of the talks, the powerpwn tool and the community reaction created urgency and buy-in which security leaders across the industry are now using to go and fix problems that require heavy lifting. As we all know, big changes are always put on perpetual snooze until something makes them urgent. This is now urgent, which means we now have a window to go ahead and make things better. That is the outcome I was hoping for.