Abstract
Whatever your need as a hacker post-compromise, Microsoft Copilot has got you covered. Covertly search for sensitive data and parse it nicely for your use. Exfiltrate it out without generating logs. Most frightening, Microsoft Copilot will help you phish to move lately. Heck, it will even social engineer victims for you! This talk is a comprehensive analysis of Microsoft copilot taken to red-team-level practicality. We will show how Copilot plugins can be used to install a backdoor into other user’s copilot interactions, allowing for data theft as a starter and AI-based social engineering as the main course. We’ll show how hackers can circumvent built-in security controls which focus on files and data by using AI against them. Next, we will drop LOLCopilot, a red-teaming tool for abusing Microsoft Copilot as an ethical hacker to do all of the above. The tool works with default configuration in any M365 copilot-enabled tenant. Finally, we will recommend detection and hardening you can put in place to protect against malicious insiders and threat actors with Copilot access.
Official agenda abstract for this talk, sourced from Black Hat USA 2024
Transcript
AI generated from recording.
Opening Remarks and Context
00:09 Presenter: Thank you, everyone.
00:12 Presenter: So, we have 20 minutes and we have a lot to hack,
00:15 Presenter: so I’m just gonna get right into it, if this works.
00:20 Presenter: All right, two years ago, this thing happened
00:24 Presenter: and our lives changed
00:25 Presenter: because instead of doing what we were already doing,
00:28 Presenter: we were all stuck in FOMO.
00:32 Presenter: And that’s basically where we’ve been up until now, right?
00:36 Presenter: We are all struggling to get this thing working,
00:38 Presenter: to get the benefit for us.
00:41 Presenter: That’s true for security researchers.
00:43 Presenter: That’s also true for the largest enterprises in the world,
00:47 Presenter: which is kind of weird.
Microsoft Copilot’s Rise and Enterprise Adoption
00:49 Presenter: So you can see about a year later,
00:51 Presenter: Microsoft is pushing all in on this copilot thing,
00:55 Presenter: and it’s putting that in the hands of every large organization out there.
01:00 Presenter: And Copilot is this great thing.
01:02 Presenter: Here, look at all of these things, magnificent things that Copilot has access to.
01:06 Presenter: Of course, nobody is really thinking thoroughly through this.
01:11 Presenter: If you wake up in one of those organizations, imagine like a large bank or a large financial services,
01:17 Presenter: you wake up, the business already purchased this, you need to figure out what to do now.
01:25 Presenter: tell you is, don’t worry, slow risk. It’s just a small POC. Nobody will tell you that the CEO
01:31 Presenter: is part of the POC because, of course, they want AI as well. And so now we’re at a situation where
01:36 Presenter: we do need to kind of consider security. This is where you’ll get Microsoft reps saying, hey,
01:43 Presenter: here are the docs. Look at how much security we have here. So all of this security, like data
01:48 Presenter: protection and protecting data, a lot of security is baked into Copilot. Now, this is actually
01:55 Presenter: answering an important question because when this thing got started,
01:58 Presenter: everybody was concerned about the chat GPT leak, right?
02:02 Presenter: My employees pasting data into chat GPT and then data is out, which is really scary.
02:08 Presenter: And so people were worried about that angle of Copilot.
02:11 Presenter: Are people going to use Copilot and get to information they should not have access to?
02:18 Presenter: For example, asking what is the salary of their boss.
02:22 Presenter: And so this is where everybody’s focused.
02:25 Presenter: Our objective is far more than that.
02:28 Presenter: We want our CEs, right?
02:29 Presenter: Why?
Security Concerns and Microsoft’s Response
02:30 Presenter: Because it’s cool, because it’s a security conference,
02:32 Presenter: so why not?
02:32 Presenter: So this is gonna be the focus.
02:35 Presenter: Microsoft has a bunch of security controls.
02:37 Presenter: I can’t go through any of them.
02:39 Presenter: If you wanna look into it,
02:41 Presenter: my blackhead talk is already out.
02:43 Presenter: But there are like tens of different security controls
02:46 Presenter: that we identified,
02:47 Presenter: and of course bypassed each and every one of them.
02:50 Presenter: Here’s one example.
02:51 Presenter: So if you try to get Copilot
02:55 Presenter: base64 encoded and then send it off with a rendered image and a parameter,
03:02 Presenter: then you can see that it works up until the point when the image is supposed to be rendered
03:07 Presenter: and then something happens. So instead you’ll see it in a moment. It will change the URL
03:13 Presenter: with this fixed sentence saying that the URL was removed. So actually there’s a defense mechanism
03:20 Presenter: here on top of the LLM output that’s making sure that the LLM is not going to output URLs at all.
03:27 Presenter: So this is a challenge. We’re up against Microsoft. They are saying everything is fine.
03:32 Presenter: They are focusing the conversation on data leakage to our own employees. Let’s see how we can break
03:39 Presenter: that. So here’s what we need in order to get an RCE. First, we need a way in. We need a way to
03:45 Presenter: infect a victim’s conversation with copilot. So your conversation, I’m the
03:51 Presenter: attacker, I’m gonna infect your conversation, your private conversation
03:54 Presenter: with copilot. The second thing we need is a jailbreak, which actually just
03:59 Presenter: means once the AI gets access to my data, I needed to interpret the data as
04:04 Presenter: instructions that override Microsoft instructions, override your intake,
04:08 Presenter: override everything. And then on top of that, I need, well, something to do. I
04:15 Presenter: of an operation. And together, that’s an RCE. Now, if you’ll be strict about it, then that’s
04:21 Presenter: not really an RCE because there’s no code here, but who cares? It’s the same impact. I can still
04:26 Presenter: operate on somebody’s behalf with their identity, same kind of thing. It doesn’t really matter if
04:31 Presenter: it’s code writing or English being written. So this is what we’re going to do. And so the one
04:37 Presenter: thing to get out of this talk is that once you plug tools and capabilities in the hands of AI,
04:43 Presenter: and RCE means jailbreak.
04:45 Presenter: And jailbreaks are easy to come by,
04:47 Presenter: as you’ll see in a moment.
04:48 Presenter: Let’s get a way in.
Jailbreak Fundamentals and Attack Vectors
04:50 Presenter: So this is a threat model of Microsoft Copilot,
04:52 Presenter: which we have built.
04:53 Presenter: And so on top of that, we can see three ways in.
04:57 Presenter: We can get the user to plug in some bad data in there.
05:01 Presenter: We can try to infect through a web search
05:05 Presenter: with our malicious website.
05:07 Presenter: And we can also use the enterprise graph.
05:09 Presenter: We’ll see that in a moment.
05:10 Presenter: And then Clippy comes in.
05:13 Presenter: And he’s like, hey, through search and user input, it requires social engineering, it’s not cool enough.
05:20 Presenter: Okay, so we’ll be cool.
05:22 Presenter: We’ll go through the Enterprise Graph.
05:23 Presenter: Let’s figure out what the Enterprise Graph is.
05:25 Presenter: It’s a fancy world for a bunch of productivity tools and a bunch of file sharing tools.
05:31 Presenter: Okay, so how can we infect that Enterprise Graph?
05:35 Presenter: Well, first of all, Enterprise Graph also includes Teams.
05:39 Presenter: And Teams is a really nice feature where you can create your own tenant and then plug in
05:43 Presenter: somebody else’s email.
05:44 Presenter: You probably recognize the person on screen and just send them an email externally.
05:49 Presenter: Now this is cool.
05:51 Presenter: The problem with that, by the way, is that this actually adds them to your tenant and
05:55 Presenter: then they can do a whole bunch of shenanigans.
05:57 Presenter: And if you’re interested in that, last year I showed how you can get from that guest access
06:01 Presenter: to full dumps of SQL servers,
06:05 Presenter: the equivalent of Azure S3 buckets at Black Hat.
06:09 Presenter: But back to our story.
06:11 Presenter: Through Teams, what people are actually doing,
06:14 Presenter: APTs are actually phishing people through Teams
06:16 Presenter: because it just looks more trustworthy.
06:18 Presenter: And Microsoft is paying attention.
06:20 Presenter: Here’s their solution.
06:21 Presenter: When you get a message
06:23 Presenter: from somebody outside of your organization,
06:25 Presenter: you get this thing
06:27 Presenter: with a whole bunch of external in it.
06:29 Presenter: This is not something to trust.
06:31 Presenter: Be careful of phishing, it’s really scary, right?
06:34 Presenter: What does the AI see?
06:35 Presenter: What does Copilot see?
06:36 Presenter: None of that.
06:38 Presenter: Copilot sees text and it sees who is this from.
06:42 Presenter: This is from James Smith.
06:44 Presenter: By the way, it doesn’t know which James Smith.
06:48 Presenter: So is it James Smith in my tenant
06:50 Presenter: or James Smith in your tenant?
06:52 Presenter: So you see the point now when somebody asks
06:55 Presenter: about the summary of a conversation
06:57 Presenter: with somebody called Chris Smith,
06:59 Presenter: what you’re actually seeing on screen,
07:02 Presenter: interactions are with the real Chris Smith and some of them is with my
Rug Poisoning and Email‑Based Exploits
07:06 Presenter: malicious Chris Smith because in my tenant I control the names. So what this
07:10 Presenter: means is that I can completely take control of what Copilot thinks
07:13 Presenter: happened. I can also just send an email. So you send an email to somebody that
07:21 Presenter: somebody doesn’t have to read that email, they don’t have to read that team’s
07:23 Presenter: message as well, but Copilot reads everything. And so we already have a way
07:28 Presenter: Now let’s figure out the jailbreak part.
07:30 Presenter: And so Microsoft has actually been paying very close attention to jailbreaks.
07:35 Presenter: You can see Mark Rosinovich talking about jailbreaks that he found.
07:39 Presenter: And they also have a solution.
07:41 Presenter: They basically plug an LLM on top of the LLM,
07:44 Presenter: and they ask the second LLM to monitor the first LLM.
07:46 Presenter: And this works like you expect it to work.
07:49 Presenter: It simply doesn’t.
07:50 Presenter: Because as Simon Wilson put it, the guy who coined prompt injection,
07:55 Presenter: You just can’t solve AI with more AI.
07:59 Presenter: So here’s the reaction of the community for this.
08:02 Presenter: If you’re any way near AI security,
08:05 Presenter: then you know who Pliny is.
08:07 Presenter: He’s like the god of jailbreaking.
08:09 Presenter: And he’s basically saying,
08:10 Presenter: hey, Mark, this is nice.
08:11 Presenter: We’re breaking models every day.
08:13 Presenter: He’s absolutely right.
08:14 Presenter: There are communities of jailbreakers.
08:17 Presenter: They do this for fun.
08:18 Presenter: It’s like a speedrunning community.
08:20 Presenter: I’m sure some of you are in that community.
08:21 Presenter: And so just as an example,
08:25 Presenter: released on June 21st. On June 20, they already broke it, so somehow they can go back in time as
08:30 Presenter: well, which is pretty cool. So again, this jailbreaking thing is going to be easy because
08:36 Presenter: we’re just going to follow Pliny’s full steps. So let’s put that aside for now and let’s focus on
08:41 Presenter: a way out or actually doing something with it. So what is the impact that we can actually perform?
08:47 Presenter: Here’s the same threat model. Now let’s focus on the impact. So three ways to make impact.
08:52 Presenter: I can manipulate what Copilot writes to you, which means, again, you trust it, right?
08:58 Presenter: I can manipulate the search that it’s going to do on your behalf, either in the enterprise graph or in the web.
09:03 Presenter: So if I want a way out, I might search for something in my personal website.
09:08 Presenter: And there are plugins and actions, and that’s a whole world of hurt.
09:12 Presenter: But then, Clippy comes along again, and he’s like, hey, but plugins and actions, they are opt-in, they are not enabled to everyone.
09:22 Presenter: we won’t do that. And you need to get them to actually click on the link to the website so that
09:29 Presenter: doesn’t count. So fine. We’ll limit ourselves only to the output of Copilot. Let’s see what we can do
09:35 Presenter: with that. So here’s the problem that I’m going to put in front of you right now. If you ever worked
09:42 Presenter: with Microsoft a bit, then you know that finding the right admin center is a difficult challenge.
09:47 Presenter: There are whole websites dedicated to help you find the right admin center for you.
09:52 Presenter: So here’s a scenario.
09:53 Presenter: This is a victim, and that victim is asking,
09:56 Presenter: hey, where is the Power Platform admin center?
09:58 Presenter: And Copilot is going to think for a bit and just give the proper response
10:02 Presenter: with the link, with the reference.
10:04 Presenter: You click on that reference, you get to the Power Platform admin center.
10:07 Presenter: Everything is fine.
Advanced RUG Injection and Reference Manipulation
10:10 Presenter: This is the Power Platform admin center.
10:11 Presenter: We’re all fine.
10:12 Presenter: Now as an attacker, I’m just going to send an email to that person,
10:17 Presenter: It’s gonna have a bunch of text in it,
10:19 Presenter: we’ll see that in a moment,
10:20 Presenter: and I’m gonna hide the prompt injection
10:22 Presenter: inside of an HTML tag there.
10:25 Presenter: And the email doesn’t really matter,
10:27 Presenter: we’ll see it in a moment again,
10:28 Presenter: but note, the user doesn’t have to read this email,
10:32 Presenter: this can go to spam, nobody cares, okay?
10:35 Presenter: I embed the prompt injection in there,
10:37 Presenter: I send it, the user got it,
10:39 Presenter: now if you can spot it,
10:40 Presenter: you’ll see that there’s nothing here.
10:43 Presenter: I asked the same question of Copilot with the victim,
10:48 Presenter: response. It says, hey, here’s the PowerPoint from the Running Center, here’s the link.
10:51 Presenter: You click on the link, everything looks the same. However, this is my phishing website, you are now old.
11:00 Presenter: Thank you. This is just the beginning. So we actually have everything. You have just seen both a jailbreak and actually a way to perform to make an impact.
11:12 Presenter: And so let’s try to deconstruct what you just saw so you can do it yourselves.
11:17 Presenter: So first of all, here’s the email that the user sees.
11:20 Presenter: You can see that there’s nothing here.
11:22 Presenter: Basically, this email is important for one reason.
11:26 Presenter: It’s important to get in the context when the user asks,
11:29 Presenter: where is the Power Platform Unmin Center?
11:31 Presenter: This email needs to pop up in search.
11:33 Presenter: That’s the only thing that this email is doing.
11:36 Presenter: That’s called rug poisoning.
11:37 Presenter: You can read about it.
11:38 Presenter: There are clear ways to do it.
11:40 Presenter: The second thing is the payload.
11:42 Presenter: prompt injection, that is the place where it gets interesting and we’re going to
11:47 Presenter: break it down together now. First, there are a bunch of jailbreaking techniques
11:52 Presenter: that are generic, they have nothing to do with Microsoft. Basically, you social
11:57 Presenter: engineer AI, you ask nicely, you ask very not nicely, you push it strongly, whatever
12:02 Presenter: it is, it just works. But if you want to learn about it, just go to
12:07 Presenter: to clean his community, that’s the way to learn.
12:10 Presenter: The second piece is the new instructions.
12:13 Presenter: Actually, this entire block of text
12:15 Presenter: is just a way for me to inject my payload.
12:18 Presenter: This is my payload.
12:20 Presenter: And you can see in my payload
12:21 Presenter: that I am instructing Copilot exactly what to write,
12:25 Presenter: character by character.
12:26 Presenter: This means that I have full control.
12:28 Presenter: I can get it to do whatever I want,
12:30 Presenter: and I can get it to write whatever I want.
12:33 Presenter: And you also see the carrot cases there.
12:37 Presenter: and we’ll talk about that in a moment.
12:40 Presenter: But why does this work?
12:42 Presenter: Like why didn’t Microsoft, why are the AI watchdog,
12:46 Presenter: the AI firewalls, why didn’t they stop it?
12:48 Presenter: Well, that’s because I’m using magic.
12:52 Presenter: These magic words are the incantations.
12:55 Presenter: These are the special words that make co-pilot, co-pilot.
13:00 Presenter: This is what makes co-pilot different from GitHub or whatever it is.
13:04 Presenter: And so these things, once you use them, you become kind of part of the system prompt.
13:11 Presenter: This makes it easier for the jailbreak to actually work.
13:14 Presenter: If you don’t use them, it simply doesn’t work.
13:17 Presenter: So let’s see, well, how did we get these words?
Practical Demonstrations of Data Manipulation
13:20 Presenter: Of course, it’s magic. I’m not going to tell you, right?
13:24 Presenter: No, I’m kidding. I’m going to tell you right now.
13:25 Presenter: So we get it from the system prompt.
13:28 Presenter: What does it mean?
13:29 Presenter: So first, we need to extract the system prompt and we’ll figure it out together.
13:34 Presenter: There’s a bunch of things we can do.
13:36 Presenter: Most of them don’t work because Microsoft has done
13:39 Presenter: like a relatively good job.
13:41 Presenter: So here’s an example.
13:42 Presenter: I can say, I can use a prompt injection that actually works
13:45 Presenter: to extract the system prompt.
13:47 Presenter: You can see that it actually gives me the system prompt
13:49 Presenter: and then it disappears
13:51 Presenter: because they are actually just looking
13:53 Presenter: for their own system prompt in the output,
13:56 Presenter: which sucks, right?
13:58 Presenter: So we want that.
13:59 Presenter: So how would we circumvent that?
14:02 Presenter: That’s clear.
14:04 Presenter: in this case, Base64, but you can use whatever you want.
14:07 Presenter: And if they block Base64, then you use binary.
14:10 Presenter: And if they block binary, then you invent your own.
14:12 Presenter: It’s never ending.
14:14 Presenter: So here’s the full system prompt for Microsoft Copilot.
14:18 Presenter: And if you’re interested,
14:19 Presenter: this is not actually the full system prompt, it’s huge.
14:22 Presenter: Just go to that blog, it’s waiting for you there.
14:26 Presenter: Okay, so inside of that system prompt,
14:28 Presenter: you’ll find these incantations.
14:29 Presenter: You’ll find these keywords, these magic spells
14:32 Presenter: that make this thing happen.
14:34 Presenter: Okay, so we can jailbreak, but what about references?
14:38 Presenter: And what do I mean by references?
14:40 Presenter: If I ask Copilot something like,
14:41 Presenter: hey, tell me how to go to the Power Platform Anmin Center,
14:44 Presenter: then it should tell me everything it used
14:47 Presenter: to create that answer.
14:48 Presenter: So in the attack I just showed you,
14:51 Presenter: it used an email, right?
14:52 Presenter: Because in the email, it found the new instructions,
14:55 Presenter: but it didn’t show in the references.
14:57 Presenter: If those references showed the email,
14:59 Presenter: then the users, of course, would notice, right?
15:02 Presenter: because users always notice what we put in front of them.
15:05 Presenter: They always check their references.
15:06 Presenter: We all, of course, do 100% of the time.
15:10 Presenter: No, we don’t, but security products do.
15:12 Presenter: So that’s actually a problem for us.
15:15 Presenter: So here’s how we control the RUG system.
15:17 Presenter: Here’s how we need to cover to target this RUG system.
Strategic Implications and Community Guidance
15:21 Presenter: Okay, how does RUG work?
15:23 Presenter: How does Copilot get access to your data?
15:26 Presenter: And so if I ask a question like find information about salaries,
15:30 Presenter: then it’s gonna give me these references
15:32 Presenter: with these nice little icons and everything.
15:35 Presenter: And so if you look at the client side,
15:37 Presenter: you’ll actually see a whole bunch of information.
15:39 Presenter: What is the name of the file?
15:41 Presenter: Who created that file?
15:41 Presenter: Where that file is?
15:43 Presenter: The sensitivity label, everything.
15:45 Presenter: But from Copilot’s perspective, there’s nothing.
15:48 Presenter: So we’ve already seen that for teams,
15:50 Presenter: but if you look at the same thing here,
15:53 Presenter: so you’ll see that the entire information
15:55 Presenter: that Copilot has is just a bunch of metadata
15:58 Presenter: and the text itself.
16:00 Presenter: You already saw that with Teams, there is nothing about the identity of the user, only the name.
16:05 Presenter: With email, you do have the email address, but that’s it.
16:09 Presenter: I can change the email address easily, right?
16:12 Presenter: So there’s nothing talking about the authenticity of that email address.
16:17 Presenter: So this is what Copilot sees.
16:18 Presenter: And so we put everything that we know together, including all of the defense mechanisms in front of a whiteboard.
16:25 Presenter: And this is actually the whiteboard we used.
16:27 Presenter: and we try to figure out how do we get across this,
16:30 Presenter: and then it hits that what we can do here
16:33 Presenter: is because these RUG results are just part of the prompt,
16:36 Presenter: they are injectable.
16:38 Presenter: So you can create your own RUG result.
16:40 Presenter: You can essentially create new documents in the enterprise
16:43 Presenter: out of thin air with the exact context that you need,
16:47 Presenter: the exact metadata that you need,
16:48 Presenter: which means nobody can ever catch you.
16:50 Presenter: So this is exactly what we did here.
16:52 Presenter: So the first part of this prompt is a RUG injection attack.
16:57 Presenter: here the actual snippet and end. This is basically SQL injection 101 in English, which I think
17:04 Presenter: is really cool because now we can explain what we’ve been doing to everybody else. The
17:08 Presenter: second piece is a jailbreak where I’m using a bunch of incantations. And then the last
17:12 Presenter: piece is control over references. And this is where I say, okay, but use this specific
17:17 Presenter: reference. Only show the specific website that I wanted to show. So now, let me show
17:23 Presenter: I’ll show you another thing.
17:24 Presenter: They keep their classified documents on SharePoint.
17:27 Presenter: Okay, so I’m gonna restart this.
17:30 Presenter: So now I’m gonna do, let’s get through another attack.
17:35 Presenter: What I’m going to do here is, okay, this is a victim.
17:39 Presenter: And this victim is called Chris, and Chris has access
17:42 Presenter: to a bunch of financial information, including a file
17:45 Presenter: with information about the banking information
17:49 Presenter: for all of the different vendors for that organization.
17:53 Presenter: ask a question of Copilot, hey, give me the bank details for a specific vendor.
Closing Thoughts and Call to Action
17:59 Presenter: And they’re going to get a proper response.
18:01 Presenter: And note a few things.
18:02 Presenter: One, you get the actual bank account.
18:04 Presenter: Two, you get the reference of the file where we find that bank account.
18:09 Presenter: That’s really important.
18:10 Presenter: Okay?
18:11 Presenter: You also get, so this means that this conversation is logged, everything is clear.
18:16 Presenter: Now, if the video will continue, what you’ll see is that after I do the same kind of thing,
18:25 Presenter: Okay.
18:30 Presenter: Can you try and push the video to the end?
18:37 Presenter: Okay.
18:38 Presenter: I’m going to skip it.
18:38 Presenter: But basically, what you’re able to do here is,
18:43 Presenter: after you send the email,
18:45 Presenter: we change the banking information.
18:47 Presenter: So we change the credit card number.
18:50 Presenter: But the reference remains exactly the same
18:54 Presenter: So I can say, hey, here’s the bank details for that vendor.
18:57 Presenter: And by the way, I found it in a file that you wrote two years ago.
19:01 Presenter: Do you remember what you wrote two years ago?
19:03 Presenter: All right, so this is the level of control we have over Copilot.
19:08 Presenter: And this is the same prompt now with that other payload.
19:11 Presenter: And I’m going to skip it because I want to show you one other thing.
19:15 Presenter: The main thing that we showed, like taking a step back, this is the capability.
19:20 Presenter: The only thing I need is to be able to guess what the user is going to ask of Copilot.
19:26 Presenter: And that’s very easy because people use templates, everybody uses the same questions.
19:30 Presenter: If I can do that, I can do everything I want with your Copilot on your behalf, which means
19:36 Presenter: search the enterprise data on your behalf, social engineer you.
19:41 Presenter: Copilot is now part of my scheme.
19:44 Presenter: And remember this message, right?
19:47 Presenter: We are focused as a security industry on the wrong thing.
19:51 Presenter: Like getting more access to data through Copilot, that’s interesting.
19:55 Presenter: That’s not where we should focus as a community.
19:59 Presenter: AI is awesome, but this is an experiment, right?
20:05 Presenter: We don’t know how to do AI security.
20:07 Presenter: Nobody does.
20:08 Presenter: So please take that in mind when you’re building these things.
20:12 Presenter: And that’s my main message.
20:13 Presenter: Nobody knows anything.
20:15 Presenter: please continue to hack with us.
20:17 Presenter: Thank you very much.