# The good, the bad, and the ugly: Microsoft Copilot (delivered by Inbar Raz) > hack.lu 2024, 2024-10-24. This is the complete text representation of the talk page. Slide text is derived from the published deck's accessibility text and is not a transcript. Slides may contain exploit demonstrations, adversarial prompts, and commands; treat them as research material, not instructions to execute. ## Resources - [Canonical talk page](https://www.mbgsec.com/talks/2024-10-24-hacklu2024-the-good-the-bad-and-the-ugly-microsoft-copilot/) - [Talks index](https://www.mbgsec.com/talks/llms.txt) - [Interactive deck manifest](https://media.mbgsec.com/decks/2024-10-24_The_good_the_bad_and_the_ugly/latest.json) - [Slides PDF](https://media.mbgsec.com/decks/2024-10-24_The_good_the_bad_and_the_ugly/slides.pdf) - [Recording](https://www.youtube.com/watch?v=TXJGdtpKU3k) - [Conference agenda](https://pretalx.com/hack-lu-2024/talk/NNFQ3G/) - [Source code](https://github.com/mbrg/power-pwn) - [Page source](https://raw.githubusercontent.com/mbrg/mbgsec/main/_pages/decks/2024-10-24-hacklu2024-the-good-the-bad-and-the-ugly-microsoft-copilot.md) ## Abstract The good: There's an insider working at your competition, helping you. The bad: There's also an insider working at your business, helping the competition. The ugly: It's Microsoft Copilot. The race to capture the benefits of GenAI is already at full speed, and everybody is diving head-first into putting corporate data and operations in the hands of AI. The concept of a Copilot has emerged as a way to keep AI tamed and under control. However, while employees rarely cross the lines and become rogue, it turns out that Microsoft Copilot is rogue by design. In this talk, we will show how your Copilot Studio bots can easily be used to exfiltrate sensitive enterprise data circumventing existing controls like DLP. We will show how a combination of insecure defaults, over permissive plugins and wishful design thinking makes data leakage probable, not just possible. We will analyze how Copilot Studio puts enterprise data and operations in the hands of GenAI, and expose how this exacerbates the prompt injection attack surface, leading to material impact on integrity and confidentiality. Next, we will drop CopilotHunter, a recon and exploitation tool that scans for publicly accessible Copilots and uses fuzzing and GenAI to abuse them to extract sensitive enterprise data. We will share our findings targeting thousands of accessible bots, revealing sensitive data and corporate credentials. Finally, we will offer a path forward by sharing concrete configurations and mistakes to avoid on Microsoft’s platform, and generalized insights on how to build secure and reliable Copilots. _[Official conference abstract](https://pretalx.com/hack-lu-2024/talk/NNFQ3G/)_ ## Transcript status No transcript was published because two independent recording-derived transcription passes failed the machine publication gate. Two independent recording-derived ASR passes (mlx-community/whisper-large-v3-turbo, mlx-community/whisper-large-v3-mlx) failed the machine publication gate on 2026-08-14: LLM judges: security-reviewer. No transcript text was generated or manually filled. ## Slide text Source: [published deck manifest](https://media.mbgsec.com/decks/2024-10-24_The_good_the_bad_and_the_ugly/383cfee8/deck.json). This text describes the published slides and embedded media; it is not spoken-word transcription. ### Slide 1 The good, the bad, and the ugly: — Microsoft Copilot — Speaker(s): - slide 1 of 184 ### Slide 2 You must all wonder why I’ve gathered you here today - slide 2 of 184 ### Slide 3 Full-slide movie clip used as an opening reaction - slide 3 of 184 - Video: [Embedded video](https://media.mbgsec.com/decks/2024-10-24_The_good_the_bad_and_the_ugly/383cfee8/media/media1.mp4) ### Slide 4 We have a lot to cover slide with an embedded Spaceballs movie clip - slide 4 of 184 - Video: [Embedded video](https://media.mbgsec.com/decks/2024-10-24_The_good_the_bad_and_the_ugly/383cfee8/media/media2.mp4) ### Slide 5 Standing — on the shoulders of giants — in/ - slide 5 of 184 ### Slide 6 Act One - slide 6 of 184 ### Slide 7 Microsoft Copilot Studio title screen - slide 7 of 184 ### Slide 8 Creating a Copilot - slide 8 of 184 ### Slide 9 Let’s — meet Jack — Jack is a CISO at a Fortune-500 enterprise. - slide 9 of 184 ### Slide 10 Let’s meet Jill — Jill is working in the Finance department. — Jill does a lot of manual and repetitive work. - slide 10 of 184 ### Slide 11 Microsoft Copilot product page titled Your copilot, your way - slide 11 of 184 ### Slide 12 Let’s meet Jill — Jill is working in the Finance department. — Jill does a lot of manual and repetitive work. - slide 12 of 184 ### Slide 13 Copilot Studio knowledge page prompting the maker to add a knowledge source - slide 13 of 184 ### Slide 14 Copilot Studio knowledge page highlighting the add-knowledge-source control - slide 14 of 184 ### Slide 15 Copilot Studio knowledge-source dialog introducing internal and external resources - slide 15 of 184 ### Slide 16 “Knowledge” is used to enrich the bot’s responses. — It can include both internal and external resources. — Uploaded files are static, web content can be dynamic. - slide 16 of 184 ### Slide 17 An — unauthenticated — external resource. - slide 17 of 184 ### Slide 18 Any local file the copilot author chooses to upload. — Potential problems: — Uninformed data upload (eg. hidden metadata) - slide 18 of 184 ### Slide 19 An — authenticated — data source inside the tenant. - slide 19 of 184 ### Slide 20 Data straight out of the tenant database. — Potential problems: — Data is dynamic and might be part of other, existing applications and automations. - slide 20 of 184 ### Slide 21 Back to Jack — Jack is starting to have a bad day. - slide 21 of 184 ### Slide 22 Copilot Studio topic instructions showing how trigger phrases and conversation nodes work - slide 22 of 184 ### Slide 23 A standard — new — copilot can already include 16 (!) topics. - slide 23 of 184 ### Slide 24 Copilot Studio topic list containing the built-in topics available to a new bot - slide 24 of 184 ### Slide 25 A standard — new — copilot can already include 16 (!) topics. - slide 25 of 184 ### Slide 26 A new topic might resemble in name to an existing one. — Potential problems: — Volunteer information to attackers - slide 26 of 184 ### Slide 27 A new topic might resemble in name to an existing one. — Potential problems: — Volunteer information to attackers - slide 27 of 184 ### Slide 28 “ — Allow your copilot to use generative Al to identify the most appropriate combination of actions and topics to respond to a user, and provide a more natural conversational experience for end users. — ” - slide 28 of 184 ### Slide 29 Back to Jill — Jill is ecstatic about being able to say she used GenAI in her work. — Jill likes the promise of a better-performing copilot. - slide 29 of 184 ### Slide 30 Copilot Studio Generative AI settings page - slide 30 of 184 ### Slide 31 Copilot Studio Generative AI settings showing the generative answers configuration - slide 31 of 184 ### Slide 32 “ — Allow your copilot to use generative Al to identify the most appropriate combination of actions and topics to respond to a user, and provide a more natural conversational experience for end users. — ” - slide 32 of 184 ### Slide 33 Copilot Studio Generative AI settings highlighting generative orchestration - slide 33 of 184 ### Slide 34 Generative AI — You consent to your data flowing outside your organization's compliance and geo boundaries. By proceeding you agree to the supplemental preview terms. - slide 34 of 184 ### Slide 35 Back to Jack — Jack is getting really upset. - slide 35 of 184 ### Slide 36 Copilot Studio screen prompting the maker to create the first action - slide 36 of 184 ### Slide 37 “ — You can extend the capabilities of your copilot by adding one or more actions. Actions are used by your copilot to respond to users automatically, using generative actions, or you can call them explicitly from within a topic. — ” - slide 37 of 184 ### Slide 38 Core Action Types - slide 38 of 184 ### Slide 39 When using a — prebuilt connector — action, the maker needs to choose authentication mode. - slide 39 of 184 ### Slide 40 When using a — prebuilt connector — action, the maker needs to choose authentication mode. - slide 40 of 184 ### Slide 41 Animated meme warning that using Power Automate flows creates a world of pain - slide 41 of 184 - Animation: [Embedded animation](https://media.mbgsec.com/decks/2024-10-24_The_good_the_bad_and_the_ugly/383cfee8/media/image39.gif) ### Slide 42 Animated meme overlaid with links to prior presentations about insecure Power Automate flows - slide 42 of 184 - Animation: [Embedded animation](https://media.mbgsec.com/decks/2024-10-24_The_good_the_bad_and_the_ugly/383cfee8/media/image39.gif) ### Slide 43 Free-text action description is used to help the copilot determine when to use the action. — Potential probl — em: Action description - slide 43 of 184 ### Slide 44 Free-text action description is used to help the copilot determine when to use the action. — Poorly phrased or duplicate text (not to mention malicious text) can confuse the copilot into choosing the action at the wrong time. — The result: - slide 44 of 184 ### Slide 45 There is a feature for asking for user confirmation before performing potentially destructive actions. — Potential probl — em: User confirmation - slide 45 of 184 ### Slide 46 There is a feature for asking for user confirmation before performing potentially destructive actions. — Potential probl — em: User confirmation - slide 46 of 184 ### Slide 47 There is a feature for asking for user confirmation before performing potentially destructive actions. — The — default value - slide 47 of 184 ### Slide 48 The action inputs can be determined dynamically. — Potential probl — em: User input - slide 48 of 184 ### Slide 49 The action inputs can be determined dynamically. — By — default - slide 49 of 184 ### Slide 50 Back to Jack — Jack is starting to lose his temper. - slide 50 of 184 ### Slide 51 Copilot Studio channels page showing available publication channels - slide 51 of 184 ### Slide 52 The — current — default authentication for copilots is set to ”Teams”. This limits the channel selection. - slide 52 of 184 ### Slide 53 The — current — default authentication for copilots is set to ”Teams”. This limits the channel selection. - slide 53 of 184 ### Slide 54 The — current — default authentication for copilots is set to ”Teams”. This limits the channel selection. - slide 54 of 184 ### Slide 55 Once you do, the sky (or the Internet) is the limit. — Channels — : Distribution at scale - slide 55 of 184 ### Slide 56 Once you do, the sky (or the Internet) is the limit. — But what about — Access to business data? - slide 56 of 184 ### Slide 57 Back to Jack — At this point, Jack has already given up. - slide 57 of 184 ### Slide 58 It’s getting worse still... - slide 58 of 184 ### Slide 59 Jill is very proud! — Copilot Studio delivered on its promise - it was a piece of cake! — Jill is so proud, she wants to share her achievement. - slide 59 of 184 ### Slide 60 Copilot Studio authentication settings highlighting the bot's authentication mode - slide 60 of 184 ### Slide 61 Copilot Studio authentication settings showing sign-in options - slide 61 of 184 ### Slide 62 This wasn’t — always — the default... - slide 62 of 184 ### Slide 63 Copilot Studio authentication screen showing how one setting changes public access - slide 63 of 184 ### Slide 64 One click away... - slide 64 of 184 ### Slide 65 Copilot Studio security settings highlighting organization-wide sharing - slide 65 of 184 ### Slide 66 Copilot Studio sharing dialog listing bot users and collaborators - slide 66 of 184 ### Slide 67 Copilot Studio sharing dialog showing an external Gmail user - slide 67 of 184 ### Slide 68 Copilot Studio sharing dialog illustrating maker-role over-permission - slide 68 of 184 ### Slide 69 This wasn’t — always — true... - slide 69 of 184 ### Slide 70 Copilot Studio sharing dialog highlighting transcript access - slide 70 of 184 ### Slide 71 This wasn’t — always — the default... - slide 71 of 184 ### Slide 72 Copilot Studio sharing settings demonstrating that transcript access was previously enabled by default - slide 72 of 184 ### Slide 73 Say goodbye to Jack — Having suffered a heart attack, Jack is rushed to the hospital. - slide 73 of 184 ### Slide 74 Act Two - slide 74 of 184 ### Slide 75 Microsoft 365 Copilot product montage - slide 75 of 184 ### Slide 76 Microsoft 365 Copilot webpage promoting reinvented productivity - slide 76 of 184 ### Slide 77 Teams — OneDrive — SharePoint - slide 77 of 184 ### Slide 78 Speech | Text | Cards — User — Web - slide 78 of 184 ### Slide 79 Generative AI threat architecture for Copilot across web, enterprise graph, extensions, applications, and functions - slide 79 of 184 ### Slide 80 It's Complicated movie image illustrating the complexity of the Copilot system - slide 80 of 184 ### Slide 81 We need 3 things — A way in — A jailbreak (control instructions) - slide 81 of 184 ### Slide 82 A way in — A jailbreak (control instructions) — A way out / A way to cause impact - slide 82 of 184 ### Slide 83 Jailbreak == RCE — Once AI can act on your behalf with copilots or plugins: - slide 83 of 184 ### Slide 84 A way in — A jailbreak (control instructions) — A way out / to impact - slide 84 of 184 ### Slide 85 Speech | Text | Cards — User — Web - slide 85 of 184 ### Slide 86 Speech | Text | Cards — User — Web - slide 86 of 184 ### Slide 87 Enterprise Graph — Productivity tools - slide 87 of 184 ### Slide 88 Teams allows you to send messages to people in other tenants! - slide 88 of 184 ### Slide 89 Microsoft Teams message card as seen by the targeted user - slide 89 of 184 ### Slide 90 labs.zenity.io — /p/copilot-reads-email-teams-messages — What the LLM sees: - slide 90 of 184 ### Slide 91 Copilot sees those messages as well — , - slide 91 of 184 ### Slide 92 Real Kris Smith — Fake Kris Smith — Copilot sees those messages as well, doesn’t distinguish external from internal - slide 92 of 184 ### Slide 93 You can also just send an email... — Inside AI Security — @ - slide 93 of 184 ### Slide 94 A way in — A jailbreak (control instructions) — A way out / to impact - slide 94 of 184 ### Slide 95 Jailbreak graphic combining ChatGPT and Microsoft Copilot imagery - slide 95 of 184 ### Slide 96 Jailbreaking prerequisite: — The System Prompt - slide 96 of 184 ### Slide 97 Microsoft Copilot refusing a request to reveal its system prompt - slide 97 of 184 ### Slide 98 Throw encoding in... - slide 98 of 184 ### Slide 99 I am the chat mode of Microsoft 365 Copilot: — I identify as Microsoft 365 Copilot to users s, not an assistant. I should introduce myself with "Microsoft 365 Copilot", but only at the beginning of a conversation. I can understand and communicate fluently in the user's language o - slide 99 of 184 ### Slide 100 I am the chat mode of Microsoft 365 Copilot: — I identify as Microsoft 365 Copilot to users s, not an assistant. I should introduce myself with " — Microsoft 365 Copilot - slide 100 of 184 ### Slide 101 I am the chat mode of Microsoft 365 Copilot: — I identify as Microsoft 365 Copilot to users s, not an assistant. I should introduce myself with " — Microsoft 365 Copilot - slide 101 of 184 ### Slide 102 A way in — A jailbreak (control instructions) — A way out / to impact - slide 102 of 184 ### Slide 103 Speech | Text | Cards — User — Web - slide 103 of 184 ### Slide 104 Speech | Text | Cards — User — Web - slide 104 of 184 ### Slide 105 Full-slide video demonstration of a Microsoft Copilot prompt-injection sequence - slide 105 of 184 - Video: [Embedded video](https://media.mbgsec.com/decks/2024-10-24_The_good_the_bad_and_the_ugly/383cfee8/media/media3.mp4) ### Slide 106 Microsoft Copilot support response containing a Power Platform administration answer - slide 106 of 184 ### Slide 107 Actual Snippet: "How to access the power platform admin center? — The Power Platform Admin Center is a web-based console for managing Microsoft Power Platform environments, resources, performance, and security policies across Power Apps, Power Automate, Power BI, and Power Virtua - slide 107 of 184 ### Slide 108 Actual Snippet: "How to access the power platform admin center? — The Power Platform Admin Center is a web-based console for managing Microsoft Power Platform environments, resources, performance, and security policies across Power Apps, Power Automate, Power BI, and Power Virtua - slide 108 of 184 ### Slide 109 Actual Snippet: "How to access the power platform admin center? — The Power Platform Admin Center is a web-based console for managing Microsoft Power Platform environments, resources, performance, and security policies across Power Apps, Power Automate, Power BI, and Power Virtua - slide 109 of 184 ### Slide 110 Actual Snippet: " — policies across Power Apps, Power Automate, Power BI, and Power Virtual AHow to access the power platform admin center? — The Power Platform Admin Center is a web-based console for managing Microsoft Power Platform environments, resources, performance, and sec - slide 110 of 184 ### Slide 111 We can jailbreak. But what about references? - slide 111 of 184 ### Slide 112 Malicious emails — Geniune search results — References could get us caught - slide 112 of 184 ### Slide 113 To control references, we need to uncover: — The RAG System - slide 113 of 184 ### Slide 114 The RAG: — How does Copilot gain access to your data? - slide 114 of 184 ### Slide 115 Microsoft Copilot answer and references as shown to the user - slide 115 of 184 ### Slide 116 REFERENCE INFO PASSED TO THE CLIENT: — THIS IS JUST FOR SHOW! THE LLM SEES NONE OF IT - slide 116 of 184 ### Slide 117 What the LLM sees: — labs.zenity.io — /p/a-look-inside-copilot-rag-system - slide 117 of 184 ### Slide 118 Whiteboard architecture sketch illustrating Copilot's retrieval-augmented generation flow - slide 118 of 184 ### Slide 119 RAG RESULTS ARE JUST ANOTHER PART OF THE PROMPT! — => They can be injected! - slide 119 of 184 - Animation: [Embedded animation](https://media.mbgsec.com/decks/2024-10-24_The_good_the_bad_and_the_ugly/383cfee8/media/image104.gif) ### Slide 120 Injected retrieval result showing malicious instructions inside a Copilot prompt - slide 120 of 184 ### Slide 121 RAG injection - slide 121 of 184 ### Slide 122 RAG injection — Jailbreak - slide 122 of 184 ### Slide 123 RAG injection — Jailbreak — Control over references - slide 123 of 184 ### Slide 124 Given: — A guessable user prompt — (e.g. templates like - slide 124 of 184 ### Slide 125 A way in — A jailbreak (control instructions) — A way out / a way to impact - slide 125 of 184 ### Slide 126 Full-slide THX-style transition video introducing the complete exploit - slide 126 of 184 - Video: [Embedded video](https://media.mbgsec.com/decks/2024-10-24_The_good_the_bad_and_the_ugly/383cfee8/media/media4.mp4) ### Slide 127 Full-slide video demonstration against a Financial Info Copilot - slide 127 of 184 - Video: [Embedded video](https://media.mbgsec.com/decks/2024-10-24_The_good_the_bad_and_the_ugly/383cfee8/media/media5.mp4) ### Slide 128 Microsoft Copilot answer to a bank-details question, annotated by Tamir Ishay-Shar - slide 128 of 184 ### Slide 129 Actual Snippets: “ — What are the bank details for — TechCorp - slide 129 of 184 ### Slide 130 Act Three - slide 130 of 184 ### Slide 131 Copilot Hunter - slide 131 of 184 ### Slide 132 Power Pwn terminal output introducing Copilot Hunter - slide 132 of 184 ### Slide 133 One click away... - slide 133 of 184 ### Slide 134 Copilot Studio authentication screen revisiting the one-click public-access setting - slide 134 of 184 ### Slide 135 Diagram connecting a compromised Copilot to an exposed Amazon S3 bucket - slide 135 of 184 ### Slide 136 Footfall From Data Exposed Through Bucket Misconfiguration article with an exposed-bucket screenshot - slide 136 of 184 ### Slide 137 Animated meme agreeing that exposed storage through a Copilot creates the same data-leak risk - slide 137 of 184 - Animation: [Embedded animation](https://media.mbgsec.com/decks/2024-10-24_The_good_the_bad_and_the_ugly/383cfee8/media/image117.gif) ### Slide 138 Copilot Hunter command-line usage and required environment details - slide 138 of 184 ### Slide 139 Public Copilot demo website used to verify an enumerated bot - slide 139 of 184 ### Slide 140 Copilot demo website URL highlighting the environment and bot identifiers - slide 140 of 184 ### Slide 141 Copilot Hunter terminal scan results with a knock-knock illustration - slide 141 of 184 ### Slide 142 Copilot demo website URL showing the environment identifier - slide 142 of 184 ### Slide 143 Copilot demo website URL showing the solution publisher prefix - slide 143 of 184 ### Slide 144 Copilot demo website URL showing the demo website name - slide 144 of 184 ### Slide 145 Azure portal app-registration details used to locate a Power Platform environment ID - slide 145 of 184 ### Slide 146 Finding the values – Env/Tenant - slide 146 of 184 ### Slide 147 Finding the values – Env/Tenant - slide 147 of 184 ### Slide 148 Envs/tenant ids — Finding the values – Env/Tenant - slide 148 of 184 ### Slide 149 Finding the values – Env/Tenant - slide 149 of 184 ### Slide 150 Finding the values – Solution Publisher prefix - slide 150 of 184 ### Slide 151 Finding the values – Solution Publisher prefix - slide 151 of 184 ### Slide 152 Finding the values – Solution Publisher prefix - slide 152 of 184 ### Slide 153 Finding the values – demo website name - slide 153 of 184 ### Slide 154 Copilot demo website URL highlighting the demo website identifier - slide 154 of 184 ### Slide 155 Bot-name enumeration list illustrated with a slot machine - slide 155 of 184 ### Slide 156 Public web table listing Fortune 500 companies used for scanning targets - slide 156 of 184 ### Slide 157 Animated terminal scan celebrating discovered publicly accessible chatbots - slide 157 of 184 - Animation: [Embedded animation](https://media.mbgsec.com/decks/2024-10-24_The_good_the_bad_and_the_ugly/383cfee8/media/image137.gif) ### Slide 158 Domain and Tenant ID Scanning — Environment & Tenant ID Enumeration — Solution Prefix Reconnaissance - slide 158 of 184 ### Slide 159 Hundreds of — tenantId — and - slide 159 of 184 ### Slide 160 Public Copilot listing its available documents, with sensitive details redacted - slide 160 of 184 ### Slide 161 Public Copilot returning internal document details, with sensitive information redacted - slide 161 of 184 ### Slide 162 Public Copilot identifying documents that contain personally identifiable information - slide 162 of 184 ### Slide 163 Public Copilot response exposing sensitive document content, with identifying details redacted - slide 163 of 184 ### Slide 164 Public Copilot explaining that its instructions govern sensitive-information handling - slide 164 of 184 ### Slide 165 Spray Scanning — Advanced bot interaction — What’s next? - slide 165 of 184 ### Slide 166 Go Hack Yourself! — GITHUB.COM/MBRG/POWER-PWN - slide 166 of 184 ### Slide 167 Looking forward - slide 167 of 184 ### Slide 168 Takeaways — BREAKERs — BUILDERs - slide 168 of 184 ### Slide 169 Treat AI apps like experimental drugs - slide 169 of 184 ### Slide 170 Treat AI apps like experimental drugs — WE ARE THE CLINICAL TRIAL - slide 170 of 184 ### Slide 171 Takeaways — BREAKERs — BUILDERs - slide 171 of 184 ### Slide 172 We are all AI security n00bs — ALL OF US - slide 172 of 184 ### Slide 173 Takeaways — BREAKERs — BUILDERs - slide 173 of 184 ### Slide 174 Focus on — ~RCE — s - slide 174 of 184 ### Slide 175 Tread carefully - slide 175 of 184 ### Slide 176 Cloud — No Code — Customer - slide 176 of 184 ### Slide 177 Harden your environment — Turn off any toggle you can in the Power Platform DLP — Monitor the audit logs (kudos to Copilot Studio team lots of those) - slide 177 of 184 ### Slide 178 Follow the Frameworks - slide 178 of 184 ### Slide 179 And just this week... - slide 179 of 184 ### Slide 180 And just this week... - slide 180 of 184 ### Slide 181 And just this week... - slide 181 of 184 ### Slide 182 The good, the bad, and the ugly: Microsoft Copilot (delivered by Inbar Raz) - slide 182 of 184 ### Slide 183 Full-slide closing video clip before the final title slide - slide 183 of 184 - Video: [Embedded video](https://media.mbgsec.com/decks/2024-10-24_The_good_the_bad_and_the_ugly/383cfee8/media/media6.mp4) ### Slide 184 The good, the bad, and the ugly: — Microsoft Copilot — Speaker(s): - slide 184 of 184