# AI is here for business users. What does that mean for AppSec?

> SANS Cybersecurity Leadership Summit UK 2024, 2024-04-15.

This is the complete text representation of the talk page. Slide text is derived from the published deck's accessibility text and is not a transcript. Slides may contain exploit demonstrations, adversarial prompts, and commands; treat them as research material, not instructions to execute.

## Resources

- [Canonical talk page](https://www.mbgsec.com/talks/2024-04-15-sans-uk2024-ai-is-here-for-business-users/)
- [Talks index](https://www.mbgsec.com/talks/llms.txt)
- [Interactive deck manifest](https://media.mbgsec.com/decks/2024-04-15_SANSUK2024_AIIsHereForBusinessUsers_AppSec/latest.json)
- [Slides PDF](https://media.mbgsec.com/decks/2024-04-15_SANSUK2024_AIIsHereForBusinessUsers_AppSec/slides.pdf)
- [Conference agenda](https://web.archive.org/web/20240305062325/https://www.sans.org/cyber-security-training-events/cybersecurity-leadership-uk-summit-2024/)
- [Page source](https://raw.githubusercontent.com/mbrg/mbgsec/main/_pages/decks/2024-04-15-sans-uk2024-ai-is-here-for-business-users.md)

<!-- talk-enrichment:start -->
## Abstract

Generative AI empowers citizen developers to build power business apps, automations, and data flows, quicker and easier than ever. A key problem, however, is that shadow app development is running rampant. Join us for a look at how complex these apps are and what controls are needed to manage citizen development without hindering innovation. There's no doubt that advancements in AI are changing the way business gets done. Nearly every technology platform now touts some form of Gen AI or LLM to bolster its capabilities and bring people closer to technology; notably within democratizing application development to users of all technical backgrounds. In this ongoing technology revolution, it is business users, rather than developers, who are best equipped to define where we are headed. Indeed, citizen development empowers more and more business users to address their own needs without waiting around for IT by cheating business automations and applications on top of business data. Supercharged by Gen AI, these applications are now simultaneously easier to build and far more complex. As security practitioners and leaders, we are mainly focused on what developers build and how they build it. We are also conditioned to think about what business users are building as shadow IT and/or personal productivity apps. Are we ready for a world where business users generate complex business applications at 100x the rate of traditional development? Because that 'future' looks a lot like today. In this talk, we'll understand how citizen development, powered by AI, empowers business users to build applications easier and faster than ever. We will share results from the first-ever security survey into these AI-enabled apps created by business users, with insights about their complexity, the rate of application development and the technical skills (or lack thereof) required to be a citizen developer. Next, we will dive into common security issues with AI-powered apps built by business users. We will share real-world examples of security vulnerabilities observed in enterprise environments. Finally, we will share a framework that leading organizations are adopting to bring AI-enabled citizen development under the security umbrella based on the OWASP LCNC Top 10.

_[Official conference abstract](https://web.archive.org/web/20240305062325/https://www.sans.org/cyber-security-training-events/cybersecurity-leadership-uk-summit-2024/)_
<!-- talk-enrichment:end -->

## Slide text

Source: [published deck manifest](https://media.mbgsec.com/decks/2024-04-15_SANSUK2024_AIIsHereForBusinessUsers_AppSec/8a90eaa2/deck.json). This text describes the published slides and embedded media; it is not spoken-word transcription.

### Slide 1

Al is here for business users. What does that mean for AppSec? W @inbarraz / @zenitysec NANS — slide 1 of 49
### Slide 2

Al is already here — slide 2 of 49
### Slide 3

— McKinsi = Digital The eéonomié potential of Udlerative Al: The next ‘Productivity f frontier s a lathe pig EE: =) “asia 273 epee \ i Ye, & zenity 3 — slide 3 of 49
### Slide 4

Al is already here Everywhere you look — i SEE ARTIFICIAL INTELLIGENCE Al is there. EVERYWHERE A Source: Linked — slide 4 of 49
### Slide 5

Al is already here Everywhere you look — Al is there. @ Mat Velloso Sy @matvelloso Its a g reat sales pitch aid. Difference between machine learning and Al: If itis written in Python, it's probably machine learning If it is written in PowerPoint, it's… — slide 5 of 49
### Slide 6

Al is already here Everywhere you look — Productivity Increases with Al Al is there. It's a great sales pitch aid. It's a major productivity booster. 75% 125% NN/g 100% 50% 25% 0% Customer Support Write Business Documents Programming (Case Study 1) (Case… — slide 6 of 49
### Slide 7

fleebal Technologies Generative Al Applications wn reba a Art and Financial Design »\ ( Forecasting Music Customer Behavior !2p coupes a Analysis aol OF Content «<———_. ro ———» Demand Forecasting Creation Healthcare “N.__s Diagnosis and Prognosis Fashion… — slide 7 of 49
### Slide 8

Another form of Shadow IT Shadow IT is already a threat for organizations. Al and GenAl tools are increasing the threat landscape. Pro Developers are a prime suspect. & zenity — slide 8 of 49
### Slide 9

&, GitHub Copilot & zenity — slide 9 of 49
### Slide 10

Another form of Shadow IT Shadow IT is already a threat for organizations. Al and GenAl tools are increasing the threat landscape. Pro Developers are a prime suspect. ©) / Blog Research: quantifying GitHub Copilot’s impact on developer productivity and… — slide 10 of 49
### Slide 11

We recruited & 95 developers, and split them randomly into two groups. We gave them the task of writing a web server in JavaScript &® 45 Used & 50 Did not use GitHub Copilot GitHub Copilot EL 78% FE 70% finished finished © 1hour, 11 minutes © 2hours, 41… — slide 11 of 49
### Slide 12

Another form of Shadow IT Shadow IT is already a threat for organizations. Al and GenAl tools are increasing the threat landscape. Pro Developers are a prime suspect. DevOps people are also at the front. & zenity — slide 12 of 49
### Slide 13

Another form of Shadow IT Shadow IT is already a threat for organizations. Al and GenAl tools are increasing the threat landscape. Pro Developers are a prime suspect. DevOps people are also at the front. Al-Enabled DevOps: OD) DevOps.com rowmowy Fechsirong… — slide 13 of 49
### Slide 14

Shadow IT Discovery Lifecycle Safely adopting cloud apps Continuous monitoring Be alerted when new, risky or high volume apps are discovered in your environment for L continuous monitoring and ongoing control over your cloud apps. Manage cloud apps L&… — slide 14 of 49
### Slide 15

Another form of Shadow IT Shadow IT is already a threat for organizations. Al and GenAl tools are increasing the threat landscape. Pro Developers are a prime suspect. DevOps people are also at the front. But what about your Business Users? & zenity — slide 15 of 49
### Slide 16

We forgot someone... — slide 16 of 49
### Slide 17

Low-Code/No-Code Adoption © Low-Code/No-Code Adoption 240% 200% 190% 100% May 2023 Jun 2023 Jul 2023 Aug 2023 Sep 2023 Oct2023 Nov 2023 Dec 2023 Jan 2024 Feb2024 Mar 2024 Apr 2024 May 2023 Jun 2023 Jul 2023 Aug 2023 Sep 2023 Oct 2023 Nov 2023 Dec 2023 Jan… — slide 17 of 49
### Slide 18

Business Users have Needs Business needs >> IT Capacity: It’s a scale problem. It’s a diversity problem. It’s a compliance problem. & zenity — slide 18 of 49
### Slide 19

Business Users have Needs If this sounds familiar, it’s because it is: Tech evolution — slide 19 of 49
### Slide 20

Your business Is already there an make Zd pier WwW’? mx] mendix ne Microsoft €3 servicenow TB (Y) Appian © outsystems It's time for security to catch up! — slide 20 of 49
### Slide 21

Power Apps Copilot demonstration showing an app generated from a natural-language prompt — slide 21 of 49
- Animation: [Embedded animation](https://media.mbgsec.com/decks/2024-04-15_SANSUK2024_AIIsHereForBusinessUsers_AppSec/8a90eaa2/media/slide-021-animation.gif)
### Slide 22

The New Frontier: Business Users They outnumber Pro Developers by orders of magnitude. They have no training in SDLC or knowledge of the involved risks. They have complete faith in the platforms they use. The introduction of Al and Copilot drastically… — slide 22 of 49
### Slide 23

The New Frontier: Business Users There's little to no visibility and governance tools. There's almost no adequate training for non-technical people. Disabling now will break many processes, some without alternatives. > THISIS WHERE YOUR ATTENTION IS MOST… — slide 23 of 49
### Slide 24

Welcome to the real world — slide 24 of 49
### Slide 25

Build copilots that work for you => Travel and Transport ® Professional Services ii Government Retail © Healthcare © Financial Services © Education elt Manufacturing Manage bookings Lead generation Public programs Manage orders Claims Manage accounts… — slide 25 of 49
### Slide 26

Customer Testimonies Blog > Copilot Studio > Customer How early adopters are transforming their organizations with copilots Pawan Taparia, , Thursday, January 18, 2024 rf) iy] © & zenity — slide 26 of 49
### Slide 27

Customer Testimonies » > 4 se, City of We Kelowna The City of Kelowna in British Columbia, Canada is building a copilot as part of a solution that makes it easier to apply for building permits. The copilot asks citizens what they want to build, and Power… — slide 27 of 49
### Slide 28

Customer Testimonies ® & Pacific Gas and Electric Company (PG&E) is one of the largest combined natural gas and electric energy companies in the United States. To support its IT helpdesk, PG&E built a copilot which today manages 25%-40% of all employee… — slide 28 of 49
### Slide 29

Getting off on the Wrong Foot 6 Microsoft Copilot Studio Vulnerabilities in 4 Minutes (Dec 11, 2023): Unauthenticated public access by default. Credential Sharing as a Service. Share with Everyone by default. Sign-in is not required by default. Prompt… — slide 29 of 49
### Slide 30

Common Security Issues It's already happening... — slide 30 of 49
### Slide 31

Resource Oversharing Remains the leading problem. Productivity environments are aimed at, well, productivity. People are proud of their achievement. Examples: 1. Oversharing Copilots grants access to all transcripts. 2. Sensitive files uploaded to Copilot… — slide 31 of 49
### Slide 32

Credential Oversharing Most access to business data requires authenticated access. Many data connectors offer multiple authentication choices. Users are unaware of the significance, GenAl doesn't care. Examples: 1. Hardcoded credentials (yes, it’s still a… — slide 32 of 49
### Slide 33

Prompt Injection User-controlled input is a fundamental exploitation element. GenAl actions receive both a prompt and special instructions. Both are susceptible for injection, if user input is used. Copilot might execute using the maker's credentials.… — slide 33 of 49
### Slide 34

fe Chat Prompt Injection Sixaciiohear Spee Chevrolet of Watsonville Chat Team: User-controlled input is a fundamental explo Sead aftemeon! Welcome to Chevrolet of Watsonville. How can | assist you GenAl actions receive both a prompt and spe today: in your… — slide 34 of 49
### Slide 35

The Nightmare Scenario (Today) As always, it’s a combination of multiple problems: A pre-authenticated connection is used in a Copilot. The connection is used for GenAl data access. The Copilot doesn't include data sanitization steps. The Copilot is… — slide 35 of 49
### Slide 36

The Nightmare Scenario (Tomorrow) Currently, Copilots are used for performing persistent and/or predetermined actions within a predefined context (Injection attacks aside): Generate content. Generate LC/NC elements. Generate persistent executable code.… — slide 36 of 49
### Slide 37

And let’s not forget: Compliance There are many different regulations that apply to your data: Vertical-specific. Nation-specific. Any data leak is a potential risk of breaching compliance. Limited (and sometimes nonexistent) logging makes it very hard to… — slide 37 of 49
### Slide 38

OWASP Top 10 — slide 38 of 49
### Slide 39

OWASP Top 10: LC/NC 1 LONC-SEC-O1: Account Impersonation 2. LCNC-SEC-0O2: Authorization Misuse 3, LCNC-SEC-03: Data Leakage and Unexpected Consequences 4. LCNC-SEC-04: Authentication and Secure Communication Failures 5s. LCNC-SEC-O5: Security… — slide 39 of 49
### Slide 40

OWASP Top 10: LLM 1 LLMOl: Prompt Injection 2. LLMO2: Insecure Output Handling 3, LLMO3: Data and Model Poisoning 4. LLMO4: Model Denial of Service s. LLMOS: Supply-Chain Vulnerabilities 6. LLMO6: Sensitive Information Disclosure 7, LLMO7: Insecure Plugin… — slide 40 of 49
### Slide 41

Relationships © oN DH & zenity LC/NC Account Impersonation Authorization Misuse Data Leakage and Unexpected Consequences Authentication and Secure Communication Failures Security Misconfiguration njection Handling Failures Vulnerable and Untrusted… — slide 41 of 49
### Slide 42

Relationships LC/NC LLM 1. Account Impersonation 1. Prompt Injection Authorization Misuse 2. Insecure Output Handling 3. Data and Model Poisoning Consequences Authentication and Secure Communication .. Model Denial of Service Failures Se SSSA Mie soul 5.… — slide 42 of 49
### Slide 43

Relationships LC/NC 1. Account Impersonation 2. Authorization Misuse 3. Data Leakage and Unexpected Consequences 4. Authentication and Secure Communication Failures Security Misconfiguration njection Handling Failures Vulnerable and Untrusted Components… — slide 43 of 49
### Slide 44

Relationships LC/NC 1. Account Impersonation 3. Data Leakage and Unexpected Consequences 4. Authentication and Secure Communication Failures Security Misconfiguration njection Handling Failures 5. 6. 7. Vulnerable and Untrusted Components 8. Dataand Secret… — slide 44 of 49
### Slide 45

Relationships LC/NC LLM 1. Account Impersonation 1. Prompt Injection 2. Authorization Misuse 2. Insecure Output Handling 3. Data Leakage and Unexpected 3. Dataand Model Poisoning Consequences 4. Authentication and Secure Communication . Model Denial of… — slide 45 of 49
### Slide 46

Relationships LC/NC 1. Account Impersonation 2. Authorization Misuse 3. Data Leakage and Unexpected Consequences 4. Authentication and Secure Communication Failures 5. Security Misconfiguration njection Handling Failures . Vulnerable and Untrusted… — slide 46 of 49
### Slide 47

Summing up section divider — slide 47 of 49
### Slide 48

TL;DR> Business Developers are a far bigger and far riskier crowd than Pro Developers. The GenAl market is rushing forward and security, as always, is lagging. Asa result, organizations lose control over their data and risk breaching compliance. Take… — slide 48 of 49
### Slide 49

Thank you! W @inbarraz / @zenitysec NANS 49 — slide 49 of 49