"We're worried about data leaks with generative AI, so we don't let people use it yet." Even with that decision in place, it's not unusual for employees to start using it with personal accounts. A ban imposed without knowing what actually leads to a leak tends not to be followed.
This article is for people in IT and general affairs who set the rules for using generative AI at their company. We take up five data leak cases that actually happened with generative AI and lay out, in order, what happened, where the cause lay, and how to prevent it.
The cases are based only on statements by the parties involved and news reports we could confirm as of November 4, 2026. Figures and sequences of events are as written in the articles listed as sources.
Bottom line: the causes fall into five types, four of which are paths for conversation content to get out
- Employee input: an employee enters confidential information. At Samsung Electronics, engineers entered internal source code into ChatGPT
- Service bug: a bug on the provider's side shows other users' information. This happened with ChatGPT in March 2023
- Sharing settings: a setting chosen by the person who shared a conversation makes it appear in search results. This happened with ChatGPT share links in July 2025
- Stolen login credentials: malware on a computer steals login credentials, and past conversations may be read along with them. A study published in 2023 found this on about 100,000 devices
- Prompt injection: instructions are planted in emails the AI reads, making it send internal information outside. This was disclosed in June 2025 as a vulnerability in Microsoft 365 Copilot
- The first four are paths by which what someone wrote in a conversation reaches people they never intended. What a company can do is decide which services to use, and mask personal and confidential information before sending
Case 1: Employees entered source code (Samsung Electronics, 2023)
What happened
According to a Bloomberg report on May 2, 2023, at Samsung Electronics in South Korea, engineers uploaded internal source code to ChatGPT and accidentally leaked it outside the company. In response, the company banned the use of generative AI on company devices and its internal network.
In its notice to employees, the company explained that data sent to generative AI is stored on external servers, making it difficult to retrieve or delete. In the company's internal survey, 65% of respondents considered generative AI a security risk. It also told employees that violating the rules could lead to disciplinary action, including dismissal.
Cause
There was no attack and no bug; employees simply entered information as part of their everyday work. The information they entered went to servers outside the company and could not be taken back afterward.
How to prevent it
Draw the line between information that may be entered and information that must not, and share it with examples. However, rules alone can't prevent an oversight in the moment of pasting.
Samsung Electronics banned use, but a total ban can also lead to "shadow AI," where employees who want to use it start doing so on personal devices and accounts. What to provide instead of a ban is laid out in Shadow AI and generative AI data leak prevention.
Case 2: A service bug showed other users' information (OpenAI, March 2023)
What happened
On March 20, 2023, some ChatGPT users saw the titles of other users' chat histories on their screens. The cause was a bug in an open-source library (redis-py) that OpenAI used.
Because of the same bug, for about 1.2% of ChatGPT Plus subscribers who were active during the nine hours from 1 a.m. to 10 a.m. Pacific Time on March 20, their name, email address, payment address, last four digits of their credit card number, and card expiration date may have been shown to other users. OpenAI said the full card number was never shown.
Cause
It was a bug in the provider's service, not something users did. It can't be prevented on the user's side.
How to prevent it
Since you can't prevent bugs on the provider's side, use the service on the assumption that anything you enter may get out someday. If no personal information is written in a conversation, the damage is small even if it's shown to someone else.
Case 3: Shared conversations appeared in search results (OpenAI, July 2025)
What happened
ChatGPT has a feature for sharing a conversation via a link. The screen for creating a share link had a checkbox, "Make this chat discoverable," and at the end of July 2025 it was reported that conversations with that box checked were appearing in Google search results.
Dane Stuckey, OpenAI's CISO (chief information security officer), withdrew the feature, saying it "introduced too many opportunities for folks to accidentally share things they didn't intend to." The company said it was also working to remove conversations that had been indexed by search engines.
Cause
It was neither a bug nor an attack but a setting the users chose. The box had to be checked by the user, but the explanation that the chat would "appear in web searches" was written in small gray text below the checkbox.
How to prevent it
Decide whether conversation share links may be given to people outside the company. If you share one, first check that no personal information or confidential material remains in the conversation.
Case 4: Stolen login credentials put past conversations at risk of being read (2022 to 2023)
What happened
On June 20, 2023, the security company Group-IB announced that it had identified 101,134 devices infected with information-stealing malware (infostealers) that had ChatGPT login credentials saved on them. These logs were being traded on dark web marketplaces. By region, Asia-Pacific had the most (40.5%), and the number of logs peaked in May 2023 at 26,802.
Group-IB noted that ChatGPT keeps all conversations by default, and pointed out that this could make it a trove of confidential information for attackers who obtain login credentials.
Cause
ChatGPT itself was not breached; the cause was malware that got into users' devices. If someone logs in with stolen credentials, they can read the account's entire conversation history.
How to prevent it
Group-IB recommends changing passwords regularly and setting up two-factor authentication. As a company, make two-factor authentication mandatory for accounts used for work. To limit the damage if history is read, it's also important not to leave personal information in it.
Case 5: Instructions were planted in emails the AI reads (Microsoft 365 Copilot, June 2025)
What happened
In June 2025, a vulnerability in Microsoft 365 Copilot called "EchoLeak" (CVE-2025-32711) was disclosed. It was found by the security company Aim Security. Simply by sending a crafted email, an attacker could make Copilot send internal information outside when it read the email, even if the recipient never opened it.
Microsoft said it had completed a fix on the server side and that no action was needed from users. No evidence of actual exploitation has been confirmed.
Cause
This is an attack called "prompt injection." AI sometimes can't tell the difference between instructions written in text it reads and instructions from the user. The more email and internal documents the AI can read, the more chances it has to encounter planted instructions.
How to prevent it
This type is hard to prevent on the user's side, and much of it depends on waiting for the provider's fix. As a company, narrow what the AI reads (email, shared folders, and so on) to what's necessary, and check the provider's security information. Masking information before sending does not prevent this type of leak.
What the five cases show
| Case | Type | What got out | Could the user side have prevented it |
|---|---|---|---|
| Samsung Electronics (2023) | Employee input | Internal source code | Preventable by not entering it |
| ChatGPT bug (2023) | Service bug | History titles, part of payment information | Not preventable |
| ChatGPT share links (2025) | Sharing settings | Shared conversations | Preventable by checking before sharing |
| Group-IB study (2023) | Stolen login credentials | Login credentials (with the risk of past conversations being read) | Can be reduced with two-factor authentication and device protection |
| EchoLeak (2025) | Prompt injection | Internal information Copilot can read | Not preventable (fixed by the provider) |
Some can't be prevented on the user's side, but cases 1 through 4 have something in common. Each is a path by which what was written in a conversation gets out. Whatever the path, if no personal or confidential information is written in the conversation, less content gets out.
What a company should decide
- Decide which services and plans may be used: use business plans for work, and don't handle work information in personal accounts
- Draw the line on what may be entered: decide by category, such as personal information, confidential material, source code, passwords, and API keys, and add examples. How to think about the categories is summarized in Is it safe to put personal information into ChatGPT?
- Decide how share links are handled: decide whether conversations may be shared outside the company, and what to check before sharing
- Protect accounts: make two-factor authentication mandatory for accounts used for work
- Narrow what the AI reads: for AI connected to email or shared folders, narrow what it can read to what's necessary
- Put a check in place before input: rules alone can't prevent an oversight in the moment of pasting
How to mask personal information before sending
Either replace names with "Person A" and email addresses with "(email)" before pasting, or make the request in masked form, such as "a thank-you email to [Name]," and fill in the details yourself at the end. For summarizing, organizing tables, or drafting emails, you almost never need the personal information itself.
When there's a lot of it, replacing by hand lets things slip through. A browser extension that checks the content the moment you paste can reduce oversights. When you introduce one, check who provides it and whether it sends the content outside for inspection.
Checking before sending, inside the browser: Samurai App Data Guard
Samurai App's Chrome extension "Samurai App Data Guard" checks the content before it's sent when you paste text or attach a file on a generative AI site such as ChatGPT, Gemini, Claude, or Copilot. It's built for moments like case 1, when someone pastes something as part of their everyday work.
- What it finds: names, email addresses, phone numbers, addresses, dates of birth, My Number IDs, credit card numbers, bank account numbers, API keys and tokens, and passwords. It finds names using cues such as table headers and labels like "Name:"
- What you can choose: Cancel, Send as is, Send partly masked, or Send masked. When masked, each found item is replaced with a category name such as
[NAME]or[EMAIL] - Checks happen only inside the browser: it doesn't use an AI model and checks by fixed rules. It never sends content outside for inspection
Personal information masked before sending doesn't remain in a conversation shown by a bug, in a shared conversation, or in the history of a stolen account. The flow from pasting a customer list into ChatGPT to masking and sending it is shown in the video on the PII detection page (45 seconds). For details, see the PII detection page.
Sources
- Samsung bans staff's AI use after spotting ChatGPT data leak (Bloomberg article republished by The Star, May 2, 2023)
- OpenAI: ChatGPT payment data leak caused by open-source bug (BleepingComputer, March 24, 2023)
- OpenAI is removing ChatGPT conversations from Google (Engadget, August 1, 2025)
- Group-IB Discovers 100K+ Compromised ChatGPT Accounts (Group-IB, June 20, 2023)
- CVE-2025-32711 (NVD, published June 11, 2025)
- EchoLeak AI Attack Enabled Theft of Sensitive Data via Microsoft 365 Copilot (SecurityWeek, June 12, 2025)