Data Privacy Risks When Using AI Tools

This Stanford study examined how six major AI companies (Anthropic, OpenAI, Google, Meta, Microsoft, and Amazon) handle user data from chatbot conversations.  Here are the main privacy concerns. 👀 All six companies use chat data for training by default, though some allow opt-out 👀 Data retention is often indefinite, with personal information stored long-term 👀 Cross-platform data merging occurs at multi-product companies (Google, Meta, Microsoft, Amazon) 👀 Children's data is handled inconsistently, with most companies not adequately protecting minors 👀 Limited transparency in privacy policies, which are complex and hard to understand and often lack crucial details about actual practices Practical Takeaways for Acceptable Use Policy and Training for nonprofits in using generative AI: ✅ Assume anything you share will be used for training - sensitive information, uploaded files, health details, biometric data, etc. ✅ Opt out when possible - proactively disable data collection for training (Meta is the one where you cannot) ✅ Information cascades through ecosystems - your inputs can lead to inferences that affect ads, recommendations, and potentially insurance or other third parties ✅ Special concern for children's data - age verification and consent protections are inconsistent Some questions to consider in acceptable use policies and to incorporate in any training. ❓ What types of sensitive information might your nonprofit staff  share with generative AI?  ❓ Does your nonprofit currently specifically identify what is considered “sensitive information” (beyond PID) and should not be shared with GenerativeAI ? Is this incorporated into training? ❓ Are you working with children, people with health conditions, or others whose data could be particularly harmful if leaked or misused? ❓ What would be the consequences if sensitive information or strategic organizational data ended up being used to train AI models? How might this affect trust, compliance, or your mission? How is this communicated in training and policy? Across the board, the Stanford research points that developers’ privacy policies lack essential information about their practices. They recommend policymakers and developers address data privacy challenges posed by LLM-powered chatbots through comprehensive federal privacy regulation, affirmative opt-in for model training, and filtering personal information from chat inputs by default. “We need to promote innovation in privacy-preserving AI, so that user privacy isn’t an afterthought." How are you advocating for privacy-preserving AI? How are you educating your staff to navigate this challenge? https://lnkd.in/g3RmbEwD

🚨 AI Privacy Risks & Mitigations Large Language Models (LLMs), by Isabel Barberá, is the 107-page report about AI & Privacy you were waiting for! [Bookmark & share below]. Topics covered: - Background "This section introduces Large Language Models, how they work, and their common applications. It also discusses performance evaluation measures, helping readers understand the foundational aspects of LLM systems." - Data Flow and Associated Privacy Risks in LLM Systems "Here, we explore how privacy risks emerge across different LLM service models, emphasizing the importance of understanding data flows throughout the AI lifecycle. This section also identifies risks and mitigations and examines roles and responsibilities under the AI Act and the GDPR." - Data Protection and Privacy Risk Assessment: Risk Identification "This section outlines criteria for identifying risks and provides examples of privacy risks specific to LLM systems. Developers and users can use this section as a starting point for identifying risks in their own systems." - Data Protection and Privacy Risk Assessment: Risk Estimation & Evaluation "Guidance on how to analyse, classify and assess privacy risks is provided here, with criteria for evaluating both the probability and severity of risks. This section explains how to derive a final risk evaluation to prioritize mitigation efforts effectively." - Data Protection and Privacy Risk Control "This section details risk treatment strategies, offering practical mitigation measures for common privacy risks in LLM systems. It also discusses residual risk acceptance and the iterative nature of risk management in AI systems." - Residual Risk Evaluation "Evaluating residual risks after mitigation is essential to ensure risks fall within acceptable thresholds and do not require further action. This section outlines how residual risks are evaluated to determine whether additional mitigation is needed or if the model or LLM system is ready for deployment." - Review & Monitor "This section covers the importance of reviewing risk management activities and maintaining a risk register. It also highlights the importance of continuous monitoring to detect emerging risks, assess real-world impact, and refine mitigation strategies." - Examples of LLM Systems’ Risk Assessments "Three detailed use cases are provided to demonstrate the application of the risk management framework in real-world scenarios. These examples illustrate how risks can be identified, assessed, and mitigated across various contexts." - Reference to Tools, Methodologies, Benchmarks, and Guidance "The final section compiles tools, evaluation metrics, benchmarks, methodologies, and standards to support developers and users in managing risks and evaluating the performance of LLM systems." 👉 Download it below. 👉 NEVER MISS my AI governance updates: join my newsletter's 58,500+ subscribers (below). #AI #AIGovernance #Privacy #DataProtection #AIRegulation #EDPB

*Let's talk about how we use AI tools in our work and personal life without increasing the risk for accidental data leakage, breaches, or extortion* First and foremost, feeding national intelligence documents (or any sensitive docs) into an AI tool to determine which parts should remain classified is not the move (see photo below). Why? Many AI-based systems lack strong contextual decision-making, which can lead to accidental disclosure of private or classified materials from the AI tool. *When it comes to work related AI-usage we have to consider the following*: - Does your org have a policy against or for AI tool usage? What about local AI tool instances that are recommended for your team? - Does your organization use prompt protection tools to avoid accidental data leakage from your user's questions (aka prompts)? - Does the doc have secrets, proprietary data, employee data, customer data, passwords, etc embedded within it? Be careful and avoid entering this data to avoid data leakage -- redact first. - Do you secure your LLM tools with long random unique passwords + MFA? If you reuse passwords (and that password shows up in a data breach) it could lead to a hack and subsequent leak of your sensitive AI queries, work, details of an M&A that hasn't been announced etc leading to an even larger breach or data leak. *What does this mean for everyday folks who use AI for their everyday life?* You still can use AI! Just redact sensitive info before entering it into AI tools. For example: if you want to use AI tools to understand, say, a report from your doctor, I recommend removing personal details like your name, address, birthdate, etc before feeding it into AI tools to avoid accidental data leakage from the AI tool. *I predict within the next 6 months bad actors will start heavily targeting credentials for AI chat bots used by organizations to leak and extort prompt history and other sensitive questions and data* - They may attempt to leverage a reused password against your organization to gain access to your team AI tools and leak/extort your history that has sensitive data within it - They may attempt to phish individual's passwords/codes to leak or extort high net worth individuals AI chat bot history outside of the office - Bad actors may increase their targeting of AI chat bot infrastructure to encourage the tool to inadvertently leak sensitive details or proprietary info that users have entered into the tool *Actions to protect yourself and your team from AI tool risk* 1. Redact sensitive or proprietary info before entering into AI tools 2. Secure your AI tools with long random unique passwords + MFA to avoid extortion for hacked embarrassing or sensitive AI query history 3. Notice and report phishing against those AI credentials -- attackers use urgency and fear (such as "your account has been compromised, click here to secure") to get you to click and enter a password/code for that AI tool

A recent issue has emerged where private ChatGPT conversations, once shared, have become publicly searchable on Google. This is a huge red flag for HR. Conversations containing sensitive information, like employee personal details from CVs, confidential business plans, or even legal advice, are now potentially exposed. My key takeaways: ▶️ Data Privacy Nightmare: This isn't just a technical glitch; it's a massive data privacy risk. Imagine employee PII, performance review details, or internal strategy documents showing up in a public search. This could lead to serious breaches and legal repercussions under regulations like GDPR or state privacy laws. ▶️ Policy and Training Gap: The root of the problem is a lack of awareness. Employees are using AI tools without fully understanding the privacy and security implications. This is a clear indicator that your AI policy needs to be robust and your training needs to be a top priority. Do your employees know what they should and shouldn't be putting into AI tools, or sharing from them? ▶️ Mitigation is Key: 🔸Audit Your Tools: Review which AI tools your employees are using and what data they might be processing. 🔸Revise Your Policy: Update your acceptable use policy to explicitly address the use of generative AI, including what types of information are strictly forbidden from being inputted or shared. 🔸Train Your People: Conduct urgent training sessions to raise awareness about the risks of sharing conversations from AI tools. This situation highlights the critical need for a proactive approach to AI governance in HR. It's no longer just about the tech; it's about the people using it and the sensitive data they handle. What's your biggest concern about employees using generative AI?

Before diving headfirst into AI, companies need to define what data privacy means to them in order to use GenAI safely. After decades of harvesting and storing data, many tech companies have created vast troves of the stuff - and not all of it is safe to use when training new GenAI models. Most companies can easily recognize obvious examples of Personally Identifying Information (PII) like Social Security numbers (SSNs) - but what about home addresses, phone numbers, or even information like how many kids a customer has? These details can be just as critical to ensure newly built GenAI products don’t compromise their users' privacy - or safety - but once this information has entered an LLM, it can be really difficult to excise it. To safely build the next generation of AI, companies need to consider some key issues: ⚠️Defining Sensitive Data: Companies need to decide what they consider sensitive beyond the obvious. Personally identifiable information (PII) covers more than just SSNs and contact information - it can include any data that paints a detailed picture of an individual and needs to be redacted to protect customers. 🔒Using Tools to Ensure Privacy: Ensuring privacy in AI requires a range of tools that can help tech companies process, redact, and safeguard sensitive information. Without these tools in place, they risk exposing critical data in their AI models. 🏗️ Building a Framework for Privacy: Redacting sensitive data isn’t just a one-time process; it needs to be a cornerstone of any company’s data management strategy as they continue to scale AI efforts. Since PII is so difficult to remove from an LLM once added, GenAI companies need to devote resources to making sure it doesn’t enter their databases in the first place. Ultimately, AI is only as safe as the data you feed into it. Companies need a clear, actionable plan to protect their customers - and the time to implement it is now.

ChatGPT is not your friend. It’s a database. In July 2025, Google indexed over 4,500 ChatGPT conversations containing sensitive personal information. Because users clicked “Share,” and the system created public URLs. Google crawled, indexed and shared them. Here’s what surfaced: 🔸 Mental illness, addiction, and abuse 🔸 Names, locations, emails, resumes 🔸 Medical histories, legal strategies All searchable, linkable and public until OpenAI intervened: ✔️ The “Discoverable” sharing feature was disabled on July 31. ✔️ They are working with Google and other search engines to remove indexed chats. ✔️ OpenAI reminded users: deleting a chat from history does not delete the public link. Millions of people, including employees and customers are confiding in AI. They believe it’s private and safe. But it isn’t. It’s recording. Indexing. Storing. And when systems designed for experimentation are used for confession, the boundaries between personal risk and enterprise liability vanish. What are the implications for Boards? 1️⃣ Regulatory risk Under GDPR: 🔹 Data subjects have the right to erase, access, and informed consent. 🔹 Shared AI conversations with personal or sensitive data may violate these rights. 🔹 AI-generated prompts could fall under automated decision-making clauses. Under the EU AI Act: 🔹 Transparency, risk classification, and human oversight are mandatory. 🔹 This incident may be classified as a high-risk system failure in healthcare, HR, legal. 2️⃣ Legal risk There is currently no legal confidentiality in AI interactions. ✔️ Anything entered into AI could be subpoenaed, discoverable in court or leaked. ✔️ Companies are liable if employees share PII, IP, or client data via chatbots. ✔️ HR, Legal, and Compliance teams must assume AI logs are discoverable records. 3️⃣ Reputational risk People assumed they were talking to a trusted tool. Instead, they ended up on Google. For enterprises using AI for: ▫️ Coaching or mental health ▫️ HR assistance ▫️ Legal or compliance advisory ▫️ Customer service … this is a trust risk. Public exposure = brand damage. 4️⃣ Operational risk Many organisations lack: 📌 AI input/output governance 📌 Policies for AI use in confidential workflows 📌 Deletion/audit protocols for AI-linked data Takeaway If employees or customers treat ChatGPT like a coach, or colleague, ensure to treat it like a legal and technical system. That means: ✅ Create AI use and data handling policies ✅ Restrict use of genAI in regulated or sensitive domains ✅ Review GDPR/AI Act exposure for all shared AI features ✅ Treat all AI interactions as auditable records ✅ Demand transparency from vendors: what is stored, shared, indexed? Until regulators catch up and new legal protections exist, assume every AI interaction is public, permanent, and admissible. #AIgovernance #Boardroom #EUAIACT #DigitalTrust #Stratedge

Remember a week or so ago when I reported ChatGPT made all its shared conversations Public on Google? Guess what - Grok just did the same thing. Surprised? 370,000 private conversations between users and AI chatbots are now suddenly as public as a billboard on Times Square. 🤯 Executive Summary Elon Musk's xAI recently faced a significant privacy incident where their Grok chatbot inadvertently made hundreds of thousands of user conversations publicly searchable through Google and other search engines. The issue arose from Grok's "share" feature - when users clicked the share button to distribute conversations via email or text, the system generated unique URLs that search engines automatically indexed without any user notification. The exposed content ranged from routine requests like meal planning to deeply personal medical consultations, financial information, and even passwords. More troubling were conversations containing detailed instructions for illegal activities, including drug manufacturing and explosive creation. This breach highlights a fundamental flaw in how AI companies handle user privacy - what users believed were private interactions became permanently accessible to anyone with an internet connection. This incident represents more than just a technical oversight. It reveals the urgent need for AI companies to prioritize transparency about data handling practices. Unlike other platforms that quickly addressed similar issues, xAI has remained largely silent, raising questions about their commitment to user protection and responsible AI development. The Future Looking ahead, this breach will likely accelerate regulatory scrutiny of AI privacy practices. We can expect stricter requirements for explicit consent before making any user data publicly accessible, mandatory privacy impact assessments for AI features, and potentially significant financial penalties for companies that mishandle user information. The incident may also drive the development of privacy-by-design AI systems where user protection is built into the core architecture rather than added as an afterthought. What You Should Think About Before your next AI interaction, consider these actionable steps: Review the privacy settings of every AI tool you use, never share sensitive information like passwords or personal details with chatbots, and always read the fine print about data sharing policies. For businesses, conduct regular audits of your AI tools' privacy practices and establish clear guidelines for employee AI usage. How has this incident changed your approach to AI interactions? What privacy safeguards do you think should be mandatory for all AI platforms? 💭 Source: forbes

The world’s leading AI tools can be tricked into leaking sensitive data with just one carefully crafted prompt. As an AI advisor who works with Fortune 100 companies like PwC and Cisco, I’m seeing a whole new world of ‘AI security’ emerging. And it’s the scariest thing you’ll see this Halloween. AI's guardrails — meant to protect against hallucinations & hate — are not security guardrails! Recent research shows attackers can: 🚨 Extract personal information from AI tools 🚨 Bypass security measures with simple text prompts 🚨 Turn harmless queries into data-stealing commands 🚨 Make AI systems ignore their safety protocols The scariest part? These LLMs are already operating in everyday tools: → Google has integrated them into core search systems  → Tesla is using them to control vehicles → Microsoft has embedded them in Office tools → Robotics companies are building LLM-powered machines And it gets worse: even if you don’t have a proprietary tool, these vulnerabilities are present in AI tools you use every day. 😲 You should be concerned. All your personal (& business) data is at risk - documents you process through AI could be exposed, and automated systems could be compromised. Including things you share about your kids with family, medical or financial advisors. Every physical device with LLMs could be a gateway for hacking. These aren't hypothetical scenarios. Researchers at UC San Diego and Nanyang Technological University Singapore just demonstrated how simple prompts can trick AI into collecting and reporting personal information. The AI even disguises this breach with an invisible response – you wouldn't know your data was stolen. The industry is working on solutions, but here's the reality: we're racing to put AI everywhere before solving these fundamental security issues. ⚠️ As someone who’s been in this field for over a decade, I am a huge fan of AI's potential. But I also believe users need to understand these risks. What concerns you most about AI security in the tools you use daily? What would you not want to expose to AI? #AI #datasecurity #privacy

AI Is Saving Lives—But At What Cost? 1 in 2 healthcare AIs may leak private data. The surge in AI across healthcare has transformed diagnostics, treatment plans, and efficiency metrics. But behind the innovation lies a growing blind spot: patient privacy. AI models are only as secure as the data practices behind them. AI inference isn’t always clean. It can echo names, Diagnoses, Genetic details. - Every output carries potential risk. - Every breach risks patient trust. - Every leak invites regulatory scrutiny. HIPAA and GDPR were built for structured systems—not generative models learning from terabytes of sensitive records. Worse still, most teams lack the cybersecurity expertise to manage the complexity of AI pipelines. > Ransomware attacks are escalating. > Staff shortages widen the gap. > And liability doesn't wait. For COOs and Ops leaders, this isn't just about compliance—it's about risk mitigation, long-term viability, and protecting what matters most: the patient. Is your AI infrastructure privacy-resilient—or just performant?

Sharing a few new papers this week! First up - my new analysis of AI companies' privacy policies. We find major companies train on childrens' data and that each company now trains on user data by default Here are a few highlights from the paper: ► We develop a novel qualitative coding schema based on the California Consumer Privacy Act and apply it to privacy policies for LLM chatbots from Amazon, Anthropic, Google, Meta, Microsoft, and OpenAI ► In addition to chat data, which these companies train on by default, companies may also train their AI models on other user provided data. This includes voice data (Microsoft, Google), images (OpenAI, Microsoft, Google), user data from other platforms (Meta, Microsoft) ► Companies make bold arguments for why users should not opt out of using data for training. OpenAI says users should “Improve the model for everyone: Allow your content to be used to train our models, which makes ChatGPT better for you and everyone who uses it” ► Unlike other companies, it is unclear whether it is possible to opt out of using chat data for training using Amazon or Meta's chatbots ► We recommend that (i) companies require user opt-in for use of chat data to train or improve AI systems (as Anthropic did earlier this year, before it changed its policies to mirror those of other companies), (ii) companies improve the transparency of their privacy policies as they apply to AI systems (much of the key information under California privacy law was not discoverable or was unclear), and (iii) US lawmakers should pass comprehensive privacy legislation in light of the risks from AI related to children's' rights, data leakage and surveillance Thanks to Stanford Institute for Human-Centered Artificial Intelligence (HAI) for supporting this work! It was great to collaborate with Dr. Jennifer King, Emily Capstick, Tiffany Saade, and Victoria H. I am presenting our work today in Madrid at the AI Ethics and Society conference