Security Considerations When Using AI Frameworks

When AI Meets Security: The Blind Spot We Can't Afford Working in this field has revealed a troubling reality: our security practices aren't evolving as fast as our AI capabilities. Many organizations still treat AI security as an extension of traditional cybersecurity—it's not. AI security must protect dynamic, evolving systems that continuously learn and make decisions. This fundamental difference changes everything about our approach. What's particularly concerning is how vulnerable the model development pipeline remains. A single compromised credential can lead to subtle manipulations in training data that produce models which appear functional but contain hidden weaknesses or backdoors. The most effective security strategies I've seen share these characteristics: • They treat model architecture and training pipelines as critical infrastructure deserving specialized protection • They implement adversarial testing regimes that actively try to manipulate model outputs • They maintain comprehensive monitoring of both inputs and inference patterns to detect anomalies The uncomfortable reality is that securing AI systems requires expertise that bridges two traditionally separate domains. Few professionals truly understand both the intricacies of modern machine learning architectures and advanced cybersecurity principles. This security gap represents perhaps the greatest unaddressed risk in enterprise AI deployment today. Has anyone found effective ways to bridge this knowledge gap in their organizations? What training or collaborative approaches have worked?

Is your AI model actually safe? ....The answer is more complicated than a simple yes or no. Many treat AI models like standard open-source software, checking the creator license and functionality. But this is a dangerous oversimplification. The term Open Source itself is misleading here. Unlike software where you can inspect the source code "open" AI models are often just open weights a massive file of numbers. You can't see the training data or the process that created them, making them a black box that's impossible to fully verify or reproduce. This opacity creates a massive attack surface. Scans have found hundreds of thousands of issues, including malicious models designed to exfiltrate data. The threats are real and evolving. So how do we secure the un-securable? Focus on three layers: The Model Itself: Source from trusted providers and rigorously evaluate for vulnerabilities like prompt injection, the number 1 security risk for LLMs according to OWASP. Continuous benchmarking is non-negotiable . The Infrastructure: The software stack running the model is a critical vulnerability. A model even if safe is only as secure as the infrastructure it runs on. Enforce strict privilege controls and secure your inference toolchain. The Integration: How does the model interact with your systems? A helpful model given excessive agency can become an unknowing accomplice, manipulated to expose system vulnerabilities or leak data. The models are innocent. It is the context they are used in that creates the risk. Security isn't a one time check, it's a continuous process of evaluation monitoring and mitigation. It's time we started treating it that way. What's your biggest concern when deploying a local AI models? #AI #Safety

🤖 𝐄𝐯𝐞𝐫𝐲𝐨𝐧𝐞’𝐬 𝐭𝐚𝐥𝐤𝐢𝐧𝐠 𝐚𝐛𝐨𝐮𝐭 𝐀𝐈 𝐚𝐝𝐨𝐩𝐭𝐢𝐨𝐧 – 𝐛𝐮𝐭 𝐡𝐚𝐫𝐝𝐥𝐲 𝐚𝐧𝐲𝐨𝐧𝐞 𝐢𝐬 𝐭𝐚𝐥𝐤𝐢𝐧𝐠 𝐚𝐛𝐨𝐮𝐭 𝐀𝐈 𝐬𝐞𝐜𝐮𝐫𝐢𝐭𝐲. 🔐 As a CISO, I see the rapid rollout of AI tools across organizations. But what often gets overlooked are the unique security risks these systems introduce. Unlike traditional software, AI systems create entirely new attack surfaces like: ⚠️ 𝐃𝐚𝐭𝐚 𝐩𝐨𝐢𝐬𝐨𝐧𝐢𝐧𝐠: Just a few manipulated data points can alter model behavior in subtle but dangerous ways. ⚠️ 𝐏𝐫𝐨𝐦𝐩𝐭 𝐢𝐧𝐣𝐞𝐜𝐭𝐢𝐨𝐧: Malicious inputs can trick models into revealing sensitive data or bypassing safeguards. ⚠️ 𝐒𝐡𝐚𝐝𝐨𝐰 𝐀𝐈: Unofficial tools used without oversight can undermine compliance and governance entirely. We urgently need new ways of thinking and structured frameworks to embed security from the very beginning. 📘 A great starting point is the new 𝐒𝐀𝐈𝐋 (𝐒𝐞𝐜𝐮𝐫𝐞 𝐀𝐈 𝐋𝐢𝐟𝐞𝐜𝐲𝐜𝐥𝐞) Framework whitepaper by Pillar Security. It provides actionable guidance for integrating security across every phase of the AI lifecycle from planning and development to deployment and monitoring. 🔍 𝐖𝐡𝐚𝐭 𝐈 𝐩𝐚𝐫𝐭𝐢𝐜𝐮𝐥𝐚𝐫𝐥𝐲 𝐯𝐚𝐥𝐮𝐞: ✅ More than 𝟕𝟎 𝐀𝐈-𝐬𝐩𝐞𝐜𝐢𝐟𝐢𝐜 𝐫𝐢𝐬𝐤𝐬, mapped and categorized ✅ A clear phase-based structure: Plan – Build – Test – Deploy – Operate – Monitor ✅ Alignment with current standards like ISO 42001, NIST AI RMF and the OWASP Top 10 for LLMs 👉 Read the full whitepaper here: https://lnkd.in/ebtbztQC How are you approaching AI risk in your organization? Have you already started implementing a structured AI security framework? #AIsecurity #CISO #SAILframework #SecureAI #Governance #MLops #Cybersecurity #AIrisks

Is your team still treating AI systems exactly like regular software when it comes to security? 🤔 I've been digging into NIST's draft Cyber AI Profile (IR 8596), which I think is essential reading for any GRC professional. The comment period closed last Friday, and this guidance confirms something many of us have felt for a while: AI challenges some of the core assumptions behind our traditional security frameworks. Unlike typical software which behaves predictably AI models are probabilistic and keep evolving. That means we face a new class of risks that require us to rethink our approach. A few takeaways for those of us in GRC: 💡 1️⃣ Static Checklists Don't Cut It: Because AI behavior is less predictable, relying solely on fixed checklists risks missing important threats. The guidance encourages adopting risk models designed specifically for AI's unique uncertainties. 2️⃣ New Threats Require New Defenses: Attacks like prompt injection, data poisoning, and model extraction aren't simply variations of traditional threats like malware or SQL injection. These AI-specific risks call for tailored mitigation strategies. 3️⃣ Seeing Beyond Vendor Reports: A SOC 2 report isn't enough anymore. To truly understand AI security, you have to trace data lineage, model origins, and base models. That means gaining much deeper insight into the AI supply chain. 4️⃣ Keep an Eye on AI Models Continuously: The draft stresses ongoing monitoring to catch things like model drift, unexpected behavior, and adversarial manipulation as soon as they happen. For those guiding AI risk and compliance programs, this is a strong nudge to update your frameworks. It also reinforces my conviction that the future belongs to practitioners fluent in both AI's technical landscape and sound governance principles. Although the comment period has closed, I encourage you to review the draft. Understanding this guidance now will help you prepare for the compliance landscape that's taking shape. If you're wrestling with how to handle AI's probabilistic risks, I'd be glad to swap notes on what I'm learning. 🤝 Find the draft here --> https://lnkd.in/gzxHSsQb #AIGovernance #GRC #Cybersecurity #AIrisk #NIST #RiskManagement

⚠️ Most companies treat AI agents like chatbots. But most of us know that this means - it’s only a matter of time before it causes a major security incident. Here’s what i experienced at an example company: An AI agent monitoring cloud infrastructure. It doesn’t just respond. It observes, reasons, and executes actions across multiple systems. That means it can: - Read logs - Trigger deployments - Update tickets - Execute scripts All without direct human prompting. My approach after years in cybersecurity & AI is to use a 5-Layer Security Model when reviewing AI agent security: 1️⃣ Prompt Layer Where instructions enter the system (user messages, docs, tickets). ⚠️ Risk: Prompt injection – hidden instructions can trick the agent into executing real commands. 2️⃣ Knowledge / Memory Layer Agents retrieve context from logs, docs, or vector databases and connects to internal resources with potential sensitive information. ⚠️ Risk: Data poisoning – malicious content can influence future decisions. 3️⃣ Reasoning Layer (LLM) Application comes in contact with you LLM - where the model decides what to do. ⚠️ Risk: Hallucinations/unintentional leakage – confident but incorrect suggestions could trigger unsafe actions. 4️⃣ Tool / Action Layer AI Agents interact with APIs, CI/CD pipelines, databases, and infra. ⚠️ Risk: Unauthorized execution – a single manipulated prompt could impact production systems. 5️⃣ Infrastructure / Control Plane The container, runtime, identities, secrets, and policy engines live here. ⚠️ Risk: Agent hijacking – compromise this layer, and attackers control every decision. 💡 Rule of thumb: Never allow an AI agent to perform an action you cannot observe, audit, or override. Curious — how are you approaching AI agent security? #aisecurity #ai

Most security frameworks were built for a world where software does exactly what you tell it to do, every time. Agentic AI breaks that assumption. Agents use LLMs to carry out actions on their own, at machine speed, with real-world consequences. And because they’re non-deterministic, the same request can produce different results each time. That’s a fundamentally different operating model, and it raises questions our industry needs to answer well. NIST’s Center for AI Standards and Innovation recently issued a Request for Information asking for industry input on how we should secure these systems. We submitted a response based on our experience building and operating agentic AI services at AWS, and we published a blog summarizing the four security principles at the core of it. A few points I’d emphasize for anyone thinking about how to secure agents at their own organization: 1. Secure foundations are more important than ever. Every traditional attack technique, including denial-of-service, man-in-the-middle, vulnerability and configuration exploitation, supply chain, log tampering, etc., remains relevant in agentic contexts. AI-specific controls must be additions to foundational security, not replacements for it. 2. Don’t rely on the agent to secure itself. Even if you tell an LLM to refuse certain requests, crafty prompt injection techniques can override those instructions. Security boundaries need to be enforced by infrastructure outside the agent that governs what it can access and do. And these controls must be deterministic. 3. Autonomy should be earned, never granted by default. Start by having humans make the final call on high-consequence operations. As you gather evidence that the agent performs reliably, expand its autonomy gradually. And be ready to pull it back when the data says you should. 4. Be thoughtful about human-in-the-loop oversight. If every action requires approval, reviewers get overwhelmed and start rubber-stamping. Focus human oversight on the decisions that genuinely carry high stakes. We’re all figuring this out in real time, and no single organization has all the answers. The more we share what we’re learning, the faster the whole industry moves forward. For more details on how to apply these principles, check out the links below. Full response to NIST: https://lnkd.in/enxE8R-V Blog post: https://lnkd.in/eRg3uc26

The Cybersecurity and Infrastructure Security Agency (CISA), together with other organizations, published "Principles for the Secure Integration of Artificial Intelligence in Operational Technology (OT)," providing a comprehensive framework for critical infrastructure operators evaluating or deploying AI within industrial environments. This guidance outlines four key principles to leverage the benefits of AI in OT systems while reducing risk: 1. Understand the unique risks and potential impacts of AI integration into OT environments, the importance of educating personnel on these risks, and the secure AI development lifecycle.  2. Assess the specific business case for AI use in OT environments and manage OT data security risks, the role of vendors, and the immediate and long-term challenges of AI integration 3. Implement robust governance mechanisms, integrate AI into existing security frameworks, continuously test and evaluate AI models, and consider regulatory compliance.  4. Implement oversight mechanisms to ensure the safe operation and cybersecurity of AI-enabled OT systems, maintain transparency, and integrate AI into incident response plans. The guidance recommends addressing AI-related risks in OT environments by: • Conducting a rigorous pre-deployment assessment. • Applying AI-aware threat modeling that includes adversarial attacks, model manipulation, data poisoning, and exploitation of AI-enabled features. • Strengthening data governance by protecting training and operational data, controlling access, validating data quality, and preventing exposure of sensitive engineering information. • Testing AI systems in non-production environments using hardware-in-the-loop setups, realistic scenarios, and safety-critical edge cases before deployment. • Implementing continuous monitoring of AI performance, outputs, anomalies, and model drift, with the ability to trace decisions and audit system behavior. • Maintaining human oversight through defined operator roles, escalation paths, and controls to verify AI outputs and override automated actions when needed. • Establishing safe-failure and fallback mechanisms that allow systems to revert to manual control or conventional automation during errors, abnormal behavior, or cyber incidents. • Integrating AI into existing cybersecurity and functional safety processes, ensuring alignment with risk assessments, change management, and incident response procedures. • Requiring vendor transparency on embedded AI components, data usage, model behavior, update cycles, cybersecurity protections, and conditions for disabling AI capabilities. • Implementing lifecycle management practices such as periodic risk reviews, model re-evaluation, patching, retraining, and re-testing as systems evolve or operating environments change.

Something feels safe about AI systems. Clean interface. Confident answers. Fast decisions. But behind the screen, a different story is unfolding. Many organizations are racing to deploy Generative AI. Few are slowing down to examine the risks hidden inside the architecture. Security teams are beginning to realize a difficult truth. The biggest risk is not the model. The biggest risk is how systems trust the model. The 𝐎𝐖𝐀𝐒𝐏 𝐓𝐨𝐩 10 𝐟𝐨𝐫 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈 highlights where things quietly break. These risks are already appearing in production environments. → 𝐎𝐯𝐞𝐫𝐫𝐞𝐥𝐢𝐚𝐧𝐜𝐞 𝐨𝐧 𝐋𝐋𝐌 𝐎𝐮𝐭𝐩𝐮𝐭 • Blind trust in generated responses • Hallucinated code entering production • Fabricated citations treated as facts → 𝐏𝐫𝐨𝐦𝐩𝐭 𝐈𝐧𝐣𝐞𝐜𝐭𝐢𝐨𝐧 • Malicious instructions hidden in prompts • System behavior manipulation • Instruction override attacks → 𝐌𝐨𝐝𝐞𝐥 𝐓𝐡𝐞𝐟𝐭 • Reverse engineering through repeated queries • Model inversion techniques • Membership inference attacks → 𝐄𝐱𝐜𝐞𝐬𝐬𝐢𝐯𝐞 𝐀𝐠𝐞𝐧𝐜𝐲 • AI agents granted broad permissions • Autonomous file deletion • Automated email execution → 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐃𝐚𝐭𝐚 𝐏𝐨𝐢𝐬𝐨𝐧𝐢𝐧𝐠 • Malicious data inserted into training pipelines • Biased or manipulated model behavior • Hidden backdoors in datasets → 𝐈𝐧𝐬𝐞𝐜𝐮𝐫𝐞 𝐎𝐮𝐭𝐩𝐮𝐭 𝐇𝐚𝐧𝐝𝐥𝐢𝐧𝐠 • Unvalidated model responses used downstream • Injection into applications • XSS, SSRF, privilege escalation, RCE → 𝐒𝐞𝐧𝐬𝐢𝐭𝐢𝐯𝐞 𝐈𝐧𝐟𝐨𝐫𝐦𝐚𝐭𝐢𝐨𝐧 𝐃𝐢𝐬𝐜𝐥𝐨𝐬𝐮𝐫𝐞 • Leakage of credentials or PII • Exposure of system prompts • Reconstruction of memorized data → 𝐌𝐨𝐝𝐞𝐥 𝐃𝐞𝐧𝐢𝐚𝐥 𝐨𝐟 𝐒𝐞𝐫𝐯𝐢𝐜𝐞 • Expensive recursive prompts • Latency spikes • Rapid cost escalation → 𝐒𝐮𝐩𝐩𝐥𝐲 𝐂𝐡𝐚𝐢𝐧 𝐕𝐮𝐥𝐧𝐞𝐫𝐚𝐛𝐢𝐥𝐢𝐭𝐢𝐞𝐬 • Compromised model weights • Risky third party datasets • Unsafe external APIs →  𝐈𝐧𝐬𝐞𝐜𝐮𝐫𝐞 𝐏𝐥𝐮𝐠𝐢𝐧 𝐃𝐞𝐬𝐢𝐠𝐧 • Weak input validation • Excessive OAuth permissions • One plugin compromising entire systems Generative AI is powerful. But power without governance creates silent vulnerabilities. Security in AI is no longer optional. It is architecture. Cyber Leadership Academy Follow Vijay Banda for more insights

𝗧𝗟;𝗗𝗥: 𝗔𝗜 𝗮𝗴𝗲𝗻𝘁 𝘀𝗲𝗰𝘂𝗿𝗶𝘁𝘆 𝗶𝘀 𝘁𝗵𝗲 𝘂𝗻𝘀𝗽𝗼𝗸𝗲𝗻 𝗽𝗿𝗲𝗿𝗲𝗾𝘂𝗶𝘀𝗶𝘁𝗲 𝗮𝗻𝗱 𝗯𝗹𝗼𝗰𝗸𝗲𝗿 𝗳𝗼𝗿 𝘄𝗶𝗱𝗲𝘀𝗽𝗿𝗲𝗮𝗱 𝗮𝗴𝗲𝗻𝘁𝗶𝗰 𝗮𝗱𝗼𝗽𝘁𝗶𝗼𝗻. Recent announcements from Cloudflare and Vercel introducing secure sandboxing solutions signal a critical industry shift — recognition that running untrusted, AI-generated code requires robust isolation mechanisms. Increasingly agents will take on tasks impersonating humans, do we trust what they are doing? Are we ready to face the consequences with bad agentic actions? I realize the image is a bit provocative but we are not far from that happening! 𝗧𝗵𝗲 𝗦𝗲𝗰𝘂𝗿𝗶𝘁𝘆 𝗚𝗮𝗽 According to OWASP's latest threat taxonomy (https://bit.ly/4m38uEW), organizations face 15 distinct threats that could compromise their AI agents. Despite proliferation of agentic frameworks, almost none tackle secure sandboxed runtime environments. 𝗖𝗼𝗺𝗽𝗮𝗻𝗶𝗲𝘀 𝗿𝘂𝘀𝗵𝗶𝗻𝗴 𝘁𝗼 𝗱𝗲𝗽𝗹𝗼𝘆 𝗮𝗴𝗲𝗻𝘁𝘀 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗿𝗼𝗯𝘂𝘀𝘁 𝘀𝗲𝗰𝘂𝗿𝗶𝘁𝘆 𝗳𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸𝘀 𝗮𝗿𝗲 𝘀𝗲𝘁𝘁𝗶𝗻𝗴 𝘁𝗵𝗲𝗺𝘀𝗲𝗹𝘃𝗲𝘀 𝘂𝗽 𝗳𝗼𝗿 𝗽𝗼𝘁𝗲𝗻𝘁𝗶𝗮𝗹𝗹𝘆 𝗰𝗮𝘁𝗮𝘀𝘁𝗿𝗼𝗽𝗵𝗶𝗰 𝗳𝗮𝗶𝗹𝘂𝗿𝗲𝘀. 𝗥𝗲𝗰𝗲𝗻𝘁 𝗮𝗻𝗻𝗼𝘂𝗻𝗰𝗲𝗺𝗲𝗻𝘁𝘀 Cloudflare's new Code Sandboxes feature (https://lnkd.in/gJe-TMMx) addresses the critical need to run untrusted, LLM-written code safely through secure, container-based environments. Vercel's Sandbox SDK (https://lnkd.in/gki56JUt) takes a complementary approach with ephemeral, isolated microVMs supporting execution times up to 45 minutes. 𝗧𝗵𝗲 𝗦𝗮𝗻𝗱𝗯𝗼𝘅 𝗜𝗺𝗽𝗲𝗿𝗮𝘁𝗶𝘃𝗲 - Both solutions demonstrate that secure AI agent execution isn't theoretical — it's becoming industry standard. 𝗔𝗰𝘁𝗶𝗼𝗻 𝗳𝗼𝗿 𝗖𝗜𝗦𝗢𝘀 & 𝗖𝗘𝗢𝘀: Treat AI agent security as a strategic imperative. Implement defense-in-depth approaches addressing all OWASP-identified threats. Reject any framework that 𝗱𝗼𝗲𝘀𝗻'𝘁 𝗿𝘂𝗻 𝗮𝗴𝗲𝗻𝘁𝘀 𝗶𝗻 𝘀𝗲𝗰𝘂𝗿𝗲 𝗶𝘀𝗼𝗹𝗮𝘁𝗲𝗱 𝗰𝗼𝗻𝘁𝗮𝗶𝗻𝗲𝗿𝘀 or 𝗹𝗮𝗰𝗸𝘀 𝗽𝗿𝗼𝗽𝗲𝗿 𝗶𝗱𝗲𝗻𝘁𝗶𝘁𝘆 𝗺𝗼𝗱𝗲𝗹𝘀. Comprehensive security should be built in or enabled by default — 𝗻𝗼𝘁 𝗿𝗲𝘁𝗿𝗼𝗳𝗶𝘁𝘁𝗲𝗱 𝗮𝗳𝘁𝗲𝗿 𝗱𝗲𝗽𝗹𝗼𝘆𝗺𝗲𝗻𝘁.

# AI Security Back to AI Security basics. I keep doing this every few months because folks aren't ready to hear it until they are ready to hear it. AI isn't magic. It isn't a "black box" from a component integration perspective. Current LLMs will take in tokens and then they will push out tokens. Everything else must be in the software that wraps the AI from actions to security controls. The LLM is an untrusted component. ## Tool Calls The core of tool calling in an LLM is conventional text processing. This is no different than the text processing that happens when evaluating an HTTP request. There will be regularized text that describe the tool being called and the parameters it should be executed with. The http server, or agentic framework or model server in our case, extracts those components and completes the action. *Security:* If you wouldn't expose it to the internet for the HTTP request, don't expose it to the model. If you wouldn't allow it to be accessed without tight scoping and good filtering that is user specific to the internet for the HTTP request, don't expose it differently to the model. *Take-away:* The browsers and internet users are not trusted entities when designing a web application. The user controlled/influenced model is not a trusted entity when designing an AI based/augmented application. ## Identity The core of identity with respects to LLMs/AI/Agents is identical to any other service, or even user - no exceptions. When designing a service, it must be restricted to least privilege actions by code designed to apply ACLs. You never let a service or user "self police" as that's the same thing as saying "Please be nice!" and just hope it works. *Security:* If you wouldn't just let a user run wild with calls on your application, you shouldn't be letting a model do it either. The LLM is an untrusted component just like the user. Treat it as such. It can never implement its own security controls via prompting. Prompting may help with user experience, but it is not a security control or a mitigation in any circumstance. *Take-away:* We MUST act according to what our frameworks and best practice say, the model is not a trusted component. We need to restrict it with conventional code that can rationally apply security controls. # Summary Do not ever trust the model. Treat it as a potentially malicious user. Add controls external to the model to remediate hazards. Prompts are *NEVER-EVER* security controls. Defensive prompting doesn't exist in a security context, only in a user experience context. # Bonus Model - the model takes in tokens and outputs tokens. The model cannot do anything on its own. Agent Framework - the software that makes prompt context and tool using capabilities via text processing possible. The conventional code that parses text for input to the model and parses text from model outputs for call tools and prompt flow. Inference Server - The software that loads the model and exposes inference calls.