Security in AI

Since the hype about ChatGPT, I see a lot of questions arise around security in AI. That such a system can and will be abused to support malicious activities is bound to happen. It is the nature of humans (at least a subset of humans). But AI offers a lot of upsides we should leverage.

The question for ordinary security people like me is more how can we understand, assess and mitigate risks to support the business to leverage the upsides. I learned a lot about AI in recent years, but I still feel insecure in some areas how to act on that from a risk and resilience perspective.

At Microsoft we developed Responsible AI principles a while ago and are enforcing them in the company. Additionally we have people doing AI red teaming to understand how our AI models could be manipulated.

In this context, there are two blogs, which you could be interested in as well. The first one is Best practices for AI security risk management linking you to an AI risk management framework, which I feel is interesting to look at. Especially if you are coming from the “classical” Cybersecurity, you will realize that a lot of good practices still apply (obviously). Controls like in IAM will not go away. How do you protect the developer, the code, the production environment etc.

There are a few points, which I feel are important, nevertheless:

• Protect and govern training and test data: Interestingly, there are two facets to that. Often, when talking to customers they look at the confidentiality of data. While this is important with the training and test data for AI as well, integrity of the data plays an important role. This does encompass altering of existing data but the trust in the data source as well.
• Model development: On the one hand side, this is a classical way of controlling a development process but there is way more. You need a threat model on the AI model, you need to think about how a bad person could abuse it. As always with such models, you need to understand the underlying technology, you need to understand AI and AI models to do that.
• Model Deployment: The same seems true to me when we look at security and compliance reviews of AI models. I am in security since a long time, but I am not sure, whether I could ask the right questions in such a review (and I feel that a checklist might not help there). Actually, I am sure that I could not… You need an AI security person to do these reviews.

Reading the AI Security Risk Management, which is linked in the blog (it has 20 pages) is definitely worth it.

In a second blog (AI security risk assessment using Counterfit), we talk about Counterfit, a tool we open-sourced to automate security testing of AI systems.

I feel that this is a a good starting point for security people wanting to learn AI – or the risks of AI.