Artificial Intelligence (AI)
Is Your AI Really Secure? Why Red Team Testing Matters in Microsoft Foundry
AI applications can be manipulated into revealing data or acting outside their remit. Red teaming shows how yours fails when challenged, and where it needs more control.
Published
AI is transforming business operations, but how confident are you that your AI applications will behave as intended when someone deliberately tries to make them do something they shouldn’t?
Organisations are increasingly using artificial intelligence to automate processes, analyse sensitive information, support customer interactions and make business decisions. Platforms such as Microsoft Foundry (formerly Azure AI Foundry) make it easier than ever to develop and deploy sophisticated AI applications and autonomous agents.
However, with these capabilities comes a new category of security risk. Traditional cybersecurity measures remain essential, but they aren’t designed to address every vulnerability introduced by generative AI.
This is where AI Red Team Testing becomes important.
What Is AI Red Team Testing?
Traditional penetration testing attempts to identify weaknesses in networks, applications and infrastructure by simulating the behaviour of malicious actors. AI red teaming applies a similar principle to artificial intelligence.
Rather than simply checking whether an application is technically secure, red teaming examines whether an AI model or agent can be manipulated into producing inappropriate responses, revealing confidential information, ignoring its operating instructions or performing actions outside its intended authority.
Consider an AI assistant designed to answer employee questions about company policies. Under normal circumstances, it might perform perfectly. But what happens when someone deliberately attempts to persuade it to reveal restricted information, disregard its instructions or access information that the user shouldn’t be able to see?
These are precisely the types of scenarios that red teaming is designed to investigate.
The objective isn’t simply to establish whether an AI application works. It’s to understand how it might fail when deliberately challenged.
The Risks Businesses Need to Understand
AI introduces potential vulnerabilities that may not be obvious during conventional development and testing. Some of the most important include:
- Prompt Injection —An attacker attempts to manipulate an AI application’s behaviour by inserting instructions into material the AI processes. This could involve a user message, an uploaded document or information retrieved from an external system.
- Sensitive Information Disclosure —An AI application connected to internal documents, databases or business systems could potentially reveal information beyond what a user is authorised to access.
- Jailbreaking —A user deliberately attempts to circumvent an AI model’s established safety restrictions or operating instructions.
- Unauthorised Actions —AI agents are increasingly capable of interacting with applications, accessing data and initiating automated workflows. An inadequately protected agent might be manipulated into performing actions outside its intended scope.
- Unreliable or Inappropriate Responses —An AI system may produce misleading, unsafe or inappropriate responses that create reputational, operational or compliance risks.
These vulnerabilities become particularly significant when AI applications are integrated with enterprise systems such as SharePoint, SQL databases, APIs and automated business processes.
How Microsoft Foundry Supports AI Red Teaming
Microsoft Foundry provides capabilities to help organisations identify and evaluate potential weaknesses in their AI applications. One of these is Microsoft’s AI Red Teaming Agent, which supports automated adversarial testing.
Rather than relying exclusively on developers to anticipate every possible misuse scenario, automated red teaming simulates challenging or malicious interactions to assess how an AI system responds. The results can help organisations understand where existing safeguards are effective and where additional controls may be necessary.
Microsoft Foundry supports activities such as:
- Adversarial testing:Simulating attempts to make AI applications behave in unintended ways.
- Risk and safety evaluation:Assessing responses against supported security and responsible AI risk categories.
- Vulnerability identification:Highlighting potential weaknesses in the behaviour of models and supported AI agents.
- Reporting and assessment:Providing measurable results that can inform remediation priorities.
- Repeated evaluation:Helping organisations reassess risks as applications, models and configurations evolve.
Microsoft’s capabilities draw on technologies including the Python Risk Identification Tool (PyRIT), developed to support structured AI security testing.
These facilities provide a valuable foundation, but automated testing alone cannot guarantee that an AI application is secure. Specialist assessment remains important, particularly where applications interact with sensitive business information or external systems.
Why Testing the AI Model Alone Is Not Enough
An important consideration is that the security of an AI application extends beyond the underlying language model. A model supplied through Microsoft Foundry may already incorporate safeguards, but these do not automatically protect every application built around it.
For example, an AI agent could be connected to:
- SharePoint libraries containing confidential business documents.
- Azure SQL databases holding commercially sensitive information.
- Power Automate workflows capable of updating business records.
- External APIs that perform transactions or exchange information.
Each integration introduces another potential point of exposure. A secure foundation model does not necessarily mean that the complete solution is secure.
Organisations need to assess the entire AI solution, including its permissions, connected data sources, automated actions and business processes.
This is particularly important as businesses move beyond conversational AI towards autonomous agents capable of taking action on behalf of employees.
Red Teaming Should Be an Ongoing Process
AI security isn’t something that should be considered only immediately before deployment. An application that behaves appropriately today may respond differently following a model upgrade, changes to its instructions, the introduction of new data sources or additional integrations.
Red teaming should therefore form part of an organisation’s wider AI governance and testing strategy. Ideally, organisations should evaluate their AI solutions during development, before production deployment and following significant changes.
Regular assessment also provides valuable evidence that potential risks are being actively identified and managed. For organisations working towards structured information security frameworks such as ISO/IEC 27001, AI red teaming can contribute to the wider processes of risk assessment, security testing and continual improvement. It does not, by itself, establish compliance with any particular standard.
How PTR Associates Can Help
Implementing an AI application is only one part of the challenge. Understanding its security, reliability and exposure to business risk is equally important.
At PTR Associates, we work with organisations across Microsoft Azure, Power Platform, data integration and business automation technologies. These disciplines become particularly important when assessing AI solutions that interact with enterprise data and operational systems.
We can help organisations approach AI red teaming as part of a broader assessment of their AI applications and supporting architecture.
An engagement can be structured to examine the intended behaviour of an AI solution, identify relevant risks, assess existing safeguards and establish practical recommendations for improving security and resilience.
This may include reviewing how AI applications interact with business information, the permissions assigned to agents, and the controls governing automated processes.
The objective is to provide organisations with a clearer understanding of their AI risk exposure and a prioritised approach to addressing identified weaknesses.
Is Your Organisation Ready to Trust Its AI?
Generative AI is creating significant opportunities for businesses, but confidence in these systems needs to be supported by appropriate testing.
The question is no longer simply whether an AI application can deliver the functionality your organisation requires. It is whether that application will continue to operate safely, securely and predictably when confronted with unexpected or deliberately malicious behaviour.
If your organisation is developing, deploying or expanding AI solutions within Microsoft Azure, now is the time to consider how those systems are being tested.
Contact PTR Associates to discuss how an AI security and red teaming assessment could help you understand potential risks, strengthen existing safeguards and move forward with greater confidence.
Secure AI isn’t just about protecting technology. It’s about protecting the business processes, information and decisions that increasingly depend upon it.
NS
Neil Skipper
Senior Business Intelligence Consultant
A talented business analyst and delivery lead, Neil has over 20 years of experience working with data analytics and business intelligence technologies and platforms.
Related Articles
Frequently Asked Questions
Couldn’t find the answer you were looking for? Feel free to reach out to us! Our team of experts is here to help.
Contact Us