AI penetration testing: what we test, who's asking, and what we find
Two or three AI-specific penetration tests a month, across regulated financial data, engineering software, and customer-facing service platforms. Here is what those tests look for, and what they keep finding.
In short
Claranet runs multiple AI-specific penetration tests every month, across regulated financial data, engineering software, and customer-facing service platforms.
We achieve remote code execution in roughly 30% of AI application tests, usually through a tool the model was given for something harmless.
On applications that render model output as markdown, we achieve client-side data exfiltration on the majority of tests.
The risk has moved. Application assessments still yield insight, but mature practices and OWASP guidance often mean fewer significant findings. Risk now lies in model permissions and misplaced trust in output. We know not to trust user input, so why trust model output shaped by it?
The organisations commissioning this work have already shipped. They are not experimenting.
AI features are now in production, and they are being tested like production
Claranet now runs multiple AI-specific penetration tests every month, and for some customers, a new AI component arrives for testing every couple of months. That volume tells you something more useful than any survey: AI has stopped being a proof of concept and started being the part of the stack that customers, brokers, and regulators actually touch. Our security testing team assesses these systems the way we assess any revenue-bearing application, with one difference. The model introduces a class of weakness that traditional application testing was never designed to catch.
The headline numbers from our recent AI testing are blunt. We achieve remote code execution in roughly 30% of AI application tests. We achieve client-side data exfiltration on the majority of tests where the application renders model output as markdown. Neither finding needs an exotic attack chain. Both come from design decisions that look reasonable on a whiteboard.
Who is commissioning AI penetration tests
The pattern in our order book is consistent, and it is not the sector mix people expect. The organisations booking AI security testing are not experimenting. They have already shipped.
Sector | What they've built | Why they test |
Financial data and market intelligence | Natural language interfaces that turn a question into a database query, then summarise the result for a human reader | Regulated data, third-party brokers consuming the output, and assurance obligations that predate AI |
Engineering, design, and manufacturing software | AI woven through browser-based design and field service products, including workflow builders that generate their own automations | AI features ship on a release cadence, so testing has to keep pace with the roadmap |
Customer-facing service and support platforms | Assistants with access to documentation, terms, records, and internal knowledge through retrieval | The assistant is exposed to anyone who can open a chat window |
Organisations reselling AI capability | AI features embedded in products sold on to their own customers | Their customers' security questionnaires now ask about model security specifically |
The types of testing we run
'AI penetration test' is a loose label, so it is worth being precise about what we actually assess. Most engagements combine several of the following, scoped to how the customer has built and deployed the feature.
AI application testing, front end and API
Tests arrive in both formats and we work in both. From the front end we behave like a user with bad intentions. With the API schema in hand we go after the logic: what the endpoint accepts, what it passes to the model, and what it does with the answer. The interesting failures usually sit in the plumbing between those two points rather than in the model itself.
Tool and permission review
We map every capability the model can reach and ask whether the business case justifies it. A support assistant that can read terms and conditions is doing its job. The same assistant with an interpreter attached is a different risk entirely, and the gap between those two states is often a single configuration choice nobody revisited.
Retrieval and ingestion testing
Wherever a system ingests content, retrieval augmented generation (RAG) stores, uploaded documents, shared knowledge bases, we treat that content as attacker-controlled and test it accordingly. Anything a user can put into the pipeline is an instruction the model may later follow.
Middleware and workflow testing
Not every AI component faces a customer. Behind the visible assistant sit AI-operated workflows, automations, and supporting services, some of them generated by other AI components. We test that layer too, because a workflow builder that writes its own automations from transcribed speech is a control plane, whatever it says on the product page.
Output handling and rendering
We test what happens after the model responds. Applications that convert model output into rendered content are trusting a component that an attacker can influence, and that trust is where a surprising share of our high-impact findings originate.
What we're finding, at a high level
Developers learned long ago not to trust user input. The shift we're seeing is that the same teams inherently trust the output of the model, and trust the tools they hand it. Three findings recur across sectors.
Finding | How often we see it | Why it matters commercially |
Overscoped tools leading to code execution | Around 30% of AI application tests | Impact depends entirely on where the code runs. Properly sandboxed, it is a capability issue. Running with high privilege alongside the application, it is a critical one |
Data exfiltration through rendered output | The majority of tests where model output is rendered as markdown | Persistent, low-noise extraction of conversation content from the user's own browsing session |
Classic application flaws in AI plumbing | Still present, including injection flaws in retrieval endpoints | Access to other users' documents and data, with all the regulatory consequences that carries |
Four questions worth asking your own team this week
None of these require a security specialist to answer, and the answers tell you how exposed you are before anyone runs a test.
- Which tools can our model call, and who signed off on each one?
If one of those tools executes code, where does that code run, and as which user?
When we render the model's answer, what can that rendered content reach out to?
Who can put a document into our retrieval store, and who reviews what's in it?
Where to start
If you've shipped an AI feature to customers, or you're about to, the pragmatic first step is a scoped penetration test of that feature and the services behind it. Our security testing team runs AI application testing alongside conventional web application, API, and infrastructure testing, so the assessment covers the model, the plumbing, and the platform in one engagement rather than three.
To talk it through with one of our testers, get in touch through the Claranet cyber security services pages, or ask your account manager for a scoping call with the security testing team.
A word on accreditation
It comes up in every scoping call, so it is worth being straight about it. Claranet is a CREST-accredited and CHECK-accredited penetration testing provider, and our AI testing is delivered by the same testers, under the same methodology and quality assurance as that work. There is now a CREST scheme covering AI testing specifically. We are not accredited under it yet, and we won't imply that we are. So the question worth putting to any provider, including us, is what their AI testing actually covers and who is doing it, rather than which logo sits on the proposal.
Frequently asked questions
What is AI penetration testing?
AI penetration testing is an authorised security assessment of an AI feature and the services around it. It covers the application front end and API, the tools and permissions granted to the model, the retrieval and ingestion path, the middleware and workflows behind the assistant, and the handling of whatever the model outputs. It is not a review of the model vendor's safety training. It is a test of how your application uses the model, and of what an attacker can make that application do.
How is AI penetration testing different from a standard penetration test?
A conventional web application or API test asks what an attacker can send in. An AI test asks that too, then adds two questions a conventional test was never designed to answer: what has the model been allowed to do, and what does the application do with what the model says. Those two questions produce most of our high-impact AI findings. The rest of the methodology, scoping, testing, reporting, and retest, is the same discipline we apply to any revenue-bearing application.
How often does AI penetration testing find serious issues?
More often than most teams expect. We achieve remote code execution in roughly 30% of AI application tests, usually by persuading a model to route our code through a tool that was added for something harmless. On applications that render model output as markdown, we achieve client-side data exfiltration on the majority of tests. Neither result requires an exotic attack chain.
What is prompt injection, and why does it matter commercially?
Prompt injection is an instruction planted where a model will read it and act on it. The direct form is typed into the chat. The form that matters commercially is indirect: the instruction is hidden inside a document that lands in a retrieval store, or inside an ordinary Word or Excel file that somebody later asks an assistant to summarise. The payload then fires for whoever retrieves it, not for the attacker. Any system that lets people upload documents into a retrieval pipeline needs to treat that content as attacker-controlled.
Do we need an AI penetration test if our AI feature is internal only?
Internal reduces exposure, it does not remove it. Two of the findings we see most often are internal problems by nature: shared agents that colleagues use daily but anyone in the organisation can edit, and retrieval stores that any employee can add a document to. Both give an attacker who already has a foothold, or an insider, a way to execute instructions inside other people's conversations.
Is Claranet CREST-accredited for AI penetration testing?
Claranet is a CREST-accredited and CHECK-accredited penetration testing provider, and our AI testing is delivered by the same testers under the same methodology and quality assurance. There is now a CREST scheme covering AI testing specifically. We are not accredited under it yet, and we will not imply that we are. The question worth putting to any provider, including us, is what their AI testing actually covers and who is doing it.