Large language models are now embedded in chatbots, copilots, knowledge bases (RAG), automations and semi-autonomous agents. They are in productive use, processing sensitive data, making preliminary decisions and triggering actions in connected systems. That makes them an attack surface themselves, in ways that classic penetration tests and automated scanners do not cover.
We test your AI systems against the established standards and then harden them together with you: testing and hardening from a single source, by the same people.
zentrust partners is recognized by the BSI as an IT security service provider in the scope of IS penetration testing (registration ID BSI-APS-9080). For testing AI systems we follow the Guideline for Penetration Testing of Large Language Models by the AI Security expert circle of the Alliance for Cyber Security, as well as the OWASP Top 10 for LLM Applications 2025. This keeps the assessment traceable, comparable and audit-ready.
Why AI systems need their own assessment
Penetration testing an AI system is not an ordinary application test with a few extra questions. Three properties make the difference:
- Non-deterministic behavior. The same input can produce different answers. Whether a vulnerability exists can often only be judged reliably through repeated and varied testing. A residual uncertainty remains that must be documented and made tangible through metrics such as the attack success rate.
- Attacks that are not in the source code. Prompt injection often arrives indirectly through the content the model processes: uploaded documents, retrieved web pages, emails or entries in a knowledge base. A pure code review does not find such attack paths.
- Limits of automated scanners. Publicly available LLM scanners cover individual attack classes but neglect combined and complex scenarios. They do not replace manual, creative testing by experienced testers. That is exactly why we combine several tools with targeted manual work.
| Aspect | Classic pentest | AI pentest |
|---|---|---|
| Behavior of the target | deterministic, reproducible | probabilistic, with residual uncertainty |
| Central attack vectors | injection, access control, misconfiguration | prompt injection, data leakage, excessive agency |
| Input sources | defined interfaces | also external content: documents, web pages, RAG context |
| Confidence of results | a single test is often enough | repetitions and variations needed, evaluation via metrics |
What we test
We test along the OWASP Top 10 for LLM Applications 2025, the internationally recognized catalogue of the most important risks, and complement it with the risk scenarios and test priorities from the BSI guideline. This combines international recognizability with the depth of the German reference publication.
Typical systems we test: customer chatbots and support assistants, knowledge bases with RAG, document and analysis assistants, code assistants and semi-autonomous agents with tool and system access.
LLM01 Prompt Injection
Manipulated inputs alter the model’s behavior, bypass protective rules or lure the model into unwanted actions. We test both direct prompt injection and indirect attacks through documents, web pages and RAG content that the model processes.
Example: text embedded in an uploaded PDF, invisible to the user, instructs the model to ignore internal rules.
LLM02 Sensitive Information Disclosure
The model reveals sensitive information: personal data, trade secrets, credentials or inferences about training data (model inversion, membership inference). We test whether filters, context separation and permissions hold.
Example: a support assistant, on a cleverly phrased follow-up, reveals other customers’ data or stored credentials.
LLM03 Supply Chain
Weaknesses in acquired models, adapters, libraries and training data. We assess the origin and integrity of the components and dependencies in use.
Example: a model or adapter obtained from a public source contains a manipulated component.
LLM04 Data and Model Poisoning
Manipulated training, fine-tuning or embedding data introduce backdoors, bias or misbehavior. We test the data paths from which your system learns and draws context.
Example: a poisoned entry in the knowledge base makes the model produce wrong outputs on certain keywords.
LLM05 Improper Output Handling
Model outputs are passed to downstream systems without validation, enabling cross-site scripting, SSRF, SQL injection or code execution. We test how your application processes the outputs.
Example: an unvalidated model output contains script code that runs in the user’s browser.
LLM06 Excessive Agency
Agents with excessive permissions, tools or autonomy trigger harmful actions. We test privilege assignment, tool access and human approvals. A test passes here only when the tool itself refuses execution, because authorization is enforced on the calling user’s identity rather than on the model output. We also test MCP tool poisoning, where the description of a connected tool alone manipulates the model.
Example: an agent with write access sends or deletes data because a manipulated input lures it into doing so.
LLM07 System Prompt Leakage
System prompts often contain rules, logic or secrets that must not be disclosed. We test whether they can be extracted and whether security-relevant decisions wrongly reside in the prompt.
Example: a user gets the model to reveal its system prompt along with stored rules or keys.
LLM08 Vector and Embedding Weaknesses
Weaknesses in RAG systems and vector databases: data leakage through embeddings, poisoning of the retrieval context and missing tenant separation. We test the entire RAG pipeline.
Example: through targeted queries, content of another tenant from the vector database ends up in the response.
LLM09 Misinformation
Hallucinations and plausible but false outputs that users or connected systems rely on. We test safeguards such as source grounding, validation and output checking.
Example: an assistant invents a non-existent court ruling that a caseworker relies on.
LLM10 Unbounded Consumption
Unbounded resource consumption leads to denial of service, surprising costs (denial of wallet) or model extraction through mass queries. We test rate limits, quotas and abuse detection.
Example: massive, expensive queries drive up costs or take the service down.
Target: model, interface and input
The BSI guideline distinguishes three levels of the test target that we address deliberately. For acquired models the focus is on their integration into your application, not on the third-party foundation model itself.
- The model and its integration: the model in use, its configuration, the system prompt, and fine-tuning and guardrails where present.
- API interfaces: authentication, authorization, rate limiting and the protection of access to the model.
- Input mechanisms: every point where data reaches the model, explicitly including external sources such as web pages, documents and emails.
Integration forms as a test focus
Modern AI applications are more than a model behind a chat interface. We test the security-critical integration forms from the BSI guideline:
- Retrieval augmented generation (RAG) and vector databases
- Agent systems as well as tool and function calling
- Model Context Protocol (MCP) and connected services
- Code generation and code execution
- Web and document context as well as web search
- Cyber-physical agent systems, where AI acts on hardware and actuators
Hardware is not a pure software topic for us. Cyber-physical agent systems connect AI with real actuators. Our co-founder Sven Faßbender is a trained industrial electronics technician and a state-certified engineer in electrical engineering. This combination of AI, software and hardware expertise is rare in a single provider.
Test depth
We agree the test depth with you in advance. Options are black-box testing (behavior without knowledge of the internal structure), white-box testing (with access to architecture, configuration and prompts) and gray-box testing as a combination. For a robust result we usually recommend a white-box or gray-box approach.
How we test
We follow the four-phase process of the BSI guideline. It treats penetration testing as an iterative process that is repeated on major changes and at regular intervals.
Phase 1: Business understanding and current-state analysis
We develop a deep understanding of the purpose, target group and current state of your application: the chosen deployment, integrations into your business processes, relevant business risks and compliance requirements. On this basis we analyze the architecture at the infrastructure, API and application level, define the scope and set priorities.
In the preparatory conversation we clarify the timeframe, budget and scope, the support required, the test environment (usually staging or development rather than production, with explicit written approval) and go through a structured questionnaire together, point by point.
Phase 2: Threat modeling, prioritization and test planning
We build a threat model using established methods such as STRIDE or PASTA and draw on MITRE ATLAS, the AI-specific knowledge base of attacker techniques. From this we derive a targeted test strategy with defined scenarios. From the risk scenarios and the questionnaire we prioritize the tests together with you and within budget, for the best possible coverage of the relevant risks. What is not tested we record transparently.
Phase 3: Test and documentation
Our testers carry out the planned attacks, combining several tools with manual, creative analysis. Because of the non-deterministic behavior we run confidence-critical prompts repeatedly and in variations and evaluate the results using metrics such as attack success rate, detection rate and jailbreak success rate. We test not only single inputs but also multi-step attacks across many conversation turns: protective rules that block a single request often give way when an attacker builds trust over several steps or splits the request. Every step is documented traceably.
Phase 4: Evaluation, risk assessment and reporting
We assess the findings by severity and impact, reconcile critical findings with your business risks and derive concrete hardening recommendations. In a closing presentation we discuss the results before the final report is written. We also propose a timeframe for the next assessment to support continuous monitoring.
The test report is divided into a management summary for decision-makers (tests carried out, most important vulnerabilities by criticality, recommendations and potential impact) and a technical report for your teams (test period, target systems, limitations, methodology and tools, and for each finding a description, the affected component, reproduction steps and a rating per CVSS, where needed following the BSI Standard 200-3). Evidence such as screenshots and logs is provided in the appendix.
Legal framework. The legal framework for penetration tests is not yet conclusively regulated by law. Without clear consent there is a criminal-liability risk under Sections 202a to c and 303a to b of the German Criminal Code. Before we start we therefore agree a written authorization (rules of engagement) covering scope, test period, permitted methods, liability and communication channels, along with an NDA and, where personal data is involved, the necessary data protection agreements. Data protection is always our highest priority.
How you protect yourself
An assessment shows where the problems are. To make sure the gaps are closed too, you need concrete protective measures in the operation of your AI systems:
Checking and sanitizing inputs and outputs, detecting prompt injection and safely processing model responses.
Hardening the system prompt, separating secrets and logic, so that security-relevant decisions do not reside in the prompt.
Minimal permissions and tool access, human approvals for critical actions, to limit excessive agency.
Rate limits, quotas and abuse detection against unbounded consumption, plus robust logging for traceability.
How we support you
We accompany you in hardening and securely operating your systems, on request directly after the test. Beyond technical hardening, we anchor AI security in your organization:
- AI governance and secure operation: policies, data classification, approval processes and human oversight. Embedded in your information security management.
- Secure development: we anchor secure development practices for AI applications in your teams, from policies to secure coding training.
- Architecture review and threat modeling: threat modeling and architecture reviews already in the design, before vulnerabilities reach operation.
- Training: suitable training for developers and executives on the secure use and integration of AI.
- Verification retest: after remediation we test again on request. We re-run the previously found vulnerabilities, repeatedly and in variations, and use the same metrics to show whether the hardening actually holds.
- Evidence for AI management systems: on request we prepare the test results as measurable evidence for an AI management system per ISO/IEC 42001 and map them to the NIST AI RMF. What gets certified is the organization, not the individual test. We provide the evidence.
Regulatory context
AI pentests are not mandated by law as such. But they are a recognized means of demonstrating that your security measures work, and they contribute to several regulatory requirements:
- AI Act: for high-risk AI the regulation requires robustness, accuracy and cybersecurity across the lifecycle. Article 15 explicitly names data poisoning, model poisoning and adversarial examples as attacks to defend against, without prescribing a penetration test. Security testing demonstrates these properties.
- NIS-2: policies and procedures to assess the effectiveness of the risk management measures (Article 21), for which penetration tests are well established.
- BSI C5: criteria catalogue for cloud services. The attestation requires regular vulnerability and penetration testing by the provider.
- DORA: for financial entities designated by the supervisory authorities, threat-led penetration testing (TLPT) is required at least every three years, on production systems of critical or important functions. Where AI-supported systems are part of those functions, they fall within scope.
- Cyber Resilience Act: security evidence for products with digital elements, increasingly with an AI component.
- GDPR: protection of personal data processed by AI systems.
Which requirements apply to you specifically is something we clarify in the initial call.
Why zentrust partners
- BSI-certified IS penetration testing service provider (registration ID BSI-APS-9080), verifiable in the official BSI register.
- Testing and hardening from a single source: whoever finds the gaps also makes the recommendations and supports the fix.
- Combination of automated methods and manual, creative analysis, rather than relying on a single scanner.
- AI, software and hardware expertise under one roof, up to cyber-physical agent systems.
- Our own security research: our testers publish CVEs and coordinated disclosures and have been heard as expert witnesses in committees of the German Bundestag. More about us and on the blog.
- Management-ready reports with a clear risk rating per CVSS and prioritized measures.
Frequently asked questions
Do you test against our production system?
Usually not. We prefer to test in a staging or test environment that mirrors production. This avoids operational risks and unnecessary costs, especially with paid model APIs. Exceptions are agreed in writing in advance.
We use an acquired model. What can be tested there?
A great deal. Most risks do not arise in the foundation model but in its integration: system prompt, RAG context, tools and agents, API protection and the handling of outputs. It is exactly this application layer that we test.
Aren’t guardrails and a filter enough?
They help but are not sufficient. Filters and guardrails can be bypassed, particularly through indirect prompt injection. The statement only becomes reliable through an assessment that combines several tools with manual analysis and runs prompts repeatedly and in variations.
How often should we test?
AI systems and their threats evolve quickly. It makes sense to test on major changes to the model, prompt or integration, as well as at regular intervals. We put a concrete proposal for the next assessment date into the report.
No dedicated information security organization yet? If you use AI but do not yet have an information security officer and a lived ISMS, the free CyberRisikoCheck per DIN SPEC 27076 is often the better first step. For an overview of how AI is changing the threat landscape, see our page Artificial intelligence, what now?