AI features are appearing in production applications before security teams even know they exist. Runtime AI visibility is how security teams close that gap by identifying AI usage, data flows and risks that code review, threat models and scanners all miss.
Someone on your product team is probably adding AI to an application right now. This introduces hidden AI security risks — often referred to as shadow AI — that are often missed by traditional AppSec tools.
Not as part of a formal AI initiative, not after an architecture review and certainly not because security approved a new model.
A product manager, ops lead, support engineer or developer has a problem to solve. They find a model. They wire up a service. They ship something that works.
Security may or may not know about it.
That is the problem with shadow AI in applications. It is not just employees pasting data into ChatGPT. It is AI functionality embedded in the software your customers use, such as a chatbot, report summary, support assistant, workflow automation or a service that calls a model behind another service.
Your policy may say one thing, but your inventory may say another. The application is the source of truth.
Runtime AI visibility is how security teams see that truth.
Runtime AI visibility is the ability to detect AI usage from inside a running application, including model calls, AI-powered services, routes and related application behavior, as it actually happens in production. It goes beyond what teams report or architecture diagrams show, revealing exactly what the application is doing in real time.
Shadow AI refers to AI usage, model calls, AI-powered services and AI-driven data flows within your applications that security has not reviewed, approved or inventoried. Shadow AI creates significant security vulnerabilities by bypassing standard AppSec review processes.
CrowdStrike's AI security team found this in every organization it assessed. One customer believed it had 150 AI agents in production. The actual count was over 500. According to IBM's 2025 Cost of a Data Breach Report, incidents involving unauthorized AI use averaged 247 days to detect and contain and cost organizations $200,000 more than standard breaches.
Your AI inventory is not a spreadsheet. It is a runtime question.
Most teams assume a code review will surface new AI functionality. In many cases, it will not.
Here is what a live test showed in a purposely vulnerable app:
A product manager at Contrast used Claude to add a shipping advisor chatbot and an AI-powered report summary to a running demo application. That’s twenty commits, a pull request and working features. But the AI call was not in the feature code. The feature called an internal service and that service handled the model interaction.
From the perspective of a code reviewer looking at the feature, there was no AI call, no SDK import, nothing flagged as AI usage. If you did not already know that an AI-capable service existed, you would not find the connection.
This is how shadow AI persists. AI functionality lives in services, dependencies and infrastructure, not always directly in the code being reviewed. A developer or non-technical builder can add a feature that calls an existing AI service without the code review ever surfacing the AI connection.
That is a structural limitation, not a failure of the code review process. Code review sees what was written, but not what the application does at runtime.
Threat models share the same limitation. They document what teams know and intend. Architecture diagrams go stale. Attestations ask people what they think is true, not what the application is actually doing.
AI enters applications through several paths that standard AppSec processes are not built to catch.
The most common is a developer or non-technical builder adding a feature that calls an existing internal AI service. The capability already exists in the infrastructure. The new feature is small. The code review focuses on feature logic, not on which downstream services it touches.
A second path is an AI SDK integrated directly, often for a prototype that gets promoted to production before a formal review. Vibe coding has made this faster. A working feature can go from idea to pull request in hours.
A third path is for an existing service to gain AI capabilities through a dependency update or configuration change. The application code does not change, but the behavior does.
In all three cases, the result is the same: The application is doing something new and security is in the dark.
A natural response is to ask whether AI-powered scanning can fill the gap. Contrast CISO Dave Lindner ran that experiment directly. The results are documented in The Hidden Cost of AI Security Scanners.
Scanning a two-million-line production codebase cost roughly $300 in token consumption and produced nearly 4,000 reported vulnerabilities. When the same model was asked to triage its own findings, it expressed confidence in only 1.1% of them. A multi-model approach narrowed the list significantly. Half of those were still false positives.
Scanning also has a timing problem. Running against a large codebase took hours, sometimes days, which is incompatible with a continuous deployment pipeline.
More importantly for this problem, scanning looks at code. It does not see AI service calls that happen at runtime through an indirect service layer. It does not observe model behavior or the data that flows through an AI-powered route. If the AI usage is indirect, buried behind services, configuration or infrastructure the reviewer was not focused on, a scanner will not find it.
The AI speed paradox is that the same tools helping developers ship faster are making scanning-based security less effective. More code, faster, with more indirect dependencies, is exactly the environment where pre-production scanning produces the most noise and the least coverage.
|
Approach |
Detects direct AI SDK calls |
Detects indirect AI service calls |
Updates as app changes |
Shows routes, services and runtime context |
Works in production |
|---|---|---|---|---|---|
|
Code review |
Sometimes |
Rarely |
No |
No |
No |
|
SAST / AI scanning |
Sometimes |
No |
No |
No |
No |
|
Runtime instrumentation |
Yes |
Yes |
Yes |
Yes |
Yes |
Runtime instrumentation works from inside the running application. It observes actual behavior, not intended behavior. When an application calls an AI service, makes a model request or routes activity through an AI-powered feature, instrumentation detects it, regardless of whether the call is obvious in the code or buried in a service dependency.
This is why Contrast's runtime security platform surfaces AI usage as part of standard application visibility. When Contrast's application security leader reviewed the application after the product manager's features were deployed, the AI activity appeared in the dashboard. She did not have to scan for it or know which service to look at. The application showed her what it was doing.
For AppSec teams, runtime AI visibility answers questions no other approach can answer reliably: Which applications in the portfolio are using AI? Which models, SDKs or services are involved? Which routes or features trigger those calls? Is this behavior in your approved AI inventory?
For SecOps teams, it provides application context that traditional security tools miss. If an application is under attack, SecOps needs to know whether the attack reached AI-powered code, whether sensitive data was exposed via an AI feature and whether the behavior can be blocked while remediation is underway.
For AI governance and compliance teams, it provides evidence rather than attestation. A policy that says only approved AI services may be used means nothing without proof of which services your applications are actually calling in production. Runtime visibility closes that gap.
Contrast's approach to runtime intelligence connects AI behavior to the full application security picture: which routes are exposed, which vulnerabilities are reachable and which attacks are being attempted. AI usage does not exist in isolation. It connects to data, to services, to the attack surface.
AI-assisted coding makes it faster to create new application behavior. Mythos-class AI systems make it faster to find and exploit weaknesses in that behavior. Security teams are stuck in the middle with the same backlog process.
Faster application development on one hand and faster vulnerability discovery on the other.
The traditional AppSec rhythm is scan, triage, prioritize, assign, patch, verify. It cannot absorb that pressure without runtime context to separate real risk from noise. Security teams do not need to know what might be vulnerable in their code. They need to know what is actually reachable, actually exploitable and actually being attacked in production.
Runtime is where those questions get answered.
For years, AppSec has been organized around one core question: What vulnerabilities are in our code?
That question still matters. But it is no longer enough.
The better question now is: What is our application actually doing?
Is it using AI? Is it calling a model security does not know about? Is it moving sensitive data through a service that bypasses review? Has the vulnerable code actually been deployed in production?
AI speeds up software creation; anyone with repository access can ship new behavior in hours. That changes the speed at which applications change and it widens the gap between what security knows and what applications are doing.
Your AI inventory is not a spreadsheet. It is a runtime question.
Runtime AI visibility is how you answer it.
What is runtime AI visibility?
Runtime AI visibility is the ability to detect AI usage within a running application, including model calls, AI-powered services, routes and related application behavior. It shows what applications are actually doing in production, not what teams intended or documented.
Can a code review catch AI features added through a service layer?
Usually not. If a feature calls an internal service that interacts with an AI model, the AI connection is not visible in the feature code. A code reviewer would need to already know that the service exists and be able to trace its dependencies to find it. Runtime instrumentation finds it automatically.
How does runtime AI visibility support AI governance?
AI governance policies define which models and services are approved. Runtime visibility verifies whether production applications are actually following those policies. It provides evidence rather than attestation.
Is shadow AI in applications different from employees using unauthorized AI tools?
Yes. Shadow AI in applications is embedded in the product. It can affect every user of the application, it can move customer data and it is often not visible to security through standard scanning. The risk profile is different and typically larger.
What should security teams do when they find unauthorized AI usage in production? First, understand what data is involved and whether it has been exposed externally. Then identify the service or model being called, whether it is reachable from untrusted inputs and whether the usage can be blocked while the team decides how to formalize or remove it. Runtime context makes all those steps faster because you already know where the behavior occurs.
Your AI inventory is only useful if it matches production reality. See how Contrast shows what your applications are actually doing at runtime.