On March 24, the most widely used open-source LLM proxy was compromised through a supply chain attack. The incident exposed a structural risk in how most enterprises consume AI infrastructure, and a design decision that determined whether the attack could reach them. 

Two malicious versions of the litellm Python package were published to PyPI, the central package registry that most Python developers pull from by default. The attack was sophisticated: a threat actor compromised a security scanner used in LiteLLM’s automated release pipeline, extracted the project’s PyPI publishing credentials, and used them to push poisoned releases that looked legitimate. For approximately three hours, anyone installing or upgrading litellm received a version that silently harvested API keys, cloud credentials, and other secrets from their environment. 

LiteLLM is the most widely used open-source LLM proxy in the Python ecosystem, with roughly 100 million monthly downloads. It sits between applications and AI model providers, routing API calls and managing credentials. That architectural position is what made this attack so consequential: a compromised LLM proxy doesn’t just expose one system. It exposes every API key, every model endpoint, and every credential flowing through it. 

The LiteLLM team responded quickly, and PyPI quarantined the packages within hours. This isn’t a story about negligence. This is a story about what happens when the AI infrastructure layer inherits the same supply chain risks that have plagued software for years. Except now the blast radius includes every AI credential in your environment. 

Why This Matters Beyond LiteLLM 

It’s tempting to treat this as a one-off incident. It wasn’t. The attack was part of a broader supply chain campaign that had already moved through Trivy, Checkmarx, and Aqua Security’s internal GitHub infrastructure before reaching LiteLLM. Threat actors follow the dependency chain upstream and target the tools that build and secure software, not just the software itself. 

For enterprise AI specifically, the implications are structural. The LLM proxy layer is the single richest target in an organization’s AI stack. It holds provider API keys. It routes every model call. It often has access to usage data, cost information, and in some configurations, the content of prompts and completions. A compromised proxy isn’t a vulnerability; it’s a skeleton key. 

The question CISOs and executive teams should be asking right now is not “were we running the affected version?” It’s: “How are we consuming AI infrastructure dependencies, and what would happen if one of them was compromised tomorrow?” 

The Architectural Decision That Matters 

Most organizations consume open-source AI infrastructure the same way they consume everything else: pull the package from a public registry, trust that the published artifact matches the source code, and update when new versions drop. It’s the default path. It’s also exactly the path this attack exploited. 

At JetStream, we took a different approach. AI-Hub is a custom-built platform for enterprise environments, shaped by deep experience building enterprise-grade systems and designed for the reliability, scalability, and fault tolerance with a secure, enterprise-grade upgrade process these deployments require.  

What This Tells Us About AI Governance 

Supply chain security is not a new problem. SolarWinds taught the industry that trusted distribution channels can become attack vectors. Log4Shell showed that a single transitive dependency can expose the entire internet. The LiteLLM incident is the same pattern applied to a new target: the AI infrastructure layer. 

But the AI layer has a characteristic that makes supply chain attacks uniquely dangerous: it concentrates credentials. A compromised web framework might expose your application. A compromised LLM proxy exposes every AI provider relationship, every API key, and potentially every prompt your organization sends. The governance question is no longer just “which AI models are we using?” It’s “who built the infrastructure between us and those models, and how do we know it hasn’t been tampered with?” 

Organizations that treat AI governance as a visibility and policy problem alone are missing a layer. Governance has to extend to the supply chain of the infrastructure itself. That means knowing exactly what code runs in your AI pipeline, controlling how it gets there, and being able to verify its integrity at any point. 

Three Questions Every Security Team Should Be Asking This Week 

These are the questions we’re hearing from security leaders we talk to, and the ones we’d be asking ourselves if we were sitting on the other side of the table. 

How are you consuming AI proxy and gateway dependencies? If the answer is “pip install from PyPI” or an equivalent, you’re trusting a public registry with the keys to your entire AI estate. Consider forking critical dependencies, building from source, and controlling your own artifact pipeline. 

Do you have an inventory of what sits between your applications and your AI providers? Many organizations have adopted LLM proxies, gateways, and routing layers without formal security review. These components often hold more sensitive credentials than any other part of the AI stack. If your security team can’t name them, they can’t govern them. 

Could you detect a compromised dependency before it exfiltrated data? The malicious LiteLLM versions were live for three hours. That’s three hours of credential harvesting. Runtime monitoring, network egress controls, and anomaly detection on your AI infrastructure aren’t optional. They’re the difference between a near-miss and a breach. 

Security-First Means Supply Chain-First 

The LiteLLM incident didn’t reveal a new class of vulnerability. It demonstrated an existing one in a new place: the AI infrastructure layer. The attackers didn’t need to find a zero-day or break encryption. They poisoned the well upstream and waited for the default installation path to do the rest. 

At JetStream, governance starts at the foundation. Not at the policy layer. Not at the dashboard. At the architectural decisions that determine whether a supply chain attack can reach your environment in the first place. The security posture is in the architecture, not bolted on after the fact. 

AI infrastructure is now critical infrastructure. It deserves the same supply chain rigor we demand from operating systems, container runtimes, and security tools. The organizations that figure this out now will be the ones who can adopt AI at speed without wondering what’s hiding inside their dependencies. 

JetStream Security is the security-first AI governance platform. The JetStream SAIG Platform™ makes every AI agent, model, workflow, and identity visible, attributed, and governed — giving enterprises the infrastructure to adopt AI at speed without surrendering control. Learn more about the platform →