As artificial intelligence becomes more autonomous, the traditional playbooks for software privacy and security are proving insufficient. In her recent keynote presentation, data scientist and security expert Katharine Jarmul challenged the industry to move beyond hype and fear, advocating for a proactive culture of ownership to tackle the unique privacy and security challenges posed by modern AI systems.
The core of her message is a shift away from blame and towards responsibility. Instead of waiting for vendors to provide perfect solutions or getting lost in abstract risk frameworks, Jarmul argues that teams must build agency and take direct ownership of the security of their AI implementations.
Here are the five key myths she debunked.
Myth 1: Guardrails Will Save Us
Guardrails—the software filters, external models, and alignment techniques like RLHF designed to control AI behavior—are a popular talking point. While useful, they are far from infallible. Jarmul demonstrated that simple adversarial tricks, such as renaming variables in code or even using ASCII art, can easily bypass these protective layers.
The reality: Guardrails are a helpful but permeable defense. They should be seen as one tool in a larger toolkit, not a complete solution. Understanding their weaknesses is the first step to building a more robust security posture.
Myth 2: Better Performance Will Save Us
There's a common assumption that as AI models become more performant, they become safer. However, the trend of "overparameterization" in Large Language Models (LLMs) tells a different story. These models often have more capacity than the data they are trained on, which can lead them to memorize specific training examples rather than purely generalizing from them. This is a significant privacy risk, as sensitive data can be inadvertently stored and later exposed.
The reality: Performance does not equal privacy. Techniques like differential privacy can help mitigate memorization, but teams must be aware that a high-performing model might also be a highly effective data leaker.
Myth 3: A New Risk Taxonomy Is All We Need
The AI industry is flooded with complex risk taxonomies and frameworks (like MITRE ATLAS, OWASP Top 10 for LLMs, etc.). While well-intentioned, these can be overwhelming and often suggest mitigations that are impractical for many organizations to implement.
The reality: Instead of trying to boil the ocean, Jarmul advises creating an "interdisciplinary risk radar." This involves bringing together people from different teams (data science, security, legal, product) to identify the most relevant threats for your specific application and organization, and then focusing on feasible, high-impact solutions.
Myth 4: We Did Red Teaming Once, So We're Fine Now
Security is not a one-and-done activity. A single red teaming exercise provides only a snapshot in time and can quickly become outdated as models and attack techniques evolve.
The reality: Security must be a continuous, iterative process integrated directly into the MLOps lifecycle. This means consistently thinking like an attacker: What are the valuable targets (data, service availability, financial cost)? How can they be compromised? Automating threat modeling, testing, and monitoring builds a durable and adaptive security practice.
Myth 5: The Next Model Version Will Fix This
It's tempting to believe that AI vendors will eventually solve all the security and privacy issues in their upcoming releases. However, Jarmul points out that vendors are primarily optimizing for their own product goals, such as user engagement, helpfulness, and code generation—not necessarily for your specific security needs.
The reality: "Only we can save ourselves." Organizations cannot afford to be passive. Jarmul urges teams to diversify their model providers, explore self-hosting open-source models (using tools like Ollama or VaultGemma), and actively test models to understand their failure modes and limitations.
Conclusion: A Call for a Culture of Care
Ultimately, Jarmul's message is a powerful call to action. The path to secure and private AI systems is not through blind trust in vendors or a quest for a magical silver-bullet solution. It's through building a culture of care, intervention, and ownership, where every team feels empowered and responsible for proactively identifying and mitigating risks.
Reference: AI Systems, Privacy and Security: A Journey Beyond the Hype