Beyond Guardrails: Meeting the AI Threat with High Assurance Software

‍Thanks to Caleb Lucas-Foley for his AI insights and contributions to this article.

In just a few short years, AI has changed the cybersecurity landscape for the worse… or at least made it much more complicated. Cyber attackers hold an edge over defenders and the advent of agentic AI appears to make this worse. According to IBM’s Cost of a Data Breach Report for 2026, bad actors are increasingly using AI tools to speed and scale their efforts, leading to an unprecedented rise in attacks that target vulnerable systems and expose private data. Exploits from both AI assisted attacks and AI introduced vulnerabilities increased 56% year over year, while the global average cost of a data breach grew by 12% to nearly $5 million. In the United States, the average breach cost reached a record high of $11.5 million. 

While the healthcare and finance industries are suffering the steepest losses, both the number of attacks and the average cost of breaches are rising across sectors, from telecom and transportation networks to educational institutions and pharmaceutical companies. Critical infrastructure is at risk as well, with cyber attacks on public water and wastewater utilities reported in more than a dozen states this year, as well as on refrigerators and cold monitoring systems across the world.  

This trend persists despite an expanding patchwork of governmental policies mandating securing citizen’s data against threats, including the EU’s General Data Protection Regulation (GDPR), California Consumer Privacy Act (CCPA), Oregon Consumer Privacy Act (OCPA), HIPAA’s healthcare privacy framework, and Title 13 Census protections. Legal protections exist on paper but are only the first step out of this Hobbesian hellscape of today’s real-world information ecosystem where citizens' privacy is still largely at risk. 

The Emerging Threat Landscape

Cyber threats manifest in response to vulnerabilities in systems that we build and use. Data privacy and security have always been at risk because of poor design decisions that do not operate on a “privacy/security-by-design” principle. Those problems are now exacerbated by the large-scale deployment of AI agents. Attackers can now deploy agents online to get a lay of the land, quickly identifying high-value systems that are critically at risk before mounting a more informed, human-led cyberattacks. Take the recent discovery of the “Php2Shell” vulnerability: Google & OpenAI’s security researchers claim to have discovered this vulnerability with AI assistance. In the hours that followed the leak and the public disclosure of “Php2Shell,” security experts sounded the alarm about attackers likely using AI assisted tools to exploit this vulnerability en masse while systems were being patched. 

This incident and many like it are causes for concern that widespread zero day vulnerabilities exist out there and are being exploited in a similar fashion because of poor security practices and engineering debt across the web. Moreover, the recent rush to integrate AI features into technological solutions without any formal analysis and measure of the risks involved has created an abundance of new vectors of cyberattacks. 

In their recent, wonderful paper “SoK: Attack and Defense Landscape of Agentic AI Systems, authors Juhee Kim, Wenbo Guo, and Dawn Song present a framework that characterizes seven key dimensions of AI agent design. These are: 

• Input Trust: The agent’s access to external data sources.
• Access Sensitivity: The agent’s access to sensitive information.
• Workflow: The authority over the sequence of actions that an agent can take and how data moves as a result.
• Action: The operations that an agent can perform.
• Tool: The tools an agent has access to.
• Memory: How an agent stores and retrieves memory
• User Interface: How the user interacts with the agent.

Each of these dimensions lies on a spectrum of flexibility and trust. The more flexibility an agent has in any particular dimension, the greater its capability to operate within a system with little to no supervision. Flexibility hence comes with deferring trust, in all the dimensions listed above, from the software stakeholder (e.g., the engineer or system administrator) to the AI agent effectively escalating its privilege in the software stack. Similarly, less flexibility corresponds to more supervision and restrictions placed by, say, an engineer, and leads to higher trust in the behavior of the system.

Thus, for example, the Input Trust dimension exists on a spectrum from an agent using no external data (highest trust, lowest flexibility) to relying on all data sources, including arbitrary untrusted data sources (lowest trust, highest flexibility). Consider the following scenario: What happens when an AI agent is allowed to use all sources on the internet, including upvoted Stackoverflow responses that advertise software vulnerabilities, as solutions to common coding questions? LLMs do not discriminate between data sources, highlighting the need for data curation and context specific datasets in the LLM’s training pipeline that many AI researchers have long advocated for. 

The same tradeoff and considerations exist across each agent design dimension.

The key takeaway here is that agent design necessarily comes with important questions regarding delimiting boundaries, and making an agent safer necessarily involves limiting their access to data and resources, restricting their actions and the flow of information, and deciding how to verify and limit their work. When these questions are not properly answered and an agent is (even inadvertently) granted more privilege and flexibility, a large vector of vulnerabilities become available to malicious actors. Many things can go horribly wrong and the consequences are often costly and irreversible. 

To list just a few of the attack types made possible by these sorts of design decisions (or indecisions):

• Data Leakage: Commercial LLMs that are available online do not formally guarantee or verify that a user’s data and conversations remain confidential and protected from unauthorized access. Anyone that can interact with said LLMs can attempt to manipulate the agent to reveal other users’ private data (see 1, 2, 3, 4, 5, 6, 7, 8). 
• Denial of Service/Unbounded Resource Consumption: Attackers can try to manipulate agents with unrestricted access to computational resources to overuse said resources, effectively leading to a denial-of-service attack. Even without the presence of an attacker, there is no guarantee that agents will not themselves misbehave and over use available compute resources at their disposal.
• Untrusted Interfaces / Unsafe Data Flow: With unrestricted access to resources available on the internet, agents can fetch malicious content tailored to allow adversaries to exfiltrate users’ systems. For example, AI-generated code can include deprecated and malicious library imports. Alternatively, agents can retrieve malicious attacker-controlled URLs that can also lead to these kinds of attacks. 
• Web agent retrieval poisoning: User generated content as simple as a small reddit post can directly influence the agent’s response to other user queries.

From Fragile Guardrails to High Assurance Foundations

The prevailing industry response to these vulnerabilities has relied on “AI guardrails” – typically in the form of post-hoc reinforcement tuning, prompts at runtime (see 1, 2), or runtime input/output filters. But such measures are not formal guarantees that can be validated and audited by security and privacy researchers. Rather, they function as probabilistic band-aids, easily bypassed by adversarial jailbreaks, brittle in novel contexts, and incapable of providing mathematical certainty (see 1, 2, 3, 4). 

While there are several, variously effective approaches for dealing with these kinds of attacks, Galois specializes in and focuses on developing and working with high assurance software. We use formal methods to ensure that software is correct by construction, operate on the cutting edge of AI and machine learning research rather than just using available AI products, and use cryptographic techniques that are backed by decades of research to design and implement systems that protect sensitive data.

In this blogpost and its follow-ons we focus our attention on our cryptographic and security work and refer readers to our other blog posts for more information on our other lines of research.

As we saw with the work of Kim, Guo, and Song, without considering the confidentiality, integrity and availability guarantees of a system and carefully designing a solution around those properties from the ground up, we are stuck reacting and catching up with emerging cybersecurity risks. The advent of AI-assisted cyber-attacks and the large-scale introduction of vulnerabilities and bugs by the haphazard integration of AI agents into software stacks has made it so that citizens' sensitive information, critical infrastructure, and business data are at greater risk than ever before. This is especially the case for users with less resources at their disposal (e.g., agencies with older software, organizations with no system administration, individuals who are not security and privacy literate [and should not be expected to be], etc.).

The cryptographic methods that we use and design at Galois allow us to rigorously define the boundaries of key dimensions for the systems we design – curbing AI agents’ potential for harm, and comprehensively sealing the cracks in the armor of systems under threat of attack. The tension between flexibility and security is inevitable, but determining the user's need and relying on mathematical precision makes this tradeoff as close to the system requirements as possible in both directions, while simultaneously making our work auditable by other researchers and engineers from design to algorithm to software. 

Privacy and security requirements do not need to be lost in the age of AI and the path to a truly secure future requires paying more attention to how we’re designing systems and what we are trying to achieve. As long as we stick to privacy as our guiding beacon, we firmly believe that many of the emerging open questions surrounding AI and privacy/security can be tackled and we at Galois are ready to do that work. 

Looking ahead, we’ll dig more into security and privacy by design, the tools and techniques of cryptography, and the path forward.