Slowing Down in a Move-Fast World: How my Galois internship reset my relationship with AI

Before I arrived at Galois, I had a complicated relationship with AI. I grew up in the relentless Bay Area “grindset” culture, which—combined with the pressures of social media—put productivity and output on a pedestal. The rise of AI code generation only amplified this culture. At college, I’ve found myself surrounded by peers who work on multiple projects simultaneously and attempt to “separate themselves from the masses.” I felt stuck in what I came to consider the modern computer science student’s dilemma: Do I move fast, stack projects, and use AI to pump out results, or risk getting left behind? Is it worth my time to learn outdated concepts while the computer science industry is continuously and quickly changing?

My internship at Galois flipped this question on its head. Working on AI systems where correctness actually mattered forced me to trade the “move fast” grind for positive skepticism. Instead of using AI to rush development, I learned how to slow down, interrogate the technology, and focus on deep conceptual understanding over quick results.

Positive Skepticism and Creative Freedom

When I started at Galois, I wasn’t sure what to expect. This was my first ever internship, and I didn't know how the company felt about AI usage for software development. I was happy to quickly observe a “positive skepticism.” Opinions varied significantly from person-to-person because each project scope was so distinct. Furthermore, we were conducting research – formulating questions, running experiments, and constantly iterating – not just rapidly building products. 

When I presented ideas for my project, I never got a straight “no.” I was always encouraged to try them out and get concrete data. This Galwegian culture of creative freedom and digging deep into complex concepts pushed me to focus more on the process than the result, and proved useful in my work. 

Making Hospitals Safer

Galois brought me on to work on UPGRADE, a project focused on automatically detecting and remediating vulnerabilities in hospital IT networks. My expertise in machine learning and generative AI made me well-suited to help the team develop an AI-assisted document extraction pipeline that supported the automatic remediation framework and hospital IT staff. The challenge I aimed to resolve centered around how to support LLMs with the right information since it is unlikely the models have seen the specific manuals and technical information exactly used by hospitals. I was excited to help solve a real-world problem with LLMs. 

Our approach centered around a document extraction pipeline leveraging RAG: Retrieval Augmented Generation, a method for supplementing an LLM's knowledge from a corpus of data. Figure 1 illustrates our approach. In UPGRADE’s context, RAG works by identifying which technical manuals and documentation are most similar to a user’s query regarding certain medical device features (e.g., hardware components, network information), and enables the LLM to use that information to better answer the request.

Preliminary results were promising, and in the past I might have mistaken that early success for proof of readiness. But now, rather than rushing into deploying initial prototypes, we investigated deeper by exploring edge cases, reading research papers, and discussing findings with teammates. We built on top of this initial framework through a multitude of features designed to improve its performance, reliability, and trustworthiness. Although it took longer to develop this second prototype, it was well thought out and easier for individual team members to test, which made further iteration much easier. 

Thus far, our assurance strategy relied on manually analyzing outputs in order to validate the RAG system’s performance. Verifying that the LLM was not hallucinating (when an LLM generates misleading information that appears confident and factual) proved more challenging. The device manuals that we ran through our extraction pipeline had no prior data or ground truth to compare the LLM’s extractions against. Thus, any improvement we made was hypothetical because we couldn’t measure it against a baseline. We solved this problem by applying a machine learning concept called natural language inference (NLI), which determines the logical relationship between two sentences: a premise (a factual statement or body of text) and a hypothesis (a claim that may be supported by the premise). An NLI model classifies the relationship in one of three ways: an entailment (the premise supports the hypothesis), a contradiction (the premise negates the hypothesis), and neutral (it can’t be determined whether the hypothesis is true or false based on the premise). 

In context, the set of technical manuals served as the premises and the LLM’s outputs as hypotheses. The NLI model evaluated each relationship according to whether or not the output was supported by the cited documentation. Ideally, all premises entailed the LLM’s hypotheses as this would signal the improved pipeline properly cites source material. When a premise contradicted the hypothesis, one of two things could be inferred: the LLM was hallucinating and we needed to further guide the model to answer the request, or the retrieved documents contained disagreeing factual material with that of the hypothesis NLI. This provided a new way to evaluate and build trust with the LLM. 

Connecting the Dots

Working so closely with LLMs on UPGRADE gave me a much clearer view of where AI fits, not only into my personal workflow, but into modern software development at large.

AI is in fact strong at performing a variety of tasks, but it also has weaknesses. It’s our responsibility to determine where we can use it to improve a workflow based on our skill sets and project scope. For me, AI proved helpful for whipping up prototypes that I could manually iterate on or implementing scaffolding code for experiments, allowing me to spend more time brainstorming more important features such as code architecture, improving our pipeline’s document retrieval, or developing future research questions. It does not replace deep thinking or complex problem solving; rather, it frees up bandwidth to allow us to focus on these things.

My exposure to formal methods at Galois and the emphasis on developing verification methods for LLMs has changed my career interests, too. I now want to work on systems where these unpredictable and largely unverifiable systems have to be correct. I’m inspired to dive deeper into trustworthy AI systems, closing the gap between newly developed tools and the wake of uncertainty they leave behind.