Evals Everywhere All The Time: A Year of Building AI Systems Responsible for Enterprise Data Privacy

I've spent the last year embedded as a research engineer at Ethyca taking AI systems from napkin sketches to production systems in the high consequence domain of data privacy. Ethyca's customers rely on their product to stay compliant with regulation and, increasingly, to mitigate the risk of data exposed to AI chats, tools, and agents.

It is one thing to build a workable prototype of an LLM-based system. It is another thing entirely to prove that its outputs can be relied upon. Is a particular output plausible-nonsense or actually-correct? We know from experience that LLMs are capable of both; under what conditions do you get one or the other?

The best practice for this sort of question is quantitative evaluation or "evals". It was clear from the start that it would be important to test features thoroughly with evals before release because of the probabilistic nature of LLMs. What was a surprise was just how central the practice of evals would be to the development of AI capabilities, playing valuable roles throughout the product development lifecycle, not just at its conclusion. I used them to:

• Decrease data classification cost by a factor of 20
• Calibrate trustworthy confidence scores
• Validate a speculative product capability (purpose of data use detection)

In the rest of this retrospective, I will dive into details about each. Taken as a whole, what I hope emerges is a vision for the framework required to deliver trustworthy AI systems, those that teams responsible for risk, security, and compliance can stake their reputations on.

Decreasing classifier cost by a factor of 20

Ethyca’s Helios subsystem classifies the data categories of fields in enterprise data warehouses. The original design classified one field at a time: one prompt, one field, one tag. For the millions of fields an enterprise would typically want to classify, this was economically feasible because of prompt caching, but only barely.

An obvious optimization to this architecture is batching: classify many fields of a table in a single inference call. A year ago, this did not work because models exhibited “laziness,” giving diligent answers for the first few fields and then skipping the rest. This tanked recall scores.

Despite its unsuitability for production, batch mode would get run from time to time in the lab because, despite its lower accuracy, it was so fast that it was convenient to use to validate if a particular change was an improvement or regression.

And then, gradually, as new generations of models were released, batch mode started to match the accuracy of field-at-a-time mode. At first, it was only frontier models (gemini-3.1-pro, claude-opus-4.6). Eventually, this capability made its way into smaller models, with claude-haiku-4.5 being the first cost-effective model to achieve high-accuracy batch mode.

I set aside the time for a full benchmarking exercise recently to identify the ideal model for batch mode as well as integrate it into the production stack. Here are some selected results:

The straightforward takeaway from these benchmark results is that it was time to roll out batch mode; it could match existing task accuracy at 1/20th the cost.

But there’s more to the story in the details:

• The laziness remains for older models. The production baseline, gemini-2.5-flash, degrades badly in batch mode (F1 drops from 0.757 to 0.557, with recall falling to 0.425). If I had swapped modes without measuring, I would have silently shipped a much worse classifier.
• claude-haiku-4.5 in batch mode is notable for extremely high precision (0.918), making it an interesting choice for deployments highly sensitive to false positives.
• Open-weight models like gemma-4-31b-it have low-throughput providers (~28 tps versus the ~120 tps of gemini-2.5-flash), but can easily make up the difference with concurrency because there are so many providers instead of just one.
• I also benchmarked frontier models (claude-opus-4.8, gpt-5.5, gemini-3.1-pro) in batch mode. While they are accurate, they don’t uniformly outperform smaller models, suggesting there’s a threshold of capability required for this task that additional model capability doesn’t help with—models only have to be so smart to classify data as well as is possible.

Now, a result as large as a 20-times cost reduction sets off alarm bells for every engineer. What am I missing here? Evals allowed me to address this anxiety. I didn’t have to trust that batch mode was fine; I measured that it was fine, across a dozen models, with metrics I knew led to successful outcomes for users.

There is a broader lesson: in a field moving this quickly, a failed experiment today may simply be waiting for the models to catch up. A resuable eval becomes, then, a probe for emergent capabilities that can take an AI system to a new level of accuracy, speed, or cost efficiency.

Calibrating trustworthy confidence scores

What do you expect would happen if you asked an LLM a difficult question about data privacy, then asked it to predict whether a human expert would agree with its answer?

Pure sycophantic bravado? Random numbers dressed up as confident prose? A loose trend that can tell trivial questions from tricky ones, but not much in between?

You can actually measure this. The standard technique is a calibration curve: bin the model’s stated confidence scores, then compute the rate at which human experts agree with the model’s prediction in each bin. A perfectly calibrated forecaster falls on the diagonal—when it says it is 80% sure of a label, then 4 out of 5 times a human expert label will match.

So what did I get when I ran this comparison for Ethyca’s Helios classifier against ~2,000 challenging, human-labeled data classification tasks? Something quite unexpected:

The LLM output was almost perfectly calibrated! Or, in other words, for this task, Helios is exceptional at forecasting human agreement with its outputs.

Now for the part relevant to evals: I did not get this calibration on the first attempt. The first implementation trended in the right direction but was systematically overconfident.

I could have bolted on a per-model correction factor but was not eager to get on that maintenance treadmill. Fortunately, with an eval in hand, I could experiment with prompt variations to find an approach to better calibrate scores.

The technique that eventually worked was borrowed from human forecasting literature, Hubbard’s How to Measure Anything in Cybersecurity Risk. A core insight from this book is that:

1. Humans are generally bad at quantitative forecasting.
2. They can be trained to get dramatically better.

It turns out the same interventions that improve human forecasters improve LLM forecasters. Most of the advice boils down to “be specific.” In the case of confidence score evaluation, calibration greatly improved when models were given specific criteria to consider when making their determinations:

A rubric anchors scores to concrete characteristics to evaluate, rather than relying on a general feeling of certainty. Because I had an eval, I could systematically explore what rubric entries improved calibration. Even more, I could test that these changes raised calibration across multiple model families rather than addressing the quirks of a single model. Before the rubric, all three models were overconfident across much of their range; with it, their agreement rates moved much closer to the diagonal.

Without an eval, it would have been impossible to tell if confidence scores output by LLM were meaningful. It is easy to imagine the machine outputting convincing nonsense. On the other hand, it looks good enough, so why not use it? Indeed, the findings from experiments with rubric design showed that both sides are correct depending on your prompt! Evals provide a way to move past this deadlock to confidence score outputs that were genuinely useful to users.

Validating purpose of data use detection

The previous two examples used evals to improve an existing system. This one used an eval to answer an earlier question: should this feature exist at all?

The capability in question is “purpose of data use” detection: given context about a system—its schemas, access logs, or source code—can an LLM automatically deduce for what purpose data is being used? Does this system use data for advertising? For contacting users? For legal obligations? Purpose of data use is a key signal in privacy regulation, and today determining it involves onerous manual research. If LLMs could pre-populate this reliably, it would transform features like RoPA generation and Purpose-Based Access Control.

Data Use Analysis — advertising.marketing.communications.email

This system appears to manage opt-outs and preferences for marketing campaigns such as newsletters and product research.

Confidence: ■■■ strong

Evidence:

• email_preferences.opt_out tracks opt-out status for newsletters, product updates, and research invitations. (Strong)
• customer_accounts.email_preferences stores each customer’s choices for the same marketing email categories. (Strong)
• email_preference_events.campaign_id records preference changes associated with a campaign. (Strong)

Referenced collections: email_preferences, customer_accounts, email_preference_events

Now, “an LLM can figure out what a company does with its data by reading table schemas” is the sort of claim that is easy to picture at demo scale, but would it stand up in production against the complexity of an enterprise environment? So, before building anything user-facing, I built the eval.

I used as much real-life enterprise data as possible, tapping a design partner that provided 173 tables and 2,342 fields, as well as an extensive human-verified system inventory to serve as ground truth. Of the 62 Data Uses in the Fideslang taxonomy, the system genuinely exhibits 22. The question for the eval was: can models find those 22 without inventing the other 40?

Here are the results against expert-labeled ground truth:

Interestingly, each model had its own personality. Gemini returned the fewest false positives (highest precision). Opus missed nothing (100% recall) but speculated the most. GPT offered the best blend, one point shy of a 90% F1. This texture of LLM intelligence maps directly to product decisions, since a suggestion-review workflow might happily trade precision for recall.

While this eval was relatively limited in scope, focusing on a single company’s systems, it contained surprises nonetheless. I expected much more disagreement with the ground truth dataset, requiring significant prompt engineering and UX consideration to resolve. Instead, with minimal prompt engineering effort, I managed to closely replicate a laborious human workflow—inventorying a system's data uses—in an automated fashion.

In fact, the biggest surprise was the system uncovering data uses that were missed by the original human reviewers! This is not the first time I have encountered this result, and it brings up a key product consideration: users must be carefully introduced to improvements from AI functionality if it will potentially reveal mistakes in their work. No one likes being told they’re wrong by a robot.

So, all together, building an eval upfront allowed me to validate the feasibility of a feature, make the case for its value, and flag downstream product considerations. With these concrete results in hand, I could confidently graduate data use detection from the lab to the product team’s purview.

Closing remarks

Evals let me get so much more out of AI systems over the last year:

• Evals let me confidently make a major architectural change to achieve a 20-fold cost reduction, and let me avoid numerous potential traps along the way.
• Without an eval, there would have been no way to tell the working techniques for confidence score calibration from the plausible-sounding duds.
• Evals validated an extremely speculative feature—purpose of data use detection—revealing product characteristics that would be critical to development.

And as useful as these evaluations were, they are far from the last word. Readers familiar with the practice of evaluation will see the many opportunities to increase rigor in the work presented here. The evaluation datasets can become larger and more diverse, labeling can become more rigorous, and the interval between benchmarking and production monitoring can become shorter. But that is the broader lesson: evaluation should not be treated as a one-time certification but instead as an operating discipline.

In this way, I found the organizational value of evals was as important as their technical value. Evals provided a discussion framing to push past general arguments about whether AI is trustworthy to specific questions: what constitutes ground truth? What failure modes matter? How much risk is acceptable? What evidence is sufficient to ship?

These are the conversations at the foundation of trustworthy AI systems, and I've had a great time advocating for these practices at Ethyca as AI projects blossom across the company. I couldn't have been happier to see an engineer share the following in a recent retrospective:

Hard-won practices

Write evals from day one. They caught regressions on every prompt and model change.