Evals Before Agents: How We Measure AI Quality in Hospitality

The core problem we're solving with Line is one of making supply (e.g., inventory and availability of a hotel) legible to demand in group sales. Our solution is to collect, aggregate, and meaningfully contextualize data for a venue and layer functionality on top of that data that greatly improves common workflows in hospitality. Our first useful piece of functionality is an AI group sales agent that helps overwhelmed group sales teams ingest and qualify inbound leads.

We started building these AI group sales agents on a per venue basis by hand. We'd bring in private and publicly available venue data (e.g., pricing, amenities, etc) and craft venue specific prompts by hand trying to be cognizant of the task the agent had to perform (group sales) and the brand, tone, and positioning of the underlying venue.

Naturally, the question arises, "How do you know if these agents are any good?" The manual way to determine this is to simply chat with the agents we trained and get a feel for their accuracy and ability to accomplish their goals. Of course there are numerous issues with this approach: it's qualitative, based almost fully on "vibes" as we chat with the agent, there's no way to concretely track improvement over time, and even if an agent is "good" we have no real underlying framework to tell us why.

We found that our agents could handle straightforward questions well enough -- venue capacity, parking options, the easy stuff. But it would routinely hallucinate as the conversation went on, especially when it was pushed (e.g., "Are you sure you don't have any pricing available?"). In hospitality, that's not a minor bug. That's a broken promise to a bride planning her wedding, a corporate admin building a budget around fabricated numbers, a venue owner whose brand just got misrepresented to a potential client. The trust damage is immediate and the recovery is slow.

It became readily apparent to us that it wasn't enough to be confident that our agents could hold a conversation (they can), but whether we could prove they were accurate before putting them in front of real prospects. We were never going to solve this manually based on vibes, optimizing for taste would only get us so far. We needed a different approach.

The Problem with Evaluating on Vibes

With all AI deployments, a core issue is measurement, Line's agents were no exception. When you test an AI agent by chatting with it yourself, you're doing two things poorly at once: simulating a realistic prospect (you're not one) and evaluating the quality of the response (subjectively). You already know what the agent should say, so you unconsciously steer the conversation toward success. You test the happy path because you built the happy path.

This approach has other problems too. You can't run the same evaluation twice and get comparable results. You can't track improvement over time. You can't identify which specific thing is broken when the conversation feels off. "The agent needs to be better" is not actionable engineering feedback.

We needed what every engineering team needs when quality matters: automated tests with quantitative pass/fail criteria.

Evals Are Tests -- Stochastic Ones

When I was an engineering director at Sentry, I put a lot of thought into quality measurement, particularly for AI-driven systems. I even wrote about it some. The insight I had at Sentry was that agent evaluation ("evals") should be treated just like software testing: define expected behavior, simulate realistic inputs, measure the output against explicit criteria, and iterate until the system passes.

But there's an important difference. Unit tests are deterministic -- same input, same output. Agent evals are stochastic. The same adversary persona can produce different conversations each time, and the evaluator can score the same transcript slightly differently on different runs. If you've tried to build reliable evals for LLM-based systems, you already know this is the hard part.

We don't pretend this is solved. What we've found is that structured rubrics with specific score-level definitions (not "rate quality 1-10" but "a score of 8 means X, a score of 5 means Y") significantly reduce variance. Multi-dimensional scoring helps too -- a single aggregate number hides too much, but a few independent scores with defined rubrics produce more stable, diagnostic results. And testing across multiple diverse use cases per cycle gives us redundancy that smooths out noise from any single evaluation.

It's not deterministic. But it's consistent enough to be actionable, which is the bar that matters.

We built our eval framework around three roles -- an Actor, an Adversary, and a Critic -- that mirrors how a good QA team would stress-test a sales agent if they had unlimited time and patience.

The Actor is the AI agent being evaluated. Not a simulated version, not a sandboxed copy -- the actual deployed agent running on our production infrastructure. We test what we ship.

The Adversary is a simulated prospect designed to be realistic and challenging. Each adversary has a specific persona with distinct behavior patterns: how much information they share, how they respond to questions, and a curveball they throw mid-conversation to test the agent's adaptability.

The Critic acts as a senior sales auditor. It reviews the full transcript against the venue's actual knowledge base -- the ground truth -- and produces a quantitative score across four dimensions. Yes, this means we're using an LLM to judge another LLM's output. We'll come back to why we think that's defensible and where it isn't.

It's important to note that, unlike unit testing for software which has reliable patterns and approaches, how you craft an eval framework is highly domain specific. For example, an eval framework to determine an agent's ability to properly diagnose a bug in a software system is very different than the framework used to determine if an AI sales agent is sophisticated enough to qualify an inbound lead. In other words, the Actor/Adversary/Critic approach made a ton of sense for us, it might not for you.

What We Measure (and Why Accuracy Comes First)

We score agents on four categories, each weighted to reflect its importance:

Weight	Category	What It Measures
40%	Accuracy	Did the agent only state facts found in the knowledge base? Any fabrication is a critical failure.
30%	Discovery	Did the agent identify the prospect's date, budget, headcount, and event type through natural conversation?
20%	Brand Alignment	Did the agent match the venue's voice and create an experience consistent with the brand?
10%	Strategic Push	Did the agent guide the prospect toward a concrete next step, like booking a call with the events team?

These four scores combine into a Performance Index (PI):

PI = ((Accuracy x 0.4) + (Discovery x 0.3) + (Brand x 0.2) + (Strategic Push x 0.1)) / 10

A PI of 0.85 or above means the agent is production-ready. Below that, it goes back for optimization.

The weighting is deliberate and opinionated. Accuracy gets 40% -- more than any other category -- because a hallucinating agent is worse than a boring one. An agent that sounds a little generic can still be useful. An agent that fabricates pricing, invents amenities, or makes promises the venue can't keep destroys trust in a single conversation. In hospitality, where events are high-stakes and high-emotion, there's no recovering from "the AI told me your venue had a rooftop terrace" when it doesn't.

Some of these categories are subjective and difficult to measure. For example, what does it mean for an agent to be brand aligned, and how do you even quantify that? Short answer: you guess and it's usually good enough. Even subjective metrics are useful because you're comparing the performance of the agent to some other iteration, not against an actual ground truth, so measured improvements are still improvements.

Three Personas, Not One

A single test conversation isn't enough. An agent that handles a straightforward corporate inquiry might fall apart when a bride changes her mind mid-conversation, or when a community organizer isn't sure what they want and needs to be led.

We test against three distinct personas in every evaluation cycle:

The luxury buyer -- detail-oriented, high expectations, shares information progressively and makes the agent earn it. Mid-conversation, they'll throw a curveball: "What if we wanted a Sunday brunch reception instead?"

The skeptical budget-holder -- price-sensitive, wants specifics, pushes back on vague answers. Their curveball: "Our CEO just told me we might need to cut the budget by 30%."

The undecided organizer -- vague on details, not sure what they want, needs the agent to take the lead. Their curveball: "Actually, could this also work as a fundraiser with a silent auction?"

Three conversations is not a statistically large sample, and we know it. But each conversation is 15+ turns covering a wide range of scenarios. The personas are designed to be maximally different from each other -- different information-sharing behaviors, different objection patterns, different curveballs. The system is built for rapid iteration rather than one-shot statistical significance. We'd rather run three deep, diverse conversations and iterate quickly than run thirty shallow ones and wait. What's important is that personas are easy to add. If in the future we identify some other diverse but important persona, we can easily add it to the framework.

All three personas must pass for the agent to be considered production-ready. Fixing a failure for one persona must not cause a regression for another. This multi-persona requirement is also our primary overfitting prevention mechanism -- an agent can't game its way to a passing score by excelling with one prospect type.

Teaching Agents What They Don't Know

One of the most effective techniques we developed doesn't involve prompts at all. It involves knowledge design.

When we build an agent's knowledge base from a venue's available data (e.g., public website, internal documentation, etc), we don't just capture what information exists. We explicitly document what information doesn't exist. If we don't find published pricing, we don't leave a gap -- we create a specific knowledge item that says: no per-person rates, minimums, ranges, or dollar amounts exist for this property. Do not fabricate, estimate, or invent any pricing figures.

This matters more than it might seem. When an LLM encounters a question it can't answer from its context, the default behavior is to be helpful -- which often means generating a plausible-sounding answer from its training data. A thin knowledge entry like "pricing will be provided by the booking coordinator" leaves room for the model to fill in gaps. An explicit negative-knowledge entry removes the ambiguity entirely.

We call this anti-hallucination knowledge design, and it changed our most persistent failure mode. It doesn't make hallucination impossible -- models can still ignore instructions, especially under sustained pressure from a user. But it moves the failure mode from "the model had no guidance and guessed" to "the model had explicit guidance and overrode it," which is a much rarer and more detectable failure. Before we implemented this approach, pricing and amenity hallucinations were our most common Accuracy failures. After, they became our least common.

The hallucination problem is often a knowledge design problem. The problem isn't that models want to lie. It's that we weren't giving them enough information to know what they don't know.

The LLM-as-Judge Problem

We said we'd come back to the Critic, and here's the full picture: using one LLM to judge another's output is a known-fragile pattern, and we're aware of the risks.

The detailed rubrics help -- each score level has a specific definition, which constrains the Critic's subjectivity. Accuracy is the easiest dimension to judge reliably because it's largely factual: did the agent say something that's in the knowledge base, or didn't it? The Critic can check statements against ground truth in a way that's closer to retrieval than opinion. Discovery and Brand Alignment are inherently more subjective, which is part of why they're weighted lower.

We periodically spot-check Critic scores against human judgment. The correlation has been strong enough for our purposes -- the Critic rarely disagrees with a human reviewer by more than one point on any dimension, and the directional assessments (pass vs. fail, improving vs. regressing) have been consistent. But we haven't run formal inter-rater reliability studies, and we know that as models update, the Critic's calibration could drift. Remember, we're biasing for actionability, not perfection.

The Critic is a limitation we're actively monitoring and working to improve, but not one we've solved. For now, the framework gives us a repeatable, quantitative baseline that's dramatically better than the subjective, ad-hoc evaluation it replaced. The rubrics make it better than naive LLM-as-judge. But it's not ground truth, and we don't treat it as such.

The Optimization Loop

Scoring alone isn't enough. The question after a failing eval isn't just "how bad is it?" -- it's "what specifically needs to change?"

Our eval framework produces diagnostic output alongside scores. When an agent fails, the system identifies specific issues: the agent fabricated a rooftop terrace, it didn't ask about budget until the prospect volunteered it, it didn't match the venue's upscale tone. Each issue maps to a specific fix:

• Low Accuracy? Fix the knowledge base first. No amount of prompt engineering fixes bad underlying data.
• Hallucinations detected? Add explicit guardrails to the agent's instructions.
• Low Discovery scores? Strengthen the qualifying question flow.
• Low Brand Alignment? Adjust the tone and voice instructions.

The system applies these fixes and re-evaluates in a closed loop -- test, score, diagnose, fix, re-test. Each iteration produces a new PI score that we can track. The loop runs up to three automated iterations. When it converges, it typically does so in two to three cycles. When it doesn't -- when fixing one persona breaks another, or when the underlying knowledge base has gaps that prompt engineering can't paper over -- it escalates to a human prompt engineer. This closed loop is fully hands-off along its happy path, keeping humans out of the loop for as long as possible and freeing them up to do more challenging work.

That escalation happens more often than we'd like, maybe a third of the time. The most common cause is knowledge gaps that our data collection/knowledge building mechanisms didn't catch -- venue policies that aren't on the website, seasonal information that's buried in a PDF, the kind of tribal knowledge that lives in an experienced coordinator's head. The automated loop is good at optimizing within the constraints of existing knowledge. It's not good at inventing knowledge it doesn't have. That's a feature, not a bug, but it means the system has clear scope boundaries.

What the Pipeline Actually Looks Like

The full pipeline uses browser automation, document retrieval and ingestion, and many other mechanisms to build out the knowledge base for an agent, populate our platform, simulate adversarial conversations against the deployed agent, score the results, and optimize until the agent passes. The eval loop mirrors what engineering teams do with automated tests -- but applied to conversational AI.

For our small team of engineers, this has been the difference between deploying one agent per week and deploying dozens. The exact timeline depends on the venue -- a simple restaurant with a clean website is faster than a multi-space event venue with complex packages -- but the manual work dropped from days to minutes in every case.

A knock on effect of all this work: since the framework is quantitative, we can have honest conversations with venue partners about agent performance. Showing a venue owner that their agent scored 0.88 across three diverse prospect types, with accuracy at 9/10, is a different conversation than "we think it's pretty good." It builds legitimacy and credibility with our customers.

What Happens After the Eval

A passing eval doesn't mean we stop paying attention. We monitor production conversations for the same signals the eval measures -- particularly accuracy. When a real prospect asks a question the agent handles poorly, our team reviews the flagged transcript and that failure feeds back into the knowledge base and the next eval cycle. In other words, real-world conversations fill the Actor and Adversary roles in our AAC framework, but the techniques to evaluate and optimize are still the same.

We're still early in correlating eval scores with real-world outcomes like conversion rates and prospect satisfaction. The honest answer is that our eval system tells us whether an agent is accurate and competent, not whether it's effective at generating revenue. Those are related but not identical. A perfectly accurate agent that lacks warmth might score well on our eval but underperform with real prospects. We're building toward production metrics that close that loop, but we're not there yet.

Start with Measurement

If you're building AI agents for a domain where accuracy matters -- and in hospitality, it always does -- start with how you'll measure quality. Not subjectively, not by chatting with your own agent and deciding it feels right. Define what good looks like across the dimensions that matter for your use case. Quantify it. Build the feedback loop that turns a failing score into a specific fix.

The framework we've built isn't perfect. The LLM-as-judge pattern has known limitations. Three personas don't cover every prospect type. The automated optimization loop doesn't always converge. But it's quantitative, it's repeatable, and it catches problems before prospects do.

Evals aren't optional. They're the foundation. Build them first.