Webinar Transcript
Getting AI Evaluation Tooling Into Your Environment
The case for an AI evaluation layer has landed. The new question is delivery: can that tooling actually get into your environment, on your terms, before the next proposal cycle?
In this session, Morgan Kostal and Martin Fay walk through what that shift looks like up close — the hidden AI tax engineers are already absorbing, the four deployment postures regulated buyers now demand, why MCP has become the wire that connects specialist tools to the AI surfaces teams already use (with a live QVscribe demo), and how security moved from a procurement checkbox to the front door. Full transcript below.
Prefer to watch?
The full recording runs about 35 minutes, including the live demo and audience Q&A.
Section 1 — Welcome & Today's Question
Lead: Morgan, supported by Martin
Hi everyone! Thank you so much for joining us live or watching the recording on demand afterwards.
I'm Morgan Kostal, Customer Success Manager here at QRA. I work with engineering teams across our customer base, helping them get the most out of QVscribe, which means I also get a pretty close view of how this kind of tooling actually plays out across an organization, from the engineers using it day-to-day to the leaders thinking about how it fits into their broader strategy.
And with me again is Martin Fay, our Product Manager. Martin works directly with customers to understand what our tools need to do to actually solve real engineering problems.
Yana set today up well, and the shift she described is real. The case we made in April — that you need an evaluation layer alongside whatever AI is generating — has landed. What's surprised me is how fast it moved after that, from “do we need this architecture” to “can it actually get into our environment.”
And we're hearing it everywhere — in nearly every customer conversation, and in what our sales team is bringing back from new prospects. Different industries, different roles, same question. That's really why we wanted to spend today on it: what the shift looks like up close, what's driving it, and what it means for anyone trying to get this tooling into their organization.
Martin, when did you first start noticing this in the conversations coming back to you?
About a year ago. Previously questions were typically feature-centric — “can the product do this particular thing.” Now it's more like: “we're already standardized on this toolchain, and we're about to commit to this platform — does your architecture fit, and what do you need from us to make that work?”
What stands out is the specificity. Prospects already have the integration architecture sketched out, with specific protocols, deployment postures, and InfoSec requirements that all have to clear before we can even begin a pilot deployment.
And we're seeing the same thing across our existing customers — teams asking how QVscribe fits what their organization is building toward, what AI tools it sits alongside, how their security teams will look at it. So whether it's a brand-new evaluation or a customer of years, the questions are converging. And if you've been the person trying to get a tool like this through, the playing field has shifted — there's good news in that, and some new things to plan for. We'll get to both. But first, the part that doesn't get talked about enough: the cost of getting this wrong. It's real, it's the hardest thing for leadership to see, and it shapes everything that comes after.
Yeah — this is perhaps the most surprising part. The cost is real, but it doesn't always look like a cost. It looks like habits.
Section 2 — The Hidden AI Tax: What Engineers Are Already Absorbing
Lead: Morgan, Martin extends
So, the cost. There's a pattern we call the hidden AI tax, and the tricky part is it doesn't look like a cost to leadership — it looks like engineers being productive, AI getting used, the license on the budget. From leadership's view, everything looks fine.
Back in April we made the case for distinct layers: the Scribe that generates, the Gavel that evaluates against your standards, and an intelligence layer underneath — your engineering knowledge, structured, that both reason against. Three different jobs, and one system shouldn't do all three. Without that separation, generation runs ahead of validation and the gap shows up downstream. The tax is what that gap costs you right now, before the architecture's in place.
Plausible output and trustworthy output look the same on first read. The only thing that can distinguish them is what they're measured against — and that's the muscle most teams haven't built yet. In practice, engineers default to visual inspection: read the output, check if it looks right, move on. That can work fine for a handful of requirements. It does not scale to a program, and it certainly does not survive an audit.
There are a handful of these patterns — we wrote up the full set in an article I'll drop in the chat — and as I describe them, see if you recognize them on your own team. The first is the whisperer: one engineer becomes the team's de facto AI expert and everyone else routes through them. From outside it looks like that person is just really good at AI — and they probably are. But the team's whole AI capability now lives in one person, and if they leave, it leaves with them.
And the fix isn't obvious. In the abstract, the advice is easy: document the prompts, write up the patterns, build a playbook. In practice, the whisperer is the busiest person on the team — and the last one with time to do any of that.
Yeah. The part that really matters underneath it is knowledge retention — the whisperer's usually a senior engineer, and when they move on, the prompt knowledge walks out with them. What I've seen actually work isn't a doc; it's carving out time for them to pair with a couple of other engineers for a few weeks. It transfers better person to person — but you have to plan for it.
And that's the longer-arc reason that architecture matters more than any individual prompt. Encoding standards in the evaluation layer means they can survive turnover. The team's institutional knowledge stops being a person and becomes a system. That matters most for the kind of programs our customers are running — long-cycle, multi-year, certification-bound. The AI investment compounds across the program, because the standards stay enforced even as the people change.
And in product discovery, this pattern shows up in a specific way. When we ask engineers to describe their goals and struggles, the whisperer will come up right away — because it's a person, there's a name attached. For the other patterns: engineers might describe them in detail but won't name them as patterns. They're just “how the work happens now.”
Which is honestly why naming these patterns matters in the first place. Once you've named a pattern, you can decide whether to change it. The diagnostic isn't really about the patterns themselves — it's about noticing what's quietly become normal.
And the second pattern — the one with the biggest downstream consequences in regulated work — is the cleanup engineer. Here's the example we keep seeing. A team tests Copilot rewrites against an existing requirements set, and what comes back has thirty extra requirements with no change record. So an engineer opens both documents side by side and goes line by line to find what actually changed. That isn't requirements work. That's forensic document reconstruction. And the person doing it is still on the org chart as a systems engineer — the job's inverted on them, they're doing data hygiene now, and nobody notices because the title hasn't changed.
Right. And this one is the easiest to miss — because it looks like someone being diligent. They're catching errors. They're doing important work. Nobody questions it. The signal that something's wrong only shows up when you look at the gap between their job title and how they spend their day — and ironically that gap is so wide that unless you're looking for it, you won't see it.
How I've come to see this one, and really all of these patterns to some extent, is that your engineers have become the integration layer. They're the ones copying requirements out of Jira, Confluence, or DOORS. They're the ones reworking those requirements into something the AI can use, and they're the ones pasting the output back into the system of record. The work the architecture should be doing, your engineers are doing it by hand.
With the “cleanup engineer” pattern, the junior staff become the “seamless” in seamless AI. The AI looks smooth because there's a person behind it making it smooth. There's a productivity cost, but more significant is the opportunity cost. The people with the most domain knowledge end up spending their time making AI usable, instead of applying that expertise where it should matter. The gap between what they should be doing and what they're actually doing is the tax.
So we've talked about what these patterns are and what they cost. The harder question — and the one that gets less attention — is why they stay invisible to leadership for so long. Part of it is what Morgan opened with: they don't look like costs from the outside, they look like normal operations.
Yeah. And the bigger AI investment kind of defends itself. When someone surfaces a tooling gap, the answer comes back as “but we already have AI.” And the conversation stops there before the comparison is even really fair. The bundled tool wins because nobody questions it, not because it's the right tool for the job.
The angle on this that I think gets underplayed is the lifecycle visibility one. Leadership doesn't have a clean view across artifacts. They have a Jira dashboard, a Polarion dashboard, a Copilot usage report, but none of those actually show them where the manual work is piling up between requirements and the downstream artifacts. So the tax is paid in places that don't appear on any of the screens leadership actually looks at.
And that's where the lifecycle argument matters. The point of the architecture from April wasn't just to evaluate requirements better. It was to clean up the requirements layer so that everything downstream can become workable too. Once your requirements are structured and traceable, the AI tools your team already has can actually do something useful with them. Right now, most teams can't get that benefit, because the input is not clean.
And the regulatory piece, for anyone in a notified-body or certification-bound environment, is that those workarounds we just walked through aren't auditable. A shadow document, a consumer LLM session, a whispered prompt that only one engineer can run — none of that stands up in a review.
And there's a clock on this. A couple of numbers, if you haven't seen them: only 7% of enterprises consider their data AI-ready — that's from Cloudera and HBR. Gartner forecasts that 60% of AI projects will be abandoned through 2026 in organizations without AI-ready data architectures. And enforcement of the European Union's AI Act is phasing in over the next year. So the architectural question is no longer theoretical. It has a calendar attached.
And here's the diagnostic I'd put back to everyone watching. If your sanctioned AI tool disappeared tomorrow, would your engineers' workflow actually change? If the answer is “not much” — because they've already routed around it — that tells you where the real workflow is. And that's where the tax is being paid.
Yeah, and from the product side, every one of these patterns has the same shape underneath. The team has good intentions, good tools, an AI investment — and the patterns still show up. Which tells you the patterns are not a tooling problem. They're a problem of how the tools relate to each other.
The case we made in April's webinar still holds. You need a specialist evaluation layer working alongside whatever AI is generating. What's surfaced since April is the next question: how does that specialist layer actually get into the environment teams are working in? Because if it can't, none of the architectural case matters. That's where the real conversation has moved.
Section 3 — The Architecture Is the Ask
Lead: Morgan, Martin co-leads [19:00]
Right. And in practice the question pulls in two directions depending on the environment. We're hearing buyers come in with very specific technical requirements already on the table — an MCP connector, API integration — before anyone on our side has raised them. What they want is the requirements intelligence layer living inside the AI tools their teams already use — Copilot, Confluence, Rovo — not competing with them.
And what's surprising is that the buyers didn't form that view from a sales conversation. They've done their own architectural research, and they're bringing in a fully formed view. That's a different posture than what we used to see — these buyers are not coming in to be sold. They're coming in to verify that what we have matches what they've already designed.
And in some cases those specs are a budget precondition — procurement won't move unless they're met. So it's not preference, it's a hard constraint. The opposite pull is showing up in other cases, like in the semiconductor industry. InfoSec in some of those organizations won't approve any external LLM path — data stays on-prem. And the tension's right there in the room: the engineers can see the AI capability in the tool, they want it, and InfoSec has already said no. So the only thing that gets a vendor in the door there is running the intelligence layer fully on-prem. And neither pull is a niche anymore — both are just part of the landscape now.
Right. And of course, hard-line demand for on-prem and air-gapped deployments is not new. This has long been the norm with defense — aerospace primes, federal research labs, defense electronics.
What's notable now is that same constraint showing up outside defense. The semiconductor case that Morgan just described is not a classified-program problem. It's InfoSec making a blanket determination: “no external path, full stop.” Distinct reasons than defense, but the same hard line.
Morgan, what I didn't see coming is how fast the migration is happening. Six months ago this still felt mostly like a defense-sector thing. When did you first notice it spreading beyond that?
Honestly, it was a couple of non-defense customers raising it within a few weeks of each other earlier this year. And it wasn't always InfoSec driving it — I've got customers in energy and nuclear work where it's the end client writing data-sovereignty terms into the contract, so they land on the same on-prem requirement by a completely different road. That's what caught me off guard — how many different doors it was coming through.
Yeah, and this makes for a “fun challenge” with product strategy. Optimizing for one of those patterns is structurally incompatible with the other. If you build cloud-only, you can't enter the on-prem environment. If you build on-prem-only, you can't ride the AI surfaces that the cloud-native team is already using. Which means — and this is the part that has shifted our roadmap thinking — we're not building a product anymore. We're building a delivery matrix, serving four postures:
Cloud-native, for organizations going digital-first. Private cloud, for accounts that have data residency requirements. True on-prem, mostly for defense but increasingly also beyond that. And finally, MCP bridging — which we'll demo in a minute — for enterprises where AI adoption is already embedded, and the intelligence layer needs to ride on top of it.
And what that means for any team watching is that the April architecture — Scribe, Gavel, intelligence layer — has to ride on any of those four. Otherwise, you don't enter the building.
And here's a question worth taking back to your own teams — if you can answer clearly which of those four postures is non-negotiable for your environment, you're ahead of where most teams are. If you can't, that's the first conversation worth having internally, because it shapes every vendor question downstream.
And the way regulated buyers are framing the ask now, when they put it cleanly, comes down to three questions. Can it run where we operate? Can it reach the AI we already use? And does it hold up under audit? That's the shape of the conversation now — and it's reshaping what we're building, too.
And the bigger pattern we're noticing is that the value conversation has settled, but the delivery conversation hasn't. How this actually gets in has gone from a procurement-checkbox afterthought to the first thing buyers want to talk about — and that's reshaping what every vendor brings to the table, us included.
Section 4 — MCP Is the Wire (with Live Demo)
Lead: Martin sets up, Morgan demos, both narrate [30:30]
Okay, let's dig into MCP — this is the part that has shifted most quickly. Recently, customers have started describing the architecture their teams are moving toward in MCP vocabulary — without being pitched anything.
To quickly frame the discussion: MCP — or “Model Context Protocol” — is an open standard that lets an AI tool discover and use specialized tools and services. The mental model has moved from “AI tools that you install” to “specialists that your AI can call.” That may sound mundane, but it's a fundamental shift in how this market thinks. The architecture has stopped being something vendors propose — it's something customers are pulling toward, often before any vendor brings it up.
And the easiest way to make it concrete is to show you. We've had a QVscribe MCP connector in active development for a few months. Before I share, let me be straight: it isn't finished, and what you see won't look identical to production six months from now. We're sharing it at this stage because the architecture is what matters to this conversation, not the polish — and the best feedback we get comes from showing it early to engineers like you. Let me share my screen.
So this is Claude Desktop — Anthropic's AI assistant. Notice we haven't opened a QVscribe surface; there's no QVscribe app. I'm in the AI tool I'd already be using, and that's the point. The QVscribe MCP connector is installed here, so Claude can call into QVscribe whenever the conversation needs it. I just ask. I've got a set of requirements in a Word document — I'll ask Claude to analyze them using QVscribe.
A couple of things. That card isn't generic AI output — it's QVscribe doing the analysis under the hood, same engine, same rules, same standards I'd apply inside QVscribe itself. Claude isn't grading the requirements; QVscribe is. Claude's just the surface. Which is exactly what Martin and I were talking about in April. The Scribe, which is Claude, and the Gavel, which is QVscribe, are doing different jobs. The fact that I can see them working together inside one surface is what MCP makes possible.
Martin, anything you'd flag here?
Yeah, the structure of this panel. The issue categories, the quality indicators, the category breakdown — none of that is invented by Claude on the fly. It comes from the MCP tool returning structured data, and that structure is what makes the audit trail real.
Exactly. And everything here is interactive — I can click into any finding and have Claude generate a follow-up prompt to dig in and start fixing it. So here's a requirement QVscribe flagged. It reads: “REQ-5: The driver will be able to use buttons on the steering wheel to set and change the maximum speed whenever the system is engaged.” That one comes back with a superfluous-infinitive alert, a passive-voice warning, and two EARS-nonconformance alerts — and just like in QVscribe, the trigger phrases are highlighted right in line, so you can see exactly which words are the problem. It scores three out of five. Let's ask Claude to suggest a rewrite using QVscribe.
And notice what Claude is not doing here. It isn't just making the sentence cleaner. It's responding to the specific flags QVscribe returned — so the rewrite is shaped by the evaluation.
Right. And here's the part that matters to me. I'm not just going to trust that the rewrite is better because it looks better. I'm going to ask Claude to run compare_requirements against the original. And that card, “Claude wants to use Compare requirements from qvscribe,” is the MCP handshake. Claude isn't pretending to do the comparison itself — it's asking QVscribe to, and I can see exactly what's being passed.
And that visibility is the part that really matters in regulated work. For a certification audit, “The AI did it” is not an answer. But “the AI asked the evaluation tool — here's what it asked, and here's what came back” — that is an answer.
And the implication for the lifecycle is worth naming. What you just saw goes beyond a rewrite. It's the requirements layer becoming clean: structured, scored, traceable across versions. Clean requirements can be reasoned over by downstream AI tools for test generation, verification, design analysis and so on. Existing downstream tools may work just fine — but they still need clean input to do so.
That's the lifecycle argument from our April webinar in working form. The evaluation layer is what makes everything downstream actually viable.
And there's a broader frame to this. Governance isn't confined to the requirements layer. Once requirements are clean and traceable, the same governance pattern has to extend across design, verification, test, certification. That's the next chapter of this work, and the architecture we're describing today is what makes it possible.
Exactly. And the practical version of this for the audience is that this isn't a new surface for your engineers to go to. It isn't another login or another tool to learn. If your team is already using Claude, Copilot, Rovo, or your own internal LLM, the intelligence layer becomes reachable from there.
The way I've started putting it is, your team doesn't come to us. We show up where they are.
What I keep hearing from engineers is that if a tool adds a step to their day, they just won't use it. They'll tell you “this is more work” and route around it. MCP removes the step.
And I want to be honest about where this still has work to do. The connector you just saw works cleanly inside Claude Desktop. The same approach across Copilot, Rovo, or your internal LLM is in progress, not finished. The architecture is portable, but the rough edges are real. We're showing it now because we'd rather have the conversation with the people in this audience while it's still shaped-able than wait until it's polished.
Pulling back from the demo for a second. The reason MCP matters now is that the AI tooling market is shifting in a fundamental way. Horizontal AI tools — Claude, Copilot, Rovo, ChatGPT — are absorbing the generic capabilities: summarizing, restructuring, drafting. The value moves to specialists that can be reached from inside those tools when the question gets serious enough to need them. What you just saw is what that architecture looks like when it's built for that future deliberately — not just another product fighting for screen time inside someone else's stack.
Yeah, and that's exactly what we're hearing on the customer side too. A year ago, the question I'd hear from teams already using us was, “do we still need this if we already have Copilot?” Now it's, “how does this work alongside Copilot?” Same teams, different question. The framing has flipped from competitive to complementary, and that's a real signal about where the market is now.
There's one distinction worth listening for, because over the next year you're going to start hearing “we have MCP” from a lot of vendors. Some are adding MCP as a feature — a single connector bolted onto an existing product to check a box. Others treat MCP as the substrate — the product is designed around being reachable by other AI tools. Those sound similar in a slide deck. In practice, they're very different.
To tell them apart: ask the vendor what happens when the AI surface you want to call them from is not on their existing connector list. If the answer is “we'd have to build that,” it's a feature. If the answer is “any MCP-aware client can already reach us,” it's substrate.
And there's a piece that matters more over time. The feature version locks the capability inside whatever tool suite the vendor already integrated with — so what that suite supports today is what you get. The substrate version travels with your team: as you adopt new AI surfaces, the same engine is reachable from them.
What we just showed is the substrate version. The same QVscribe engine is reachable from anything that speaks the protocol.
Before we move on — the air-gap question. The demo we just walked through doesn't apply to everyone; some of you can't use any external LLM path at all.
Right — this one is real. Every cloud-first product strategy starts to break down at this question.
For defense and classified-program audiences, the LLMs you're allowed to use inside your environment are usually older and smaller than what's available externally. That's structural — no tool can wave it away. But our approach is fundamentally different here, because the evaluator is deterministic. It applies your standards the same way every time, regardless of which model is on the other end of the MCP connection. So where model quality is constrained, the older, smaller model on the air-gapped side can still do the generation — and the deterministic evaluator is what makes the result defensible.
The architecture works whether the model is bleeding-edge or two generations behind. Same standards, same evaluation, same record.
“Bring your own model” sounds great until you realize the model you can actually bring is a generation behind — so getting that into a true air-gapped environment is a specific conversation, not a one-liner. Okay, one more thing before we hand off. There are four questions worth taking back to your organization after today.
First: does our requirements substrate have the structure an AI could actually reason against, or are we asking AI to interpret prose?
Second: is the engineer who knows how to prompt our tool documented anywhere, or does the team's AI capability live in one person?
Third: can a certifier trace an AI-assisted decision back through our evidence chain, end to end?
And fourth: if we said yes to an MCP-based purchase tomorrow, which AI surface would we want it reachable from first?
If you can't answer those clearly today, that's the architecture conversation worth starting next week.
Section 5 — Security as the Front Door
Lead: Morgan, Martin supports
Okay — we've covered the cost, the architecture, and the wire. The last piece is how this actually gets into your environment. The short version: the security conversation has moved from procurement to pre-demo. Sometimes it lands before anything can even be scheduled, because IT or security has put a hold on the whole vendor category, and nobody moves until that clears. And the speed has surprised us — even six months ago security was a back-half conversation; now it's often the first one, before engineering has even weighed in on whether they want the tool. Martin, what's that look like on the product side?
The biggest shift is who's in the first conversation. It used to be engineering; now it's security and engineering together, or security first. And the questions are architectural — how data flows, where it sits, what's auditable, who has access.
To make it concrete: two patterns you may recognize. For medical devices, the previous generation of tools ran locally and the rules were inspectable — an engineer could trace exactly what the tool did. The newer AI tools route data through models whose behavior isn't always visible — and that deserves an architectural answer, not a “trust us.” It's not so much that they don't trust AI; it's that they can't tell what's happening with their data. And for automotive, it's a hard gate: no pilot, no scoped test, no trial of any kind, until a Data Processing Agreement is signed. Until then, no engineer even touches the product.
And we're seeing that across our existing customers too — we have customers right now whose tooling is blocked by category-wide holds that have nothing to do with the individual product, so even renewals hit the same gate. But the pattern underneath all of it is the same: the bottleneck isn't conviction anymore. The teams already want the tooling. What's holding them up is the architecture clearing security first. And that's why the framing has to shift. Security isn't the obstacle. It's the audience.
Right. And what that means architecturally — for any vendor, us included — comes down to a few specific things. Tenant isolation has to be structural, not configurable: data segregation built into how the system works, not per-customer settings that drift. Customer inputs can't feed shared model training — an architectural property, not a written policy you can change later. And that's the distinction InfoSec is testing for: policies can change, architecture can't — not without rebuilding the system. The rest is the same family — real deployment options, controlled-data handling, DPA-ready. And the audience for all of that is the security function, not engineering. That's who you're writing for now.
So security stops being the last thing a vendor talks about and becomes the first. The architecture diagrams, the deployment story, the data-handling answer — all of it has to be ready upfront. And for anyone evaluating tools, that's the flip side: the material you'd normally see late should be the first thing you ask for.
Section 6 — Wrap-Up & Key Takeaways
Lead: Morgan, Martin closes
Okay, here's what it comes down to. The value conversation — whether evaluation infrastructure matters — is settled. What we spent today on is the new question: can it get into your environment, on your terms, before the next proposal cycle? And three things I'd love you to take away.
First, look at the patterns inside your own team. Find your whisperer, notice your cleanup engineer, watch for the workarounds nobody's named yet. They're the honest map of where your AI investment is sitting on top of integration work your engineers are doing by hand. Read the workarounds as data.
Second, the decision usually happens in a room you're not in. The decision-maker has done their own research and won't sit through a deck with you — so whatever you hand them has to stand on its own: the architecture story, the deployment options, the security answer, clear enough to evaluate without you there.
And third, treat deployment as a first-class question, not a last one. Where can it live — cloud, private cloud, on-prem, air-gapped? Which AI surfaces can it reach? Does the audit trail hold up end to end? Those are the front-door questions now, for any vendor, us included.
And depending on where you sit — the rest of the series is in the chat for going deeper; engineering leaders, reach out and we'll build a business case for your environment; and for security or IT, we've got an architecture and data-handling walkthrough ready, because that's a different conversation and we built it that way on purpose. Martin, anything to add before questions?
Just one thing, and it ties back to where we started in April.
There's a line that's stuck with me lately: AI can be the right answer and still never make it into your environment. That's the actual problem we've been describing today.
The April architecture — Scribe, Gavel, and the intelligence layer — that case has already been made. What we're working on now is the harder version: making that architecture actually deliverable into environments with constraints we didn't have to think about a year ago. And we're early in that work ourselves. The MCP connector we showed today is not a finished product — but it is the direction we're committed to.
Once specialist tools are reachable across every AI surface, the next question becomes: which specialist gets called, in what order, for which kinds of work. Supporting that is a deeper ambition — systems that detect conflicts automatically, visualize dependencies across artifacts, and maintain continuous alignment as things change. That's where the conversation seems to be moving. We're not there yet. But the architectural separation we're describing today — the surface, the substrate, and how they reach each other — is what makes that next layer reachable at all.
And whether the eventual answer for your team is us or someone else, the shape of the question is the same. AI that reaches the layer that makes its output trustworthy, AI that runs in the environment you actually operate in, and AI that leaves a record someone can defend.
That's the conversation we wanted to start with everyone today.
Love that. Alright everyone, let's open it up.
Audience Q&A
Yeah, this comes up almost every time, and the demo actually makes the answer easier. Those tools are doing one job, which is generating. They draft, summarize, restructure, and they're good at it. What they don't do is evaluate the output against your standards in a way that's repeatable and auditable. What you saw in the demo wasn't Claude doing the evaluation, it was Claude calling QVscribe to do the evaluation and surfacing the result. The architecture you need is both layers working together, and the evaluation layer is what those tools don't have built in.
And the design choice is intentional — it's exactly what we made the case for in April. The same system that generates an artifact shouldn't be the one grading it. That's architectural, not merely competitive positioning. If your evaluation and your generation are the same model, you don't have an audit trail that holds up.
Yes, and arguably more relevant. If your AI strategy is still forming, the worst outcome is making architectural decisions now that you can't unwind later. The teams I see in the strongest position are the ones who set up the evaluation layer first and then layer in generation capability on top of it. Doing it in that order means whatever AI tool you adopt down the road plugs into a foundation that already enforces your standards.
And the architectural shape doesn't depend on the size of your AI investment. A team with one engineer using ChatGPT informally has the same evaluation problem as a team with a fully rolled-out Copilot deployment. They just feel it at different scales.
This is a near-verbatim transcript prepared from the hosts' presentation. Live delivery included minor natural variations in wording; audience questions shown were posed during the live session. Timestamps are approximate.