Evaluations | The .NET Blog

Model Router Evals Are the Step Too Many Teams Skip

Emiliano Montesdeoca — Fri, 29 May 2026 00:00:00 +0000

Automatic model routing sounds great right up until you realize you still have to prove it is the right choice for your workload.

That is why the new model router evaluation repo is useful.

It gives teams a more concrete way to answer the questions that actually matter:

does routing preserve quality?
does it improve cost?
what does it do to latency?
what changes if I restrict the model subset?

The source article asks the right questions

One thing I really like about the original post is that it does not treat the model router as self-evidently good.

Instead, it asks the uncomfortable but correct questions:

“On my prompts, does the model router’s auto-selected model match or beat the single model I’d otherwise pick?”
“Am I actually saving money end-to-end, or just shifting spend around?”

That is exactly the right attitude.

Because automatic routing is attractive, but it is still a systems decision. And systems decisions should be measured, not admired.

Why this repo matters more than it first sounds

At one level, this is just an evaluation repo.

At another level, it is a sign of maturity.

It says: if you want to adopt automatic routing, here is a more disciplined way to test:

quality
cost
latency
subset trade-offs
model distribution behavior

That is much better than treating routing as a black box with good branding.

My take

This is a good example of the kind of tooling AI platforms need more of: not more magic, but more ways to validate the magic before you trust it.

That is how teams avoid building expensive confidence on top of untested assumptions.

Original post: How to run evals for the model router

The Hard Part of AI Development Is No Longer Access. It Is Operating the Right Model Well

Emiliano Montesdeoca — Tue, 26 May 2026 00:00:00 +0000

We are well past the phase where simply having access to a powerful model is enough.

That is what this new Foundry guide to managing models, cost, and quality gets right.

The real challenge now is operational:

choosing the right model per workload
validating it against your own data
managing latency and spend
governing upgrades and regression risk

That is what serious teams need to get good at.

The source article gets the problem definition right

One sentence from the original post captures the shift very well:

“The hardest part of building AI systems today is no longer getting access to a capable model. It is knowing how to choose, validate, optimize, and operate the right model across the full lifecycle of a real application.”

That is exactly the right diagnosis.

Too many teams still think model selection is the main decision.

It is not.

Model operation is the bigger problem:

what workload gets which model?
how is quality verified?
what cost shape is acceptable?
what happens when a new model appears or an old one drifts?
how do you test a change without breaking real workflows?

That is the real engineering work now.

Why this Foundry piece is useful

I like this article because it talks about AI systems the way experienced platform engineers actually have to think about them.

Not as “pick the smartest model and move on.”

But as systems that live under trade-offs:

capability
latency
cost
safety
governance
upgrade pressure

That is much more useful than benchmark-driven optimism.

The most important shift is criteria-first thinking

The source article recommends defining success criteria before opening the model catalog.

I think that is one of the most important habits teams can adopt.

If you open the catalog first, you anchor on reputation.

If you define criteria first, you anchor on workload reality.

That is a healthier process.

Because the model that wins a benchmark is not automatically the model that wins:

on your prompts
with your latency budget
within your cost guardrails
inside your governance requirements

That distinction is where mature AI engineering begins.

The multi-model story is becoming a real advantage

Another thing I like is the explicit model-agnostic framing.

The article presents Foundry not as a single-model destination, but as an operating surface across:

Microsoft models
partner models
open-source models
post-trained variants
routing and optimization strategies

That matters because model flexibility is not a luxury anymore. It is part of risk management.

If quality shifts, prices move, or quota becomes constrained, teams need options.

Cost control is not a secondary concern

The article is also right to frame cost as an architectural concern.

This is not a “we will optimize later” problem.

If you send every task to the heaviest model by default, that may work beautifully in a demo and collapse under production economics.

That is why I think the sections on:

routing
batching
caching
provisioned throughput
quota management

are more important than a lot of people might assume.

The teams that treat cost discipline as part of system design will age much better than the teams that treat it as cleanup.

My take

This is a useful Foundry piece because it talks about AI systems the way experienced engineers actually have to run them.

Not as demos. Not as one-off prototypes. Not as leaderboard tourism.

But as operating systems for workloads, constraints, trade-offs, and continual change.

That is the level the conversation needs to keep moving toward.

And if you are building production AI systems, this is exactly the mindset I would want teams to internalize early.

Original post: A Developer’s Guide to Managing Models, Cost and Quality in Microsoft Foundry

Foundry’s Observability-to-ROI Story Is What Serious Agent Platforms Need

Emiliano Montesdeoca — Mon, 25 May 2026 00:00:00 +0000

If AI agents are going to live in production, observability cannot stop at logs and traces.

That is why the new Foundry observability to ROI story feels important.

The real message is not “we added more dashboards.”

The real message is that serious agent platforms need a continuous operating loop:

trace what happened
evaluate whether it was good
optimize what needs work
connect the outcome to business value

That is a much stronger story than the usual platform hand-waving.

The key sentence in the source article says it all

The original post opens with a line I think every team building agents should pay attention to:

“Shipping an AI agent is the easy part. Keeping it accurate, safe, and accountable in production is where teams get stuck.”

That is exactly right.

We have already passed the phase where “can I make an agent do something cool?” is the main question.

The harder and more valuable question is:

can I operate the thing once it starts interacting with real users, real tools, and real costs?

That is where Foundry is trying to push the conversation.

Why this matters more than another agent demo

A lot of AI agent announcements still focus on creation: build the agent, wire the tools, route the tasks, ship the interface.

That is all fine.

But the operational questions are where most serious systems either become sustainable or become expensive experiments:

what is the agent actually doing in production?
did it do the right thing?
is it getting worse over time?
is it too expensive for the value it creates?
which configuration changes actually improved quality?

This is why I think the Foundry announcement is more important than a typical feature roundup. It is trying to define an Agent DevOps loop, not just an agent creation story.

The four-part loop is the real product here

The article basically organizes the platform around four capabilities:

Trace
Evaluate
Monitor
Optimize

That is the right shape.

I would actually argue that any platform that wants to be taken seriously for agent production workloads eventually needs all four.

Tracing alone is not enough. Evaluation alone is not enough. Optimization without evidence is just guesswork. And ROI discussion without telemetry is usually theater.

The interoperability angle is especially smart

One of the strongest choices in the announcement is that Foundry is not pretending every agent will be built in one framework.

The source post explicitly talks about tracing and evals extending across:

LangChain
LangGraph
OpenAI SDK
Microsoft Agent Framework
custom frameworks via OpenTelemetry

That is important.

Because platform lock-in is one of the fastest ways to make an otherwise useful operations story less attractive.

If teams can keep their framework choices and still gain production-grade telemetry and evaluation surfaces, that lowers friction considerably.

Rubric evaluation might end up mattering more than people expect

The rubric evaluator part is also worth calling out.

I think this is one of the most practical additions in the whole post.

Why? Because “good” is contextual.

The article says rubric evaluation generates “context-aware evaluation criteria from your agent’s intended behavior.” That is exactly the direction these systems need.

Generic quality scoring is useful.

But eventually teams need to score agents against their own standards:

tone
task completion
policy adherence
latency expectations
cost boundaries
domain-specific business rules

That is where evaluation starts becoming operationally meaningful instead of academically interesting.

ROI is the most uncomfortable part, which is why it matters

I also think the ROI portion of the announcement is important precisely because it is uncomfortable.

The source asks the question directly:

“is this agent worth what it costs?”

That question gets dodged a lot in AI conversations.

But it is the right question.

If the platform can really connect cost, task completion, time saved, and production traces in one place, that gives engineering and leadership a much better shared language.

And honestly, that shared language is badly needed.

My take

This is one of the better platform-level announcements in the batch because it focuses on operating agents, not just building them.

And that is where the hard work really starts.

The strongest AI platforms over the next couple of years will not just be the ones with access to more models or more demos. They will be the ones that help teams trace behavior, evaluate outcomes, optimize safely, and justify cost with evidence.

This Foundry story is trying to move in exactly that direction.

That makes it worth taking seriously.

Original post: Build 2026: From observability to ROI for AI agents on any framework