// Naive approach - What NOT to do: the endpoint talks to the model vendor directly
app.MapPost("/products/{id:int}/description", async (int id, IProductCatalog catalog, IHttpClientFactory httpClientFactory) =>
{
    var product = await catalog.FindAsync(id);
    if (product is null)
        return Results.NotFound();
 
    var http = httpClientFactory.CreateClient();
    http.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", "YOUR-API-KEY");   // a secret in the source code
 
    var response = await http.PostAsJsonAsync("https://api.example.com/v1/chat/completions", new
    {
        model = "the-biggest-model",                                                                      // chosen once, for every request
        messages = new[] { new { role = "user", content = $"Write a nice description for: {product.Name}. Notes from the supplier: {product.SupplierNotes}" } }
    });
 
    var json = await response.Content.ReadAsStringAsync();                                               // no status check, no timeout chosen, no retry
    var text = JsonDocument.Parse(json).RootElement
        .GetProperty("choices")[0].GetProperty("message").GetProperty("content").GetString();
 
    return Results.Ok(new { description = text });                                                       // published as it is, nobody reads it first
});

It works, today. The result?

• The vendor leaks into the endpoint: its URL, its JSON, its model names, its key.
• The prompt is a string inside the endpoint, and the supplier’s notes are pasted into it without a thought. Whoever writes those notes can write instructions to the model.
• When the provider is slow or fails, nobody has decided what the customer sees.
• To test it, you either call the real service, and pay for it, or fake the vendor’s HTTP and JSON by hand.

And you have two more features to go, each one with its own copy of this code.

This is exactly the kind of problem AI architecture was born to solve: treat the model as one more dependency, with the same discipline we give to a database or a payment provider, and a few extra rules for the ways it surprises us.

This article is the map of a new series. It follows “Modern Software Architecture”, where we rebuilt the same order endpoint in many architectures without a single language model in sight. (If you want to find out more about that series, click here.) Now we add the new dependency, and ask where it belongs.

A Different Kind of Dependency: Six ways a model surprises you

A database, a payment gateway and a language model are all external services. But the model breaks some habits we take for granted:

• Latency: usually slower and less predictable than the other calls of the same request.
• Cost per use: with most hosted models it grows with the size of the prompt and of the answer, so “just call it again” is a decision.
• Non-determinism: the same input can give different outputs, so a test asserting an exact sentence will fail one day for no reason.
• Silent failure: besides exceptions and timeouts, a model can answer fluently, confidently and wrongly. No status code tells you.
• Data leaving the building: the prompt may carry customer data outside your boundary.
• Change without you: the provider can update or retire a model, and the behavior you tested moves.

None of these is a prompt problem. They are architecture problems: where the call lives, who may make it, what happens around it.

The Origins: From a hidden debt to a middleware pipeline

Software with a learned model inside is older than today’s chat assistants, and so is the lesson.

• 2015, the debt: D. Sculley and nine colleagues at Google presented “Hidden Technical Debt in Machine Learning Systems” at NIPS. Their message: a machine learning system is quick to build and expensive to maintain, and many of the risks are in the system design, with names like boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies and configuration issues. Almost every item of that list will come back in this series.
• End of 2022, the wave: chat assistants built on large language models reached the general public, and teams rushed to call the same models from their own applications. The first integrations looked like the code above.
• March 2023, the SDKs: Microsoft published Semantic Kernel on GitHub, an open source SDK that brought language models into C# applications.
• October 8, 2024, the abstraction: the .NET team previewed Microsoft.Extensions.AI: shared interfaces for AI services, plus middleware for logging, caching and telemetry.
• May 21, 2025, the standard: Microsoft.Extensions.AI, its abstractions package and Microsoft.Extensions.VectorData.Abstractions reached general availability.

It was clear that something familiar was happening. ASP.NET Core had given HTTP a pipeline of middleware, each piece wrapping the next. Now AI calls got the same treatment: one interface, IChatClient, and a chain of components around it. (If you want to find out more about the Decorator pattern, click here.) The provider-specific part sits behind an Adapter, which is what the client libraries do. (If you want to find out more about the Adapter, click here.)

The Shape: Where the model sits

The rule that will guide the whole series is short: the use case asks for a result, and the pipeline decides how to get it. The use case doesn’t know which provider answers, which model, whether the answer came from a cache, or how many times the call was retried.

In a .NET solution this shows up as a few projects and a few arrows. The tree is illustrative, and the names are ours:

Shop.slnx
src/
  Shop.Core/             use cases: PlaceOrder, SummarizeReviews, ...         -> Microsoft.Extensions.AI.Abstractions
  Shop.Ai/               the pipeline: telemetry, guardrails, cache, routing  -> Microsoft.Extensions.AI
  Shop.Infrastructure/   EF Core, payment client, email                       -> Shop.Core
  Shop.Web/              minimal API, Program.cs                              -> all of the above
tests/
  Shop.Core.Tests/       the model is a mock                                  -> Shop.Core

What does the port look like? Should the core know IChatClient, or a smaller interface that says "describe this product"? That's the subject of the first article. For now, the pipeline is where the cross-cutting concerns live, and in .NET it takes a few lines in the composition root. (If you want to find out more about Dependency Injection, click here.)

// Composition root: the model is one more dependency, wired in one place
builder.Services.AddDistributedMemoryCache();
 
builder.Services.AddChatClient(_ =>
        new OpenAIClient(
                new ApiKeyCredential(builder.Configuration["Ai:ApiKey"]!),                              // from user secrets or a vault, never from the source
                new OpenAIClientOptions { Endpoint = new Uri(builder.Configuration["Ai:Endpoint"]!) })
            .GetChatClient(builder.Configuration["Ai:Model"]!)
            .AsIChatClient())
    .UseOpenTelemetry()
    .UseDistributedCache()
    .UseLogging();

Three details matter here. First, the endpoint, the model and the key come from configuration, not from the code: moving to another model or another compatible endpoint is a deployment decision, and replacing the client library is a change in this one place. Second, each Use... call wraps the client in a new layer, and the first one registered is the outermost: here the telemetry sees every request, even those answered by the cache, while the logging only sees the calls that reach the provider. (We checked the order with a test.) Third, writing your own layer is just a class that derives from DelegatingChatClient: this is how guardrails and tenant rules will be built (routing and failover get base classes of their own).

One caution, from the package itself: UseOpenTelemetry follows the OpenTelemetry conventions for generative AI, which its documentation still calls experimental. What you measure may change names, so we'll handle it with care.

The Use Case Stays Boring: Testing without a model

Now look at what the use case becomes when the model is a dependency like any other. For now it depends on IChatClient itself, from the small Microsoft.Extensions.AI.Abstractions package:

public sealed class SummarizeReviewsHandler(IChatClient chat)
{
    public async Task<string> HandleAsync(IReadOnlyList<Review> reviews, CancellationToken cancellationToken)
    {
        if (reviews.Count == 0)
            return string.Empty;                                    // nothing to summarize: no call, no cost, no risk
 
        List<ChatMessage> messages =
        [
            new(ChatRole.System, "Summarize the customer reviews in two sentences. Use only what the reviews say."),
            new(ChatRole.User, string.Join("\n", reviews.Select(r => $"{r.Rating}/5: {r.Text}")))
        ];
 
        var response = await chat.GetResponseAsync(
            messages,
            new ChatOptions { Temperature = 0.2f, MaxOutputTokens = 200 },
            cancellationToken);
 
        return response.Text;
    }
}

There is plenty of logic worth testing here, and none of it needs a real model. The model is an interface, so a mock stands in for it:

[Test]
public async Task HandleAsync_NoReviews_DoesNotCallTheModel()
{
    // Arrange
    var chat = new Mock<IChatClient>();
    var handler = new SummarizeReviewsHandler(chat.Object);
 
    // Act
    var summary = await handler.HandleAsync([], CancellationToken.None);
 
    // Assert
    Assert.That(summary, Is.Empty);
    chat.Verify(c => c.GetResponseAsync(
        It.IsAny<IEnumerable<ChatMessage>>(), It.IsAny<ChatOptions?>(), It.IsAny<CancellationToken>()), Times.Never);
}

The test runs in milliseconds, costs nothing and never flakes. It checks what we own: when the model is called, what we send, what we do with the answer. What the model says is another matter: a mock proves that your code is correct, not that the answers are good.

Part B: The patterns that belong to AI

The first nine topics each take one problem that appears as soon as a model enters the application, and give it an architectural answer. Three of them, routing, RAG and guardrails, need two articles each. They are neutral: Microsoft.Extensions.AI, OpenTelemetry, ASP.NET Core and open libraries, no product attached.

• Model Port (Nov 14): which interface does the application own? IChatClient is already shared by the ecosystem, but a use case wants "describe this product", not "messages in, text out".
• AI Gateway (Nov 16): one place where every call passes, with the keys, limits, logs and policies: inside the application or as a service in front of many.
• Model Routing and Fallback (Nov 18, 20): not every request needs the same model, and every provider fails sometimes. Microsoft.Extensions.AI has experimental pieces for routing and failover; we'll look at them and at a hand-written version.
• Semantic Cache (Nov 22): the exact cache is one line of the pipeline above. A semantic one uses embeddings (IEmbeddingGenerator), and the hard part is deciding when two questions are really the same.
• RAG as an architecture (Nov 24, 26): retrieval-augmented generation as components with owners: an ingestion pipeline, a store, and a retrieval step on the request path.
• Guardrails (Nov 28, 30): checks before and after the model. The output is untrusted input, and the application needs a plan for when a check fails.
• Observing AI calls (Dec 2): traces and metrics with OpenTelemetry, and what you must never write in a log.
• Prompts as versioned configuration (Dec 4): a prompt changes the behavior of the application as much as code does, so it deserves review, versions and rollback.
• Multi-tenant AI (Dec 6): isolating data, context, limits and cache, so that the answer prepared for one tenant never reaches another.

What do you think? Which of these nine problems hurts most in your application today, or is the one you’re hoping to avoid? Leave a comment with your opinion.

Part A: The same architectures, with AI inside

The second part goes back to the architectures of the first series, one at a time, and asks the question that each one makes interesting:

• Layered: which layer may call the model?
• MVC and Razor: where does a streamed answer live?
• MVVM: what changes when a view model calls the model, on a device?
• Hexagonal and Clean: is the chat an input port and the model an output adapter?
• Vertical Slice: one AI feature per slice, one shared pipeline.
• CQRS: what may a model read, and what may it change?
• Event-Driven: reacting to events with a call that is slow, costly and not repeatable.
• Microkernel: AI features as plugins.
• Modular Monolith: one AI module for everybody, or one per module?
• Microservices: where the gateway and the boundaries go.
• Serverless: functions, long answers and timeouts.
• A reference architecture that puts the pieces of both parts together.

Each one starts where the classic version of that architecture leaves off. Part A starts on November 23 and runs on the days in between.

So What?: When AI belongs in the architecture (and when not)

A model deserves its own place in the architecture because its calls are costly, slow, uncertain or sensitive. That doesn’t mean every problem needs one. A rule you can write down should stay a rule:

// You do NOT need a model for this: the rule is known, so write the rule
public static decimal ApplyCoupon(decimal total, string? couponCode) =>
    couponCode == "BENVENUTO10" ? total * 0.9m : total;

Asking a model whether a cart qualifies for a coupon would be slower, costlier, and sometimes wrong. The model is for work with no exact rule: writing, summarizing, classifying, answering in natural language. Three more criteria:

• Consequence of a mistake: a wrong summary of a review is an annoyance; a wrong refund is a problem. The more it costs to be wrong, the more checks, and the more a human in the loop.
• Number of features: with one AI feature, a single well-written client may be enough. With three, the duplication starts to hurt, and the pipeline pays for itself.
• Data and rules: if the prompt carries personal data, or customers expect isolation, the gateway and the tenant rules stop being optional.

Modern Best Practices (2026): Five ideas that return in every article

Whatever the pattern or the architecture, the same ideas keep coming back:

1. Own the interface, rent the model. Your code depends on a port you control; the provider is a detail.
2. One place for the rules of every call. Limits, logs, policies and retries live in the pipeline, not in forty endpoints.
3. Assume the answer can be wrong. Check it, bound it, and design what happens when it is.
4. Test what you own. Mock the model, and evaluate the quality of the answers as a separate activity.
5. Measure before you tune. Without traces and metrics, “it feels slow” is an opinion.

What’s Next

AI in software architecture is not a new religion with new commandments. It is the old craft of drawing boundaries around something unpredictable, applied to a component that happens to speak our language. The patterns of part B are the first boundaries, and the architectures of part A show where they fit.

The next article starts with the interface itself: the Model Port. Because before you decide how to call a model, you need to decide what your application is allowed to know about it.

The key to success is not adding AI everywhere, but knowing exactly where it belongs and what must surround it.

Thanks for reading.

If you found this article helpful, follow me for the next ones or just clap.

Back to article Software Architecture in .NET C#