Setting up the test foundations using Testcontainers and transactions for AI-enabled applications with C# and .NET

Summary

• An effective testing strategy is part of the scaffolding for a foundational codebase that allows agents to operate with guardrails and checks.
• Testcontainers and automatic test-scoped transactions allow agents to write and execute database level integration tests in a stateless manner with isolation and parallelism.
• To help short circuit per-run agent discovery of how to operate the test harness, we interactively produce a skill with the agent on how to use write and operate the test infrastructure.
• Part 1 will introduce two under-the-radar components of the .NET stack that make it surprisingly amenable to building software with agents: Aspire and CSharpRepl.
• Part 2 will dive into a hands-on implementation from the ground up to scaffold an open-source template for teams to build on top of.
• Part 3 will implement the next layer of the application including a simple streaming interface to the Copilot SDK agent.
• Part 4 (👈 you are here) will extend the application with Testcontainers to demonstrate how to simplify test execution for agents with stateless containers.
• Part 5 will configure the application with logging, telemetry, and observability before diving into building the actual application using your coding agent.

The full repo: https://github.com/zeeq-ai/zeeq-tmpl/tree/feat-part4-Part-4-code-changes (note the branch; see the PR for the diff)

What we’re building

The previous part of this series walked through the construction of an AI enabled application that connects a web-based chat frontend to an instance of the Copilot Agent via the SDK that allows us to work with the codebase from any web API. This can be a foundation for building your own collaborative agent meta-harness on top of a model agnostic agent harness.

The exercises have already walked through examples of how to leverage Aspire and CSharpRepl, but the focus for this series is still on understanding the mechanical scaffolding that a platform team has to undertake to build a solid foundation for an agentic engineering team to accelerate. The posts intentionally focus on how to build each piece and how the pieces are connected together as these are the core components of a platform.

There are two major pieces left to scaffold before we let loose with agents. In this part, we’ll scaffold the testing infrastructure which will let the agents verify their code using a stateless containerized database and transaction rollback to ensure that the tests are isolated. In the next part, we’ll add logging, telemetry, and observability to the application so that we can see what the agents are doing in real time.

Scaffolding for storage

For local use, SQLite or even the local file system are suitable for storing the user specifications, but to demonstrate how to use Postgres and Testcontainers, we’ll add a Postgres database to the application.

Adding database access to the application

First is to add the packages necessary to connect to Postgres and set up the connection string

cd src/server
dotnet add package EFCore.NamingConventions
dotnet add package Microsoft.EntityFrameworkCore.Design
dotnet add package Microsoft.EntityFrameworkCore.Relational
dotnet add package Npgsql.EntityFrameworkCore.PostgreSQL

The Entity Framework Core DbContext is first:

// src/server/Data/ZeeqContext.cs
using Microsoft.EntityFrameworkCore;

namespace Zeeq.Tmpl;

public class ZeeqContext(DbContextOptions<ZeeqContext> options) : DbContext(options)
{
    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        // 👇 Applying the entity configurations only from this assembly; modify as needed
        modelBuilder.ApplyConfigurationsFromAssembly(typeof(ZeeqContext).Assembly);
    }
}

And add this into the DI container at setup:

// src/server/Program.cs

// 👇 This will be injected by Aspire when we connect the resources
var connectionString = Environment.GetEnvironmentVariable("ConnectionStrings__zeeq");

// Set up the database context
builder.Services.AddDbContext<ZeeqContext>(options =>
    options
        .UseNpgsql(connectionString)
        .EnableDetailedErrors(true)
        .EnableSensitiveDataLogging(true)
        .UseSnakeCaseNamingConvention()
);

Setting up the model

Now we’ll set up the model for storing the specifications:

// src/server/Data/Models.cs
using System.ComponentModel.DataAnnotations;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Metadata.Builders;

namespace Zeeq.Tmpl;

/// <summary>
/// A pure domain model class for storing the specification
/// </summary>
public class Specification
{
    public Guid Id { get; set; }

    [MaxLength(255)] // 👈 Note the validators here
    public string Name { get; set; } = string.Empty;

    [MaxLength(10000)]
    public string Content { get; set; } = string.Empty;
    public long TokenCount { get; set; }
    public DateTime CreatedAtUtc { get; set; } = DateTime.UtcNow;
    public DateTime? UpdatedAtUtc { get; set; }
}

/// <summary>
/// The database storage configuration for the model.  This is separated from the
/// pure domain model and represents the storage mapping for the ORM.  When scaling
/// the codebase, this can be placed with a dedicated DB project instead of in the
/// same codebase
/// </summary>
public class SpecificationConfiguration : IEntityTypeConfiguration<Specification>
{
    public void Configure(EntityTypeBuilder<Specification> builder)
    {
        builder.HasKey(s => s.Id);
        builder.Property(s => s.Name).IsRequired();
        builder.Property(s => s.Content).IsRequired();
        builder.Property(s => s.TokenCount).IsRequired();
        builder.Property(s => s.CreatedAtUtc).IsRequired();
        builder.Property(s => s.UpdatedAtUtc);
    }
}

Setting up Postgres in Aspire

To add Postgres into the stack, we’ll add a new resource using the Aspire CLI:

# From root to add the Postgresql package to /host
aspire add postgresql

After adding this, we can add the resource to the stack:

// host/AppHost.cs

// Add the resource first
var username = builder.AddParameter("username", "zeeq", secret: true);
var password = builder.AddParameter("password", "P@ssw0rd", secret: true);
var postgres = builder
    .AddPostgres("postgres", userName: username, password: password)
    .AddDatabase("zeeq")

// Wire it to the backend resource so it gets the connection string
var backend = builder
    .AddProject<Projects.server>("app-backend")
    // 👇 Wire up CSharpRepl environment variables to the runtime.
    .WithEnvironment("DOTNET_STARTUP_HOOKS", csrHook)
    .WithEnvironment("ASPNETCORE_HOSTINGSTARTUPASSEMBLIES", "CSharpRepl.InjectedHook")
    // 👇 Connect the postgres instance
    .WithReference(postgres);

After we wire up Postgres to the backend, Aspire automatically injects the connection string into the backend as an environment variable.

Create the database migration

Before moving on, we’ll create the database migration that will be needed by the test fixture in the next section.

cd src/server
dotnet ef migrations add Zeeq_Planner_Init

Our database scaffolding is ready; next is to scaffold the test infrastructure that we need to allow agents to run integration tests in an isolated manner using Testcontainers.

Adding Testcontainers for integration testing

For testing the application, we’ll add a new test project with Testcontainers and TUnit.

Adding the Testcontainer and TUnit packages

We’ll add the packages and add a smoke test to ensure that everything is working and wired up:

cd src
mkdir tests
cd tests
dotnet new console
dotnet add package TUnit
dotnet add package Testcontainers.PostgreSql
dotnet add reference ../server
mv Program.cs SmokeTest.cs

cd ../../
dotnet sln add src/tests

In global.json, we’ll need to instruct the runtime that we’re using Microsoft Testing Platform (MTP) mode:

// global.json
{
  "sdk": {
    "version": "10.0.302",
    "rollForward": "latestFeature"
  },
  "test": {
    "runner": "Microsoft.Testing.Platform"
  }
}

Add a smoke test:

// src/tests/SmokeTest.cs
public class SmokeTest
{
    [Test]
    public async Task TestSmoke()
    {
        await Assert.That(true).IsTrue();
    }
}

Now we can run the test:

dotnet test

Test run summary: Passed!
  total: 1
  failed: 0
  succeeded: 1
  skipped: 0
  duration: 717ms

Wiring up a database fixture and transaction rollback

Before we let the coding agent loose to build the backend, we’re going to manually scaffold the Testcontainer test fixture and also set up the transaction rollback so that integration tests remain isolated.

First up is setting up the Testcontainer:

// src/tests/PgDatabaseFixture.cs
using DotNet.Testcontainers.Builders;
using Microsoft.EntityFrameworkCore;
using Testcontainers.PostgreSql;
using TUnit.Core.Interfaces;
using Zeeq.Tmpl;

/// <summary>
/// TUnit fixture for setting up the Postgres Testcontainer database
/// </summary>
public class PgDatabaseFixture : IAsyncInitializer, IAsyncDisposable
{
    /// <summary>
    /// The container instance for this fixture.
    /// </summary>
    private PostgreSqlContainer? _container;
    private DbContextOptions<ZeeqContext> _options = null!;

    private DbContextOptions<ZeeqContext> EnsureOptions()
    {
        if (_container is null)
        {
            throw new InvalidOperationException("Postgres container is not initialized.");
        }

        _options ??= new DbContextOptionsBuilder<ZeeqContext>()
            .UseNpgsql(_container.GetConnectionString())
            .EnableDetailedErrors(true)
            .EnableSensitiveDataLogging(true)
            .UseSnakeCaseNamingConvention()
            .Options;

        return _options;
    }

    public ZeeqContext CreateContext() => new(EnsureOptions());

    public async Task InitializeAsync()
    {
        // 👇 Initialize the Postgres container
        _container = new PostgreSqlBuilder("postgres:18")
            .WithPortBinding(5432, true)
            .WithPassword("password")
            .WithUsername("username")
            .WithDatabase("zeeq")
            .WithWaitStrategy(
                Wait.ForUnixContainer()
                    .UntilMessageIsLogged("database system is ready to accept connections")
                    .UntilInternalTcpPortIsAvailable(5432)
            )
            .WithAutoRemove(true)
            .Build();

        await _container.StartAsync();

        // 👇 Run the migrations to set up the database schema
        using var context = new ZeeqContext(EnsureOptions());

        await context.Database.EnsureCreatedAsync();
    }

    public async ValueTask DisposeAsync()
    {
        if (_container is not null)
        {
            await _container.DisposeAsync();
        }

        GC.SuppressFinalize(this);

        await ValueTask.CompletedTask;
    }
}

This will create one container per test project (only one in this case) and can help scale test execution throughput.

To wrap the tests with a transaction, we’ll use a base class that has two methods that wrap the execution of each test method:

// src/tests/PgTransactionalTestBase.cs
using Microsoft.EntityFrameworkCore.Storage;
using Zeeq.Tmpl;

/// <summary>
/// A base class that provides access to creating a context and transaction scoping
/// around the operation.
/// </summary>
public abstract class PgTransactionalTestBase(PgDatabaseFixture pg)
{
    private IDbContextTransaction? _transaction;

    // 👇 Each test instantiates a new class instance so we can use class-level state
    protected ZeeqContext _context = pg.CreateContext();

    [Before(Test)] // 👈 Start a transaction before the test
    public async Task Before()
    {
        _transaction = await _context.Database.BeginTransactionAsync();
    }

    [After(Test)] // 👈 Roll it back so nothing is committed to state
    public async Task Cleanup()
    {
        if (_transaction != null)
        {
            await _transaction.RollbackAsync();
            await _transaction.DisposeAsync();
        }

        await _context.DisposeAsync();
    }
}

Now our test classes inherit from this base class and every test is wrapped in a transaction that is rolled back after the test completes.

A basic CRUD test to make sure everything is working:

// src/tests/SpecificationTests.cs
using Microsoft.EntityFrameworkCore;
using Zeeq.Tmpl;

// 👇 Wire the fixture and forward to the base class
[ClassDataSource<PgDatabaseFixture>(Shared = SharedType.PerTestSession)]
public class SpecificationTests(PgDatabaseFixture pg) : PgTransactionalTestBase(pg)
{
    [Test]
    public async Task Specification_Basic_WriteRead()
    {
        _context.Set<Specification>().Add(new() { Name = "Test Spec" });

        await _context.SaveChangesAsync();

        _context.ChangeTracker.Clear(); // 👈 Clear EF change tracking and reload from DB

        var spec = await _context.Set<Specification>().FirstOrDefaultAsync();

        await Assert.That(spec).IsNotNull();
        await Assert.That(spec!.Name).IsEqualTo("Test Spec");
        await Assert.That(spec.Id).IsNotEqualTo(Guid.Empty); // 👈 Set on write
    }
}

Now if we run dotnet test:

Test run summary: Passed!
  total: 2
  failed: 0
  succeeded: 2
  skipped: 0
  duration: 3s 198ms

Looks good!

Training the agent to use the test harness

By default, the agent will try to use XUnit commands because TUnit is a relatively newer framework with less representation in the model training. Additionally, the Microsoft Testing Platform (MTP) is also a relatively new test runner that is not as widely represented in the model training.

To help the agent understand how to run the tests, we’ll go through the same exercise as before with CSharpRepl and guide the agent through some experiments with some starting points for it to read so it can materialize a skill for testing.

<!-- Give it some basics about TUnit and specifically mention MTP -->
We are working on writing a skill in .agents/skills/unit-integration-testing/SKILL.md for .NET and C# code
Our goal is to keep this skill short, terse, to the point with key examples broken down into sections.
The test project is in src/tests using TUnit (NOT XUnit) and Microsoft Testing Platform
See: https://tunit.dev/docs/execution/test-filters
See: https://tunit.dev/docs/assertions/getting-started
See: https://tunit.dev/docs/migration/xunit
See: https://tunit.dev/docs/reference/command-line-flags
Write a quick introductory section and then a table of the key basic commands.
When done, I'll prompt the next experiment

<!-- Document key assertion patterns -->
Document the key assertion patterns supported by TUnit
See: https://tunit.dev/docs/assertions/exceptions
See: https://tunit.dev/docs/assertions/combining-assertions
See: https://tunit.dev/docs/assertions/equality-and-comparison
When done, I'll prompt the next experiment

<!-- Specifically call out the PostgresDbFixture and transactional behavior -->
See: PgDatabaseFixture.cs, PgTransactionalTestBase.cs to understand how the test database is set up
Document a bullet list of notes for the test scaffolding that is relevant to understanding how to write integration tests
Capture information about test isolation behavior for integration tests that touch the database
When done, I'll prompt the next experiment

<!-- Now have it run a filtered test (if it did not already produce this) -->
Use treenode filters to select a single test file to run
Use treenode filters to select a single test method to run
Document your findings and fill any gaps in the document
When done, I'll prompt the next experiment

<!-- Add instructions for how to write good tests -->
Document best practices for writing good tests:
- High signal, low noise
- Focus on invariants
- Use class-level setup for shared resources; each test runs as class instance
- Use three part naming convention for test methods: EntityOrScopeName_StateUnderTest_ExpectedBehavior

We’ll add a short note in AGENTS.md for both the CSharpRepl skill as well as the test skill.

<!-- AGENTS.md -->
## Skills

- Use the skill `unit-integration-testing` when writing and running .NET C# tests
- Use the skill `csharprepl` to directly manipulate the running C# application, wrap methods, replace methods, and inspect the runtime state of the application

This way the agent knows exactly when to use these two skills.

Closing thoughts

This stage of the exercise continues to focus on the key choices that human operators need to make when scaffolding the foundational codebase that agents will code on top of. Because these decisions will ultimately affect correctness, iteration speed, and operational runtime correctness, each one has been made intentionally to understand the guardrails that are in place to help agents work effectively.

Testcontainers and transaction rollback are a key part of this by providing parallelism for large projects as well as test isolation. Using Aspire and EF Core, it’s easy to wire together a fully type-safe data layer with a domain model isolated from the underlying storage mapping.

In the last and final part, we’ll wire up logging, telemetry, and observability to the application and build the final specification writing surface that will allow the agent to give continuous feedback on the specification as it runs research on the underlying code as the specification is being written.\

If you are curious to see a real-world setup, check out the Zeeq.ai repo: https://github.com/zeeq-ai/zeeq-app

The Unexpected AI Stack: C# + .NET (Part 4) was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Summary

• The application makes several key architectural decisions to make it easier for agents to work with and iterate like using OpenAPI (stable, easy for agents to understand and work with, contract-based) with minimal customization.
• This is wired using standard .NET tooling and Kubb.dev on the frontend to generate the TypeScript API clients
• Using the Copilot SDK allows us to build a powerful agentic application with minimal code (just a few lines of setup). Using .NET’s built in System.Threading.Channels allows the application to queue inputs and stream outputs to the frontend — fully typed end-to-end.
• To build the skill for CSharpRepl, use your coding agent and “teach” it how to use it correctly rather than re-discovering this each time. YMMV based on your model, your harness, and the complexity of your application DI setup so it is better to use this approach than to provide a canned example.

What we’re building

This exercise will build a simple chat application connected to an instance of a Copilot agent via the Copilot SDK.

The Copilot SDK is a good choice for building agentic applications because:

• The SDK supports multiple language bindings including TypeScript, Python, Go, C#, Rust, and Java
• The harness is model agnostic and allows using different models for different tasks and BYOK or use the GitHub Copilot subscription (we’ll be using BYOK for this exercise)
• The SDK is well documented and comes with advanced features for working with and navigating source code.

For the front-end, we’ll build a chat interface using a Vite + Nuxt UI web app using a streaming API. We’ll use Kubb.dev for client generation from OpenAPI specs.

Wiring OpenAPI

Before continuing on and letting the agents loose, we still have a bit of scaffolding to do (this can be moved into a skill, but the purpose of this exercise is to understand the underlying wiring before building a skill template).

Update the .csproj to produce the OpenAPI schema

We’ll need to add a package to the src/server/server.csproj to emit the OpenAPI spec:

cd src/server
dotnet add package Microsoft.Extensions.ApiDescription.Server

A few tweaks are also needed to the .csproj

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <RootNamespace>Zeeq.Tmpl</RootNamespace>
    <UserSecretsId>db43be79-6f94-4d75-9ee0-45d496cfae4d</UserSecretsId>
    <!-- 👇 OpenAPI config -->
    <OpenApiDocumentsDirectory>..\app\src\api</OpenApiDocumentsDirectory>
    <OpenApiGenerateDocumentsOptions>--file-name zeeq-tmpl-api</OpenApiGenerateDocumentsOptions>
  </PropertyGroup>
  <!-- 👇 Don't generate in the Release build (should already be present) -->
  <PropertyGroup Condition="'$(Configuration)' == 'Release'">
    <!-- Suppress OpenAPI document generation in Release; the target checks this property -->
    <OpenApiGenerateDocuments>false</OpenApiGenerateDocuments>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.OpenApi" Version="10.0.11" />
    <!-- 👇 Added via the `dotnet add package ...` -->
    <PackageReference Include="Microsoft.Extensions.ApiDescription.Server" Version="10.0.11">
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>
    <PackageReference Include="Scrutor" Version="7.0.0" />
  </ItemGroup>
</Project>

Now running dotnet build src/server will produce the OpenAPI schema in src/app/src/api/zeeq-tmpl-api.json.

Wire up Kubb.dev to generate the API client from the schema

Kubb is a TypeScript code generator that uses a plugin architecture to produce client bindings, fakes, validators, and more using the OpenAPI schema.

We’ll use this to generate the API client for the frontend.

cd src/app

# Note: this is using 5.0.0-beta.103
pnpm add -D kubb @kubb/plugin-ts @kubb/plugin-fetch

Now to configure it:

// src/app/kubb.config.ts
import { defineConfig } from "kubb";
import { pluginTs } from "@kubb/plugin-ts";
import { pluginFetch } from "@kubb/plugin-fetch";

export default defineConfig({
  input: "./src/api/zeeq-tmpl-api.json",
  output: {
    path: "./src/api/generated",
    clean: true,
  },
  plugins: [
    pluginTs({
      output: {
        path: "./types",
        mode: "directory"
      },
      enum: {
        type: "asConst"
      },
      optionalType: "questionTokenAndUndefined",
    }),
    pluginFetch({
      baseURL: "http://localhost:5138", // 👈 From the launch profile for the backend
      output: {
        path: "./clients",
        mode: "directory",
      },
    }),
  ],
});

The package.json will need two scripts:

// src/app/package.json
{
  "scripts": {
    "generate": "kubb generate",
    "generate:watch": "kubb generate --watch"
  }
}

Now if we run:

# Manually generate to smoke test the wiring
cd src/app
pnpm generate

This will produce an updated API client in src/app/src/api/generated (recommend adding to your .gitignore).

Automating with Aspire

To automate this generation when the schema updates on build, add another resource to Aspire that watches the OpenAPI schema file and runs the Kubb generation command.

// host/AppHost.cs
var kubb = builder
    .AddExecutable(
        name: "kubb-generate-watch",
        command: "pnpm",
        workingDirectory: "../src/app",
        args: ["generate:watch"]
    )
    .WaitFor(frontend); // 👈 Wait for this because it installs dependencies

Scaffold the application flow

The application will flow like this:

1. The browser application initiates a connection to a streaming API endpoint to read responses from the agent
2. When the user sends a prompt from the browser, it goes into a web API endpoint and then into a System.Threading.Channels.Channel that queues the prompt for the agent; the user can keep sending prompts and they will be queued in the channel
3. The agent is an instance of the Copilot harness that is running in a background service connect to the inbound channel to receive the prompt and an outbound channel to stream the responses

First, let’s add the .NET SDK for GitHub Copilot:

cd src/server
dotnet add package GitHub.Copilot.SDK

This SDK will allow our backend to operate an instance of the Copilot harness and directly work with any codebase!

With only a few lines of code, we’re going to build the foundations of an agentic application.

(As an alternative, you can also use the Agent Framework AgentHarness if you need more control)

Add AgentServiceWorker to run the agent in the background

The AgentServiceWorker is a background service that will run the agent and handle requests from the API.

The service initializes the Copilot SDK client and session, and then listens for incoming messages on a channel. When a message is received, it sends the message to the agent and writes the response to an outbound channel.

// src/server/AgentServiceWorker.cs
using System.Threading.Channels;
using GitHub.Copilot;
using Microsoft.Extensions.Options;

namespace Zeeq.Tmpl;

public class AgentServiceWorker(
    IOptions<AppSettings> options,
    [FromKeyedServices("inbound")] Channel<string> inboundChannel,
    [FromKeyedServices("outbound")] Channel<string> outboundChannel
) : BackgroundService
{
    // 👇 Change this to some directory on your local disk
    private const string WorkingDirectory = "/Users/cchen/code/zeeq/zeeq-app";
    private CopilotClient? _client;
    private CopilotSession? _session;

    /// <summary>
    /// 1️⃣  Set up the Copilot client and session with BYOK
    /// </summary>
    public override async Task StartAsync(CancellationToken cancellationToken)
    {
        // See: https://github.com/github/copilot-sdk/blob/main/docs/auth/byok.md
        // See: https://github.com/github/awesome-copilot/blob/main/cookbook/copilot-sdk/dotnet/recipe/managing-local-files.cs
        _client = new(new() { WorkingDirectory = WorkingDirectory });
        // 👇 It's easy to create a session with different models and different capabilities
        _session = await _client.CreateSessionAsync(
            new()
            {
                Model = "gpt-5.6-luna",
                OnPermissionRequest = PermissionHandler.ApproveAll,
                Provider = new()
                {
                    Type = "azure",
                    // 👇 This is an Azure OpenAI endpoint; use your own
                    BaseUrl = "https://zeeq-open-ai.openai.azure.com",
                    WireApi = "responses",
                    // 👇 Key set in previous step via `dotnet user-secrets set...` (or hardcode here)
                    ApiKey = options.Value.LlmApiKey,
                },
            },
            cancellationToken
        );

        await base.StartAsync(cancellationToken);
    }

    /// <summary>
    /// 2️⃣  Start the loop that pulls the incoming prompts off of the channel and executes
    /// </summary>
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        if (_session is null)
        {
            throw new InvalidOperationException("Copilot session is not initialized.");
        }

        if (_client is null)
        {
            throw new InvalidOperationException("Copilot client is not initialized.");
        }

        // Start the streaming output that writes to the outbound channel
        _session.On<SessionEvent>(evt =>
        {
            if (evt is AssistantMessageEvent messageEvent)
            {
                // 👇 Streams messages out to the SSE endpoint via the channel
                outboundChannel.Writer.TryWrite(messageEvent.Data.Content);
            }
        });

        // Start the message loop here
        while (await inboundChannel.Reader.WaitToReadAsync(stoppingToken))
        {
            while (inboundChannel.Reader.TryRead(out var message))
            {
                // 👇 Reads messages from the inbound channel and forwards to the agent
                await _session.SendAsync(message, cancellationToken: stoppingToken);
            }
        }
    }

    /// <summary>
    /// 3️⃣  Clean up resources when the service is requested to stop.
    /// </summary>
    public override async Task StopAsync(CancellationToken cancellationToken)
    {
        if (_client is not null)
        {
            await _client.DisposeAsync();
        }

        if (_session is not null)
        {
            await _session.DisposeAsync();
        }

        await base.StopAsync(cancellationToken);
    }
}

We’re intentionally keeping it simple here, but keep in mind that this is the full SDK for the GitHub Copilot harness so you can really go wild and build some interesting agentic applications on top of this.

The Copilot SDK supports multiple programming languages (TypeScript/Node, Go, Rust, Java, C#) so it is a solid platform to build agentic applications on top of. It is also likely to have good longevity and stability since it is the same harness that powers GitHub Copilot.

See more docs here:

• GitHub Copilot SDK
• BYOK
• Multiple sessions
• Managing local files

Microsoft Agent Framework Agent Harness is a lower-level primitive that may be more suitable for some applications, but requires a bit more work to wire up (exercise left to the reader).

Wire it via DI with channels

In Program.cs, we can wire up the channels and the AgentServiceWorker:

// src/server/Program.cs
// Add the channels; keyed for inbound and outbound.
builder
    .Services.AddKeyedSingleton(
        "inbound",
        (sp, key) => Channel.CreateUnbounded<string>()
    )
    .AddKeyedSingleton(
        "outbound",
        (sp, key) => Channel.CreateUnbounded<string>()
    )
    .AddHostedService<AgentServiceWorker>();

This is a single-user, local app so using a singleton for the channels makes sense. In a multi-user, stateless, multi-server application, the design would have to be more nuanced and back the endpoints with an actual messaging layer.

While we’re here, let’s also add the CORS configuration since our frontend and backend are on different ports:

// src/server/Program.cs BEFORE the app has been built
builder.Services.AddCors(options =>
    options.AddDefaultPolicy(policy =>
        policy
            .AllowAnyOrigin()
            .AllowAnyHeader()
            .AllowAnyMethod()
        )
);

// src/server/Program.cs AFTER the app has been built
app.UseCors();

Expose via API endpoints

Now we need to wire both the inbound and outbound API endpoints:

// src/server/Endpoints/Agent.cs
using System.Net.ServerSentEvents;
using System.Runtime.CompilerServices;
using System.Threading.Channels;

namespace Zeeq.Tmpl;

// Inbound endpoint mapping...
public class AgentEndpoint : IEndpoint
{
    public void MapEndpoints(IEndpointRouteBuilder endpoints)
    {
        endpoints.MapPost(
            "/send-prompt",
            (AgentHandler handler, string prompt) => handler.Handle(prompt)
        );
    }
}

// ... and the handler
public class AgentHandler(
    [FromKeyedServices("inbound")] Channel<string> inboundChannel
) : IEndpointHandler
{
    public bool Handle(string prompt) => inboundChannel.Writer.TryWrite(prompt);
}

// Outbound endpoint mapping...
public class AgentResponseEndpoint : IEndpoint
{
    public void MapEndpoints(IEndpointRouteBuilder endpoints)
    {
        endpoints.MapGet(
            "/read-response",
            async (AgentResponseHandler handler, CancellationToken cancellation) =>
                TypedResults.ServerSentEvents(handler.HandleAsync(cancellation))
                // 👆 The response type is SSE; this translates at the OpenAPI spec!
        );
    }
}

// ...and the streaming handler
public class AgentResponseHandler(
    [FromKeyedServices("outbound")] Channel<string> outboundChannel
) : IEndpointHandler
{
    // 👇 Note the typed `SseItem<T>`; in C# 15, this is even nicer with unions
    public async IAsyncEnumerable<SseItem<string>> HandleAsync(
        [EnumeratorCancellation] CancellationToken cancellation
    )
    {
        while (await outboundChannel.Reader.WaitToReadAsync(cancellation))
        {
            while (outboundChannel.Reader.TryRead(out var response))
            {
                yield return new SseItem<string>(response);
            }
        }
    }
}

Here, you might wonder why the separation of the HTTP endpoint from the handler since the code is quite simple.

This is an intentional pattern because the handler is injected into the DI container. This means that the application can attach authentication and input validation (including deserialization) at the app endpoint, but the agent can use CSharpRepl to directly manipulate the handler without going through the HTTP layer.

Note how nicely everything is typed. When the client is generated via Kubb, the type information flows through to the frontend as well.

In the upcoming C# 15 release, this story gets even better with union types which map cleanly at the boundary and translate directly into TypeScript union types.

Building a simple frontend

The frontend in its current state is cloned from the Nuxt UI template.

Some adjustments are needed to build a simple chat interface to the agent.

Add packages for Markdown and code rendering

We’ll need two packages to help with the display of Markdown and code snippets

cd src/app
pnpm add @comark/vue shiki

A small update to the vite.config.ts is needed to add the @comark/vue plugin:

// src/app/vite.config.ts
export default defineConfig({
  plugins: [
    vueRouter({
      dts: 'src/route-map.d.ts'
    }),
    vue(),
    ui({
      prose: true, // 👈 Need to add this (see the Nuxt UI docs)
      ui: {
        colors: {
          primary: 'green',
          neutral: 'zinc'
        }
      }
    })
  ]
})

You can learn more about Nuxt UI Markdown rendering and typography in the docs.

Implement the chat agent composable

Building the backend produces an updated OpenAPI schema which the Kubb watcher should pick up and create new stubs.

We can use these stubs in a Vue composable to implement the interface to the backend API.

import { ref } from 'vue'
// 👇 Import the API clients from the Kubb generated output
import { postSendPrompt } from '../api/generated/clients/postSendPrompt'
import { getReadResponse } from '../api/generated/clients/getReadResponse'

export type ChatStatus = 'ready' | 'submitted' | 'streaming' | 'error'

export interface ChatTextPart {
  type: 'text'
  id: string
  text: string
}

export interface ChatMessage {
  id: string
  role: 'user' | 'assistant'
  parts: ChatTextPart[]
}

/** No explicit end-of-turn signal comes from the server, so a response is
 * considered finished once no new chunk has arrived for this long. */
const READY_DELAY_MS = 800

export function useAgentChat() {
  const messages = ref<ChatMessage[]>([])
  const status = ref<ChatStatus>('ready')

  let readyTimeout: ReturnType<typeof setTimeout> | undefined
  let assistantMessage: ChatMessage | undefined
  let responseStreamStarted = false

  function scheduleReady() {
    clearTimeout(readyTimeout)
    readyTimeout = setTimeout(() => {
      status.value = 'ready'
      assistantMessage = undefined
    }, READY_DELAY_MS)
  }

  async function consumeResponseStream() {
    const { stream } = await getReadResponse()

    for await (const event of stream) {
      const chunk = typeof event.data === 'string' ? event.data : ''
      if (!chunk) continue

      if (!assistantMessage) {
        assistantMessage = {
          id: crypto.randomUUID(),
          role: 'assistant',
          parts: [{ type: 'text', id: crypto.randomUUID(), text: '' }],
        }
        messages.value.push(assistantMessage)
      }

      assistantMessage.parts[0].text += chunk
      status.value = 'streaming'
      scheduleReady()
    }
  }

  async function sendPrompt(prompt: string) {
    const trimmed = prompt.trim()
    if (!trimmed || status.value === 'submitted' || status.value === 'streaming') return

    messages.value.push({
      id: crypto.randomUUID(),
      role: 'user',
      parts: [{ type: 'text', id: crypto.randomUUID(), text: trimmed }],
    })

    status.value = 'submitted'
    assistantMessage = undefined

    if (!responseStreamStarted) {
      responseStreamStarted = true
      consumeResponseStream().catch(() => {
        status.value = 'error'
      })
    }

    try {
      await postSendPrompt({ query: { prompt: trimmed } })
    } catch {
      status.value = 'error'
    }
  }

  return { messages, status, sendPrompt }
}

Use the composable in the UI

And now we update the UI to implement a simple chat interface once again using Nuxt UI components:

<template>
  <div class="flex h-full justify-center p-4">
    <UCard
      class="flex h-full w-full max-w-md flex-col"
      :ui="{ body: 'flex-1 min-h-0 overflow-y-auto', footer: 'shrink-0' }"
    >
      <UChatMessages
        :messages="messages"
        :status="status"
      >
        <template #content="{ message }">
          <template
            v-for="part in message.parts"
            :key="part.id"
          >
            <Markdown
              v-if="part.type === 'text'"
              :value="part.text"
              :plugins="[shiki()]"
              :streaming="message.role === 'assistant' && status === 'streaming'"
              unwrap
            />
          </template>
        </template>
      </UChatMessages>

      <template #footer>
        <UChatPrompt
          v-model="input"
          :status="status"
          @submit="onSubmit"
        >
          <UChatPromptSubmit :status="status" />
        </UChatPrompt>
      </template>
    </UCard>
  </div>
</template>

<!-- Medium doesn't allow script tag?? -->
<_script setup lang="ts">
import { ref } from 'vue'
import { Markdown } from '@comark/vue'
import shiki from '@comark/vue/plugins/shiki'
import { useAgentChat } from '../composables/useAgentChat'

const input = ref('')
const { messages, status, sendPrompt } = useAgentChat()

function on_Submit() {
  const prompt = input.value
  input.value = ''
  sendPrompt(prompt)
}
</_script>

Here, we can see Nuxt UI paying huge dividends by providing not only a very rich, comprehensive set of components, but also agent-friendly docs to build rapidly.

Nuxt UI includes a nice set of components for this, out of the box:

• UChatMessages
• UChatPrompt

With plenty more to build a fully-featured agent interface.

Adding a skill for CSharpRepl

Agents can figure out CSharpRepl by themselves since the CLI itself has good instructions, but it will help A LOT if we provide the agents examples of how to work with CSharpRepl. We can effectively capture the key scenarios that the agents need to know about by walking the agent through how we want it to use the CSharpRepl CLI.

Rather than provide the finished skill, I’ll share the prompts that you can use to get the agent to build the skill.

Create the skill directory and placeholder file

Different harnesses will acquire local skills in different ways (for now) so we will need to do some basic scaffolding of the skill for different harnesses.

# Add the .agents directory which most harnesses will use
mkdir .agents
mkdir .agents/skills
mkdir .agents/skills/csharprepl
touch .agents/skills/csharprepl/SKILL.md

# Add the .claude directory and symlink the skills
mkdir .claude
cd .claude
ln -s ../.agents/skills skills

# Codex should already be present; symlink the skills
cd .codex
ln -s ../.agents/skills skills

This should cover all mainstream harnesses.

Prompt the agent to write the key instructions for the skill

We’ll use the agent to build the skill directly by having the agent run the CLI.

Your results may vary by harness, model, and your prompts so here’s my guide to help the agent discover and build the key elements of the skill.

(Note: best to paste each prompt one-by-one to see the agent iterate on building the skill)

<!-- Prime the agent so it knows what we're doing -->
We're working on writing a skill in .agents/skills/csharprepl/SKILL.md for the CSharpRepl CLI.
CSharpRepl is a REPL that allows agents to execute C# code in a running .NET application.
Our goal is to keep this skill short, terse, to the point with key examples broken down into sections.
Each section should have one code block that demonstrates the different commands relevant to that section.
Use comments above the command in the fenced code block to explain the command and use case.
Keep writing dense and terse; to the point and instructional with notes for key operational details.
We'll do this iteratively and refactor this document as we learn.
Start by running `csharprepl connect -h` to learn the commands.
Write a quick introductory section and then a table of the key basic commands.
When done, I'll prompt the next experiment

<!-- Now let's ask it to get an instance of the health service and replace it -->
Read: https://fuqua.io/blog/2026/06/injecting-a-csharp-repl-into-a-running-net-process/
Read: https://github.com/waf/CSharpRepl/wiki/Connecting-to-a-Running-Process
Acquire an instance of the `HealthHandler` and use `#replace` to replace it with a different implementation.
<example>
printf '%s\n' \
  '#replace Zeeq.Tmpl.HealthHandler.Handle with (instance) => "Healthy (replaced!)"' \
  'var health = Get<Zeeq.Tmpl.HealthHandler>(); var msg = health.Handle(); msg' \
  | NO_COLOR=1 csharprepl connect 92260 --streamPipedInput
<example>
Use `aspire logs` to see the output and verify that the replace worked.
Document your findings and how to use #replace in the skill file.
When done, I'll prompt the next experiment

<!-- Now let's ask it to get an instance of the health service and wrap it -->
Read: https://fuqua.io/blog/2026/06/injecting-a-csharp-repl-into-a-running-net-process/
Read: https://github.com/waf/CSharpRepl/wiki/Connecting-to-a-Running-Process
Acquire an instance of the `HealthHandler` and use `#wrap` to wrap it with a console log.
Use `aspire logs` to see the output and verify that the wrap worked.
Document your findings and how to use #wrap in the skill file.
When done, I'll prompt the next experiment

<!-- We also want to prompt it to learn how to use .csx files for more complex ops-->
Now do both exercises again using `--eval-file` and write a `.csx` file to perform the same action.
Update and document any findings necessary in the file
When done, I'll prompt the next experiment

<!-- Tighten up the notes to make it more dense -->
Write notes more dense
Compact but no loss of clarity
Keep key details only
Terse notes and comments

<!-- Experiment with the `-u` flag to make code more terse-->
Test -u --using <namespace> and document
This makes scripts more terse (no namespace inline)
Update and document any findings necessary in the file

By providing the agent reference docs and a few examples with clear instructions for discovery, the agent is able to self-materialize the skill with the key information we need it to have.

Closing thoughts

At this point, we have a solid foundation for building an agentic application on top of Aspire using .NET, C#, the Copilot SDK, and a Vue + Nuxt UI frontend.

This set of exercises moves very intentionally so human operators can have a clear view of what’s happening in the code and we’ve made very specific design decisions on how to wire up our endpoints to make it more friendly for agents to work with.

The setup is now built for coding agents to work with the codebase and iterate on the application itself.

In the next part of this series (Part 4), we’ll wire up the the data model as well as Testcontainers for isolated, transactional integration testing to let agents move autonomously as they build and verify correctness. Then in the final part, we’ll wire up logging, telemetry, and observability before building the final prototype specification writing application that will use the Copilot SDK to write a technical specification as we type out modify the business requirements.

We’ll still move with purpose so that each technical decision is clear and explained in the context of building an agentic foundation for an application.

If you are curious to see a real-world setup, check out the Zeeq.ai repo: https://github.com/zeeq-ai/zeeq-app

The Unexpected AI Stack: C# + .NET (Part 3) was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Summary

• This exercise will set up the foundational scaffolding for a typical Vite frontend app connected to a standalone backend via OpenAPI. The objective is to focus on the DX and getting key pieces in place so that agents a team can operate on top of this codebase and iterate rapidly with agents.
• Mise is used to simplify the installation of required tooling and the runtime environment for development
• Aspire is used to orchestrate the runtime components of the application
• We’ll also wire up CSharpRepl and give it a test run, but there’s not much for it to do yet!

What we’re building

This exercise will build a chat frontend application connected to an instance of a Copilot agent via the Copilot SDK.

The Copilot SDK is a solid choice for building agentic applications because:

• The SDK supports multiple language bindings including TypeScript, Python, Go, C#, Rust, and Java
• The harness is model agnostic and allows using different models for different tasks and BYOK or use the GitHub Copilot subscription (we’ll be using BYOK for this exercise)
• The SDK is well documented and comes with advanced features for working with and navigating source code.

For the front-end, we’ll build a chat interface using a Vite + Nuxt UI web app using a streaming API. We’ll use Kubb.dev for client generation from OpenAPI specs.

Part 3 will focus more on the build out; here, we’re setting up the foundation first.

Scaffolding the runtime configuration

The runtime will require configuring Mise to install the tooling that we’ll need. This makes it easier for teams to onboard new developers by simplifying the installation process and ensuring developers have the right versions of tools installed with minimal friction.

Configuring Mise

Install Mise for your platform following the instructions

# Create the config directory and place the mise.toml file there
mkdir .config
cd .config
touch mise.toml

The config file needs to specify our dependencies. The easiest way to do this from the command line is to type mise use which has auto-complete for popular packages:

This will add entries into the mise.toml file like this:

[tools]
dotnet = "10.0.302"

To add tools:

# After the @, the versions will resolve from Nuget
mise use dotnet:Aspire.Cli@

Here’s the full file of the tools we’ll need:

[tools]
dotnet = "10.0.302"
"dotnet:Aspire.Cli" = "13.4.6"
"dotnet:CsharpRepl" = "0.9.2"
"dotnet:Csharpier" = "1.3.0"
"dotnet:dotnet-ef" = "10.0.11"
node = "26"
pnpm = "11.18.0"

(Update the versions as necessary)

When pulling the stack, run mise install to configure a local environment with the correct versions of the specified tools.

Wire up AGENTS.md and CLAUDE.md as a symlink

# At the root, create the AGENTS.md file and symlink it to CLAUDE.md
touch AGENTS.md
ln -s AGENTS.md CLAUDE.md

Here’s the simple AGENTS.md we’ll use to describe how the agent should work in this repo:

# .NET 10, C# 14 Web API with Nuxt UI and Vite Frontend

- `src/server`: .NET 10, C# 14; use modern language features (primary constructors, deconstruction, pattern matching, switch expressions, named tuple types, etc.)
- `src/app`: Vue 3, Nuxt UI, Vite, TypeScript, Tailwind CSS

## Tooling

### Nuxt UI Rules

- Prefer Nuxt UI components first before writing new components
- Use Nuxt UI **component slots** first when suitable
- Use Nuxt UI **component props** before writing custom layout or style
- <https://ui.nuxt.com/llms.txt> has dense listing of components available to check
- <https://ui.nuxt.com/llms-full.txt> (download and search)

### Icons, Text, Tailwind

- Use `lucide` for Nuxt UI (`i-lucide-*`)

### Aspire CLI

```shell
aspire -h

# Get resources if needed; but usually just use the rebuild command below
aspire ps

# Avoid restarting the full stack; just rebuild the server
aspire resource app-backend rebuild
```

### Playwright MCP

Use the Playwright MCP to run the app in a browser and test the UI.

For UI work, get access first thing **before** you start!

Set up the Nuxt UI and Playwright MCP

The Playwright MCP will give the agent access to operate on the UI to perform UI layer E2E testing.

# Nuxt UI
claude mcp add --scope project --transport http nuxt-ui-remote https://ui.nuxt.com/mcp

# Playwright
claude mcp add --scope project playwright -- npx -y @playwright/mcp@latest --extension

This creates the following .mcp.json

{
  "mcpServers": {
    "playwright": {
      "type": "stdio",
      "command": "npx",
      "args": [
        "-y",
        "@playwright/mcp@latest",
        "--extension"
      ],
      "env": {}
    },
    "nuxt-ui-remote": {
      "type": "http",
      "url": "https://ui.nuxt.com/mcp"
    }
  }
}

Codex/ChatGPT doesn’t (yet?) have a project local CLI command for MCP so we’ll manually create .codex/config.toml:

[mcp_servers.nuxt-ui-remote]
enabled = true
url = https://ui.nuxt.com/mcp"

[mcp_servers.playwright]
command = "npx"
args = ["@playwright/mcp@latest", "--extension"]

Install the Chrome extension for Playwright MCP. This extension is useful because it allows the agent to drive an existing browser session so you can manually log in and bypass the complexity of auth flows.

In practice, with CSharpRepl, Playwright becomes only necessary to address UI layout since the agent can directly interface with the application at the API interface layer for E2E integration testing (bypassing the need to perform auth or use slower UI interactions altogether). But applications that implement auth will still benefit from using Playwright with the extension when iterating on the UI.

Setting up the applications

Scaffold the backend minimal web API

For this application, we’re going to use a slightly tweaked minimal web API that I think makes it easier for agents to work with.

mkdir src
mkdir src/server
cd src/server
dotnet new webapi --no-https
dotnet user-secrets init
dotnet user-secrets set AppSettings:LlmApiKey "YOUR_API_KEY_HERE"
cd ../../
dotnet new sln              # Add solution file
dotnet sln add src/server   # Add the project to the solution
dotnet build                # First build

At this point, the server can already run, but it’s only serving the default weather forecast endpoint.

We’re going to add two helper interfaces: IEndpoint and IEndpointHandler to make it easier to parcel out the endpoint implementations into separate files. We’ll also add the configuration object as well.

// src/server/IEndpoint.cs
public interface IEndpoint
{
    void MapEndpoints(IEndpointRouteBuilder endpoints);
}

// src/server/IEndpointHandler.cs
public interface IEndpointHandler { }

// src/server/AppSettings.cs
public record AppSettings
{
    public string LlmApiKey { get; init; } =
        "dotnet user-secrets set AppSettings:LlmApiKey YOUR_API_KEY_HERE";
}

We’ll add the Scrutor package to make the wiring a bit more terse:

cd src/server
dotnet add package Scrutor

Then in src/server/Program.cs, replace with:

var builder = WebApplication.CreateBuilder(args);

// Add the settings into the DI container
builder
    .Services.AddOptions<AppSettings>()
    .Bind(builder.Configuration.GetSection(nameof(AppSettings)))
    .ValidateOnStart();

// Wire up endpoints into the DI container
builder
    .Services.AddOpenApi()
    .Scan(scan =>
        scan.FromApplicationDependencies()
            .AddClasses(classes => classes.AssignableTo<IEndpoint>())
            .AsImplementedInterfaces()
            .WithTransientLifetime()
            .AddClasses(classes => classes.AssignableTo<IEndpointHandler>())
            .AsSelf()
            .WithTransientLifetime()
    );

var app = builder.Build();

if (app.Environment.IsDevelopment())
{
    app.MapOpenApi();
}

// Connect the endpoints
var endpoints = app.Services.GetRequiredService<IEnumerable<IEndpoint>>();

foreach (var endpoint in endpoints)
{
    endpoint.MapEndpoints(app);
}

app.Run();

To test this, we’ll add a simple health check endpoint and handler.

// src/server/Health.cs
public class HealthEndpoint : IEndpoint
{
    public void MapEndpoints(IEndpointRouteBuilder endpoints)
    {
        // 👇 DI at the endpoint to separate the HTTP binding from the app code
        endpoints.MapGet("/health", (HealthHandler handler) => handler.Handle());
    }
}

public class HealthHandler : IEndpointHandler
{
    // 👇 This simple example looks like overkill, but scales well!
    public string Handle() => $"Healthy @ {DateTime.UtcNow}";
}

This design is intentional even though the health handler is simple enough to implement directly inline with the MapGet. By injecting handlers as a dependency, we’ll be able to directly manipulate it at runtime later using CSharpRepl. While this example is simple, a key ingredient to successful agentic engineering is consistency. So a goal here is to establish the pattern that the agent will follow when building the next set of handlers.

In practice, I generally create separate files for the endpoint and the handler so the agents follow the pattern.

An important note with minimal API endpoints is that it is possible to directly access the user as a DI injected parameter:

// src/server/SomeAuthenticatedEndpoint.cs
public class SomeAuthenticatedEndpoint : IEndpoint
{
    public void MapEndpoints(IEndpointRouteBuilder endpoints)
    {
        // 👇 DI at the endpoint to separate the HTTP binding from the app code
        endpoints.MapGet(
            "/some-authenticated-endpoint",
            (
                ClaimsPrincipal user, // 👈 DI will resolve and inject this
                SomeAuthenticatedHandler handler
            ) => handler.Handle(user) // 👈 Pass forward to the handler
        );
    }
}

This distinction is very useful since it means when CSharpRepl is connected, it can easily simulate different users by pulling out the injected handler and directly injecting a different ClaimsPrincipal to simulate different users and test the behavior of the handler without going through auth.

Now we can quickly run and curl this as a smoke test:

# Run the server
dotnet run

# Check the endpoint
curl http://localhost:5138 -v # Healthy @ 8/11/2026 8:46:50 PM

Scaffold the frontend Nuxt UI Vite app

The Nuxt UI Vite app setup mostly follows this doc: https://ui.nuxt.com/docs/getting-started/installation/vue

We’ll use the starter Vue + Vite template: https://github.com/nuxt-ui-templates/starter-vue

mkdir src/app
cd src/app
git clone https://github.com/nuxt-ui-templates/starter-vue.git .
# Remove the .git directory and .github directory since we don't need these
rm -rf .git .github

This gives us a basic starter template (see the template) so we can skip the manual wiring of Nuxt UI itself.

cd src/app
pnpm dev # App served at http://localhost:5173/

Wiring into Aspire

Now that we have both the backend and frontend scaffolded, we can wire them into Aspire so that one command orchestrates the stack (albeit a simple stack at the moment!).

Create the Aspire host

Because the application has .slnx file at the root, Aspire will create a project-based setup:

# From root
aspire init --language csharp

# Rename for convenience
mv zeeq-tmpl.AppHost host
mv host/zeeq-tmpl.AppHost.csproj host/host.csproj

# Add to the solution and add a reference to the server project
dotnet sln add host
dotnet add reference --project host src/server

# Add the JavaScript module that will give us pnpm and Vite support
aspire add # Then type and select "JavaScript"

At this point, you can use the aspireify skill to wire up the app, but it’s also instructive to set it up manually.

Add the Aspire resources

Now we can add the two application resources to the host.

// host/AppHost.cs
var builder = DistributedApplication.CreateBuilder(args);

var backend = builder.AddProject<Projects.server>("app-backend");

var frontend = builder
    .AddViteApp(name: "zeeq-planner", appDirectory: "../src/app")
    .WithPnpm()
    // 👇 Note: I fixed a port here to make it easier to find
    .WithEndpoint(name: "http", port: 7321, scheme: "http")
    .WaitFor(backend);
builder.Build().Run();

The stack is ready to run 🚀:

# Start the stack
➜  zeeq-tmpl aspire run --detach --non-interactive
Finding AppHosts...
host/host.csproj
🛑 Stopping previous instance (AppHost PID: 85558, CLI PID: 85469)
✅ Running instance stopped successfully.
Starting Aspire AppHost in the background...

AppHost:  host/host.csproj

   Dashboard:  https://localhost:17245/login?t=8555e27c92c928953486d41d2a3d0db8
        Logs:  /Users/cchen/.aspire/logs/cli_20260811T211920737_detach-child_7987c3fad30b
               4bc88b08ced7272136b8.log
         PID:  87818

✅ AppHost started successfully.

# Stop the stack
aspire stop

This same pattern can be used to wire up Postgres, Redis, Kafka, and many other modules that Aspire supports which makes it easy to build full-stack, distributed applications that start up with a single command and provides agents with the right interfaces to rapidly iterate on the application.

Setting up CSharpRepl and a quick smoke test

CSharpRepl’s wiring requires running csharprepl connect init to list the location of the CSharpRepl installation. To automate this, we’ll need to run the command and parse it out of the output.

using System.Diagnostics;

var builder = DistributedApplication.CreateBuilder(args);

var csrHook = ResolveCSharpReplHook(); // 👈 Extract the local hook path

var backend = builder
    .AddProject<Projects.server>("app-backend")
    // 👇 Wire up CSharpRepl environment variables to the runtime.
    .WithEnvironment("DOTNET_STARTUP_HOOKS", csrHook)
    .WithEnvironment("ASPNETCORE_HOSTINGSTARTUPASSEMBLIES", "CSharpRepl.InjectedHook");

var frontend = builder
    .AddViteApp(name: "app-frontend", appDirectory: "../src/app")
    .WithPnpm()
    .WithEndpoint(name: "http", port: 7321, scheme: "http")
    .WaitFor(backend);

builder.Build().Run();

// ⭐️ A helper function that runs `csharprepl connect init` and extracts the path to
// the hook .dll from the output.
string ResolveCSharpReplHook()
{
    var process =
        Process.Start(
            new ProcessStartInfo
            {
                FileName = "csharprepl",
                Arguments = "connect init",
                RedirectStandardOutput = true,
                UseShellExecute = false,
                CreateNoWindow = true,
            }
        ) ?? throw new InvalidOperationException("Failed to start csharprepl process.");

    // Full output from the command; need to extract just the hook .dll
    return process
        .StandardOutput.ReadToEnd()
        .Split('\n')
        .Select(line => line.Trim())
        .First(line => line.StartsWith("export DOTNET_STARTUP_HOOKS="))
        .Split('"')[1];
}

csharprepl connect list will show the running processes:

➜  zeeq-tmpl git:(main) ✗ csharprepl connect list

  PID   │ Process
 ───────┼─────────────────────
  71304 │ dotnet
  71441 │ dotnet
  92260 │ server

And we can connect directly to the repl with csharprepl connect <PID> and manipulate the runtime application.

Coding agents will be able to use the same REPL via --eval <code> or --eval-file <file> to manipulate the runtime state of the application.

Here’s an example of replace the health endpoint to change the logging behavior:

printf '%s\n' \
  '#replace Zeeq.Tmpl.HealthHandler.Handle with (instance) => "Healthy (replaced!)"' \
  'var health = Get<Zeeq.Tmpl.HealthHandler>(); var msg = health.Handle(); msg' \
  | NO_COLOR=1 csharprepl connect 92260 --streamPipedInput

What I hope is impressed here is that with CSharpRepl connected, agents can directly work with the fully wired up DI container and effectively simulate failure modes more realistically than possible with only integration tests. This gives agents a powerful tool to iterate and diagnose, build, or perform E2E regression tests in more realistic scenarios.

Closing thoughts

In part 2 of examining how to use C#, .NET, and Aspire to build agent-friendly applications, we’ve set up the foundation to build and extend the application.

What should be evident at this stage is that Aspire and CSharpRepl provide a powerful combination of tools that allow agents to operate autonomously and iterate on an application rapidly. This same scaffolding will make it easier to reproduce issues, reproduce production failure modes, check for regressions, and experiment with new features rapidly.

In the next section, we’ll:

• Add a skill to help agents understand how to effectively use CSharpRepl
• Wire up the API client generation from the backend to the frontend
• Start to incorporate the Copilot SDK and build a simple streaming chat interface to the agent now that the foundation of the application is ready.

If you are curious to see a real-world setup, check out the Zeeq.ai repo: https://github.com/zeeq-ai/zeeq-app

The Unexpected AI Stack: C# + .NET (Part 2) was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Summary

• C# and .NET as a stack are probably flying under the radar for many teams building new apps in the agentic era
• For teams already using legacy C# and .NET, it may not be clear how to best leverage the modern .NET stack tooling to speed up agentic development
• Aspire tooling provides agents a programmable, isolated runtime orchestration layer that improves agent autonomy and facilitates agentic patterns like worktrees and parallel development.
• CSharpRepl lets agents interact with an instance of the running application and manipulate the runtime state of the application including wrapping and replacing existing functions.
• Combined, this set of tooling gives agents more autonomy to iterate and build stable, correct, production-ready code that is proven in runtime configurations efficiently.

Why C#?

In an age when agents can write code in any programming language, it’s fair to ask: why have agents write code in C#?

Last year, I was a technical lead on the effort at Motion ($500m valuation, series C, post YCombinator startup, ~40 engineers) to move the entire backend off of TypeScript + Node and shift to a combination of C# and F# for all future backends.

The reasons are numerous and I’ve written about this extensively before, but it’s worth summarizing this again in mid 2026 framed in the context of building in an agent-first reality:

• Build- and run-time type checks provide a foundational layer of safety and correctness to check agent outputs
• Clean, easy to understand build-time error traces; compare this to typical TypeScript type error stacks…
• Roslyn static analyzers that allow for custom static analysis for agent guardrails
• Roslyn source generators for terse runtime code to remove boilerplate and reduce context for agents on the read path
• Extensively and richly documented with broad first-party libraries means that LLMs have very good coverage in their training data. While Python and JavaScript may have more representation, the lack of a standard library and BCL means that there are many variants of how to accomplish the same task.
• EF Core (mature, powerful .NET ORM) provides build-time checks for database schemas and decreases the likelihood of runtime errors from schema changes.
• Mature, stable, consistent, easy to use tooling and tool chain. Just dotnet. Again, easy for agents to work with, minimizing the need for skills or explanatory text for the agents.

Aside from that, if developers are no longer writing code, why not choose a platform that has:

• Built-in primitives for in-process asynchronous coordination (System.Threading.Channels, Orleans actor model)
• Higher performance and throughput where it matters (boundary serialization for both JSON and gRPC/Protobuf, etc.)
• Better story around ecosystem security and dependency management (NPM vs Nuget)
• A whole professional organization continuously patching and fixing the platform and libraries

Given the benefits of adding both build and runtime type safety as well as a mature, well-documented, and broad first party platform, C# is a surprisingly strong choice for building real software with AI.

The objective of this two-part series is to shed some light on how C# and the .NET ecosystem is an ideal foundation for building production software with agents.

• Part 1 (you are here) will introduce two under-the-radar components of the .NET stack that make it surprisingly amenable to building software with agents: Aspire and CSharpRepl.
• Part 2 will dive into a hands-on implementation from the ground up with an open-source template for teams to build on top of.
• Part 3 will implement the next layer of the application including a simple streaming interface to the Copilot SDK agent.
• Part 4 will extend the application with Testcontainers to demonstrate how to simplify test execution for agents with stateless containers as well as telemetry for runtime visibility.

The full repo: https://github.com/zeeq-ai/zeeq-tmpl

Encapsulating the runtime with Aspire

If you’re using Docker Compose or Tilt for dev runtime orchestration, you may have occasionally wished that it was just a bit more programmable. That’s exactly the gap that Aspire fills: a programmable orchestration layer that makes it easy to build an isolated runtime stack that agents can control while building software.

The best analogy is to consider the difference between Terraform and Pulumi or CDK. Terraform is a declarative tool for building infrastructure, while Pulumi and CDK are programmable tools that allows teams to build infrastructure with code. Aspire is the same thing for orchestrating your runtime stack.

Early iterations of Aspire were focused more on building distributed, microservices systems. While Aspire is still great for that, it is even better as an agent-friendly runtime orchestration layer.

It has several key features that make it a foundational component of an agent-friendly software stack:

• It has a built-in internal network loop that isolates individual runtime stacks. This is important for agent development and worktrees because this allows agents to run the full stack in isolation.
• It has a built-in, queryable OpenTelemetry target that gives agents the richness to access not just logs, but logical flows through traces and spans with rich metadata attributes on logs, traces/spans, and metrics.
• The aspire CLI supports searching and filtering console logs as well as OTEL structured logs and traces by resource.
• It is deeply programmable and supports C#, TypeScript, and Python for building runtime orchestration logic. And of course, it can run any code and has built-in extensions for handling most common runtime stack components like Vite powered JS frontends, databases, messaging, caching, etc.

Built in telemetry and observability

A key piece of the story is Aspire’s easy-to-use and built-in telemetry and observability backend.

This screenshot of the Zeeq Aspire dashboard shows what a typical local runtime looks like:

It doesn’t look too different from Tilt or Docker Desktop, but peek the telemetry:

Most importantly, the telemetry sink also surfaces gen.ai attributes that give visibility into standard agent interactions: materialized prompts, tool calls, etc.

You will certainly get much, much richer telemetry via specialized tools like Langfuse (not mutually exclusive since it becomes just another OTEL sink), but Aspire’s built-in, searchable telemetry sink let’s agents autonomously iterate with visibility into the runtime state.

aspire otel spans zeeq-server --search "BEGIN CHANGES" \
  --non-interactive -nologo \
  --dashboard-url http://localhost:15010

Searches through the list of spans that the agent can then then read for a full trace of the execution flow (assuming your code has been well-instrumented!):

Agent interaction with runtime resources

The aspire CLI becomes a primary interaction point with agents for runtime resources including:

• Reading logs via aspire logs
• Query the state of running resources via aspire describe <resource>
• Controlling resource lifecycle via aspire resource <resource> start|stop|rebuild

# Search logs for a given resource
aspire logs zeeq-server --search "principal" \
  --non-interactive --nologo

# Allow agent to further filter via jq
aspire logs zeeq-server --search "principal" \
  --non-interactive --nologo --format json

# Check application status with all environment variables
aspire describe zeeq-server --format json

# Check application status and query for specific properties
aspire describe zeeq-server --apphost ./host --format json | \
  jq -c '.resources[] | {
    name,
    displayName,
    state,
    healthStatus,
    commands: (.commands // {} | keys)
  }'

# Rebuild and restart a compiled resource like a .NET or Rust backend
aspire resource zeeq-server rebuild

# Restart a Node frontend resource
aspire resource zeeq-web restart

Port isolation for worktrees

The inner network loop becomes a key feature if you’re using worktrees and you want to run the full application stack in isolation to allow for agents to work in parallel.

# Run with randomized ports and isolated user secrets
aspire run --isolated

Wiring CSharpRepl into the running application

CSharpRepl (CSR) is a command line REPL that allows interactive C# scripting.

In the past, this has been generally useful for quickly trying out C# code without having to create a project or compile and run. However, the real unlock with CSR is that it can be wired into a running instance of the application.

Direct access to runtime dependencies

Imagine the following dependency graph:

If the agent can only test the web API handler entrypoint via curl, then it cannot easily diagnose issues with Service α since it must always pass through three layers of dependencies before it can test an input scenario with Service α. While some of this can be covered with unit and integration tests, in some cases — especially with AI-enabled apps — there’s no substitute for being able to test the runtime application directly like a human might with a debugger.

This is precisely what CSR allows. Will Fuqua (author of CSharpRepl) has a writeup that shows the gist of this:

# Get the installation location of the CSharpRepl tool hooks
csharprepl connect init

# This yields two environment variables (example) that can be wired up via Aspire
export DOTNET_STARTUP_HOOKS=".../tools/csharprepl/.../connector/CSharpRepl.InjectedHook.dll"
export ASPNETCORE_HOSTINGSTARTUPASSEMBLIES="CSharpRepl.InjectedHook"

# Then list the candidate processes for CSR to connect to
csharprepl connect list

# And connect to the PID
csharprepl connect 6580

Once connected, the agent can then directly manipulate Service α and any other runtime dependencies without having to go through the API entrypoint. This allows agents to:

• Diagnose issues and iterate on fixes much more quickly since it can wrap the running functions,
• Test different input scenarios,
• Replace the current implementation and run a live test reaching directly into the DI container,
• Bypass the authentication and authorization layers at the boundary to simulate different users without having complicated agent authorization paths or weakened surface areas to create an affordance to let agents access the runtime.

Think of it like a blood test versus directly examining an organ or tumor. The blood test is still useful, but being able to directly probe the organ or tumor will give a more complete picture of underlying “operational runtime behavior” and make it easier for diagnosing issues.

Wrapping and replacing functions at runtime

But it goes beyond that:

• #replace allows agents to intercept and replace a running function with another implementation
• #wrap allows agents to intercept and wrap a running function with additional logic (like logging, telemetry, etc.)

Together, this toolset gives agents a huge unlock because the agents can reach deep into the runtime configuration of the application to test different scenarios. Three aspects that I find this boosts:

1. Agents can better diagnose production issues by directly interacting with a local running instance and simulate the failure conditions or attempt to recreate the failure conditions from a production trace.
2. Agents can build more stable runtime systems and regression test backends without the need to use slower frontend browser/client manipulation.
3. Agents can experiment and iterate faster while building more stable runtime/production code.

Closing thoughts

While it is true that a lot of the innovation in the AI space originates from the Python ecosystem, for most teams building business applications using AI, the C# and .NET stack is a highly underrated and perhaps surprising option, especially for folks that haven’t dabbled in C# or .NET since the .NET Framework era.

Modern .NET and C# are well-suited for teams that want to build real products and real software on a stable base in a programming language and stack that offers agents strong build-time guardrails as well as throughput and performance at scale where it matters (consider the API boundary serialization of JSON and gRPC/Protobuf, for example — both areas where C# and .NET excel).

In Part 2, we’ll build a practical template for a C# + .NET agent-enabled application that you can use as a starting point for your own projects (wiring EF core for build time checked database queries, minimal web API endpoints, test containers for integration testing, etc.). We’ll also go into real-world examples of how to let agents operate the stack effectively when building autonomously including a skill that guides agents on getting the most out of CSharpRepl.

If you are curious to see a real-world setup, check out the Zeeq.ai repo: https://github.com/zeeq-ai/zeeq-app

The Unexpected AI Stack: C# + .NET (Part 1) was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Summary

• There is currently a social media and industry zeitgeist dialed-in on CLIs…just as there was a moment for MCP but just a few short months ago
• While it is true that there are token savings to be had by using a CLI, many folks have not considered how agents using custom CLIs run into the same context problem as MCP, except now without structure and many other sacrifices
• In much of the discourse, there is a lack of distinction between local MCP over stdio versus server MCP over HTTP; the latter is a very different use case
• Many folks are also only familiar with MCP tools, but overlook MCP prompts and resources as an important org- and enterprise-level mechanism for moving from cowboy vibe-coding to organizationally aligned agentic engineering.
• The importance of MCP auth is also commonly misunderstood as is the role and importance of telemetry in understanding org-wide tool use
• The oversight made by many is that individual usage of coding agents looks very different from organizational adoption of coding agents where there is an emphasis on visibility, telemetry, security, quality, and being able to operationalize and maintain agent-coded systems by a team of varying degrees of skill and experience.
• For enterprise and org-level use cases, MCP is the present and future and teams need to be able to cut through the hype of the moment.

The Influencer Driven Hype Cycle

Just 6 months back, Model Context Protocol (MCP) was all that anyone could talk about and it seemed that everyone was in a frenzy to ship MCP-related offerings and tools. At Motion, where I work, we were in the thick of it with seemingly every single vendor trying to get us to buy or use their MCP product.

“It’s just an API; why do I need a wrapper around that when I can just call the API directly.”

This was my common response to these vendors as they tried to pitch usage of their MCP offerings (often at a premium!). My guard was up as I looked on from the sidelines at the hype surrounding MCP with skepticism; we entirely skipped the MCP hype cycle because it wasn’t clear what the added value was.

But in just 6 short months, it seems as if the industry discourse around MCP has completely shifted. What happened?

First, most folks realized that in most use cases, MCP is just overhead to calling APIs directly because MCP as a wrapper for APIs do not make sense. Instead of using MCP, we simply wrote small tool wrappers around REST API endpoints.

Second, it is instructive to step back and understand just how much of this field is now driven by individuals marketing their companies and marketing themselves. So much of the AI landscape requires creating a sense of FOMO and hype and has been that way for years (while those of us adopting and adapting these tools on a daily basis often look on scratching our heads). These influencers constantly need some content to rant about to stay relevant and push themselves, their products, their companies, or their services so we see constant re-orientation around the zeitgeist of the moment.

In that sense, I view many of these influencers (and even people that should know better like Garry Tan and Andrew Ng) as no different than influencers peddling Ivermectin as a cure-all or anti-vax conspiracies; it’s simple ignorance.

The influencer-driven discourse has now shifted towards hating on MCP and praising CLIs as the current zeitgeist. Not without reason (to be clear) because in many cases, it is true that the CLI makes much more sense as an interface for agentic tool use…except when it does not.

Understanding the Misunderstanding

Before we dive any deeper, I would like to make clear that I think MCP is indeed the wrong choice for a class of use cases, but as the title might suggest, there is a deep misunderstanding of where and how CLI tools can yield savings and the important role of Model Context Protocol in moving orgs and enterprises towards more structured engineering with AI coding agents.

There are two topics at the core of the debate of CLI vs MCP and the foundation of much of the discussion:

1. Token savings and efficiency with CLIs.
2. The bloat and uselessness of MCP and its complexity.

Do Token Savings Exist?

Yes, they do. There are three modalities in token savings to be had, but it might not be as dramatic as social media would have you believe.

CLI Tools in the Training Dataset

First is that CLI utilities in the models’ training dataset like jq, curl, git, grep, psql, aws, s3, gcloud, etc. benefit tremendously from the underlying models already having encountered innumerable examples of how to use these tools. Because of this, the agent does not need additional instruction, schemas (not true when calling custom REST APIs, though), or context on how to use these tools; it can simply one-shot the tool in many cases. This can be a significant savings over MCP because in MCP, the tools must be declared up front in the tools/list response.

For CLI tools that will already be in the agent’s training dataset, absolutely always prefer them over MCP (for custom REST APIs, that’s a “maybe”).

However, this is not true of a custom, bespoke CLI tool. The LLM has no way of knowing which CLI to use and how it should use it…unless each tool is listed with a description somewhere either in AGENTS|CLAUDE.md or a README.md. You must provide the LLM some instruction on how and when to use a bespoke CLI tool that it has never seen before.

It is possible to point it to a directory called /cli-tools and rely on descriptive naming to have the agent pick and choose the tool, but anyone that works with agents day in and day out already knows that agents are often not very good at this without more explicit instructions. It will make mistakes and you will have to update your AGENTS|CLAUDE.md or other docs somewhere and fill it with more and more descriptions each time you find the agent misbehaving with your bespoke CLI tools.

Aside from that, even with a tool like curl, any token savings are lost the moment the agent has to understand a bespoke OpenAPI schema to correctly call the API since the entire OpenAPI schema may need to be loaded into context or extensive examples given to the agent in context to instruct it on how to use it. Oops; there goes all of your hyped up context savings.

Chained Extraction and Transform

In some workflows, it is possible to use a chain of CLIs to perform retrieval and then a transform or use extraction to reduce the data that makes it into the context window. In this scenario, there can be significant savings to be had, but it is also not unique to CLIs since most content that follows a format (HTML, JSON, XML, etc.) can be extracted using standard libraries via selectors (DOM/CSS, JSONPath, XPath, etc.). I don’t see this as a unique capability of a CLI based approach, but rather an underutilized approach because it is generally easier to just stuff it into context and let the LLM figure it out and optimize later.

Progressive Context Consumption

It is true that CLIs allow for progressive context consumption as an agent tries to deduce usage. Whereas MCPs will state the toolset and schema up front (the “bloat”), CLI tools can progressively load their --help into context.

// Sample from the Model Context Protocol site
{
  name: "searchFlights",
  description: "Search for available flights",
  inputSchema: {
    type: "object",
    properties: {
      origin: { type: "string", description: "Departure city" },
      destination: { type: "string", description: "Arrival city" },
      date: { type: "string", format: "date", description: "Travel date" }
    },
    required: ["origin", "destination", "date"]
  }
}

But nonetheless, the reality is that unless it is a well known tool in the LLM’s training dataset, the agent will need to progressively (and over multiple turns) descend the CLI tools’ help content to understand the available commands, sub-commands, and parameters.

There is simply no getting around this because the agent has no way of knowing how to use a bespoke CLI tool otherwise.

# What it might look like as --help output

command: searchFlights   Search for available flights
                         input: JSON object with origin, destination, date
                         example:
                         {
                           origin: "(string; required) departure city",
                           destination: "(string; required) arrival city",
                           date: "(date:yyyy-MM-dd; required) travel date"
                         }

I don’t know about you, but this sure looks like the MCP schema…just without any structure.

It is true that this could be progressively loaded instead by first listing all of the commands and then having the agent only --help the desired command to disclose the costly payload descriptor for the tool:

# Progressively loading instead of loading the entire schema

For usage:

flights <command> [--help]

commands: searchFlights      Search for available flights
          bookFlight         Book a flight
          ...

However, I would make five points here:

1. For a sufficiently complex flow, the agent will end up traversing most of the tree, regardless.
2. The likely context savings will end up pretty minimal if the MCP toolset is smartly designed in the first place; the agent just ends up taking more turns with a CLI while progressively discovering commands and parameter descriptors.
3. Without giving the agent the full schema up front, the chance of the agent using the toolset correctly will go down. In the same way that Vercel found agent usage of docs improved when they placed the full doc index into AGENTS.md, our intuition should tell us that if the agent is aware of all of the tools and parameters at the outset, it will be better equipped to select the right one.
4. Don’t give your agent complex, useless MCP tools in the first place? CLI and MCP are not mutually exclusive. Be selective in both cases.
5. With Anthropic also offering 1m context window now, is this still the defining argument to make?

If you’re still not convinced that a lot of this discourse lacks nuance and is just hype, congrats on buying into the current AI-influencer FOMO hype cycle; see you in 6 months when the influencers move on to the next revelation of the moment to stay relevant and get your eyeballs and dollars.

The Duality of MCP

When the MCP hype first started, I was as much a skeptic as anyone. Motion, at the time, was building out our “AI Employees” (like every other hype-chasing team) and building integrations.

While vendors were trying to sell us on their MCP implementations of integrations, we simply wrote tool wrappers around REST API calls and passed API keys or auth info in headers using standard Bearer tokens. The hype seemed entirely unjustified and I looked on with skepticism that MCP had any future whatsoever.

Especially in stdio mode, MCP felt excessive and useless. Indeed, in most use cases, MCP over stdio is probably not needed and adds complexity over writing a simple CLI.

But MCP over streamable HTTP? This is an absolute game changer and will be a key linchpin in organizational and enterprise adoption shifting from vibe-coding to agentic engineering.

Why Centralization is Key

(I preface that this is primarily relevant for orgs and enterprises; it really has no relevance for individual vibe-coders)

Most influencers talking about MCP fail to make a distinction between how MCP communicates locally over stdio mode versus streamable HTTP.

In stdio mode, the MCP server runs locally with the agent and indeed, why bother with this over writing a simple CLI?

But when accessed over streamable HTTP transport, it is possible to run that same logic in a centralized server and there are several unlocks when deployed and accessed by the agents under this modality.

Richer Underlying Capabilities

What if the use case would benefit from having access to a Postgres instance? With Apache AGE enabled for Cypher graph queries over indexed context? This is very straight forward when the tooling is centralized on a server and accessed by a thin client. It is possible to implement richer platform capabilities for the tooling because distribution is as simple as pointing an agent to an HTTP endpoint and adding an auth token.

Yes, a local database like SQLite may also do the trick, but there is a limit to what’s possible and sharing state across an org becomes more difficult.

Ephemeral Agent Runtimes

Remote MCP servers over HTTP also offer big advantages depending on the runtime context.

For example, when exposing remote tools and APIs to agents running in GitHub. Here, MCP makes it trivially easy to use tools that could require complex backends without any install and without restrictions of operating in the ephemeral nature of the GitHub Actions runtime environment.

It offloads management of stateful workloads in stateless, ephemeral contexts to a centralized server.

Auth and Security

Moving workloads into a server also improves the story around auth and security. CLIs that need to access secured API endpoints using curl, as an example, require that every developer have access to keys to those APIs (or proxy calls through some server). It’s easy to see why this is bad and a pain in the ass for an ops teams.

Centralizing this behind MCP allows each developer to authenticate via OAuth to the MCP server and sensitive API keys and secrets can be controlled behind the server like any other bog-standard server REST API using (more or less) bog-standard OAuth. The exposure of secrets is controlled, limited, and easy to audit.

An engineer leaves your team? Revoke their OAuth token and access to the MCP server; they never had access to other keys and secrets to start with.

Telemetry and Observability

For teams, another big win with centralized streamable MCP is the story around telemetry and observability. Which tools are having an impact? Which agent runtimes are the team using? Which tools are low value? Where and how are tools failing? Without centralized, standardized telemetry, it is exceedingly difficult as an engineering org to understand what’s working and what’s not.

With a centralized MCP server, this is simply a matter of emitting OpenTelemetry traces and metrics and collecting them using standard, off-the-shelf tooling.

While this is achievable with CLI tools, it is far easier to do so when deploying a single server because local delivery requires consumers to update and a lot of the scaffolding that happens centrally now has to be reproduced on each CLI tool that is shipped by a team.

Standardized, Instant Delivery of Up-To-Date Content

Distributed tooling (e.g. via packages) has much the same problem as working with distributed apps: keeping deployments up to date with the latest release and API compatibility. While it is relatively straightforward to update a server to add new capabilities, if tool capabilities are delivered as local CLI tools that interact with those APIs, now API version compatibility becomes a concern.

MCP includes provisions for this: subscriptions and notifications which allow servers to call back to the client notifying them of updates.

While most folks are aware of MCP tools, relatively fewer are aware of MCP prompts and MCP resources.

Look carefully and you can see:

• MCP Prompts are effectively server-delivered SKILL.md
• MCP Resources are effectively server-delivered /docs

Why would an org, team, or enterprise want this?

Well, it’s easy to see several benefits.

1. Dynamic content. While *.md files in a repo are static text files that need to be manually updated, synced, and maintained, server produced prompts and resources have no such restrictions. It is possible to dynamically generate the text on the fly to construct a skill. What about docs? What if some docs are useful in some contexts, but not others? What if docs could benefit from dynamic injection of content/context without a tool call (pricing data, current system status, etc.)?
2. Automatic and consistent updates. When *.md files are delivered as part of the repo or as part of a package import, the downside is that they can become out of sync and require explicit syncs on the local system. This is not the case with server-delivered skills and resources via MCP prompts as these are always up to date. What about third party, official sources of docs and skills? Do you manually reproduce and update these in your repo? Or would it be easier to simply proxy it through a server?
3. Org-wide knowledge. Some content applies org-wide. For example, standard best practices for security or telemetry in apps or infrastructure deployment considerations for apps. What about orgs that use microservices where one team may need to have docs for another service. Or what if a service team could provide skills for their service dynamically each time they ship? Does it make sense to reproduce these docs into every repo? How does an org keep them up-to-date and in sync?

MCP is the answer to all of these.

If you want to deliver standard planning prompts (skills) and standard docs across all repositories, all workflows, all teams, and all agent front-ends (Claude Code, Codex, VS Code Copilot, GitHub Agents, OpenCode, etc.) consistently and always up-to-date with server telemetry to measure adherence and usage across an org, then MCP prompts and resources IS that delivery mechanism.

Here is an MCP delivered prompt in OpenCode:

And a set of MCP delivered resources:

The same in Claude Code CLI:

Each of these resources is a dynamically generated, “virtual” index of documents that are available (similar to Vercel’s index in AGENTS.md) and we have the benefit of bringing these documents into any project, always up-to-date, and with full server telemetry on which documents are being accessed.

In all cases, delivery of this capability requires only configuring the MCP client; it is fire and forget with no need to keep things updated on the client.

Closing Thoughts

I get it; the industry moves fast now and these social media influencers need to keep chasing something new to keep their audience engaged. 6 months ago, MCP was the hot ticket. Now it’s a has-been; blamed for context bloat while often not even considering the tradeoffs and similar traps with custom CLIs. The masses following along have seemingly lost all critical thought in so far as engineering discipline goes.

But a simple thought exercise of how teams can move engineering orgs from vibe-coding towards agentic engineering would land one pretty squarely on the design and mission of the Model Context Protocol. In any scenario beyond a solo vibe-coder, MCP’s telemetry, simplified considerations for managing security, automatic content synchronization, schema + standards based approach, and ease of observability (how can you tell which tools are effective otherwise?) mean that teams that buy into the current zeitgeist will make a mistake when selecting an approach for delivering the scaffolding that enables agentic engineering.

We are still relatively in the early days of AI agents taking a leading role in software engineering, and because the field moves quickly, there is an emphasis on speed at all costs. But as we’ve seen with Amazon’s recent challenges in their AWS division, teams eventually have to operationalize and maintain these software systems produced by AI agents.

Take advice and anecdotes on agents from the likes of Garry Tan and Andrew Ng with a grain of salt: what works for them *individually* in a homogeneous environment likely will not hold true for teams and orgs in a heterogeneous environment where the objective is to ensure a team of diverse skill and experience levels converges upwards in their output and production of software.

If the goal is to ship software that is dependable and maintainable once operationalized, we still need an engineering discipline that ensures consistency, security, high quality, and correctness — even when the producer of that software is an AI agent. Organizations need architectures and processes that start to move beyond cowboy, vibe-coding culture to organizationally aligned agentic engineering practices. And for that, MCP is currently the right tool for orgs and enterprises.

Long live MCP!

MCP is Dead; Long Live MCP! was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.

Just over a decade ago, Satya Nadella took over the helm at Microsoft from Steve Ballmer and instrumented perhaps one of the biggest recent turnarounds in Big Tech.

Prior to Nadella’s turn at the helm, Microsoft’s business model was still tightly focused on Windows licensing with cloud seemingly an afterthought. Over the last decade, Microsoft’s transformation has been nothing short of spectacular as Nadella drove a rapid pivot from the desktop era to the cloud computing era while playing catch-up with AWS. Had Nadella not righted the ship at that moment in time, it’s safe to say that Microsoft could have gone the way of IBM before them.

At first blush, it may seem odd to correlate Microsoft’s trajectory with startups; after all, what does a multi-trillion dollar behemoth like Microsoft have in common with startups?

It turns out that many startups can probably learn from Nadella’s embrace of growth mindset. In an interview with Inc., Nadella shared that successful people are defined by 3 words: Learn-It-All

[Say] you have two students — one of them has more innate capability, and the other has less. The person who has less, but is a learn-it-all, will ultimately [become] better. That applies to CEOs, and that applies to companies. I think it has been a helpful cultural metaphor for us to say that you can’t act like a know-it-all; you have to be a learn-it-all.

This he had learned through the work of a Stanford psychologist: Carol Dweck.

Growth Mindset

Dweck’s book Mindset: The New Psychology of Success introduced this term and defined the difference between a “growth mindset” and a “fixed mindset”.

Through her research, Dweck differentiates individuals who view traits like ability and intelligence as innate and immutable (fixed mindset) and those who view these traits as flexible (growth mindset). While it may seem a simple and trivial distinction, in fact, it is probably one of the key discriminators of those who will thrive at startups and those who will wither.

Listen to enough episodes of the How I Built This podcast, and it is clear that successful startups and founders are those that embraced the growth mindset. Intuitively, this makes sense: just about every successful startup zig-zags their way through a maze of failures, rejections, pivots, and crises. Those who approach this with a fixed mindset will capitulate before finding traction.

What This Means for Engineers

One of the challenges for engineers when joining a startup is that it can often be quite chaotic and overwhelming. Unlike Big Tech where there tends to be more well-defined processes, isolated teams, stable tooling and platforms, and carefully planned on-boarding flows, startups more often than not require individuals to jump right into the fire.

For many engineers (especially more senior ones), this can often lead to fast burnout and feelings despair with a heavy onset of imposter syndrome that can lead down an irrecoverable path of self-defeat. One hypothesis why this might have a stronger effect on more senior engineers is that they might have moved on from their growth mindset phase. This is why individually having a growth mindset (and organizationally embodying a growth mindset) is key: because it helps otherwise talented engineers find their footing.

When Lefan Tan Jia Hui reached out after joining Motion, he sought guidance on how to be productive and how to cope with the incredibly complex, multi-million line codebase at Motion. He had already been a high performing engineer (despite his short resume) that had tackled multiple complex projects across a variety of stacks (Next.js, React, Vue, Astro.js, C#, Postgres — he tackled them all with aplomb!). But at Motion, he found himself struggling to gain his footing; he started to question his own ability and competency.

To help right his mindset, I sent him the following LinkedIn post and left him with some simple advice: “embrace looking like an idiot.”

This advice embodies what it means to have a growth mindset and is the key to finding success in almost any undertaking, but holds especially true for startups because there is simply less of a support apparatus in most cases. To get a footing and rapidly contribute requires simply letting go of ego and putting in the work while reflecting on the feedback.

Scott Brown — a carpenter with a fantastic YouTube channel— has one of my favorite clips on this that I often share as well when folks ask for advice on how to get better:

That’s not how you get better. The way you get better is by putting something out and going “Well, I’ll do better on the next one”. And then you do that week after week, month after month. Before you know it, your first video and your most recent video don’t look anything alike.

This perspective is important as engineers can view their current state and their current expertise as “fixed intrinsics” and that their value is in this existing expertise whereas thriving in a startup actually requires engineers to frequently step out of their comfort zone, learn new skills, deal with new complexities, and grow rapidly.

What This Means for Startup Teams

Many teams fail to foster this kind of environment and end up creating the opposite: an environment driven by the need to protect one’s ego. When this takes hold across the team, it will find itself accidentally oriented around a fixed mindset driven by ego — the exact opposite of what is desirable in a fast-paced, high-growth environment.

Gather a group of smart founders and talented engineers and it can be easy to create an environment where “I don’t know” or “Why? How?” become sacrilegious phrases when, in fact, these are some of the most powerful phrases in a team’s common grammar — so long as the team is oriented around a growth mindset.

Motion’s Culture Document explicitly emphasizes this mindset:

We encounter a lot of hard problems, most of which don’t have obvious answers. We aren’t constrained to any particular solution, and sometimes the best solution may sound crazy at first. We want an environment where every member of the team feels excited sharing new ideas, which will be respected and challenged by others.

There are many systematic ways that teams can right the ship and foster this culture:

• Leaders have to embody this and be ready to say “I don’t know” and always be openly seeking the expertise of others. This sets the tone that “not knowing” is not the same as “not capable”. “Not knowing” is always the first step on the path to true knowledge.
• Shape the onboarding process so that new hires understand that they are expected to “embrace looking like an idiot” for the first few weeks. Candidates may not have the right perspective that they were selected through an expensive and time consuming process and the team is making a bet on them. They may hyper-focus on their current ability rather than their ability to grow as the key reason for their selection. Make it clear that the goal is for them to learn fast and the best way to learn fast is by asking questions and making some mistakes along the way rather than protecting their ego. The more senior a hire, the more important it is to focus on this messaging because for these hires, they likely have their own high expectations.
• Foster a low-ego culture where even the most senior and capable resources (perhaps especially these) openly probe with genuine curiosity rather than probing maliciously with an attitude of self-preservation of the ego.
• Focus on a culture of documentation and sharing of responsibility (rather than singular ownership) as this creates an environment where knowledge is not siloed. Siloed knowledge can become a precursor to an ego-driven culture because it inflates the importance of the individual rather than the prosperity of the team. Documentation and transferability of knowledge are systematic ways to curtail this. Some engineers take the wrong view on this; they see siloing as protecting their job security and self-importance by creating a closed region of the company. Instead, they are simply falling into a fixed mindset of guarding their small empire. Successful leadership will ensure to equate growth with transferability of responsibility; to take on new and impactful work necessarily requires handing off established and stable work.

Teams and leaders should actively foster a culture of open comms, asking for help, and visible collaboration (especially important for remote teams) as this creates an operating environment where individuals focus on moving the ball forward rather than protecting their ego.

Shifting Hiring

One key mistake that many startups make is to confuse “badges” for “technical competency”. This is not to disparage the many supremely talented and capable engineers at Big Tech, but a caution that competency at a Big Tech may not transfer to a fast-paced startup environment because of the mindset shift.

(Author’s aside: I call this “badge dropping” and is one of my biggest pet peeves as badges can sometimes be a sign of being a good interviewer rather than being a great engineer.)

In fact, Harj Taggar, a YC partner dedicated a slide to this in one of the Startup School sessions where he discussed his experience coaching hundreds of startup teams:

Harj’s key point: many FAANG engineers will not thrive in a startup environment not because they do not have the technical ability, but because they have the wrong expectations and mindset.

Intuitively, there are several reasons why this approach of badge-driven hiring can be a flawed approach for high growth startups:

• In Big Tech, there is a lot of scaffolding already in place for most engineers to take advantage of. Many engineers are thus not broadly skilled in various areas of software delivery (e.g. CI, databases, actual implementation of systems architecture, etc.). In startups, everyone needs to be able to pitch in, even if outside of their technical comfort zone. To do so successfully requires one to oriented their mindset towards growth.
• Big Tech generally has many processes and procedures in place that are shaped for their specific operating environment. Those processes and procedures are more often than not antithetical to how high-growth companies like startups need to operate and individuals can bring that culture with them.
• Career growth in Big Tech often demands siloing — a dangerous trait for startups.
• Big Tech approaches to solving engineering challenges are often too heavy and complex for the needs of fast moving startups where teams should favor flexibility and seek approaches that bias towards adaptiveness.

Indeed, at one e-commerce startup which received an $8 million seed round partially predicated on hiring an all-Amazon engineering team, it quickly became clear within the first two weeks of joining that half of the engineering team simply had the wrong mindset to thrive at a startup. While some individuals embraced the autonomy, speed, and occasional chaos, others were clearly lost in that transition and required either a lot of hand-holding or would end up creating overly complex solution approaches that lacked flexibility and adaptability often required for meandering paths that startups undertake.

The other half? These were the engineers that embraced a growth mindset and quickly adapted while learning new skills, new platforms, and new technologies. Case in point: the team started with cumbersome Elastic Beanstalk based deployments driven by AWS CodeBuild because it was what the team had used in Amazon. But in a highly agile environment, a container-based deployment approach on Elastic Container Service better suited the fast-changing needs of the infrastructure as the product bounced between pivots.

Many hiring teams perhaps over-index on “badges” and under-index on “growth mindset” (which carries with it some other character traits like low ego and high adaptiveness). For Motion, a key objective is to find the individuals who are not only talented, but also bring with them a low-ego, growth mindset to help rapidly ship products to our customers.

Closing Thoughts

Motion’s Culture Document is one of the key reasons I made the choice to join the team as it aligned with my own ideals of an operating culture. Having speed run multiple startups over the last 5 years, one key lesson I take away from those experiences is the importance of “vibes” or culture.

As Brian Chesky of Airbnb opined in his essay Don’t Fuck Up the Culture,

Why is culture so important to a business? Here is a simple way to frame it. The stronger the culture, the less corporate process a company needs. When the culture is strong, you can trust everyone to do the right thing. People can be independent and autonomous. They can be entrepreneurial.

For Motion and other startup engineering teams, this can often be one of the biggest hurdles to get right and one of the easiest to stray from. Even a few bad hires or the wrong growth framework can lead to a virulent degradation of culture. Fostering a culture focused around a growth mindset and actively seeking “learn-it-alls” can right that ship.

For candidates and new hires joining startup teams, simply “embrace looking like an idiot” for the first few weeks and get as much feedback as you can. If an organization doesn’t foster, support, and embody that mindset? Then you’re not going to have a good time and better to cut your losses early!

If you have a growth mindset and you want to work with a low-ego, high-autonomy team, check out Motion’s Careers page and hit the apply button! We are growing rapidly; building greenfield projects in C#, TypeScript, React, Postgres; and always exploring creative (Grug-brained) engineering approaches to solving high-value business problems!

Something I’ve come to appreciate over my career in software engineering is simplicity and how it is an enabler of speed and creation.

Scott Carey wrote a great essay on this: Complexity is killing software developers:

“Complexity kills,” Lotus Notes creator and Microsoft veteran Ray Ozzie famously wrote in a 2005 internal memo. “It sucks the life out of developers; it makes products difficult to plan, build, and test.”

We have all felt this friction at some point in our careers where seemingly simple tasks are met with painful papercuts wrought by complexity. (It often reminds me of that scene in the movie Saw where one of the captives has to dive into a pit of needles to find a key.)

Fred Brooks famously made the distinction between complexity of the “accidental” and “essential” kind thinking, at the time, that many advancements had rooted out much of the accidental complexity in software engineering. Brooks might be quite disappointed at the reality that we face today.

From that same article, Carey quotes:

Justin Etheredge, cofounder of the software agency Simple Thread, helpfully differentiates between essential and accidental complexity. He told InfoWorld, “Essential is the complexity in the business domain you are working in, the fact that enterprises are extremely complicated environments, so the problems they are trying to solve are inherently complex. The other area is accidental; this is the complexity that comes with our tooling and what we layer on top when solving a problem.”

This distinction probably spells out the reason you’ve read this far: because you too, know the frustration of having to deal with accidental complexity while trying to solve the valuable problem of the essential complexity.

The challenge is that when a group of smart engineers are gathered, it can often be a difficult exercise of discipline to keep them from reaching for complexity of the accidental kind to get their highs.

Complexity is the Enemy of Value

When foundational systems have been addled by accidental complexity, the net result is that it:

• Unnecessarily creates drag on overall team velocity
• Creates an unstable foundation to build upon
• Focuses the efforts of smart engineers on work that does not correlate to business value
• Creates a culture of gatekeeping by making it difficult if not impossible (or at least highly discouraged) for the broader team to contribute

These systems are often brittle and difficult to own, understand, and scale because well, that was actually the intent from the start…

The Darker Side of Accidental Complexity

While accidental complexity can arise innocently enough from well-intentioned engineers, the darker side is that it can manifest from ego and self-preservation. For you see: building a simple, stable, easy-to-use, easy-to-own, and easy-to-operate system means that such a system does not need nearly as much manpower, attention, and of course budget to own. If there are no fires to fight, then there is no need for a firefighter.

How else shall the complexity merchant demonstrate their value and build their fiefdom within the enterprise but to create the context for which ever more complexity is layered?

To make things simple from the get-go is not exciting to the complexity merchant and is also not profitable. How else shall the complexity merchant demonstrate their value and build their fiefdom within the enterprise but to create the context for which ever more complexity is layered to solve the previously added complexity? Of course the complexity merchant seeks to command more resources because can’t you see? How else will all of this complexity be solved?

For the complexity merchant to operate, they must always introduce a bit of friction and pain (this self-inflicted injury is how you will know that you are dealing with a complexity merchant). It is the presence of this friction and pain that grants them the justification to peddle their solutions (without it, how else could they expand their influence if everything were easy and smooth?). If you only bring on this tool or hire more engineers or add this vendor, then everything will be solved!

All along, they will tell you that such complexity is unavoidable and essential; such problems, they will say, have no simple solutions. They will tell tales of Airbnb and Facebook and Google and how yes, you — your team — also needs this complexity. You want to be Facebook, don’t you?

Even in cases where there is no malicious intent, it can be driven by the need to feel accomplished by summiting that peak of complexity.

These are individuals that have let their ego drive their decisionmaking.

A Path to Undoing Complexity

As a parent, I often grumble and reprimand my kids if they start playing with something new without cleaning up and putting away what they were playing with previously. It is a simple rule to keep one’s space clean and organized and also imparts a basic sense of responsibility upon the child.

One simple solution to solving for new complexity? Make the engineers clean up after themselves first. No new complexity until the old complexity has been stabilized or simplified so that anyone can operate and maintain it and keep it in a stable state. No toys left in a state of disarray; all things orderly in their place where they belong at the end of the day. No half-finished promises and half-delivered (of course undocumented) solutions.

Even better to be skeptical of Silver Bullets and favor tried, true, and boring solutions and approaches for these rob the complexity merchant of their grip. Where complexity is necessary or actually adds business value, it must be contained and there must be a path to transition ownership of that complexity that includes handing it off in a stable, well-documented state.

Demand documentation and transferability of systems since obfuscation and gatekeeping are key tools of complexity merchants. The more push-back encountered for these basic demands, the more sure you can be that you’ve encountered a complexity merchant.

Closing Thoughts

I get it: simple, boring, proven platform solutions just aren’t exciting or thrilling to build. They do not create the conditions for those concerned with self-importance, self-preservation, and simultaneously the desire to lord over a fiefdom. If the system is simple, well-documented, and transferable, then anyone can own and operate it, the complexity merchant has no foundation on which to build their empire.

Why build following boring, tried, true, and established patterns and solutions following paths where the hard challenges have been solved for? Where’s the thrill in that? How shall the moat of self-importance be created if these systems are boring, easy, and straightforward to build upon, operate, and maintain?

So a word of warning to you, dear traveler: beware of the complexity merchants as you endeavor to create value and seek to gain velocity towards whatever your destination shall be. You will know them by the trail of once new and shiny toys, only half played with, that they leave in their wake — packaging strewn to the side. You will feel the friction that they create and the control that they wish to exert. Do not fall, time and again, for their promises of a Silver Bullet only half kept.

• Many folks who have not worked with LLMs to produce some complex output tend to see most AI startups as simple wrappers around third party models and APIs.
• However, this is the wrong way to think about LLMs and AI. A more accurate analogy is to think of LLMs and AI as an (admittedly magical) oven — an appliance that transforms the input mixture of ingredients into some intermediate product.
• In other words, when evaluating any AI startup, focusing on the oven is the wrong approach. Focus instead on how well the team understands the input ingredients (the data) and how adept they are at preparing that data long before it goes into the “oven” (the LLM).
• Teams that do not invest in deeply understanding, preparing, and structuring the input data sources to the LLM are simply baking supermarket box cake mix; yes of course these are simple wrappers because the effort on the input was low and cheap.

“It’s Just Another Wrapper”

In a recent Hacker News thread launching an AI code generation CLI tool Codebuff, I saw this comment at the bottom:

I don’t really find it exciting at all, I’m sorry. It’s just another wrapper for a 3rd party model. — Hacker News Poster

This is a sentiment that is quite common to come across in the current tech landscape of AI and LLMs where venture capital has started a gold rush in AI-for-{X} with countless startups in the wake of this boom trying to capitalize. Most early “AI startups” really were just thin wrappers for OpenAI, but — believe it or not — some AI startups really can create a moat and that moat is probably not what you think.

Before continuing, it is important to differentiate startups that work on AI (and AI adjacent services) — like OpenAI, Anthropic, Mistral, HuggingFace, et al — and startups that work with AI that use LLMs and gen AI to transform and create content (be that text/code, data, images, or video). Here, let us consider only startups that work with AI.

Having an Oven Does Not a Baker Make

Folks who have not worked heavily with gen AI — day-in, day-out — severely underestimate how difficult it is to get consistently good output from any LLM at the moment. The team at Ownit AI (where I work) first started working with OpenAI around May of 2023 and we’ve spent 18 months learning how to utilize LLMs in an e-commerce context.

What is clear through the many different iterations of services we’ve built around LLMs is just how lacking they are. Let me iterate the ways:

• Without retrieval augmented generation (RAG), the LLM will hallucinate
• A related problem is that, until web search was integrated, the LLM had no way of knowing about knowledge that came after its training data
• Giving it access to web search creates another problem: search has become “enshittified” with ads and spam results so even giving it access to search can be risky
• Of course, it also won’t have knowledge about information that never showed up in its training dataset or search (e.g. private enterprise data)
• Basic RAG has its limits so teams need to continuously find clever ways to improve the RAG like using graph-based RAG
• The LLM’s understanding of instructions for complex tasks is quite “robotic” in that it prefers following very specific instructions and any ambiguity risks undesired/useless output
• Even when given precise instructions on the output format, LLMs often tend to generate incorrect output formats. This was especially bad with JSON early on, but still present today and crops up when generating output in batches of thousands (you will see failures).
• Even with extremely large context windows, LLMs tend to suffer in their comprehension of the context as the window gets larger so there is a need to create as reasonably focused a context as possible not only for speed and cost, but also for accuracy.

In almost all cases, the solution to getting the desired output from the LLM is to fix the input by making the prompt and context data more precise, more targeted, or transformed in a way that makes it more likely to get the desired output from the LLM.

In a sense, this is not unlike baking: the AI is an appliance — the most fantastical and amazing appliance man has made, but yet still an appliance — and the artistry and science all happens before the ingredients go into the oven and we hit “bake”.

Indeed, there are many “AI startups” that quickly fizzled because having an oven does not a baker make and many startups were just content putting supermarket box mix cake batter into the oven and calling it a day.

How AI Startups Build Their Moat

In my 20’s, Alton Brown’s Good Eats series was the first time I had the thought that cooking is equal parts art and science. The preparation and combination of ingredients followed by the application of energy (usually in the form of heat or mixing) fundamentally changes the chemistry and physical properties of the ingredients themselves (I actually think it was specifically his episode on mayonnaise and how the different ingredients play their roles in creating an emulsion).

Watching the recent Netflix cooking competition Culinary Class Wars, I was often reminded of this as I watched the competitors meticulously transform a set of ingredients into incredibly creative dishes. Every detail from how “Triple Star” (a nickname of one of the competitors) finely sliced wonton wrappers to achieve the correct texture when fried to “Napoli Matfia” creating a Tiramisu that wowed the judges using only ingredients available in a convenience store (even extracting the cream filling from a baked pastry).

It is really striking just how deeply the top contestants bring not only their culinary skills, but also their deep understanding of the ingredients and how to use each for the desired effect like changing texture, adding acidity, complementing another flavor, masking some pungent note, and so on. In one challenge, the judge notes that “Triple Star” even meticulously chopped vegetables to be the same size as the caviar for more consistent visual presentation and better mouth feel.

Startups that want to succeed working with AI similarly must understand that the moat is actually in the data:

• How well does the team understand what data is needed and what datasets are available to achieve the desired output.
• How creative the team is in transforming and using that data given the limitations of LLMs.
• How the team thinks about combining different pieces of data to feed context into the LLM.
• How the team puts the finishing touches on the output and presents the results to the consumer (often requiring parsing and transforming once again).

It is not unlike the culinary arts: the best chefs and bakers deeply understand the input ingredients be it their flavor and texture profiles, how they should be shaped for different applications, how they have to be prepared, how one ingredient will complement another, how to combine it all, and how to present the finished product tastefully and creatively. Hitting “bake” on the oven is simply one step that applies heat to combine and transform the ingredients; all of the artistry and science is actually in the preparation steps before and the presentation steps afterwards.

For most AI startups, the moat probably really comes down to how well the team can do ETL and how well the team understands the domain. Does the team have some insight into datasets in a particular domain? Have they come up with a novel or creative way to use that data? Do they understand how to shape and prepare that data given the constraints of LLMs? Can they combine the right pieces of data to achieve the desired output?

These two videos from Ownit AI’s work with retailer catalogs shows just how good the results can be when the underlying datasets have been extracted, transformed, loaded, and prepared for the LLM.

In Safeway’s case, the delta between the native Google-powered search results and the Ownit AI generated shelves is striking:

In Ulta’s case, the native platform misses entire classes of products when answering the query:

To achieve both, the process involved extracting and transforming the retailer’s catalog ahead of the selection of the correct context (indeed, it would be impossible to select the correct context without this initial transformation into a graph).

Closing Thoughts

What’s clear is that AI startups like Codebuff are anything but just another “wrapper for a 3rd party model”. That third party model is simply an oven; everyone has an oven, but few could claim to be world-class patissiers because at the end of the day, the oven is but an appliance that transforms the input ingredients to the output product. If the input is low effort with cheap ingredients, the chances of the output being high quality and world class — at least with the current magical ovens — are quite slim.

In the case of Codebuff, the art and science is how the team prepares, resolves, and structures the correct and relevant pieces of the codebase to feed into the LLM; it is not just a simple matter of shoving all of the codebase into the context. For Ownit, it is an analogous challenge (though perhaps a bit more abstract than a codebase) where we’ve invested an incredible amount of effort into acquiring, transforming, and mapping out product catalog data from retailers — classic ETL — so that we can accurately resolve the correct products.

For both companies — and other AI startups that seek to have a moat — a good portion of the effort and resultant software system is dedicated to designing and building the pipeline that performs the extract-transform-load and the smaller part of the effort is interfacing with the LLM. If you want to build an AI startup moat, everything starts with the ingredients (the data) and how adept you are in your preparation of the data before hitting “bake”.

• Content production and distribution has always been constrained by “bandwidth”. For example, the early days of broadcast TV meant that with 3 channels, there could only be 72 hours of programming in a 24 hour day because of limited bandwidth.
• Advancements in technology (e.g. printing press, WWW) have always expanded that bandwidth and along the way, increased the ratio of producers to consumers.
• Generative AI is the next engine for that expansion of bandwidth and it has the potential to push that ratio closer to 1.
• This expansion has always followed a reliable pattern of text → images → audio → video and we see this same pattern with OpenAI and current LLMs.
• We are still quite far from the endpoint and the hype and high valuations are justified; generative AI will create “infinite bandwidth” for content.

Are We in the Trough of Disillusionment?

Spend enough time on the Internet, and there will be no lack of doubters about the state of AI and specifically generative AI (gen AI). Some bemoan the “AI slop” that has pervaded everything from Facebook feeds to Google Image search results for terms like “baby peacock”. Some think that Nvidia’s stock is overvalued and that there is a glut of GPUs — who needs more? Some are already in the trough of disillusionment with AI.

I beg to differ; I think there’s not nearly enough hype around gen AI and we’re barely getting started.

Content and the Concept of “Bandwidth”

Some time in the mid 2000’s, I was speaking with an older coworker about Al Gore’s TV channel, Current TV. Gore’s Current TV would air pieces produced by citizen journalists; a novel idea at the time. My coworker shared with me an interesting perspective: the history of content, media, and information is one of “bandwidth” — he called it — and Gore’s Current TV represented an expansion of that valuable bandwidth which was once reserved for the likes of Walter Cronkite, Edward R. Murrow, and David Brinkley.

When broadcast television first started, there were a handful of channels; the bandwidth was limited, he said. Audiences watched more or less the same few shows on the tiny number of channels because there was only so much “bandwidth”. He meant this word in every sense: from the broadcast capacity to the production capacity. 3 channels and 24 hours a day meant, at most, 72 hours of television programming in any given day and there was only so much capacity for professional crews to create that content.

Then came cable which created more bandwidth and more content followed to fill that capacity. That transition created enough bandwidth to deliver content for a variety of special interests whether that was cooking (Food Network), nature and documentaries (Discovery, History), DIY (HGTV), or even whole channels dedicated to kids shows (Nick, Cartoon Network).

Of course, the mid 2000’s is also around the time that YouTube started. YouTube represents an exponential explosion in the number of “channels”; it represents seemingly infinite “bandwidth”. There is a channel or a creator for nearly any niche interest one could think of and more content is generated and published every day at an astounding rate; everyone with a phone in their pocket can be a creator.

Aside: it is natural, then, that the overall quality of that content decreases, but there is a broadening of the the space by enabling new producers and publishers.

This same evolution has happened in nearly every content industry from publishing to music to games. At each step of advancement, there is a common pattern: the ratio of producers to consumers increases.

Viewed from this perspective, gen AI is the next explosion and we are just barely off of the starting line.

The Pattern of Advancement

Looking at this historically, there has been a repeatable pattern of advancement in terms of the creation and dissemination of content and information. First is always text, then still images. Once we had the ability to reproduce audio and video, then audio and moving images followed text and still images.

In every era of advancement of content and information, this exact progression has repeated because each step requires more capacity and bandwidth. Think about the early Internet and how it progressed from text to images. Eventually came MP3s and Napster. And finally, we are in the era of YouTube, Zoom, and Netflix; we can start a video call with anyone in the world seemingly free of cost or choose from a catalog of thousands of shows and movies to watch any time we want for less than $20 per month.

It is important to acknowledge this because I think this gives context to where we are at this moment with gen AI. OpenAI’s ChatGPT first launched with text. Then came multi-modal models. Most recently, OpenAI released their realtime API that supports bi-directional audio streaming. Could it be more obvious what the next leap will be?

Gen AI as “Infinite Bandwidth”

Today, YouTube content already feels infinite; but the ratio of producers to consumers on the platform is still asymmetric and there can only be as much content that is as tailored to a specific audience as there are producers of that content and those producers are constrained by real-life constraints like time and the cost of generating certain types of content such as 3D animated films, for example.

Gen AI promises something else: a channel of content for every consumer generated specifically for that consumer; “infinite bandwidth”. Effectively, gen AI will accelerate the push of the ratio of producers to consumers ever closer to 1 where every consumer can produce content tailored specific to their niche, interests, preferences, and tastes.

At Ownit (where I work), this is precisely the route that we took. In the early days of our efforts with AI (just a year ago 🤣), we were pre-generating content targeting long-tail SEO terms for our retail customers — dozens to thousands of articles at a time.

After encountering a Hacker News thread on benchmarking LLMs in May, I started to experiment with multi-stream concurrent content generation. The dramatic increase in throughput (this was before GPT-4o) meant that we could deliver all new experiences which were previously unthinkable because of the poor UX caused by low throughput. So instead of pre-generating a set of articles targeted to a set of questions, what if we could literally generate high quality answers to every question on the fly?

Of course, for anyone following the industry, Google came to the same conclusion and released AI Search Overviews around the same time. While Google has taken their punches for some questionable responses from their search overviews, I have no doubt that they are on the right path because it is effectively moving the producer-consumer ratio towards 1: for every question, a custom and tailored chunk of content generated specifically for the consumer to answer their question.

Perspective for the Doubters

It is easy to be a doubter in this moment because much of the tooling to generate high quality results is still quite niche and complex while often being inaccessible to lay people without the hardware and know-how to pull everything together. The more accessible parts of the ecosystem are producing the low quality content and, frankly, there are a lot of low-quality efforts to leverage gen AI.

It is easy to look at the low quality, obviously AI-generated content and come to the conclusion that gen AI is a dud; destined to aggravate the enshiffitication of the once pure web.

But pay attention and there are some folks out there producing really jaw dropping content and we’re seeing the state of the art advancing quite rapidly. 10 years ago, this was the state of the art research in image generation from text:

And just two months ago (click the link for video):

Companies like Meta are investing heavily in this space with research projects like Movie Gen. It’s obvious to see how this tooling could one day be accessible to anyone right in Instagram or Facebook.

Reddit’s /r/aivideo sub is one community to keep an eye on. Examples like this AI generated fashion show or fictional depiction of how ancient Egypt was built give a peek into what’s possible and how quickly, week-over-week, the capabilities are advancing.

Adobe’s Project Turntable is also quite eye opening as the implication is that the underlying model can extrapolate a 3D model from a set of 2D vectors:

Teams are also experimenting with playable AI generated games (also video):

The implications for industries from special effects to film making to game design are immense. They are yet to be disrupted by gen AI, but there is no doubt that as the capabilities expand, accessibility increases, and prices decrease, many industries will have to adjust.

What’s at Stake?

To understand just how high the stakes are at the moment, the three major cloud providers — Amazon, Google, and Microsoft — have all recently announced investments in nuclear energy to power a massive expansion of compute capacity to fulfill this endpoint of “infinite bandwidth”.

Here are some headlines from just the last few weeks:

• Amazon buys stake in nuclear energy developer in push to power data centres
• Amazon and Google have plans for fueling their data centers: Nuclear power
• Three Mile Island nuclear plant will reopen to power Microsoft data centers

This is a signal that the gen AI era is still early in its infancy. The build-out of this capacity and continued advancements in the algorithms, tooling, and ecosystem will push us closer to that 1:1 ratio of producers to consumers (one might still consume content or a model produced by a creator, but producing a personalized version of it will be a prompt away).

Meta invested billions in creating the next content platform — virtual reality — only to have the Metaverse largely fizzle. But what if a Star Trek inspired holodeck-lite experience isn’t far in the future? What if the Meta Quest could deliver that virtual experience, tailored to each consumer’s specific interests and tastes? What if it could render any world and any experience from just a few lines of text?

The stakes are incredibly high because just as Google and Meta became trillion dollar advertising companies with the transition of content consumption from broadcast and cable to the Internet — if the future of content is AI generated — the platform that can deliver this future of content to the consumer will enrich themselves with trillions more.

Closing Thoughts

While I do not yet have concrete thoughts on artificial general intelligence (AGI) and “singularity”, my framework for thinking about generative AI is drawn from this concept of “bandwidth” that my coworker shared with me almost two decades ago.

The pattern is obvious: at each stage of bandwidth expansion, the ratio of producers to consumers increases and generative AI seems to be the engine that will push this ratio ever closer to 1. The expansion always moves from text to images to audio to video to interactive. To power this expansion, expect to see continued investments by the likes of Nvidia, Microsoft, Meta, Google, and Apple into generative AI.

It is easy to look at the middling state of generated content today and dismiss “AI” as a dud; an over-hyped promise that has under-delivered. But history has shown that technology that expands bandwidth for content and decreases the friction for production and distribution — from the printing press to the World Wide Web — will ultimately prevail and deliver riches.

• When working with large dataset ETL, it can be useful to “checkpoint” your output to reduce costs if something fails.
• Google Cloud Storage offers a nifty feature called object composition for merging multiple objects into one
• This feature has some limits, but ultimately provides a useful way to perform ETL steps with a “checkpoint” to reduce the cost of recovery.

Google Cloud Storage Object Composition

Google Cloud Storage has a really nifty feature that allows you to “compose” multiple storage objects into one.

This can be really useful if you’re performing ETL-like tasks across a large dataset and you want to add resiliency to your process by allowing you to “checkpoint” your work by simply dumping out state into a storage object. For example, imagine that you are moving data from an API and processing a large resultant CSV file (think tens of thousands of records) or perhaps processing a stream from a large video file.

That API call or retrieval may be costly and may be subject to rate limiting. So what happens if your process fails? To solve for this, we can simply periodically dump out a fragment of our extract. For example, imagine that we are extracting 30 days of data from some API. We can simply dump out 1 day at a time and then at the end, merge all 30 days into one large object for the next step of processing.

Let’s explore how we can achieve this with C# and .NET

Getting Set Up

First, let’s set up our project:

mkdir gcp-storage-compose
cd gcp-storage-compose
dotnet new console
dotnet add package Google.Cloud.Storage.V1

Next, you’ll want to grab your account credential key (.json file) for authenticating. As far as I’m aware, there is no storage emulator and the Firebase storage emulator does not support this feature (your API call will fail).

Creating the Fragments

Let’s start by creating the fragments:

using System.Text;
using Google.Apis.Auth.OAuth2;
using Google.Apis.Storage.v1.Data;
using Google.Cloud.Storage.V1;

var credential = GoogleCredential.FromFile("credentials.json");
var storage = StorageClient.Create(credential);

var run = Random.Shared.Next(100, 10000000);
var bucket = "__YOUR_BUCKET_HERE__";
var count = 10;

var fragments = Enumerable
  .Range(0, count)
  .Select(i => $"{run}/fragment-{i}.csv")
  .ToArray();

var tasks = fragments.Select(async path => {
  using var stream = new MemoryStream();

  var content = Encoding.UTF8.GetBytes(
    $"LINE FOR: {path}{Environment.NewLine}"
  );

  await stream.WriteAsync(content);
  await storage.UploadObjectAsync(bucket, path, "text/csv", stream);
});

Task.WaitAll([.. tasks]);

The run identifies a unique folder into which we’ll place 10 objects which represent individual fragments of a .csv file.

Then we can merge it into 1:

// Merge into 1
await storage.Service.Objects.Compose(
  new() {
    SourceObjects = fragments
      .Select(path => new ComposeRequest.SourceObjectsData() {
        Name = path
        })
      .ToList(),
    Destination = new Google.Apis.Storage.v1.Data.Object {
      ContentType = "text/csv"
    }
  },
  bucket,
  $"{run}/__merged.csv"
)
.ExecuteAsync();

If we execute this, we’ll see that we generate 10 files and dump out a merged file! Sweet!

How far can we take this?

Probing the Limits

Let’s up this to 1500 objects:

var count = 1500;

And our first stumbling block:

Unhandled exception. The service storage has thrown an exception.
HttpStatusCode is BadRequest.
Google.Apis.Requests.RequestError
The number of source components provided (1500) exceeds the maximum (32) [400]
Errors [
  Message[The number of source components provided (1500) exceeds the maximum (32)] Location[ - ] Reason[invalid] Domain[global]
]

So the maximum number of individual items we can compose is limited to 32.

We’ll need to change how we do the composition into chunks.

Refactoring for Chunking

Let’s first extract a function that encapsulates the composition:

var compose = (string[] paths) => {
  storage.Service.Objects.Compose(
    new() {
      // 👇 Always stuff the first fragment into the paths
      SourceObjects = ((string[])[fragments[0], .. paths])
        .Select(path => new ComposeRequest.SourceObjectsData() {
          Name = path
        })
        .ToList(),
      Destination = new Google.Apis.Storage.v1.Data.Object {
        ContentType = "text/csv"
      }
    },
    bucket,
    fragments[0] // 👈 Overwrite the first object
  )
  .Execute();
};

Our strategy is simple: whatever the first fragment is, we’ll just compose everything into it iteratively.

Now we need to aggregate our fragments into one:

fragments
  .Skip(1)            // Skip the first one since we'll always insert this.
  .Chunk(31)          // Take in chunks of 31 since we'll always insert one.
  .ToList()           // Get all of our chunks
  .ForEach(compose);  // Compose each chunk in turn

And here is our output file:

LINE FOR: 1729868/fragment-0.csv
LINE FOR: 1729868/fragment-1.csv
LINE FOR: 1729868/fragment-2.csv
LINE FOR: 1729868/fragment-3.csv
LINE FOR: 1729868/fragment-4.csv
LINE FOR: 1729868/fragment-5.csv
....
LINE FOR: 1729868/fragment-37.csv
LINE FOR: 1729868/fragment-38.csv
LINE FOR: 1729868/fragment-39.csv

Nice!

With this mechanism in place, then we can easily “checkpoint” and dump out a partial CSV as we process a large dataset.

If we need to restart, simply check if the dataset already exists in Cloud Storage and add the path if it does. Otherwise, process the chunk and dump out a CSV.

If we fail at row 40,000 of 100,000, we don’t have to start at 0 and instead, we can just pick up at 40,001 instead.

Then on a successful run, merge everything into the first file.

Deleting the Excess Objects

Once we’ve merged the objects, we can actually simply delete them.

Let’s add that in:

var compose = (string[] paths) => {
  storage.Service.Objects.Compose(
    new() {
      SourceObjects = ((string[])[fragments[0], .. paths])
        .Select(path => new ComposeRequest.SourceObjectsData() {
          Name = path
        })
        .ToList(),
      Destination = new Google.Apis.Storage.v1.Data.Object {
        ContentType = "text/csv"
      }
    },
    bucket,
    fragments[0]
  )
  .Execute();

  // 👇 We can delete concurrently.
  var deletionTasks = paths.Select(
    path => storage.DeleteObjectAsync(bucket, path)
  );

  Task.WaitAll([.. deletionTasks]);
};

Now our contents after 1 run:

Only one file remains; keeping everything tidy.

Limited Chunks

What if we actually don’t want 1 massive file? What if we want to compose say every n files in to 1? An example might be a case where each file has 1,000 records and we want resultant files to max out at 10,000 records.

Let’s update our logic here so that we dump out a file for each subset:

var compose = (string[] paths) => {
  storage.Service.Objects.Compose(
    new() {
      SourceObjects = paths // 👈 No longer inserting here
        .Select(path => new ComposeRequest.SourceObjectsData() {
          Name = path
        })
        .ToList(),
      Destination = new Google.Apis.Storage.v1.Data.Object {
        ContentType = "text/csv"
      }
    },
    bucket,
    paths[0] // 👈 Compose into the first file
  )
  .Execute();

  var deletionTasks = paths
    .Skip(1) // 👈 Skip the first file for deletion
    .Select(
      path => storage.DeleteObjectAsync(bucket, path)
    );

  Task.WaitAll([.. deletionTasks]);

  return paths[0]; // 👈 Return the path so we can print it
};

var chunkSize = 10;

var paths = fragments
  .Chunk(chunkSize) // 👈 No skip here
  .ToList()
  .Select(compose);

foreach (var path in paths) {
  Console.WriteLine(path);
}

Here is our output:

➜  gcp-storage-compose dotnet run
5056692/fragment-0.csv
5056692/fragment-10.csv
5056692/fragment-20.csv
5056692/fragment-30.csv
Completed run: 5056692

And in storage:

The contents for the first chunk:

LINE FOR: 5056692/fragment-0.csv
LINE FOR: 5056692/fragment-1.csv
LINE FOR: 5056692/fragment-2.csv
LINE FOR: 5056692/fragment-3.csv
LINE FOR: 5056692/fragment-4.csv
LINE FOR: 5056692/fragment-5.csv
LINE FOR: 5056692/fragment-6.csv
LINE FOR: 5056692/fragment-7.csv
LINE FOR: 5056692/fragment-8.csv
LINE FOR: 5056692/fragment-9.csv

And the second:

LINE FOR: 5056692/fragment-10.csv
LINE FOR: 5056692/fragment-11.csv
LINE FOR: 5056692/fragment-12.csv
LINE FOR: 5056692/fragment-13.csv
LINE FOR: 5056692/fragment-14.csv
LINE FOR: 5056692/fragment-15.csv
LINE FOR: 5056692/fragment-16.csv
LINE FOR: 5056692/fragment-17.csv
LINE FOR: 5056692/fragment-18.csv
LINE FOR: 5056692/fragment-19.csv

Nice; we can imagine that each of these files had 1,000 rows and now file 1 and file 2 have 10,000 rows each.

Closing Thoughts

If you’re in Google Cloud, this is a neat platform feature that allows you to do interesting things with your ETL type workloads without involving a database for what is ostensibly temporary data.

By check pointing each set of data, it’s possible to avoid potentially costly retries.

Full Code Listing

using System.Text;
using Google.Apis.Auth.OAuth2;
using Google.Apis.Storage.v1.Data;
using Google.Cloud.Storage.V1;

var credential = GoogleCredential.FromFile("credentials.json");
var storage = StorageClient.Create(credential);

var run = Random.Shared.Next(100, 10000000);
var bucket = "__YOUR_BUCKET_HERE__";  // 👈 Replace this
var count = 40;

var fragments = Enumerable
  .Range(0, count)
  .Select(i => $"{run}/fragment-{i}.csv")
  .ToArray();

var tasks = fragments.Select(async path => {
  using var stream = new MemoryStream();
  var content = Encoding.UTF8.GetBytes($"LINE FOR: {path}{Environment.NewLine}");
  await stream.WriteAsync(content);
  await storage.UploadObjectAsync(bucket, path, "text/csv", stream);
});

Task.WaitAll([.. tasks]);

var compose = (string[] paths) => {
  storage
    .Service.Objects.Compose(
      new() {
        SourceObjects = paths
          .Select(path => new ComposeRequest.SourceObjectsData() {
            Name = path
          })
          .ToList(),
        Destination = new Google.Apis.Storage.v1.Data.Object {
          ContentType = "text/csv"
        }
      },
      bucket,
      paths[0]
    )
    .Execute();

  var deletionTasks = paths
    .Skip(1)
    .Select(
      path => storage.DeleteObjectAsync(bucket, path)
    );

  Task.WaitAll([.. deletionTasks]);

  return paths[0];
};

var chunkSize = 10;

var paths = fragments
  .Chunk(chunkSize)
  .ToList()
  .Select(compose);

foreach (var path in paths) {
  Console.WriteLine(path);
}

Console.WriteLine($"Completed run: {run}");

Merging Objects in Google Cloud Storage with Compose and C# was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.