Add ability to switch between LLM Gateway and Agent Builder

[reakaleek]

Ask AI: provider switch, unified streaming, and improved UX

This PR introduces a simple way to choose the AI provider, unifies streaming across providers, and improves reliability and UX.

Overview

Implemented a provider-agnostic AskAI architecture using the Adapter and Factory patterns, allowing seamless switching between Agent Builder and LLM Gateway without frontend code changes.

Architecture

Backend (Abstraction Layer)

• AskAiProviderResolver: Routes requests based on X-AI-Provider HTTP header (defaults to LLM Gateway)
• AskAiGatewayFactory: Dynamically instantiates the correct AI gateway (Agent Builder or LLM Gateway)
• StreamTransformerFactory: Selects the appropriate stream transformer for the chosen provider
• Gateway Implementations:
  • LlmGatewayAskAiGateway - Handles Google Cloud LLM Gateway communication
  • AgentBuilderAskAiGateway - Handles Kibana Agent Builder communication
• Stream Transformers:
  • LlmGatewayStreamTransformer - Translates LLM Gateway SSE format
  • AgentBuilderStreamTransformer - Translates Agent Builder SSE format
• Canonical Format: AskAiEvent - Single event format both transformers output

Frontend (Provider Agnostic)

• aiProviderStore: Zustand store managing user's provider selection
• useAskAi hook: Sends X-AI-Provider header, receives canonical events
• AiProviderSelector: UI component for switching providers
• AskAiEvent types: Frontend mirrors backend's canonical event schema

Key Benefits

✅ Zero coupling: Frontend has no knowledge of provider-specific implementations
✅ Hot-swappable: Switch AI providers via a single header
✅ Extensible: Add new providers by implementing two interfaces
✅ Maintainable: Provider API changes only affect their respective transformer
✅ Testable: Each component can be tested in isolation

Technical Highlights

• Backend handles all complexity (auth, routing, translation)
• Frontend only understands canonical AskAiEvent format
• Single API endpoint serves all providers: /docs/_api/v1/ask-ai/stream
• Real-time SSE streaming maintained across both providers

Screenshots

[Mpdreamz]

Use Task.Factory.StartNew(.., TaskCreationOptions.LongRunning);

https://github.com/davidfowl/AspNetCoreDiagnosticScenarios/blob/master/AsyncGuidance.md#avoid-using-taskrun-for-long-running-work-that-blocks-the-thread

I also wonder if we can just use Asp.NET HttpResponse.BodyWriter which is a PipeWriter already to directly write the transformed data to the outgoing SSE stream to avoid copying data further.

See also our very own @stevejgordon's blog post:

https://www.stevejgordon.co.uk/using-the-bodyreader-and-bodywriter-in-asp-net-core-3-0

[Mpdreamz]

This would also remove the need the need to return the stream so eagerly and just use async/await on the reader/writer.

[reakaleek]

I think I was able to address both concerns with replacing Task.Run with _ = ProcessPipeAsync(reader, pipe.Writer, cancellationToken);

and using PipeReader correctly.

[Mpdreamz]

❤️ this! nice work, my comments are not blocking we can tackle that in a follow up.