Skip to content

Integration BigQuery Extension

aryehcitron@gmail.com edited this page May 24, 2026 · 14 revisions

The Kronikol.Extensions.BigQuery package adds Google BigQuery REST API operation tracking to your test diagrams. Instead of showing raw HTTP requests like POST /bigquery/v2/projects/my-project/queries, your sequence diagrams show classified operations like Query: /query or Insert: /mydataset/table/mytable.

Using a shared library or abstraction layer? If your code doesn't use the BigQuery SDK directly — e.g. it goes through a shared data-access library, wrapper, or custom abstraction — this extension won't be able to intercept the underlying calls. See Tracking Custom Dependencies for alternative approaches including RequestResponseLogger.LogPair(), TrackingProxy<T>, and MessageTracker.


How It Works

The Google BigQuery .NET SDK (Google.Cloud.BigQuery.V2) translates all operations into REST HTTP requests. BigQueryTrackingMessageHandler is a DelegatingHandler that intercepts these HTTP requests, classifies each one into a BigQuery operation (Query, Insert, Read, List, Create, Delete, Update, Cancel) using the HTTP method and URL path, then logs it to RequestResponseLogger with a human-readable label.

Because it logs to the same RequestResponseLogger as the standard TestTrackingMessageHandler, BigQuery operations appear alongside your HTTP API calls in the same sequence diagram — showing the complete flow from test → API → BigQuery.


Install

dotnet add package Kronikol.Extensions.BigQuery

Verbosity Levels

The extension supports three verbosity levels that control how much detail appears in the diagrams:

Level Method shown URI shown Headers Request body Response body
Raw HTTP method (POST, GET, etc.) Full BigQuery REST API URI All except excluded set Full JSON Full JSON
Detailed Classified operation (Query) Clean path with dataset/resource context (/mydataset/table/mytable) Filtered (noisy Google API headers excluded) Full JSON Full JSON
Summarised Classified operation (Query) Resource type and name only (/table/mytable) None None None

The default is Detailed.

Diagram Label Examples

Operation Raw Detailed Summarised
Run a query POST: /bigquery/v2/projects/proj/queries Query: /query Query: /query
Get query results GET: /bigquery/v2/projects/proj/queries/job123 Query: /query/job123 Query: /query/job123
Streaming insert POST: /bigquery/v2/projects/proj/datasets/ds/tables/tbl/insertAll Insert: /ds/table/tbl Insert: /table/tbl
Read a table GET: /bigquery/v2/projects/proj/datasets/ds/tables/tbl Read: /ds/table/tbl Read: /table/tbl
List tables GET: /bigquery/v2/projects/proj/datasets/ds/tables List: /ds/table List: /table
Create a dataset POST: /bigquery/v2/projects/proj/datasets Create: /dataset Create: /dataset
Delete a table DELETE: /bigquery/v2/projects/proj/datasets/ds/tables/tbl Delete: /ds/table/tbl Delete: /table/tbl
Create a job (load) POST: /bigquery/v2/projects/proj/jobs Create: /job Create: /job
Cancel a job POST: /bigquery/v2/projects/proj/jobs/job456/cancel Cancel: /job/job456 Cancel: /job/job456
SDK metadata GET: /discovery/v1/apis Other: ... (skipped)

Classified Operations

The classifier recognises these BigQuery operations from the SDK's HTTP traffic:

Operation HTTP Pattern
Query POST /projects/{proj}/queries or GET /projects/{proj}/queries/{jobId}
Insert POST /datasets/{ds}/tables/{tbl}/insertAll
Read GET on a specific resource (table, dataset, job, model, routine)
List GET on a collection (/tables, /datasets, /jobs, /models, /routines, /tables/{tbl}/data)
Create POST on a collection (/datasets, /tables, /jobs)
Delete DELETE on a specific resource, or POST /jobs/{id}/delete
Update PUT or PATCH on a specific resource
Cancel POST /jobs/{id}/cancel
Other Unrecognised paths (e.g. discovery API, service account metadata)

In Summarised mode, Other operations are silently skipped.

Supported Resource Types

The classifier extracts and tracks these resource types:

Resource Type URL Path Segment
query /queries
dataset /datasets/{id}
table /datasets/{ds}/tables/{id}
tabledata /datasets/{ds}/tables/{id}/data
job /jobs/{id}
model /datasets/{ds}/models/{id}
routine /datasets/{ds}/routines/{id}

The upload endpoint variant (/upload/bigquery/v2/projects/...) is also recognised for data loading jobs.


Setup

Option A: With BigQueryClientBuilder (recommended)

Use the WithTestTracking() extension method on BigQueryClientBuilder:

var trackingOptions = new BigQueryTrackingMessageHandlerOptions
{
    ServiceName = "BigQuery",
    CallerName = "My API",
    Verbosity = BigQueryTrackingVerbosity.Detailed,
    CurrentTestInfoFetcher = CurrentTestInfo.Fetcher,
    HttpContextAccessor = httpContextAccessor // optional: enables dual-resolution from HTTP headers
};

var client = new BigQueryClientBuilder
{
    ProjectId = "my-project",
    // Credential = ... (for real BigQuery)
}
.WithTestTracking(trackingOptions)
.Build();

WithTestTracking sets the builder's HttpClientFactory to a TrackingBigQueryHttpClientFactory that wraps the default HTTP handler with the tracking handler. The pipeline becomes:

BigQueryClient → ConfigurableMessageHandler (auth, retry) → BigQueryTrackingMessageHandler → HttpClientHandler

Option B: In ASP.NET Core Integration Tests

For WebApplicationFactory-based integration tests where BigQuery clients are registered in DI:

builder.ConfigureTestServices(services =>
{
    // Remove existing BigQueryClient registration
    var descriptor = services.SingleOrDefault(d => d.ServiceType == typeof(BigQueryClient));
    if (descriptor != null) services.Remove(descriptor);

    // Register tracked client
    services.AddSingleton(sp =>
    {
        var trackingOptions = new BigQueryTrackingMessageHandlerOptions
        {
            ServiceName = "BigQuery",
            CallerName = "My API",
            Verbosity = BigQueryTrackingVerbosity.Detailed,
            CurrentTestInfoFetcher = CurrentTestInfo.Fetcher
        };

        return new BigQueryClientBuilder
        {
            ProjectId = "test-project",
        }
        .WithTestTracking(trackingOptions)
        .Build();
    });
});

Option C: Manual Handler Wiring

If you need full control over the HTTP pipeline, use BigQueryTrackingMessageHandler directly:

var trackingOptions = new BigQueryTrackingMessageHandlerOptions
{
    ServiceName = "BigQuery",
    CallerName = "My API",
    Verbosity = BigQueryTrackingVerbosity.Detailed,
    CurrentTestInfoFetcher = () => ("My Test", testId)
};

var handler = new BigQueryTrackingMessageHandler(trackingOptions, new HttpClientHandler());
// Use handler with your own HTTP pipeline setup

Configuration Reference

BigQueryTrackingMessageHandlerOptions

Property Type Default Description
ServiceName string "BigQuery" The participant name shown in the diagram for the BigQuery service
CallerName string "Caller" The participant name shown for the service making BigQuery calls
Verbosity BigQueryTrackingVerbosity Detailed Controls how much detail appears in the diagram (Raw, Detailed, Summarised)
CurrentTestInfoFetcher Func<(string Name, string Id)>? null Returns the current test's name and ID. Required — if null, requests are forwarded but not logged
CurrentStepTypeFetcher Func<string?>? null Optional — returns the current BDD step type (Given/When/Then)
HttpContextAccessor IHttpContextAccessor? null Optional — enables dual-resolution of test identity from HTTP headers
ExcludedHeaders HashSet<string> See below Headers to exclude from diagrams in Raw/Detailed mode
SetupVerbosity BigQueryTrackingVerbosity? null Verbosity override for the Setup phase. See Phase-Aware Tracking
ActionVerbosity BigQueryTrackingVerbosity? null Verbosity override for the Action phase. See Phase-Aware Tracking
TrackDuringSetup bool true When false, tracking is suppressed during Setup. See Phase-Aware Tracking
TrackDuringAction bool true When false, tracking is suppressed during Action. See Phase-Aware Tracking

v2.23.0+ Dual-Resolution: BigQueryTrackingMessageHandler accepts an optional IHttpContextAccessor? httpContextAccessor constructor parameter for resolving test identity from HTTP request headers when running inside the SUT's request pipeline. See HTTP Tracking Setup#Dual-Resolution Test Identity (v2.23.0+) for details.

Default Excluded Headers

The following Google API headers are excluded by default (they add noise without diagnostic value):

  • Authorization
  • User-Agent
  • x-goog-api-client
  • x-goog-request-params
  • google-cloud-resource-prefix
  • Cache-Control

Override ExcludedHeaders to customise:

new BigQueryTrackingMessageHandlerOptions
{
    ExcludedHeaders = ["Authorization"] // Only exclude Authorization
}

Option D: Direct Tracker Usage (v2.24.0+)

If you don't need HTTP-level interception — for example, you want to log BigQuery operations from a gRPC-based pipeline, a shared data layer, or a custom abstraction — use BigQueryTracker directly:

using Kronikol.Extensions.BigQuery;

var tracker = new BigQueryTracker(new BigQueryTrackingMessageHandlerOptions
{
    ServiceName = "BigQuery",
    CallerName = "OrdersApi",
    Verbosity = BigQueryTrackingVerbosity.Detailed,
    CurrentTestInfoFetcher = CurrentTestInfo.Fetcher
});

// Classify and log
var request = new HttpRequestMessage(HttpMethod.Post, "/bigquery/v2/projects/my-project/queries");
var op = BigQueryOperationClassifier.Classify(request);
var (reqId, traceId) = tracker.LogRequest(op, "SELECT * FROM dataset.table");

// ... execute BigQuery operation ...

tracker.LogResponse(op, reqId, traceId, "42 rows returned");

BigQueryTracker implements ITrackingComponent, supports phase-aware tracking, and uses the same BigQueryTrackingMessageHandlerOptions as the handler.

Option E: DI Registration (v2.24.0+)

Register a BigQueryTracker instance in your DI container:

services.AddBigQueryTestTracking(options =>
{
    options.ServiceName = "BigQuery";
    options.CallerName = "OrdersApi";
    options.Verbosity = BigQueryTrackingVerbosity.Detailed;
    options.CurrentTestInfoFetcher = CurrentTestInfo.Fetcher;
});

Then inject BigQueryTracker where needed in your test setup or custom middleware.

Note: Options D and E are complementary to Options A–C. The HTTP handler (Options A–C) intercepts all BigQuery SDK traffic automatically. The direct tracker (Options D–E) is for scenarios where you intercept at a different layer or want to log custom BigQuery-related events.


Bridging Test Info From HTTP Tracking

If you already use TestTrackingMessageHandlerOptions for HTTP tracking and want to share the same test info fetchers with BigQuery tracking:

var httpOptions = new TestTrackingMessageHandlerOptions
{
    CurrentTestInfoFetcher = () => (testName, testId),
    CurrentStepTypeFetcher = () => currentStepType,
    CallerName = "My API"
};

var bqOptions = new BigQueryTrackingMessageHandlerOptions
{
    ServiceName = "BigQuery",
    CallerName = httpOptions.CallerName,
    CurrentTestInfoFetcher = httpOptions.CurrentTestInfoFetcher,
    CurrentStepTypeFetcher = httpOptions.CurrentStepTypeFetcher,
    Verbosity = BigQueryTrackingVerbosity.Detailed
};

Framework Compatibility

The extension works with all Kronikol framework adapters:

Framework Test Info Fetcher Pattern
xUnit3 TestContext.Current.Test(TestDisplayName, UniqueID)
xUnit2 ITestOutputHelper + ambient storage
NUnit4 NUnit.Framework.TestContext.CurrentContext
MSTest MSTest.TestContext
TUnit TUnit.Core.TestContext.Current
LightBDD LightBddTestTrackingMessageHandlerOptions bridge
ReqNRoll ReqNRollTestTrackingMessageHandlerOptions bridge
BDDfy BDDfyTestTrackingMessageHandlerOptions bridge

Invocation Validation

BigQueryTrackingMessageHandler implements ITrackingComponent and auto-registers with TrackingComponentRegistry on construction. At report generation time, unused components are automatically detected and surfaced as console warnings and in the diagnostic report (when DiagnosticMode=true). This never throws or fails tests.

See Diagnostics and Debugging for full details on the TrackingComponentRegistry API.

Home


Demo


Getting Started

Common Tasks

Integration Guides

Extensions

Configuration

Features

Reference

Clone this wiki locally