This folder contains benchmarks that represent common scenarios to look at for the .NET team.

These jobs can be executed using the .NET Crank global tool. .NET Core 3.1 is required to install the global tool.

Install crank with the following command:

dotnet tool install Microsoft.Crank.Controller --version "0.1.0-*" --global

Alternatively, update crank with the following command:

dotnet tool update Microsoft.Crank.Controller --version "0.1.0-*" --global

Each profile defines a set of machines, private IPs and ports that are used to run a benchmark.

For testing purpose only, the local profile requires a local agent to run:

dotnet tool install Microsoft.Crank.Agent --version "0.1.0-*" --global

crank-agent

The source code for these benchmarks is located here. These scenarios return a "Hello World" string and the client uses HTTP pipelining with 16 requests.

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/master/scenarios/plaintext.benchmarks.yml --scenario plaintext --profile aspnet-perf-lin

• plaintext: Middleware implementation
• https: Middleware implementation, using HTTPS
• endpoint: Middleware implementation with Endpoint routing
• mvc: Controller implementation
• connectionclose: Middleware implementation, the connection is closed after each request. Pipelining is disabled.

The source code for these benchmarks is located here. These scenarios serialize and return a { "message": "Hello World" } string.

The serialization is done with System.Text.Json.

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/master/scenarios/json.benchmarks.yml --scenario json --profile aspnet-perf-lin

• json: Middleware implementation
• https: Middleware implementation, using HTTPS
• mvc: Controller implementation

The source code for these benchmarks is located here. These scenarios execute some database requests and return either HTML or Json.

The database server is PostgresQL.

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/master/scenarios/database.benchmarks.yml --scenario fortunes --profile aspnet-perf-lin

The following scenarios are implemented from a middleware (no MVC)

• fortunes
• fortunes_ef
• fortunes_dapper
• single_query
• single_query_ef
• single_query_dapper
• multiple_queries
• multiple_queries_ef
• multiple_queries_dapper
• updates
• updates_ef
• updates_dapper

The following scenarios are using ASP.NET CORE MVC

• fortunes_ef_mvc_https

The suffixes represent different database access strategies:

• No suffix: Raw ADO.NET
• "ef" suffix: Entity Framework Core
• "dapper" suffix: Dapper

The source code for these benchmarks is located here. These scenarios are highly optimized to provide the best performance, in detriment of extensibility and code complexity.

The database server is PostgresQL.

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/master/scenarios/platform.benchmarks.yml --scenario fortunes --profile aspnet-perf-lin

• plaintext
• json
• fortunes
• single_query
• multiple_queries
• updates
• caching

These scenarios are running several web proxies, including YARP.

The downstream service returns a variable size content. By default the result is 10 bytes.

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/master/scenarios/proxy.benchmarks.yml --scenario proxy-httpclient --profile aspnet-perf-lin

• proxy-yarp
• proxy-httpclient
• proxy-nginx
• proxy-haproxy
• proxy-envoy
• proxy-baseline: This scenario doesn't go through a proxy

The size of the payload can be changed by adapting the path of the requested url:

--variable path=/?s=100

The server and downstream protocols can be changed to http (default), https and h2. The following example shows how to use "h2 - h2":

--variable serverScheme=https --variable downstreamScheme=https --load.variables.transport http2 --downstream.variables.httpProtocol http2

Custom bodies can be used with the bodyFile and verb variables like this:

--variable bodyFile=https://raw.githubusercontent.com/aspnet/Benchmarks/master/scenarios/assets/100B.txt --variable verb=POST

Local and remote files can be used.

As for many other scenarios, the existing web load clients (wrk, bombardier, ...) are configured to support predefined headers:

• none (default)
• plaintext: "Accept: text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7"
• json: "Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7"
• html: "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"

The variable presetHeaders is used to select one of these:

--variable presetHeaders=plaintext

These scenarios measure the performance of different other frameworks

• NodeJs
• Actix (Rust)
• FastHttp (Go)
• Netty (Java)
• ULib (C++)

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/master/scenarios/te.benchmarks.yml --scenario plaintext_nodejs --profile aspnet-perf-lin

• plaintext_nodejs
• json_nodejs
• fortunes_nodejs
• plaintext_actix
• json_actix
• fortunes_actix
• plaintext_fasthttp
• json_fasthttp
• fortunes_fasthttp
• plaintext_ulib
• json_ulib
• fortunes_ulib
• plaintext_netty
• json_netty

These scenarios measure the performance of different Grpc server and clients implementations.

• Go
• Native (C)
• ASP.NET

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/master/scenarios/grpc.benchmarks.yml --scenario grpcaspnetcoreserver-grpcnetclient --profile aspnet-perf-lin --variable streams=70 --variable connections=1

• grpcaspnetcoreserver-grpcnetclient
• grpccoreserver-grpcnetclient
• grpcgoserver-grpcnetclient
• grpcaspnetcoreserver-grpccoreclient
• grpccoreserver-grpccoreclient
• grpcgoserver-grpccoreclient
• grpcaspnetcoreserver-grpcgoclient
• grpccoreserver-grpcgoclient
• grpcgoserver-grpcgoclient
• grpcaspnetcoreserver-h2loadclient
• grpccoreserver-h2loadclient
• grpcgoserver-h2loadclient

• Number of streams:
  • --variable streams=1
  • --variable streams=70
• Protocol:
  • --variable protocol=h2c
• Call types:
  • Unary: --variable scenario=unary
  • Server streaming: --variable scenario=serverstreaming
  • Ping ping streaming: --variable scenario=pingpongstreaming

Middleware based application that serve static files of any size.

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/master/scenarios/staticfiles.benchmarks.yml --scenario static --profile aspnet-perf-lin

• static

The filename and size can be changed by adapting these variables:

--variable sizeInBytes=1024
--variable filename=file.txt

This scenario can easily reach the max network bandwidth. To verify that use --load.options.displayOutput true which will display the wrk output, including the transfer rate. Example of saturated network for a 40Gb/s NIC (Citrine).

[load] Transfer/sec:      4.37GB

These scenarios are running various SignalR benchmarks. The transport and serialization methods can be configured.

crank --config https://raw.githubusercontent.com/aspnet/Benchmarks/master/scenarios/signalr.benchmarks.yml --scenario signalr --profile aspnet-perf-lin --variable scenario=echo --variable transport=websockets --variable protocol=messagepack

• signalr

• Scenario:
  • --variable scenario=broadcast
  • --variable scenario=echo
  • --variable scenario=echoAll
• Transport:
  • --variable transport=websockets
  • --variable transport=serversentevents
  • --variable transport=longpolling
• Protocol:
  • --variable protocol=json
  • --variable protocol=messagepack

Note: MessagePack is not supported with ServerSentEvents

The following command lines assume that the job to configure is named application which should be the name used in most of the configuration defined in this document.

By default the pre-configured scenarios use what is called the current channel of .NET, which represents the latest public release, to ensure that these scenarios almost always work.

Other custom channels can be used:

• latest: which will use whatever SDK and runtime versions were used by the latest ASP.NET builds
• edge: which will use the latest available SDK and runtime versions and can potentially contain breaking changes that will make the builds to fail (though very rare).

Example:

Using the daily builds of .NET, and targeting net5.0.

--application.channel latest --application.framework net5.0

--application.options.outputFiles c:\build\System.Private.CoreLib.dll