Skip to content

2.6.1: OpenSSL handshake fails with QUIC_STATUS_TLS_ERROR when ServerName is an IP literal or not set (regression from #6274) #6288

Description

@externl

Describe the bug

Since 2.6.1, on the OpenSSL backend, a client connection whose ServerName is an IP literal (e.g. 127.0.0.1) — or not set at all, in which case msquic falls back to the remote IP address and hits the same path — fails the handshake with QUIC_STATUS_TLS_ERROR. The regression appears to come from #6275 (cherry-pick of #6274).

DNS ServerNames still connect and defer certificate validation to the application callback. An IP or unset ServerName fails before the callback ever runs — even though the certificate's SAN includes IP Address:127.0.0.1.

Affected OS

  • Windows
  • Linux
  • macOS
  • Other (specify below)

Additional OS information

Ubuntu 24.04 (libmsquic from packages.microsoft.com; verified by A/B on identical containers where only the libmsquic version changed) and macOS (Homebrew libmsquic). Windows/Schannel not tested — the OpenSSL backend is the affected path.

MsQuic version

v2.6.1 (v2.5.9 and v2.6.0 are not affected)

Steps taken to reproduce bug

  1. Save Program.cs (below; .NET 10, generates its own self-signed certificate with IP:127.0.0.1 and DNS:localhost SANs)
  2. Linux: apt install libmsquic && dotnet run Program.cs — macOS: brew install libmsquic && DYLD_LIBRARY_PATH=$(brew --prefix)/lib dotnet run Program.cs
Program.cs
// msquic 2.6.1 regression (OpenSSL backend): the handshake fails with QUIC_STATUS_TLS_ERROR when the
// client's ServerName is an IP literal, or is not set at all (msquic then falls back to the remote IP).
// With 2.6.0 all four cases below connect and certificate validation is deferred to the app callback.
using System.Net;
using System.Net.Quic;
using System.Net.Security;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;

using var key = RSA.Create(2048);
var request = new CertificateRequest("CN=repro", key, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
var san = new SubjectAlternativeNameBuilder();
san.AddIpAddress(IPAddress.Loopback);
san.AddDnsName("localhost");
request.CertificateExtensions.Add(san.Build());
using X509Certificate2 cert = request.CreateSelfSigned(
    DateTimeOffset.UtcNow.AddDays(-1), DateTimeOffset.UtcNow.AddDays(30));

foreach (string? targetHost in new string?[] { "127.0.0.1", null, "localhost", "wrong.example" })
{
    Console.Write($"TargetHost = {targetHost ?? "(null)"} ... ");
    try
    {
        await using var listener = await QuicListener.ListenAsync(new()
        {
            ListenEndPoint = new IPEndPoint(IPAddress.Loopback, 0),
            ApplicationProtocols = [new SslApplicationProtocol("test")],
            ConnectionOptionsCallback = (_, _, _) => ValueTask.FromResult(new QuicServerConnectionOptions
            {
                // Both error codes are required by System.Net.Quic: the app-defined codes sent to the
                // peer when a connection or stream is closed without an explicit code.
                DefaultCloseErrorCode = 0,
                DefaultStreamErrorCode = 0,
                ServerAuthenticationOptions = new()
                {
                    ApplicationProtocols = [new SslApplicationProtocol("test")],
                    ServerCertificateContext = SslStreamCertificateContext.Create(cert, null)
                }
            })
        });

        ValueTask<QuicConnection> accepted = listener.AcceptConnectionAsync();
        bool callbackInvoked = false;
        await using var client = await QuicConnection.ConnectAsync(new()
        {
            RemoteEndPoint = listener.LocalEndPoint,
            DefaultCloseErrorCode = 0,
            DefaultStreamErrorCode = 0,
            ClientAuthenticationOptions = new()
            {
                ApplicationProtocols = [new SslApplicationProtocol("test")],
                TargetHost = targetHost,
                RemoteCertificateValidationCallback = (_, _, _, _) => { callbackInvoked = true; return true; }
            }
        }).AsTask().WaitAsync(TimeSpan.FromSeconds(10));

        await using var server = await accepted.AsTask().WaitAsync(TimeSpan.FromSeconds(10));
        Console.WriteLine($"OK (validation callback invoked: {callbackInvoked})");
    }
    catch (Exception ex)
    {
        Console.WriteLine($"FAILED: {ex.Message}");
    }
}

Expected behavior

All four cases connect and certificate validation is deferred to the application callback, as with v2.6.0:

TargetHost = 127.0.0.1 ... OK (validation callback invoked: True)
TargetHost = (null) ... OK (validation callback invoked: True)
TargetHost = localhost ... OK (validation callback invoked: True)
TargetHost = wrong.example ... OK (validation callback invoked: True)

Actual outcome

TargetHost = 127.0.0.1 ... FAILED: Authentication failed: Status code: QUIC_STATUS_TLS_ERROR.
TargetHost = (null) ... FAILED: Authentication failed: Status code: QUIC_STATUS_TLS_ERROR.
TargetHost = localhost ... OK (validation callback invoked: True)
TargetHost = wrong.example ... OK (validation callback invoked: True)

Additional details

Hit via .NET's System.Net.Quic in icerpc/icerpc-csharp CI (icerpc/icerpc-csharp#4906); pinning libmsquic to 2.6.0 restores all tests.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions