Closed
Description
First time observed on #62072.
E2E tests appear green, even though they did not run, they finish in ~15 min with logs like:
Generating script.
Script contents:
.dotnet/dotnet test ./src/Components/test/E2ETest -c Release --no-build --filter 'Quarantined!=true|Quarantined=false' -p:VsTestUseMSBuildOutput=false --logger:"trx%3BLogFileName=Microsoft.AspNetCore.Components.E2ETests.trx" --logger:"html%3BLogFileName=Microsoft.AspNetCore.Components.E2ETests.html" --results-directory /home/vsts/work/1/s/artifacts/TestResults/Release/Unquarantined
========================== Starting Command Output ===========================
/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/d72768ca-7d59-4680-845c-8f412e1a9f2f.sh
Test run for /home/vsts/work/1/s/src/Components/test/E2ETest/bin/Release/net10.0/Microsoft.AspNetCore.Components.E2ETests.dll (.NETCoreApp,Version=v10.0)
VSTest version 17.15.0-preview-25272-112 (x64)
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
No test matches the given testcase filter `Quarantined!=true|Quarantined=false` in /home/vsts/work/1/s/src/Components/test/E2ETest/bin/Release/net10.0/Microsoft.AspNetCore.Components.E2ETests.dll
Results File: /home/vsts/work/1/s/artifacts/TestResults/Release/Unquarantined/Microsoft.AspNetCore.Components.E2ETests.trx
Html test results file : /home/vsts/work/1/s/artifacts/TestResults/Release/Unquarantined/Microsoft.AspNetCore.Components.E2ETests.html
Finishing: Run E2E tests
or
/usr/bin/dotnet --version
6.0.428 [/usr/share/dotnet/sdk]
8.0.410 [/usr/share/dotnet/sdk]
9.0.203 [/usr/share/dotnet/sdk]
The command could not be loaded, possibly because:
* You intended to execute a .NET application:
The application '--version' does not exist.
* You intended to execute a .NET SDK command:
A compatible .NET SDK was not found.
Requested SDK version: 10.0.100-preview.6.25272.112
global.json file: /home/vsts/work/1/s/global.json
Installed SDKs:
Install the [10.0.100-preview.6.25272.112] .NET SDK or update [/home/vsts/work/1/s/global.json] to match an installed SDK.
Learn about SDK resolution:
https://aka.ms/dotnet/sdk-not-found
Result Attachments will be stored in LogStore
Run Attachments will be stored in LogStore
No Result Found to Publish '/home/vsts/work/1/s/artifacts/TestResults/Release/Unquarantined/Microsoft.AspNetCore.Components.E2ETests.trx'.
Async Command Start: Publish test results
Async Command End: Publish test results
Finishing: Publish E2E Test Results
Known workarounds:
Use eng/build.sh -test or eng/build.ps1 -test script instead of dotnet test. Trying to leverage it in #62222.
Investigation:
One of the hypothesis about the offender is code xunit that reads test's deps.json:
https://github.com/xunit/visualstudio.xunit/blob/e1ace41fc55f2b92fff7779ac9bdcd0b7b2265c8/src/xunit.runner.visualstudio/VsTestRunner.cs#L346-L363
We can see a few differences between the working deps.json and failing ones. Packages: "xunit.analyzers", "xunit.runner.console" and "xunit.runner.visualstudio" are missing in the failing file, see diff1, diff2. When comparing obj/project.assets.json, they look the same for the mentioned packages diff.
When we run the tests on CI using the workaround, we are getting an error:
artifacts
The application to execute does not exist: '/home/vsts/work/1/s/.packages/xunit.runner.console/2.9.2/tools/netcoreapp2.0/xunit.console.dll'
=== COMMAND LINE ===
"/home/vsts/work/1/s/.dotnet/dotnet" exec --depsfile "/home/vsts/work/1/s/src/Components/test/E2ETest/bin/Release/net10.0/Microsoft.AspNetCore.Components.E2ETests.deps.json" --runtimeconfig "/home/vsts/work/1/s/src/Components/test/E2ETest/bin/Release/net10.0/Microsoft.AspNetCore.Components.E2ETests.runtimeconfig.json" "/home/vsts/work/1/s/.packages/xunit.runner.console/2.9.2/tools/netcoreapp2.0/xunit.console.dll" "/home/vsts/work/1/s/src/Components/test/E2ETest/bin/Release/net10.0/Microsoft.AspNetCore.Components.E2ETests.dll" -noautoreporters -xml "/home/vsts/work/1/s/artifacts/TestResults/Release/Microsoft.AspNetCore.Components.E2ETests_net10.0_x64.xml" -html "/home/vsts/work/1/s/artifacts/TestResults/Release/Microsoft.AspNetCore.Components.E2ETests_net10.0_x64.html" -notrait "Quarantined=true" -nocolor > "/home/vsts/work/1/s/artifacts/log/Release/Microsoft.AspNetCore.Components.E2ETests_net10.0_x64.log" 2>&1
and in the binlog none of projects that we want to import exists:
<ImportGroup Condition=" '$(ExcludeRestorePackageImports)' != 'true' ">
<Import Project="$(NuGetPackageRoot)xunitxml.testlogger/2.1.26/build/netstandard1.0/XunitXml.TestLogger.props" Condition="Exists('$(NuGetPackageRoot)xunitxml.testlogger/2.1.26/build/netstandard1.0/XunitXml.TestLogger.props')" />
<Import Project="$(NuGetPackageRoot)xunit.runner.visualstudio/2.8.2/build/net6.0/xunit.runner.visualstudio.props" Condition="Exists('$(NuGetPackageRoot)xunit.runner.visualstudio/2.8.2/build/net6.0/xunit.runner.visualstudio.props')" />
<Import Project="$(NuGetPackageRoot)xunit.runner.console/2.9.2/build/xunit.runner.console.props" Condition="Exists('$(NuGetPackageRoot)xunit.runner.console/2.9.2/build/xunit.runner.console.props')" />
<Import Project="$(NuGetPackageRoot)xunit.core/2.9.2/build/xunit.core.props" Condition="Exists('$(NuGetPackageRoot)xunit.core/2.9.2/build/xunit.core.props')" />
<Import Project="$(NuGetPackageRoot)microsoft.extensions.configuration.usersecrets/10.0.0-preview.6.25302.103/buildTransitive/net8.0/Microsoft.Extensions.Configuration.UserSecrets.props" Condition="Exists('$(NuGetPackageRoot)microsoft.extensions.configuration.usersecrets/10.0.0-preview.6.25302.103/buildTransitive/net8.0/Microsoft.Extensions.Configuration.UserSecrets.props')" />
<Import Project="$(NuGetPackageRoot)microsoft.testplatform.testhost/17.1.0-preview-20211109-03/build/netcoreapp2.1/Microsoft.TestPlatform.TestHost.props" Condition="Exists('$(NuGetPackageRoot)microsoft.testplatform.testhost/17.1.0-preview-20211109-03/build/netcoreapp2.1/Microsoft.TestPlatform.TestHost.props')" />
<Import Project="$(NuGetPackageRoot)microsoft.codecoverage/17.1.0-preview-20211109-03/build/netstandard1.0/Microsoft.CodeCoverage.props" Condition="Exists('$(NuGetPackageRoot)microsoft.codecoverage/17.1.0-preview-20211109-03/build/netstandard1.0/Microsoft.CodeCoverage.props')" />
<Import Project="$(NuGetPackageRoot)microsoft.net.test.sdk/17.1.0-preview-20211109-03/build/netcoreapp2.1/Microsoft.NET.Test.Sdk.props" Condition="Exists('$(NuGetPackageRoot)microsoft.net.test.sdk/17.1.0-preview-20211109-03/build/netcoreapp2.1/Microsoft.NET.Test.Sdk.props')" />
</ImportGroup>
Reverting changes from dotnet/sdk#46218 fixes the issue.