Problem matcher support for Context errors

[tillig]

PR Summary

Fixes #2438 - allows errors that occur in Pester test context to be shown in the Problems window just like other test failures. Solution is a simple regex fix in the problem matcher for line number/location.

PR Checklist

Note: Tick the boxes below that apply to this pull request by putting an x between the square brackets.
Please mark anything not applicable to this PR NA.

• PR has a meaningful title
• Summarized changes
• N/A - PR has tests
• This PR is ready to merge and is not work in progress

I didn't see that there were any existing problem matcher tests. If there are, I'd be happy to try adding some updates to them.

[tillig]

For reference, I tested this on MacOS on Pester tests that have both context and test failures. The console log looks like this:

> Executing task: Invoke-Pester -PesterOption @{IncludeVSCodeMarker=$true} <

    ____            __
   / __ \___  _____/ /____  _____
  / /_/ / _ \/ ___/ __/ _ \/ ___/
 / ____/  __(__  ) /_/  __/ /
/_/    \___/____/\__/\___/_/
Pester v4.9.0
Executing all tests in '.'

Executing script /Users/tillig/dev/powershell-test/has space/Failing.Tests.ps1

  Describing Reproduce Issue

    Context Failing BeforeAll in Context block
Get-Command: /Users/tillig/dev/powershell-test/has space/Failing.Tests.ps1:9:30
Line |
   9 |              $pathToCommand = Get-Command nosuchcommand.exe | Select-O …
     |                               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The term 'nosuchcommand.exe' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling
     | of the name, or if a path was included, verify that the path is correct and try again.

      [-] Error occurred in Context block 0ms
        At /Users/tillig/dev/powershell-test/has space/Failing.Tests.ps1:10 char:14
        The term 'othermissingcommand' is not recognized as the name of a cmdlet, function, script file, or operable program.
        Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
    [-] has two failures - context and test 76ms
      at <ScriptBlock>, /Users/tillig/dev/powershell-test/has space/Failing.Tests.ps1: line 14
      14:         $False | Should -Be $True
      Expected $true, but got $false.

Executing script /Users/tillig/dev/powershell-test/Failing.Tests.ps1

  Describing Reproduce Issue

    Context Failing BeforeAll in Context block
Get-Command: /Users/tillig/dev/powershell-test/Failing.Tests.ps1:9:30
Line |
   9 |              $pathToCommand = Get-Command nosuchcommand.exe | Select-O …
     |                               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | The term 'nosuchcommand.exe' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling
     | of the name, or if a path was included, verify that the path is correct and try again.

      [-] Error occurred in Context block 0ms
        At /Users/tillig/dev/powershell-test/Failing.Tests.ps1:10 char:14
        The term 'othermissingcommand' is not recognized as the name of a cmdlet, function, script file, or operable program.
        Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
    [-] has two failures - context and test 7ms
      at <ScriptBlock>, /Users/tillig/dev/powershell-test/Failing.Tests.ps1: line 14
      14:         $False | Should -Be $True
      Expected $true, but got $false.
Tests completed in 686ms
Tests Passed: 0, Failed: 4, Skipped: 0, Pending: 0, Inconclusive: 0 

The failing test fixture looks like this:

Describe "Reproduce Issue" {
    Context "Failing BeforeAll in Context block" {
        BeforeAll {
            # In a "real" scenario, this is
            # $pathToNuGet = Get-Command nuget.exe | Select-Object -ExpandProperty Source
            # &mono $pathToNuGet do-some-things-with-nuget.exe
            # to execute NuGet on a Linux/OS X box. The failure I'm seeing in the context
            # happens when mono isn't available, and possibly nuget.exe is also not there.
            $pathToCommand = Get-Command nosuchcommand.exe | Select-Object -ExpandProperty Source
            &othermissingcommand $pathToCommand
        }
    }
    It "has two failures - context and test" {
        $False | Should -Be $True
    }
}

I had two instances of this, one in the top-level folder, one in a folder called has space so I could verify spaces in filenames wouldn't mess things up.

I did have to create a tasks.json and run the Pester tests using that because the current "Run Pester tests" from the extension doesn't have the "IncludeVSCodeMarker" part.

    {
      "command": "Invoke-Pester -PesterOption @{IncludeVSCodeMarker=$true}",
      "group": {
        "isDefault": true,
        "kind": "test"
      },
      "label": "test",
      "options": {
        "cwd": "${workspaceFolder}"
      },
      "problemMatcher": "$pester",
      "type": "shell"
    }

[tillig]

It appears the builds are failing due to API rate limits being exceeded. I'm not sure there's anything I can do about that.

[TylerLeonhardt]

Yeah that CI failure can be ignored.

[TylerLeonhardt]

LGTM thanks for fixing this!

@nohwnd this is what we will need to change for Pester 5. We can probably just add another problemMatcher and call it pester5 instead of pester or something.

[nohwnd]

Cool, I think Pester v5 already prints errors like this, but I will have to check if it triggers the same problem matcher. Is there an environment variable that can be used to detect we are running in VSCode? I would like to do that rather than passing the parameter (even though it is implemented already). That would align better with modifying the output for azure devops / other build services, which is planned later.

[SeeminglyScience]

@nohwnd

Is there an environment variable that can be used to detect we are running in VSCode?

# Any terminal in VSCode
$env:TERM_PROGRAM -eq 'vscode'

# Specifically this extension's terminal
$env:TERM_PROGRAM -eq 'vscode' -and $psEditor

[TylerLeonhardt]

Since Tasks don't run in the PS Integrated Console, you should just check the TERM_PROGRAM