microsoft / vstest-action

GitHub Action equivalent to the Azure DevOps VS Test Task

VSTest GitHub Action

This action was created to run tests using VSTest framework and to easily migrate a pipeline using Azure DevOps VSTest task. Most of the commonly used properties in the Azure DevOps VSTest task map to properties of this GitHub action. Like the Azure DevOps VSTest task, this action only supports Windows but NOT Linux.

Due to the unavailability of a Test results UI, test results are displayed in the console logs of the action.

Usage

See action.yml

Example

jobs:
  my_action_job:
    runs-on: windows-latest
    name: A job to run VSTest
    steps:
      - name: Download tests binary zip
        run: powershell Invoke-WebRequest -Uri "https://localworker.blob.core.windows.net/win-x64/tests.zip" -OutFile "./tests.zip"
      - name: Unzip tests binary
        run: powershell Expand-Archive -Path tests.zip -DestinationPath ./
      - name: Run tests
        uses: microsoft-approved-actions/vstest@master
        with:
          testAssembly: CloudTest.DefaultSamples*.dll
          searchFolder: ./tests/
          runInParallel: true

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines.

Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

vstest-action's Issues

Action fails when assembly path contains spaces.

Action fails when assembly path contains spaces. This is because the paths are not surrounded by quotes before provided to vstest.

https://github.com/microsoft/vstest-action/blob/main/src/index.ts#L36

Error: Unable to resolve action microsoft-approved-actions/vstest, repository not found

Running this action twice in the same job generates a huge number of errors

We're using this action twice in a GH job to run two sets of tests: https://github.com/Azure/bicep/blob/ebead5597e42d9c227a845fc86047ef6d609cb99/.github/workflows/build.yml#L177-L190

The second time it runs, it appears that it is trying to overwrite the original folder, which generates an error for every file:

Downloading test tools...
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe Invoke-WebRequest -Uri https://aka.ms/local-worker-win-x64 -OutFile D:\a\_actions\microsoft\vstest-action\v1.0.0\dist\win-x64.zip
Unzipping test tools...
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe Expand-Archive -Path D:\a\_actions\microsoft\vstest-action\v1.0.0\dist\win-x64.zip -DestinationPath D:\a\_actions\microsoft\vstest-action\v1.0.0\dist
ExpandArchiveHelper : Failed to create file 
'D:\a\_actions\microsoft\vstest-action\v1.0.0\dist\win-x64\VsTest\v140\ActivateApplication.exe' while expanding the 
archive file 'D:\a\_actions\microsoft\vstest-action\v1.0.0\dist\win-x64.zip' contents as the file 
'D:\a\_actions\microsoft\vstest-action\v1.0.0\dist\win-x64\VsTest\v140\ActivateApplication.exe' already exists. Use 
the -Force parameter if you want to overwrite the existing directory 
'D:\a\_actions\microsoft\vstest-action\v1.0.0\dist\win-x64\VsTest\v140\ActivateApplication.exe' contents when 
expanding the archive file.
At 
C:\Windows\system32\WindowsPowerShell\v1.0\Modules\Microsoft.PowerShell.Archive\Microsoft.PowerShell.Archive.psm1:3[9](https://github.com/Azure/bicep/runs/7781476281?check_suite_focus=true#step:8:10)7 
char:17
+ ...             ExpandArchiveHelper $resolvedSourcePaths $resolvedDestina ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (D:\a\_actions\m...Application.exe:String) [Write-Error], IOException
    + FullyQualifiedErrorId : ExpandArchiveFileExists,ExpandArchiveHelper
 
ExpandArchiveHelper : Failed to create file 'D:\a\_actions\microsoft\vstest-action\v1.0.0\dist\win-x64\VsTest\v[14](https://github.com/Azure/bicep/runs/7781476281?check_suite_focus=true#step:8:15)0\A\B\
PrivateAssemblies\DataCollectors\Microsoft.VisualStudio.TraceCollector.dll' while expanding the archive file 
'D:\a\_actions\microsoft\vstest-action\v1.0.0\dist\win-x64.zip' contents as the file 'D:\a\_actions\microsoft\vstest-ac
tion\v1.0.0\dist\win-x64\VsTest\v140\A\B\PrivateAssemblies\DataCollectors\Microsoft.VisualStudio.TraceCollector.dll' 
already exists. Use the -Force parameter if you want to overwrite the existing directory 'D:\a\_actions\microsoft\vstes
t-action\v1.0.0\dist\win-x64\VsTest\v140\A\B\PrivateAssemblies\DataCollectors\Microsoft.VisualStudio.TraceCollector.dll
' contents when expanding the archive file.
At 
C:\Windows\system32\WindowsPowerShell\v1.0\Modules\Microsoft.PowerShell.Archive\Microsoft.PowerShell.Archive.psm1:397 
char:[17](https://github.com/Azure/bicep/runs/7781476281?check_suite_focus=true#step:8:18)
+ ...             ExpandArchiveHelper $resolvedSourcePaths $resolvedDestina ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (D:\a\_actions\m...ceCollector.dll:String) [Write-Error], IOException
    + FullyQualifiedErrorId : ExpandArchiveFileExists,ExpandArchiveHelper

To solve this, could we either have an option to:

• Silence the errors
• Unzip to a unique location each time the action runs

Doesn't fail if no tests are found

If there are no tests found, it spits out a warning, but doesn't fail.

It seems somewhat obvious to me that if I'm running the action, there are some tests. So if it's not finding any, it should fail.

There is the failOnMinTestsNotRun but that defaults to false, it should really default to true.

No matched test files! for project compiled with msbuild (.NET 4)

In my workflow file I have defined the following step for test execution, but I get back an "No matched test files" error back:

- name: vstest-action
  uses: microsoft/[email protected]
  with:
    # Run tests from the specified files
    # testAssembly: # default is **\\*test*.dll\n!**\\*TestAdapter.dll\n!**\\obj\\**
    # Folder to search for the test assemblies
      searchFolder: "D:\\a\\<myapplication>\\source\\clientapp\\<mytestproject>\\bin\\Release\\"

all other parameters contain default values.

I have verified the test .dll file is compiled and available by looking at the logs.
To make sure, I have also fully specified the absolute path where the test .dll is located.

I am wondering if the reason is that the test assembly is .NET 4.0 and it was built using microsoft/[email protected] action.

Any ideas on what is the problem or how can I further isolate it?

Thanks!

Update VS test tools used by action

The VS test tools used by this action are a fixed download from https://aka.ms/local-worker-win-x64. This needs to be updated to support as the version for VS 2019 is missing assemblies.

Running the same dll twice

I'm using this action, but my dll is running twice where it contains the tests.
When I analyzed the execution I verified that the DLL is twice

D:\a\_actions\microsoft\vstest-action\v1.0.0\dist\win-x64\VsTest\v160\Common7\IDE\Extensions\TestPlatform\vstest.console.exe 
D:\a\web-application-full-framework\web-application-full-framework\src\WebApplicationFullFramework.UnitTest\bin\Release\WebApplicationFullFramework.UnitTest.dll 
D:\a\web-application-full-framework\web-application-full-framework\src\WebApplicationFullFramework.UnitTest\obj\Release\WebApplicationFullFramework.UnitTest.dll /Logger:TRX

My action is configure like bellow

- name: Run tests
    uses: microsoft/[email protected]
    with:
      testAssembly: '**\\WebApplicationFullFramework.UnitTest.dll'
      searchFolder: src\WebApplicationFullFramework.UnitTest\

Bug: passing path to installed VsTest.console.exe produces error.

As a workaround to the newer version of VsTest not being used by this action, I resorted to attempting to use the vstestLocation input argument.

VSWHERE correctly returns a path of:

C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe. 

Because the long path names include spaces, passing this value through vstestLocation input argument produces error:

Error: Unable to locate executable file: C:\Program. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also verify the file has a valid extension for an executable file.

Release Version: 1.0.0

How to reproduce:

1. In your workflow add a powershell step containing:

- id: setup
  run: |
    $vswhere=vswhere -latest -products * -requires Microsoft.VisualStudio.Workload.ManagedDesktop     Microsoft.VisualStudio.Workload.Web -requiresAny -property installationPath
    $vswhere=join-path $vswhere 'Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe'
    if (test-path $vswhere) {
      write-output "vstest-path=$vswhere" >> $env:GITHUB_OUTPUT
    } else {
      write-output "::warning::Unable to locate VSTest.console.exe"
    }
  shell: pwsh

2. use vstest-action passing vstestLocation argument

      - name: Run vstests
        uses: microsoft/[email protected]
        with:
          searchFolder: '${{github.workspace}}/out'
          vstestLocationMethod: location
          vstestLocation: '${{ steps.setup.outputs.vstest-path }}'
          testAssembly: |
              **/*test*.dll
              !./**/*TestAdapter.dll
              !./**/obj/**

3. run workflow and observe error

Typo in code sample in README.md

The code example contains the line

uses: microsoft-approved-actions/vstest@master

Shouldn't that be

uses: microsoft/vstest@master

... ?

Deprecation warnings due to use of node12

Using this action results in a deprecation warning:

Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/. Please update the following actions to use Node.js 16: microsoft/vstest-action

I have opened #10 with a suggestion to fix