Suggest introducing tags about dscresource.tests

[KarolKaczmarek]

Can you provide more details about your proposition? Perhaps including scenario where you need to target particular set of tests.

[mgreenegit]

A small example (please don't put too much in to this specific case though) would be separating the script analyzer tests from project-specific unit tests. A larger example in the future would be separating unit tests that can be run in persistent VM's from integration tests that require ephemeral VM's. This very well might just be 'planning ahead'.

[KarolKaczmarek]

If it's common tests vs unit tests vs integration, those are all in separate directories, so all you need to do is to call invoke-pester inside of it. I was wondering if you meant some more granular distinction, but if it's just that, I think we don't need tags, as it's very easy to target any of those currently.

[KarolKaczmarek]

That being said, looks like pester has tag support implemented already, so if we come up with need for targeting more specific sets of tests, that'd be very easy to do:
https://github.com/pester/Pester/wiki/Describe#tags

[PlagueHO]

I think I understand the reason for this - and agree the pester tags would be useful here.

An example:
With Distributed File System (DFS) DSC resources the integration test currently only cover testing a replication group with a single node. Ideally a proper integration test would create a rep group with two or more nodes. But clearly this isn't possible on Appveyor, but may be possible when testing locally.

Currently I have these tests defined, but they are commented out and you have to uncomment them for them to run. A better way to handle them would be to have define them with a tag say "NoAppVeyorRun" (or something similar) and change the AppVeyor.yml so that when it runs pester it passes the parameter -ExcludeTag "NoAppVeyorRun".

@mgreenegit - is this sort of what you had in mind? If so, @KarolKaczmarek is right, I don't think it needs any changes to implement.

[johlju]

I have started to add tags on some of the tests, those tests that are opt-in. But the tags are more for debugging purposes. So I don't have to run all the tests when changing a particular test, but instead can run that specific group of tests. We could add tags on the rest of the describe blocks as well. for debugging purposes.

Adding tags for bigger scenarios without have an actual use case seems unnecessary?

@PlagueHO We have started to use if ($env:APPVEYOR -eq $true) for those tests that should not run in AppVeyor. Is that enough? That could be used for both the common tests and tests for each repository.

To close this issue, should we add tags on the rest of the common tests describe blocks (for debugging purposes), or should we just close this issue, because it is not relevant anymore?

[PlagueHO]

@johlju - I think this issue is still relevant - but it has been a while? It would be good to get a user story together for this to figure out what is best.

Perhaps what @mgreenegit might have been talking about was defining a list of common tags like:

• Full
• Smoke

And then specifying which groups each of the core test should be included in?

But I'm not 100% sure. This was over a year ago after all (I can barely remember last week) 😁

[johlju]

Today we have ExcludeTag parameter on Invoke-AppveyorTestScriptTask. it would be easy enough to add a Tag parameter as [System.String[]] that is also added to the Pester arguments. While we are at it we could add TestName as [System.String[]] to, then there are options to run whatever you would like depending how the AppVeyor environment is configured.

[johlju]

The unit test template has -Tag added to each Describe-block. Since there are no clear scenario to cover above that, I'm closing this as 'not fixed'.