powershell / dscconfiguration.tests

Test automation scripts for evaluating the quality of DSC Configurations using Azure virtual machines and Azure Automation DSC.

DSCConfiguration.Tests

This repository provides test automation scripting that are intended to accelerate iterative authoring of DSC Configurations by hosting tests on Azure.

Versions

Unreleased

• README.MD:
  • Fixed markdown rule violations.
  • Added Change Log.
• Added support for specifying the Azure Data center location to use by defining the $ENV:Location environment variable. Will default to 'EastUS2' if not specified.
• NUnit Pester Test results uploaded to AppVeyor as artifacts.
• Updated New-ResourceGroupandAutomationAccount to automatically register Microsoft.Automation resource provider.

dscconfiguration.tests's Issues

Test reference wrong path for property WindowsOSVersion in the module manifest

On this line 428 in DSCConfiguration.Tests.ps1 it reference .PrivateData.PSData.WindowsOSversion.

$OSVersion = (Import-PowerShellDataFile $env:BuildFolder\$ProjectModuleName\$ProjectModuleName.psd1).PrivateData.PSData.WindowsOSVersion

Instead I think is should be

$OSVersion = (Import-PowerShellDataFile $env:BuildFolder\$ProjectModuleName\$ProjectModuleName.psd1).PrivateData.WindowsOSVersion

Can only load one required module from module manifest

When having a configuration with more than one required module, only the first required module is loaded.

This is because on line 47 we single out only the first.

foreach($RequiredModule in $ManifestData.RequiredModules[0])
{
    ...
}

This should be

foreach($RequiredModule in $ManifestData.RequiredModules)
{
    ...
}

Need administrator rights when calling Install-Module

When running locally the user can not install modules if the user is not running the PowerShell console in elevated mode.

Instead of require administrator right, the cmdlet Install-Module should be called with -Scope CurrentUser.

Need to Ensure Microsoft.Automation is Registered in Azure Subscription

Although it is easy enough to register Microsoft.Automation in a subscription it would be better if New-ResourceGroupandAutomationAccount in TestHelper.psm1 did this automatically.

@mgreenegit - is it worth me submitting PR's to fix some of these issues I'm finding?

Change so that the Azure AD Application account only need access to the resource group

I suggest that this repository do not create the resource group, but instead expects that the resource group is already created with the correct Contributor permission in the resource group.

Creating the resource group and setting the correct permission can be added to the helper script that @PlagueHO suggested to add here PowerShell/DscConfigurations#5.

Make the dev branch the default pull request branch

Currently the master branch is set to default when making pull request. I suggest making the dev branch default instead.

Question: The "acceptance" tests, when/how are they suppose to be run?

I can't see these "acceptance" tests are being run. I talking about these tests in the template here; https://github.com/PowerShell/TemplateConfig/tree/master/TemplateConfigModule/Tests

I added a dummy test to the repository here;
https://github.com/johlju/xFailOverClusterIntegration/blob/dev/xFailOverClusterIntegrationModule/Tests/Acceptance/xFailOverClusterIntegration.Acceptance.Tests.ps1

But can't see that it was run during testing.
https://ci.appveyor.com/project/johlju/xfailoverclusterintegration/build/1.0.5.1

Is it meant to be run with the other tests?
Or was the idea that these test are run by the "configuration module"? Is so, was there an idea on how to run these tests?

Add a test for WindowsOSVersion in manifest

I had accidentally removed the WindowsOSVersion from the manifest of the DSC Configuration I was testing with. A test should be added to ensure it is there. It could be added in Common Tests - Configuration Module Requirements.

I can submit a PR for this.

Use Push-AppveyorArtifact to upload test results to AppVeyor

Is it possible to use the Push-AppVeyorArtifact cmdlet (available in AppVeyor) to publish the results of testing artifacts instead of the WebClient with a URL specified in the AppVeyor.yml?

This would slightly simplify the code and eliminate the need for the TestResultsUploadURI in the AppVeyor.yml file in each DSC Configuration repo.

The only reason to keep this as WebClient upload would be to allow uploading from outside AppVeyor or to another destination, but I don't know if this was a requirement.

I can submit a PR with this change if acceptable.

Add ability to change Azure resource deployment location

Some Azure subscriptions don't allow VM's in some locations depending on the tier.

For example:
https://ci.appveyor.com/project/PlagueHO/dockerenterprisedscconfig/build/1.0.6.0#L186

Therefore it would be good to be able to allow overriding of the Azure Resource Group location by specifying an $env: parameter which could be set via AppVeyor.yml.

I can submit a PR for this as well.

Unable to run locally without setting environment variables

To be able to run this locally the environment variables need to be set to mimic the behavior of AppVeyor.

I ended up creating a script Start-Build.ps1 in the root folder that sets the variables before running Invoke-Build.
This is not an optimal idea since this script generate a varning in the unit test that Invoke-Build is an alias, and should be change to point to the full script path 'C:\Program Files\WindowsPowerShell\Modules\InvokeBuild\3.6.4\Invoke-Build.ps1'. Since the version can change, it's not possible to change to the full path.

$env:TenantID = '<guid>'
$env:SubscriptionID = '<guid>'
$env:ApplicationID = '<guid>'
$env:ApplicationPassword = '<key/password>'
$env:Location = 'westeurope' # Used by @PlagueHO's branch (PR #21).

$env:BuildFolder = $PSScriptRoot
$env:ProjectName = Split-Path -Path $PSScriptRoot -Leaf
$env:ProjectID = New-Guid # This is used to create a folder in $env:TEMP.
$env:BuildID = 1001 # Any number

Invoke-Build -File .\DscConfiguration.Tests\.build.ps1 -Summary

Installing module Pester fails in Windows Server 2016 AppVeyor build worker

When using a AppVeyor build worker with Windows Server 2016 then Pester must be installed using the parameter -SkipPublisherCheck.

PackageManagement\Install-Package : The version '4.0.5' of the module 'Pester' being installed is not catalog signed. Ensure that the version '4.0.5' of the module 'Pester' has the catalog file 'Pester.cat' and signed with the 
same publisher 'CN=Microsoft Root Certificate Authority 2010, O=Microsoft Corporation, L=Redmond, S=Washington, C=US' as the previously-installed module '4.0.5' with version '3.4.0' under the directory 'C:\Program 
Files\WindowsPowerShell\Modules\Pester\3.4.0'. If you still want to install or update, use -SkipPublisherCheck parameter.
At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:1772 char:21
+ ...          $null = PackageManagement\Install-Package @PSBoundParameters
+                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (Microsoft.Power....InstallPackage:InstallPackage) [Install-Package], Exception
    + FullyQualifiedErrorId : ModuleIsNotCatalogSigned,Validate-ModuleAuthenticodeSignature,Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackage

Pester test result file is not cleaned up and fails to upload test result file

When running locally, the Pester test result is not removed, which results in unit tests failing on second run.
Also when running locally there is no point in trying to upload the Pester test result file to AppVeyor.

I suggest that the lines 107-108, lines 215-216 and the lines 243-244 is replaced respectively with

if ($env:APPVEYOR -eq $true)
{
    (New-Object 'System.Net.WebClient').UploadFile("$env:TestResultsUploadURI", `
    (Resolve-Path $testResultsFile))
}

Remove-Item $testResultsFile

When a build task fails the console window is closed

When running locally and a build task fails, for example when failing to logon to Azure correctly, then the console window closes and the error message disappears.

This is a little annoying behavior when trying to problem solve. 😄

I managed to take a screenshot before the window closed.