Remove PS5.0 requirement in Meta.Tests to run PSScriptAnalyzer test about dscresource.tests

[iainbrighton]

Yeah it's a PITA until they block publishing when there are any errors; then it's a problem! I'll ask the product team where this needs to be logged..

[iainbrighton]

@PlagueHO I've logged it here: https://windowsserver.uservoice.com/forums/301869-powershell/suggestions/13353099-pssa-errors-with-dsc-example-files

[KarolKaczmarek]

Thanks @iainbrighton ! We discussed this exact issue last week and there were different opinions on how this should be handled, so this is definitely a valuable feedback.

[iainbrighton]

Agreeable! Ideally we'd be testing on both v4 and v5?

[PlagueHO]

That would be ideal, but I have to admit I usually only test on v5.

I think all the repos are set to test on WMF5 AppVeyor boxes too. I haven't played with it much but I think AppVeyor can support dual environments - so possibly all resources could be validated in V4 and V5 environments. But I guess that is a bit off topic 😄

[KarolKaczmarek]

Currently only few repos test on WMF 5.
Please go ahead and send PR to remove this requirement though, as it's no longer needed.
The main task here would be to verify that PSSCriptAnalyzer test passes for all repos before merging it. Could you run it on your dev box first?

[PlagueHO]

Sure thing @KarolKaczmarek - I'll pull all Community repos onto my dev box and them a bulk test with the new version once completed.

[KarolKaczmarek]

thanks!

[PlagueHO]

Just batch running these tests now with the new version of DscResouce.Tests against all submodule of DSCResources (which is taking a long time).

I thought I'd post the script I used to batch run tests against all community DSC Resources in case it is useful to someone else (e.g. Issue #31, Issue #28):

Set-Location -Path $PSScriptRoot
git.exe submodule init
git.exe submodule update
$ResourcePath = "$PSScriptRoot\xDscResources"

$Resources = Get-ChildItem -Path $ResourcePath
foreach ($Resource in $Resources)
{
    # Pull the DSCResource.Tests repo to each Resource Module
    Set-Location -Path $($Resource.FullName)
    git.exe clone https://github.com/PowerShell/DscResource.Tests.git
    Write-Verbose -Verbose "Invoking tests on $($Resource.Name)"
    Invoke-Pester -TestName 'PowerShell DSC resource modules' -Script "$($Resource.Fullname)\DscResource.Tests\Meta.Tests.ps1" -Verbose
}

Put the above script into the DSCResources Repo root and run it - although in my case I was cloning my fork of the DscResource.Tests repo (not the PowerShell Org version).

[PlagueHO]

Unfortunately some of the existing resources are failing the PSSA tests:

So far It seems to be the PSAvoidUsingUserNameAndPassWordParams error only.

I'd actually suggest that this error is not actually valid for DSC resources anyway (well not valid for the *-TargetResource functions) and should be disabled - @iainbrighton, @KarolKaczmarek ? This error could still be valid for "helper" functions however. But the rule is either applied to all functions or none at all.

So, were this change to go in as it is some of the modules would need to be changed. Otherwise the PSAvoidUsingUserNameAndPassWordParams rule would need to be disabled.

[iainbrighton]

@PlagueHO I think the PSAvoidUsingNameAndPasswordParams rule will be/was tweaked in the next/a recent release of the PSSA?! This update was due for release in Feb by the looks of it. If it has been introduced and is still tripping the PSSA, someone needs to log a bug there /cc @KarolKaczmarek

My understanding is that if a parameter is named Password and of either a System.Security.SecureString or System.Management.Automation.PSCredential type, then that's OK (and rightly so).

[KarolKaczmarek]

@raghushantha Do you know what's the current state of PSAvoidUsingNameAndPasswordParams rule?

@PlagueHO As a temporary solution, feel free to disable this rule. We should get some basic set of rules passing for all resources and expand from there (that's why we don't report occurences of warnings currently)

[PlagueHO]

@iainbrighton - I'll check on which version that was running and confirm what the situation is with that one.

@KarolKaczmarek - once I have confirmed that I was running using the latest version I'll disable the rule (if still a problem). I have created a list of proposed warnings to ignore in issue #28.

[PlagueHO]

@KarolKaczmarek, @iainbrighton - I've had some time to do some more work on this one.

The most common PSSA rule that is being violated by resources is:

PSAvoidUsingConvertToSecureStringWithPlainText

In 90% of the cases with this violation are in Samples code in the samples folder. However, there are some violations in the Modules themselves (for example https://github.com/PowerShell/xWebAdministration/blob/dev/DSCResources/MSFT_xWebAppPool/MSFT_xWebAppPool.psm1#L95).

There are several ways to fix this:

1. Modify all samples in all affected repos to use an alternate method of generating a secure string that doesn't violate the rule:

$Password = New-Object -Type SecureString
[char[]] 'MyPassword' | % { $Password.AppendChar( $_ ) }

Or if the password is irrelevant/not actually used (e.g. as is the case in some unit tests):

$Password = New-Object -Type SecureString

1. Disable this specific rule in DSCResource.Tests
2. Exclude the Samples folder from PSSA scanning
   Are there any other options?

A summary of the repos where rules are violated:

• xWordPress - PSAvoidUsingConvertToSecureStringWithPlainText time
• xWebAdministration - 5 PSAvoidUsingConvertToSecureStringWithPlainText
• xSQLServer - 3 PSAvoidUsingConvertToSecureStringWithPlainText
• xSharePoint - 32 PSAvoidUsingConvertToSecureStringWithPlainText + 8 PSAvoidUsingNameAndPasswordParams errors
• xSCVMM - 5 PSAvoidUsingConvertToSecureStringWithPlainText
• xSCSR - 4 PSAvoidUsingConvertToSecureStringWithPlainText
• xSCSPF - 6 PSAvoidUsingConvertToSecureStringWithPlainText
• xSCSMA - 6 PSAvoidUsingConvertToSecureStringWithPlainText
• xSCOM - 8 PSAvoidUsingConvertToSecureStringWithPlainText
• xSCDPM - 6 PSAvoidUsingConvertToSecureStringWithPlainText
• xSafeHarbor - 4 PSAvoidUsingConvertToSecureStringWithPlainText
• xRemoteDesktopAdmin - 5 PSAvoidUsingConvertToSecureStringWithPlainText
• xPSDesiredStateConfiguration - 2 PSAvoidUsingConvertToSecureStringWithPlainText + 1 PSAvoidUsingComputerNameHardcoded These violations are in Modules
• xMySql - 6 PSAvoidUsingConvertToSecureStringWithPlainText
• xJea - 3 PSAvoidUsingConvertToSecureStringWithPlainText
• xExchange - 1 PSAvoidUsingConvertToSecureStringWithPlainText This error is in integration tests
• xDnsServer - 2 PSAvoidUsingConvertToSecureStringWithPlainText These violations are in Modules
• xComputerManagement - 1 PSAvoidUsingConvertToSecureStringWithPlainText This error is in unit tests - I have fixed this one because I was working on the resource anyway

Other notes:

• xFailOverCluster - has schema validation errors.
• xSqlPs - has schema validation errors.

I also note that some of these repos (xWebAdministration for example) are already failing AppVeyor builds because of these issues.

Any thoughts?

[iainbrighton]

@PlagueHO I can't find any SecureString references in the xDnsServer module? When running the latest version of PSSA, this is what I'm seeing:

[10] C:\.....\Resources\xDnsServer [dev]\> Invoke-ScriptAnalyzer -Path .\ -Verbose
VERBOSE: Analyzing file: C:\Users\Iain\Dropbox\PowerShell\DSC\Resources\xDnsServer\xDnsServer.psd1
VERBOSE: Running PSAvoidUsingCmdletAliases rule.
VERBOSE: Running PSAvoidDefaultValueSwitchParameter rule.
VERBOSE: Running PSAvoidGlobalVars rule.
VERBOSE: Running PSAvoidUsingEmptyCatchBlock rule.
VERBOSE: Running PSAvoidInvokingEmptyMembers rule.
VERBOSE: Running PSAvoidNullOrEmptyHelpMessageAttribute rule.
VERBOSE: Running PSAvoidUsingPositionalParameters rule.
VERBOSE: Running PSAvoidShouldContinueWithoutForce rule.
VERBOSE: Running PSReservedCmdletChar rule.
VERBOSE: Running PSAvoidDefaultValueForMandatoryParameter rule.
VERBOSE: Running PSReservedParams rule.
VERBOSE: Running PSAvoidUsingComputerNameHardcoded rule.
VERBOSE: Running PSAvoidUsingUserNameAndPassWordParams rule.
VERBOSE: Running PSAvoidUsingInvokeExpression rule.
VERBOSE: Running PSAvoidUsingPlainTextForPassword rule.
VERBOSE: Running PSAvoidUsingWriteHost rule.
VERBOSE: Running PSAvoidUsingConvertToSecureStringWithPlainText rule.
VERBOSE: Running PSAvoidUsingWMICmdlet rule.
VERBOSE: Running PSMisleadingBacktick rule.
VERBOSE: Running PSUseOutputTypeCorrectly rule.
VERBOSE: Running PSUseBOMForUnicodeEncodedFile rule.
VERBOSE: Running PSPossibleIncorrectComparisonWithNull rule.
VERBOSE: Running PSProvideCommentHelp rule.
VERBOSE: Running PSUseApprovedVerbs rule.
VERBOSE: Running PSUseCmdletCorrectly rule.
VERBOSE: Running PSUseDeclaredVarsMoreThanAssigments rule.
VERBOSE: Running PSUsePSCredentialType rule.
VERBOSE: Running PSShouldProcess rule.
VERBOSE: Running PSUseShouldProcessForStateChangingFunctions rule.
VERBOSE: Running PSUseUTF8EncodingForHelpFile rule.
VERBOSE: Running PSUseSingularNouns rule.
VERBOSE: Running PSAvoidUsingDeprecatedManifestFields rule.
VERBOSE: Running PSMissingModuleManifestField rule.
VERBOSE: Running PSUseToExportFieldsInManifest rule.

RuleName                            Severity     FileName   Line  Message
--------                            --------     --------   ----  -------
PSUseToExportFieldsInManifest       Warning      xDnsServer 27    Do not use wildcard or $null in this field. Explicitly
                                                 .psd1            specify a list for FunctionsToExport.
PSUseToExportFieldsInManifest       Warning      xDnsServer 30    Do not use wildcard or $null in this field. Explicitly
                                                 .psd1            specify a list for CmdletsToExport.

No mention of the PSAvoidUsingConvertToSecureStringWithPlainText rule?!

[PlagueHO]

@iainbrighton - Sorry, you're completely right. I was manually going through the errors on resources from Z to A and so xDNSServer was one of the last ones so I think I was getting a little bit cross-eyed near the end.

There errors are these:

C:\Users\Daniel\Source\GitHub\xDnsServer [dev]> Invoke-ScriptAnalyzer -path . -Recurse -Severity Error

RuleName                            Severity     FileName   Line  Message
--------                            --------     --------   ----  -------
PSAvoidUsingComputerNameHardcoded   Error        MSFT_xDnsA 68    The ComputerName parameter of cmdlet
                                                 Record.psm       'Add-DnsServerResourceRecordA' is hardcoded. This will
                                                 1                expose sensitive information about the system if the script
                                                                  is shared.
PSAvoidUsingComputerNameHardcoded   Error        MSFT_xDnsA 72    The ComputerName parameter of cmdlet
                                                 Record.psm       'Remove-DnsServerResourceRecord' is hardcoded. This will
                                                 1                expose sensitive information about the system if the script
                                                                  is shared.

Also, I am using PSSA 1.5.0.0.

I'm also using the script I mentioned earlier to perform a bulk validation on all the modules using the actual Pester tests. I wasn't running PSSA directly (except for the text I pasted above).

[iainbrighton]

Is there no way to exclude a directory from PSSA scanning? Should this be added as an issue?

As a workaround can we not enumerate the files we want checking with Get-Item and exclude the \Examples folder rather than scanning the whole directory structure?

[PlagueHO]

For the specific xDNSServer it is actually easy enough to fix as it isn't in samples like I first though (been a long day) - get rid of the -ComputerName "Localhost" from these lines:
https://github.com/PowerShell/xDnsServer/blob/dev/DSCResources/MSFT_xDnsARecord/MSFT_xDnsARecord.psm1#L68
https://github.com/PowerShell/xDnsServer/blob/dev/DSCResources/MSFT_xDnsARecord/MSFT_xDnsARecord.psm1#L72

I'm fairly certain they are not required anyway.

@iainbrighton, I agree, we could prevent scanning of the Samples and Examples folders. That would eliminate a large number of these issues. It would be easy enough to enumerate the core folders/files and then call PSSA on each folder individually. Definitely no problem there! @KarolKaczmarek - would this be an acceptable solution to this?

We could then see what was left over after that (which should be under 20 issues).

[KarolKaczmarek]

We want examples to be as high quality as possible, so we would like to keep scanning those as well.
@mbreakey3 is looking into PSSA failures and will be fixing them in the following days.

[PlagueHO]

@KarolKaczmarek - Nice! Thanks!

[iainbrighton]

I seem to have problems with example DSC configurations that use the Import-DscResource with external resources. These raise errors when the example DSC resources aren't available.

[PlagueHO]

@iainbrighton - is that rule Warning level or Error level? I get e-mailed those from PSGallery for my submitted modules and from memory they were Warning level (might be wrong though).

[PlagueHO]

I've attached the full output from the PSSA run on all modules for easy reference (rather than relying on my summary). Sorry about the formatting.
Output.txt

[iainbrighton]

Yes - the PS Gallery and presumably the MS DSC resources suffer the same fate too? They look like errors to me:

Parse error in file ...\Documents\WindowsPowerShell\Modules\Lability\0.9.8\DSCResources\xHyper-V\Examples\Sample_xVhdFileExamples.ps1: The DSC engine could not load the module 'xHyper-V'. It was not found on the system at line 27 column 9

I'm not sure that this needs to logged here though. Where do PS Gallery issues needed to be logged?! User voice?

[PlagueHO]

@iainbrighton - Yes, those are the ones I get too. PITA, but I guess with good intention. All my sample DSC configs throw so I get pages of them on every release.

There needs to be a method to acknowledge and override some PSSA rules in PSGallery for specific projects.

Good question about where to log enhancements - MSFT should release the PSGallery source on GitHub so we can have at it 😉

[PlagueHO]

Awesome @iainbrighton - have upvoted!

[johlju]

@PlagueHO What's the status of this issue? Still a problem?

[PlagueHO]

@johlju - I think we can close this one - pretty sure it is resolved.