gaelcolas / datum Goto Github PK

This has been done but not released yet.

The GitHub PSD1 file says the module version is 0.0.1.
https://github.com/gaelcolas/datum/blob/master/datum/datum.psd1

And when I type
Find-Module -Name Datum
PowerShell Gallery answers me that the module version is 0.0.38
but also the description says it is an experimental module…

The README.md says that v1.0.0 is aimed to be released for April 2018, or earlier…

Where can I find the latest stable version for production usage?

For certain scenarios it is required to access an environment variable to create the ResolutionPrecedence:

ResolutionPrecedence:
  - AllNodes\$($Node.Environment)\$($Node.NodeName)
  - ...
  - LCM\LCM_$($env:BuildLcmMode)
  - ...

This results in the following error when creating the RSOP:

Generating RSOP output for 13 nodes.
Join-Path : Cannot find drive. A drive with the name 'LCM\LCM_$($env' does not exist.
At D:\Git\DatumTest\DSC\BuildOutput\Modules\Datum\0.40.0\datum.psm1:1418 char:26
+         $CurrentSearch = Join-Path $SearchPrefix $PropertyPath
+                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (LCM\LCM_$($env:String) [Join-Path], DriveNotFoundException
    + FullyQualifiedErrorId : DriveNotFound,Microsoft.PowerShell.Commands.JoinPathCommand

ERROR: Exception calling "Replace" with "4" argument(s): "Value cannot be null.
Parameter name: input"
At D:\Git\DatumTest\DSC\BuildOutput\Modules\Datum\0.40.0\ScriptsToProcess\Resolve-NodeProperty.ps1:58 char:18

It is not documented but one would expect that removing values using the knockout prefix would work with base type arrays.

Windows Features defined in the role:

WindowsFeatures:
  Names:
    - File-Services
    - Feature1

Windows Features defined on the node:

WindowsFeatures:
  Names:
    - --Feature1

Actual RSOP:

WindowsFeatures:
  Names:
  - File-Services                                                                               AllNodes\Dev\DSCFile01
  - --Feature1                                                                                  AllNodes\Dev\DSCFile01
  - Feature1                                                                                          Roles\FileServer
  - -Telnet-Client                                                                                  Baselines\Security

Expected RSOP:

WindowsFeatures:
  Names:
  - File-Services                                                                               AllNodes\Dev\DSCFile01
  - -Telnet-Client                                                                                  Baselines\Security

The datum.yml definition for merging and knockout looks like this:

WindowsFeatures:
  merge_hash: deep
WindowsFeatures\Names:
  merge_basetype_array: Unique
  merge_options:
    knockout_prefix: --

I am currently working on code to make this working.

Even more important than knocking out keys would be knowing out hash tables from a hash table array, similar like removing knocking out values from a base type array. This behavior is not available in Puppet.

Lower Level:

HashTables:
  Items:
    - Key1: h3
      Key2: 3
      Key3: v
    - Key1: h4
      Key2: 4
    - Key1: h5
      Key2: 5

Higher Level:

HashTables:
  Items:
    - Key1: h1
      Key2: 1
    - Key1: h2
      Key2: 2
    - Key1: --h3
      Key2: 3
      Key3: v3

Actual RSOP:

HashTables:
  Items:
  - Key1: h1                                                                                  AllNodes\Dev\DSCFile01
    Key2: 1                                                                                   AllNodes\Dev\DSCFile01
  - Key1: h2                                                                                  AllNodes\Dev\DSCFile01
    Key2: 2                                                                                   AllNodes\Dev\DSCFile01
  - Key3: v3                                                                                  AllNodes\Dev\DSCFile01
    Key1: --h3                                                                                AllNodes\Dev\DSCFile01
    Key2: 3                                                                                   AllNodes\Dev\DSCFile01
  - Key3: v                                                                                         Roles\FileServer
    Key1: h3                                                                                        Roles\FileServer
    Key2: 3                                                                                         Roles\FileServer
  - Key1: h4                                                                                        Roles\FileServer
    Key2: 4                                                                                         Roles\FileServer
  - Key1: h5                                                                                        Roles\FileServer
    Key2: 5                                                                                         Roles\FileServer

Expected RSOP:

HashTables:
  Items:
  - Key1: h1                                                                                  AllNodes\Dev\DSCFile01
    Key2: 1                                                                                   AllNodes\Dev\DSCFile01
  - Key1: h2                                                                                  AllNodes\Dev\DSCFile01
    Key2: 2                                                                                   AllNodes\Dev\DSCFile01
  - Key1: h4                                                                                        Roles\FileServer
    Key2: 4                                                                                         Roles\FileServer
  - Key1: h5                                                                                        Roles\FileServer
    Key2: 5                                                                                         Roles\FileServer

Datum.yml

  HashTables:
    merge_hash: deep
  HashTables\Items:
    merge_hash_array: UniqueKeyValTuples
    merge_options:
      tuple_keys:
        - Key1
        - Key2
      knockout_prefix: --

Instead of knocking out the hash table with the with Key1 == h3, Datum treats it as a new hash table to merge as Lower Level->Key1 is different from Higher Level->Key1.

• How does knockout work with more than two layers / how is it supposed to work?
• Is it sufficient to allow a knockout only on the first key in merge_options->tuple_keys

If you have strings in an array that are catched up by a Datum handler, you get this error, as the Datum handler expects a string.

WARNING: Error using Datum Handler Datum.InvokeCommand::InvokeCommand, returning Input Object

The result is pretty malformed data in the RSOP. The following array looks like this then:

TestArray1:
  - '[x={ Get-Random }]'
  - '[x={ Get-Random }]'

TestArray1:
- '[x={ Get-Random }]'
- value:
  - 196783108
  - 685218656
  Count: 2
- value:
  - 1621033775
  - 327351527
  Count: 2

In almost any projects we want to use the handler Datum.InvokeCommand also on arrays.

I have published a working demo of the issue: https://github.com/raandree/DscWorkshop/tree/test/DatumHandlerArrayBug
The same demo with the fix added to Datum: https://github.com/raandree/DscWorkshop/tree/test/DatumHandlerArrayBugFix

I have example files that demonstrate this issue at https://github.com/thomwatt/datum

If you have a look at Datum.ps1 and once run, look at the $TableData object. The output should return a collection of Windows shares with properties:
•StartupType
•Name
•State

If you look at the output below, it returns objects for the services taken from \Tibco\FTL.yml but for the service taken from \Tibco\Base.yml it is returning key/value pairs (the last 3 objects in the YAML below).

This behaviour only happens if the base.yml file contains one service definition. If it has 2 or more, the output is perfect

• StartupType: Automatic
  Name: tibJettySvc
  State: Running
• StartupType: Automatic
  Name: tibrealmserver
  State: Running
• StartupType: Automatic
  Name: tibrealmserver-as
  State: Running
• StartupType: Automatic
  Name: tibstore
  State: Running
• Key: StartupType
  Value: Automatic
• Key: Name
  Value: Base Service
• Key: State
  Value: Running

Following our discussion you thought that this may be a bug that you had spotted but not yet released the fix for.

Hi,
Is there a complete example of how to use the DatumHandler feature? We need to be able to inject variable in the yaml at built time (we need to be able to use a function that get passwords out of SecretServer and inject the result in the yml.) and the Handler seems the way to do it, but I admit it's a bit advanced for my Powershell skills and so far I never got it to work.

This is really a minor thing:

In the readme it says:

A stable v1 release is expected for March 2018, while some concepts are thought through, and prototype code refactored.

Today it is november 2019, which is faaaar after march 2018 ;)
Perhaps it would make sense to update this to reflect what the reality of today is? (what is the latest version etc...).
Or remove that sentence?

Hi Gael

I've noticed an error after the CU June for my 2016 Server, and i took me some time to find out, what possibly cause this error.

After the (update)reboot i was not longer able to build the mof files. The error message was this one:

===============================================================================
			COMPILE ROOT CONFIGURATION

-------------------------------------------------------------------------------
  /./Compile_Root_Configuration
  C:\Temp\DSC\.build\DSC\ConfigData.build.ps1:63

-----------------------
FilteredNode: 
-----------------------
PSDesiredStateConfiguration\Configuration : The term 'Set-PSTopConfigurationName' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a 
path was included, verify that the path is correct and try again.
At C:\Temp\DSC\RootConfiguration.ps1:7 char:1
+ configuration "RootConfiguration"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Set-PSTopConfigurationName:String) [Configuration], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : CommandNotFoundException,Configuration
 
ERROR OCCURED DURING COMPILATION
Done /./Compile_Root_Configuration 00:00:15.2377078

As mentioned before, this happened only on a fresh patched buildserver.

PS C:\Temp\DSC> $PSVersionTable
Name                           Value                                                                                                                                                                 
----                           -----
PSVersion                      5.1.14393.2312                                                                                                                                                  
PSEdition                      Desktop                                                                                                                                                            
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                                                                                                               
BuildVersion                   10.0.14393.2312                                                                                                                                                
CLRVersion                     4.0.30319.42000                                                                                                                                                
WSManStackVersion              3.0                                                                                                                                                                     
PSRemotingProtocolVersion      2.3                                                                                                                                                                     
SerializationVersion           1.1.0.1 

I copied the Module PSDesiredStateConfiguration from an unpatched prod server to this server with the error to "C:\Windows\System32\WindowsPowerShell\v1.0\Modules\PSDesiredStateConfiguration" and replaced this module. The feshly started build of the mof files went through as usual.

I found that in Azure cloudshell the module would not load unless targeting the PSD1, and I think it's because of a case issue between the folder it installs to: datum and the PSD1/PSM1 Datum.

I haven't found where the case is dropped, but I suspect it's not PSDeploy but maybe Publish-Module.

Keeping lower case might avoid future issues anyway.

To implement some additional logic/filtering, I am using the Lookup function in a composite resource. When compiling the MOF files, I am getting a lot of warnings (several hundred):

WARNING: /CompileRootConfiguration C:\src\DscWorkshop\DSC\BuildOutput\Modules\Datum\0.39.0\ScriptsToProcess\Resolve-NodeProperty.ps1:98
	No Datum store found for DSC Resource

This is caused because the following check fails:

The code I am using in the composite resource is:

    $currentEnvironment = $AllNodes.Environment
    $currentDepartment = $AllNodes.Department
    $currentTier = $AllNodes.Tier

    $servers = @{}
    foreach ($node in $datum.AllNodes.$currentDepartment.$currentEnvironment.$currentTier.ToHashTable().GetEnumerator())
    {
        $nodeObj = $datum.AllNodes.$currentDepartment.$currentEnvironment.$currentTier.$($node.Name)
        $servers.$($node.Name) = Lookup -Node $nodeObj -Datum $datum -PropertyPath 'SharePointFarmConfig/RunCentralAdmin' -WarningAction SilentlyContinue
    }

    $firstAppServer = $servers.Keys | Where-Object -FilterScript { $servers[$_] -eq $true } | Sort-Object | Select-Object -First 1

To work with Datum (during dev) it'd be easier to do a lookup by using a nodename (i.e. SRV01) instead of $Node, and let the lookup function resolve the $Node via the Configuration Data.

While trying a Get-DatumRsop I found that some error are thrown because of my Datum.yml not specifying lookup_options.

When specifying $false as the DefaultValue for Resolve-NodeProperty it is treated as if no DefaultValue was passed in. If the specified PropertyPath does not exist instead of $false being returned instead we throw the following error:
"The lookup of path '$PropertyPath' for node '$($node.Name)' returned a Null value, but Null is not specified as Default. This is not allowed."

Some Config Data should not or cannot be saved at the correct place of the hierarchy (i.e. a config file, or certificate used as a parameter to a Configuration):

MyResource:
  CN: MyCN
  Certificate: "<blob of file content>"

Managing the certificate in-line is not convenient, the file should probably be in it's own 'provider' or mounted somewhere else in the Datum Tree ($Datum.Files.certificates.MyResourceCert).

That means the data structure for MyResource should have the certificate file 'linked' to the Certificate key of the config Data, like so (in a similar pattern that of Secure Datum):

MyResource:
  CN: MyCN
  Certificate: "[REF=[Files\certificates\MyResourceCert]]"

Now, during a lookup $Node 'MyResource', Datum should resource the following hashtable, resolving under the hood the file content (or following what looks like a redirection):

@{
    CN = 'MyCN'
    Certificate = 'blob of file content'
}

If I've got a structure like below is is possible to say I'd want yml -> psd1 -> json or something similar to let me choose which files should be added to the datum structure?

Nodes
|
|--Node1.yml
|--Node1.json
|--Node1.psd1

The main use case here is that I've got some folders with nodes in them that also contain ARM templates named after the node type, along with config data (which is what I want) and a DSC config. Since all the files match in name Datum is grabbing the json file first, which isn't much use since it's an ARM template.

If I could set a precedence for what file types it should look for first then that would be great, or being able to filter out (or in) file types would be another useful solution.

I personally think that any string beginning with a dollar sign ($) should be ran through Invoke-Expression. That would cover several use cases:

• $Node.Name
• $env:ComputerName
• $((Get-CimInstance Win32_ComputerSystem).Name)

That last two would have to be compiled on the node, but serves as an example. If you really want a string beginning with a $, do: $('$pecial$tring').

Thoughts?

A JSON that worked without any issue on version 0.39.0 doesn't work anymore in version 0.40.1

{
    "ResolutionPrecedence": [
      "AllNodes\\$($Node.NodeName)"
    ],
    "DatumHandlersThrowOnError": true,
    "DatumHandlers": {
      "Datum.ProtectedData::ProtectedDatum": {
        "CommandOptions": {
          "PlainTextPassword": "SomeSecret"
        }
      },
      "Datum.InvokeCommand::InvokeCommand": {
        "SkipDuringLoad": true
      }
    }
  }

This json for example doesn't work with 0.40.1, but it works with 0.39.0. If I convert this json to yml it works with both versions.

New-DatumStructure -DefinitionFile .\Datum.json returns

MetadataError: Cannot convert value "@{ResolutionPrecedence=System.Object[]; DatumHandlersThrowOnError=True; DatumHandlers=; __File=C:\tmp\DSCTest\Datum.json}" to type
"System.Collections.Hashtable". Error: "Cannot convert the "@{ResolutionPrecedence=System.Object[]; DatumHandlersThrowOnError=True; DatumHandlers=;
__File=C:\tmp\DSCTest\Datum.json}" value of type "System.Management.Automation.PSCustomObject" to type "System.Collections.Hashtable"."
InvalidOperation: You cannot call a method on a null-valued expression.
InvalidOperation: The property 'DatumDefinitionFile' cannot be found on this object. Verify that the property exists and can be set.
InvalidOperation: You cannot call a method on a null-valued expression.
InvalidOperation: You cannot call a method on a null-valued expression.

Similar to the GPO's RSOP, it'd be very useful to be able to 'Compile' the Configuration Data per Node.
This could be used for testing, or to actually drive the MOF compilation.

Now that the Merge behaviours can be specified per Path in Datum, it should be possible to start from $Node and merging all layers down to produce the Resultant Set of Config Data.

Hey Gael,

with Resolve-DatumPath line 34:

$PathItem = ($CurrentNode.$($ExecutionContext.InvokeCommand.ExpandString($StackItem)))

This is breaking my merges as I want to add a single hashtable to an array of hashtables.

Maybe I'm doing something wrong but when I run Get-FileProviderData in PowerShell, it maintains the object as an array. However, when I run the Resolve-DatumPath or Resolve-NodeProperty then it loses its original type.

I've isolated the problem to when this command it run but with strictmode on/off json/yaml/psd1. It doesn't make a difference.

Do you know what this could be or how to solve it?

Thanks

The current version only supports merging Hashtable, array of hashtables, and basic types.
PSCustomObjects or arrays of PSCustomObjects would throw an exception or silently ignore the required merges.

This is making impossible to merge data that spans across files (as a File/Folder objects returns PSCustomObject by the Datum File Provider), so you can't split a Role as a folder with keys as Files.

The Merged data should be a hashtable, ready to be used for splatting.

This is a clone of issue dsccommunity/DscWorkshop#94. I don't have the rights to transfer the issue.

LcmConfig data cannot be captured with the standard composition key 'Configurations'. But the data can be easily added in case it is there.

When running the Build script, the RSOP output only contains the LcmConfig data that is included in the YAML file of the node itself. The data from the DscBaseline.yml file is missing:
https://github.com/dsccommunity/DscWorkshop/blob/935d168cad1546c4bf37db23f0735380d2d9866d/DSC/DscConfigData/Roles/DscBaseline.yml#L38-L54

Example of DSCFile01.yml

  MaxLcmRuntime: 00:30:00
LcmConfig:
  ConfigurationRepositoryWeb:
    Server:
      ConfigurationNames: DSCFile01
PSDscAllowDomainUser: true

Would be great if these config items would also be included in the RSOP data.

Currently the store must be of a specific provider, but that's adding more complexity for not much benefit.

The idea is to merge the Secure provider with the non secure, and handle when there's no security params.

Since upgrading to Datum 0.0.36 the CommonTasks project does no longer build on AppVeyor. When starting the build on the AppVeyor host via RDP manually, it work in PowerShell 5 and 6.

The error is this one: https://ci.appveyor.com/project/AutomatedLab/commontask/builds/21816268.

The 'Microsoft.PowerShell.Management' module was not imported because the 'Microsoft.PowerShell.Management' snap-in was already imported.
The 'Microsoft.PowerShell.Management' module was not imported because the 'Microsoft.PowerShell.Management' snap-in was already imported.
PSDesiredStateConfiguration\Configuration : The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)
At C:\projects\commontask\BuildOutput\Modules\CommonTasks\DSCResources\FilesAndFolders\FilesAndFolders.schema.psm1:1 char:1
+ Configuration FilesAndFolders {
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Configuration], FileLoadException
    + FullyQualifiedErrorId : System.IO.FileLoadException,Configuration
 
    [-] FilesAndFolders Compiles 2.67s

Knockouts don't work as expected / produce unexpected results assuming that 'knockout_prefix' is implemented similar like in Puppet. There is not much documentation and descriptions in Puppet and the best summary of that feature is on StackOverflow: Using knockout_prefix in puppet hiera hierarchy:

Lower Level:

Car:
  Brand: VW
  Model: Golf
  Engine:
    Volume: 2.0

Higher Level:

Car:
  --Model:
  Color: Red
  Engine:
    Type: Gasoline

Actual RSOP:

Car:
  Color: Red                                                                                    AllNodes\Dev\DSCFile01
  --Model: 
  Engine:
    Type: Gasoline                                                                              AllNodes\Dev\DSCFile01
    Volume: 2                                                                                         Roles\FileServer
  Brand: VW

Expected RSOP:

Car:
  Color: Red                                                                                    AllNodes\Dev\DSCFile01
  Engine:
    Type: Gasoline                                                                              AllNodes\Dev\DSCFile01
    Volume: 2                                                                                         Roles\FileServer
  Brand: VW

Datum.yml

  Car:
    merge_hash: deep
    merge_options:
      knockout_prefix: --

The key model was knocked out but the key for knocking out is added to the result. This is not expected and results in errors when splatting the hashtable to the DSC resource.

• Is there a reason why the knocking out key is added to the result?
• When should it been removed? After the merge, while merging?
• How does knockout work with more than two layers / how is it supposed to work?

Similar to Redirections, it would be useful to use variable interpolation in Datum similar to Hiera:

'#{user}@#{domain}'

That would do a lookup for User and a lookup for Domain, and then aggregates them.

Seen when loading a Datum Tree without a defined Data handler.
The tree seemed to load fine, the DatumFileProvider looked fine, but the data was actually missing (i.e. $Datum.AllNodes.DEV.SRV01 was empty (but existing).

Validation should be done in New-DatumStructure to make sure everything will work as expected, or raise an exception and stop.

I think the way to do it is change the provider to use a class for Node (as in Demo3).
Add a Hidden property to reference Datum there so that the data is always accessible.
Extend the data with Merge behaviour by path.

It would be nice, if there is a stage or task to build only the modified server configurations or all affected servers from a role.

Server1 - role appserver
Server2 - role appserver
Server3- role webserver

Change server1.yaml -> new *.mof for Server1
Change of role appserver -> new *.mof for Server1 and Server2

Is there an easy way to do this?

When you merge an object containing only one element with other objects containing more elements you get the following error message:

WARNING: Cannot merge different types in path 'xPSDesiredStateConfiguration\xRegistry' REF:[baseType_array] | DIFF:[hash_array]System.Object[] , returning most specific Datum.

or another example:

WARNING: Cannot merge different types in path 'cChoco\cChocoPackageInstaller' REF:[baseType_array] | DIFF:[hash_array]System.Object[] , returning most specific Datum.

I have found a workaround by adding a fake element to the object with only one element and by ensuring it is absent.

For example:

xPSDesiredStateConfiguration:
xRegistry:
- Key: 'HKLM:\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate'
ValueName: TargetGroup
ValueData: Group1
ValueType: String
Ensure: Present
Force: True
#Fake element as a workaround
- Key: 'HKLM:\SOFTWARE\DatumFakeKey'
ValueName: RandomNumber_654321654
ValueData: 0
ValueType: Dword
Ensure: Absent
Force: True

However, unfortunately all objects don't have an Ensure: Absent|Present property.
Thus this workaround doesn't work for everything...

Usually every bit of config data is visible in the RSOP output. But the LCM settings are missing the pull server URI and the RegistrationKey which is in the Meta MOFs (so it does exist) but not in the RSOP.

A repo is here: https://github.com/raandree/DatumTest

The RSOP output for node PreServer1

Environment: Pre
Description: CoreDC in CSPre
Name: PreServer1
PSDscAllowPlainTextPassword: true
LcmConfig:
  ConfigurationRepositoryWeb:
    Server:
      ConfigurationNames: PreServer1
SoftwarePackages:
  Package:
  - Name: Password Policy Enforcer 9.1
    Ensure: Present
    Path: D:\Packages\PPE\PPE910.msi
    ProductId: 55EBFC31-1031-463C-A870-416E2C312A64
PSDscAllowDomainUser: true
Configurations:
- FilesAndFolders
- SoftwarePackages
- WindowsFeatures
Role: CoreDC
WindowsFeatures:
  Name:
  - -Telnet-Client
FilesAndFolders:
  Items:
  - DestinationPath: C:\Test\Frankfurt
    Type: Directory
Location: Frankfurt
NodeName: PreServer1

The meta.mof with the content that is missing in RSOP.

/*
@TargetNode='PreServer1'
@GeneratedBy=randr
@GenerationDate=08/30/2019 13:24:26
@GenerationHost=RAANDREE0
*/

instance of MSFT_WebDownloadManager as $MSFT_WebDownloadManager1ref
{
SourceInfo = "::6::2::ConfigurationRepositoryWeb";
 ServerURL = "https://PrePull.contoso.com/PSDSCPullServer.svc";
 ResourceID = "[ConfigurationRepositoryWeb]Server";
 RegistrationKey = "fbc6ef09-ad98-4aad-a062-92b0e0327562";
 ConfigurationNames = {
    "PreServer1"
};

};

instance of MSFT_WebReportManager as $MSFT_WebReportManager1ref
{
SourceInfo = "::6::2::ReportServerWeb";
 ServerURL = "https://PrePull.contoso.com/PSDSCPullServer.svc";
 ResourceID = "[ReportServerWeb]ReportServerWeb";
 RegistrationKey = "fbc6ef09-ad98-4aad-a062-92b0e0327562";

};

instance of MSFT_DSCMetaConfiguration as $MSFT_DSCMetaConfiguration1ref
{
RefreshMode = "Pull";
 AllowModuleOverwrite = True;
 ActionAfterReboot = "ContinueConfiguration";
 RefreshFrequencyMins = 30;
 RebootNodeIfNeeded = True;
 ConfigurationModeFrequencyMins = 30;
 ConfigurationMode = "ApplyAndMonitor";

  ReportManagers = {
  $MSFT_WebReportManager1ref  
 };
  ConfigurationDownloadManagers = {
  $MSFT_WebDownloadManager1ref  
 };
};

instance of OMI_ConfigurationDocument
{
 Version="2.0.0";
 MinimumCompatibleVersion = "2.0.0";
 CompatibleVersionAdditionalProperties= { "MSFT_DSCMetaConfiguration:StatusRetentionTimeInDays" };
 Author="randr";
 GenerationDate="08/30/2019 13:24:26";
 GenerationHost="RAANDREE0";
 Name="RootMetaMOF";
};

Is this by design or does something go wrong?

Creating and reviewing PRs for Datum is a challenge as there is not coherent style guideline used. #91 is just one example where most of the changes are just the result of the auto-format function in VSCode.

Proposal: Add a 'settings.json' and 'analyzersettings.psd1' to the project and then re-format each file.

Hi,
I've a problem with the merging when using non unique roles (multiples rules assigned to one server). I've created some demo code here
"ResolutionPrecedence": [ "AllNodes\\$($Node.Name)", "Role\\<%= $CurrentNode.PSObject.Properties.where{$_.Name -in $Node.Packages}.Value %>" ],

When using the "dynamic" Datum.json section "firewallrules" then contains two times "items",

{
  "FirewallRules": [
    {
      "Items": [
        {
          "Action": 2,
          "Description": "Test123",
          "Direction": 1,
          "DisplayName": "Test123",
          "RemotePort": "Any",
          "Name": "Test123",
          "LocalAddress": "Any",
          "LocalPort": "123",
          "Service": "Any",
          "Protocol": "TCP",
          "Profile": 0,
          "Program": "Any",
          "RemoteAddress": "Any",
          "Enabled": 1
        }
      ]
    },
    {
      "Items": [
        {
          "Action": 2,
          "Description": "Test456",
          "Direction": 1,
          "DisplayName": "Test456",
          "RemotePort": "Any",
          "Name": "Test456",
          "LocalAddress": "Any",
          "LocalPort": "456",
          "Service": "Any",
          "Protocol": "TCP",
          "Profile": 0,
          "Program": "Any",
          "RemoteAddress": "Any",
          "Enabled": 1
        }
      ]
    }
  ],
  "Configurations": [
    "FirewallRules",
    "FirewallRules"
  ],
  "NodeName": "SRV01",
  "Packages": [
    "Test123",
    "Test456"
  ],
  "Name": "SRV01"
}

when using Datum1.json (that has just hardcoded the two dynamic linked JSON's it just contains one "items" under "firewallrules".

{
  "Configurations": [
    "FirewallRules"
  ],
  "Packages": [
    "Test123",
    "Test456"
  ],
  "FirewallRules": {
    "Items": [
      {
        "Service": "Any",
        "Name": "Test123",
        "RemoteAddress": "Any",
        "LocalPort": "123",
        "RemotePort": "Any",
        "LocalAddress": "Any",
        "Enabled": 1,
        "Description": "Test123",
        "Action": 2,
        "Profile": 0,
        "Protocol": "TCP",
        "Program": "Any",
        "DisplayName": "Test123",
        "Direction": 1
      },
      {
        "Action": 2,
        "Description": "Test456",
        "Direction": 1,
        "DisplayName": "Test456",
        "RemotePort": "Any",
        "Name": "Test456",
        "LocalAddress": "Any",
        "LocalPort": "456",
        "Service": "Any",
        "Protocol": "TCP",
        "Profile": 0,
        "Program": "Any",
        "RemoteAddress": "Any",
        "Enabled": 1
      }
    ]
  },
  "Name": "SRV01",
  "NodeName": "SRV01"
}

Currently Datum is loaded via New-DatumStructure, but the structure path are either aboslute, or relative to the calling script.

By making a different parameterSet where the New-DatumStructure could load its configuration from a file (the Datum.yml), or from a parent folder of a file it'd be easier to load the structure with relative path from that file (and less params to deal with).

find a way to declare how to get the nodes and load their RSOP in the configdata variable.

So far, the best I found is this:

Variables:
  ConfigurationData:
    AllNodes:
	  Path: Datum:\AllNodes\#{Environment}\#{Name}
	Datum:
	  Path: Datum:\
	  
ConfigurationData:
  AllNodes:
    - 

The main goals are to:

• define where to find the $Nodes in the hierarchy
• what path metadata to be loaded (#{property} or * to not include it)

There's currently no help available.
It'd be good to add some help and get those QA tests passed :)

Hey Gael,

Just some feedback following your presentation since we didn't get to chat properly. I might have missed your intentions but for me as a DSC user I would have thought it simpler and more intuitive to take all the data as you do, apply the ResolutionPrecedence as you do, and then present the resultant values as part of $AllNodes.

A few points why i think this:

• It will let people access variables using $Node.VariableName which is simple and familliar
• It pushes the configuration author to respect the fact that a particular value is "preferred" based on the ResolutionPrecedence (but you may want to allow them to override this as you said)
• It abstracts from the structure of the data which is quite complex, what i mean by this is a lot of data in different formats from different sources is collected. This is likely to result in a complex structure, you yourself had to index into a fair few variables to access the value you wanted. If you flattened this structure respecting the ResolutionPrecedence then it would ease the burden of the author having to figure out where his values are
• It will be less attractive for authors to make exceptions at a whim
• Finally its more fitting with the definition of Datum. I would say that each $Node is the "datum". I wouldn't term a complex structure as "datum", that is more "data".

Just some thoughts, otherwise i really like the idea. Don't think ill get much of a chance to use this inside my black box but maybe this will come in handy when i start working on some personal DSC projects.

This code is vulnerable to command injection from the data:
https://github.com/gaelcolas/Datum/blob/master/Datum/classes/SecureDatum.ps1#L44

e.g.

PS C:\> $datum = @{normal=1;evil='[ENC=s";Write-Host "HACKED";$e="]'} | ConvertTo-ProtectedDatum -UnprotectOptions @{ClearTextPassword='doesntmatter'}
PS C:\> $datum
HACKED
HACKED

normal evil
------ ----
     1

This is not great, there are cases where config data will be updated by less trusted persons - and that should not result in execution on the build machine. A quick fix may be to be more explicit with your regex to only accept base64 data but I'm not sure it should be relied on.

The current merging parameters is a bit messy, and could be improved to look like this:

lookup_options:
  MergeTest1:
    merge_options:
      knockout_prefix: --
      PropertyNames:
        - name
        - version
    merge_hash: 
      strategy: MostSpecific|Hash|Deep
    merge_basetype_arrays: 
      strategy: Unique|Sum
    merge_hash_arrays:
      strategy: UniqueTuple|DeepTuple|Sum

That would also allow to simplify and clarify the code underneath.
As this will be a breaking change, it should be done before v1.0 release.

Datum is not meant to be DSC Specific, although it's been built mainly to be used within a DSC Pipeline.
It was convenient at that time to bundle function such as Get-DscSplattedResource, but was never intended to stay beyond the initial experimentation.

In an effort to move closer to a v1.0.0, some functions will be removed, including:

• Get-DscSplattedResource which should naturally live in the DscBuildHelpers module (See @VertigoRay 's effort in this PR already gaelcolas/DscBuildHelpers#4).

In Hiera, they allow a specific hierarchy for module independent of the global/env ones, yet overridable in the latter.

I really like this idea and it would allow to bundle some data specific to the resource or composite resource, that could be either defined at a layer above of the hierarchy or at runtime on the Node for a resource (not Composite as they're evaluated at MOF compilation time).

To support something similar, DSC Resources could:

• use a locally defined Datum
• feed non-mandatory parameters with Datum lookup (if defined above, they're overridden)

Needs to do a PoC of this...

When trying to create a new Datum structure in AzureCloudshell, we're greeted by an error as below.
It turns out that Get-FileProviderData returns an array of PSObject instead of a single Item.
It's most likely an issue with the -NoEnumerate.

Needs further investigation

PS /home/gael/data> $datum = New-DatumStructure -DefinitionFile /home/gael/data/datum.yml
DEBUG: File /home/gael/data/datum.yml found. Loading...
VERBOSE: Getting File Provider Cache for Path: /home/gael/data/datum.yml
Cannot convert the "System.Management.Automation.PSObject[]" value of type "System.Management.Automation.PSObject[]" to type "System.Collections.Hashtable".
At /home/gael/data/datum/0.0.35/Datum.psm1:917 char:17
+ ...             $DatumHierarchyDefinition = Get-FileProviderData $Definit ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidArgument: (:) [], RuntimeException
+ FullyQualifiedErrorId : ConvertToFinalInvalidCastException

Datum needs different Merge behaviours when searching for datum property in the hierarchy.
Hiera defines those here: https://docs.puppet.com/puppet/5.3/hiera_merging.html

In Datum, Resolve-Datum should be changed here: https://github.com/gaelcolas/Datum/blob/master/Datum/Public/Resolve-Datum.ps1#L58-L65

And it should support:

• Most Specific (already implemented)
• All Values (already implemented)
• Array Merge (all unique items from all arrays)
• Hash merge (all keys merged, not recursive)
• Deep merge (all keys merged, if Key's value is hash also merge, if array, do Array Merge)

Other nice to have features:

• Data source can override lookup options: a configuration uses an option, but the option can be overridden by a user in their config data in control repo.

Configure Many Lookup key with a regex:

• Litteral wins over Regex
• first Regex wins

Think about Priority when merging, Hiera has low pri first for hash merge.

During some testing I found that the Merge-DatumArray function doesn't always return an array, but an OrderedDictionary. This results in an error when the next level is merged and Datum throwing the error:
Write-Warning -Message "Cannot merge different types in path '$StartingPath' REF:[hashtable] | DIFF:[hash_array] , returning most specific Datum."

I tracked down this issue to this line:

By adding a , (comma) in front, the function will return an array:

    , $mergedArray

@gaelcolas, can you have a look at brach https://github.com/AutomatedLab/DscWorkshop/tree/HashTableMergeIssue of DscWorkshop. I have added the following configuration to the DSCFile01 node which leads to the conflict:

FilesAndFolders:
  Items:
    - DestinationPath: C:\TestFolderInDev
      Type: Directory

and that's the merge behavior:

lookup_options:
  Configurations:
    merge_basetype_array: Unique
  
  FilesAndFolders:
    merge_hash: deep

  FilesAndFolders\Items:
    merge_hash_array: DeepItemMergeByTuples

Am I doing something wrong or is this a bug?

When you use a hash_array to configure data and then edit data on three or more different levels, Datum throws a warning and only uses the data from the highest two levels.

SSLCertificateImport:
    merge_hash: deep
  SSLCertificateImport\Certificates:
    merge_basetype_array: Deep
    merge_hash_array: DeepTuple
    merge_options:
      tuple_keys:
        - Name

If you then edit the data of this composite resource on level 1 and add data on level 2 and 3, the following error is displayed:

WARNING: Cannot merge different types in path 'SSLCertificateImport\Certificates' REF:[hashtable] | DIFF:[hash_array]System.Object[] , returning most specific Datum.

Troubleshooting info

Data Level 1

SSLCertificateImport:
  Certificates:
    - Name: SSLCertificate
      Location: LocalMachine
      Store: My
      Exportable: True
      Ensure: Present

Data Level 2

SSLCertificateImport:
  Certificates:
    - Name: SSLCertificate
      Ensure: Absent

Data Level 3

SSLCertificateImport:
  Certificates:
    - Name: SSLCertificate
      Thumbprint: A1B2C3D4E5F6

Actual result

SSLCertificateImport:
  Certificates:
    - Name: SSLCertificate
      Ensure: Absent
      Thumbprint: A1B2C3D4E5F6

Expected result

SSLCertificateImport:
  Certificates:
    - Name: SSLCertificate
      Location: LocalMachine
      Store: My
      Exportable: True
      Ensure: Absent
      Thumbprint: A1B2C3D4E5F6

Reproduction
If you download the following zip file and run the below commands, the first command will work (SPApplication.yml does not contain data) and the second with show the issue (SPSearchBackEnd.yml does contain data.

Level 1: SharePointServer.yml
Level 2: SPSearchBackEnd.yml
Level 3: StandardAcceptancePrimary.yml

$datum = New-DatumStructure -DefinitionFile .\Datum.yml

$srvApp = $datum.AllNodes.Standard.Acceptance.Primary.ServerApp
$dataApp = Lookup -Node $srvApp -DatumTree $datum -PropertyPath 'SSLCertificateImport'
$dataApp.Certificates

$srvSbe = $datum.AllNodes.Standard.Acceptance.Primary.ServerSbe
$dataSbe = Lookup -Node $srvSbe -DatumTree $datum -PropertyPath 'SSLCertificateImport'
$dataSbe.Certificates

Solution
I found that this issue is caused by the fact that PowerShell doesn't return an array when it only contains one item. This is causing the module later on to try to compare a hashtable with an array. By changing line 606 in Datum.psm1 by the following code the issue is fixed, forcing the item to be stored as an array:

                    if ($clonedReference.$currentKey -is [System.Array])
                    {
                        [System.Array]$clonedReference[$currentKey]  = Merge-Datum @MergeDatumParams
                    }
                    else
                    {
                        $clonedReference[$currentKey]  = Merge-Datum @MergeDatumParams
                    }

Is this a good solution or do you see a different (better) solution?