subpointsolutions / metapack Goto Github PK

Metapack API should be able to provision both SPMeta2/PnP models without NuGet packages. SolutionPackageDeploymentService should be able to provide such option.

That need is driven by 3rd party tooling that would be using only model deployment options without NuGet packaging.

That said, API should still run provision in isolation not interfering with the current application domain. That ensures API's ability to run provision under particular tool version, API and runtime.

As mentioned, DefaultUpdateCommand needs to be covered with regression tests.

MetaPack CLI should have a concept of "metapackfile" and "metapackfile.lock" similar, for instance, to Ruby's "Gemfile" and "Gemfile.lock"

Both "metapackfile" and "metapackfile.lock" aim to support the following scenarios:

Package deployment orchestration

metapackfile describes mappings between SharePoint site urls (or connections) and packages to be deployed. JSON or Yaml can be seen as following:

galleries:
    - url: http://metapackgallery.com/api/v2
    - url: https://packages.nuget.org/api/v2
    - url: http://www.myget.org/F/subpointsolutions-staging/api/v2

deployments:
    - TeamSite:
        name: team-site        
        url: https://contoso.sharepoint.com/sites/intranet
        spversion: sp2013 | sp2016 | o365
        spapi: csom | ssom
        packages:
            - id: contoso.metadata
              version: 1.0.0
            - id: contoso.teamsite
              version: 1.2.0-beta1
    - LeaveRequestSite:
        name: leave-request-site
        url: https://contoso.sharepoint.com/sites/intranet/leave-request
        packages:
            - id: contoso.leaverequest
            - id: contoso.teamsite
              version: 1.2.0-beta1          

Such yaml aims to define:

• NuGet galleries to be used
• Mappings between target SharePoint sites (urls) and packages to be deployed

JSON representation can be seen as following:

{
  "galleries": [
    {
      "url": "http://metapackgallery.com/api/v2"
    },
    {
      "url": "https://packages.nuget.org/api/v2"
    },
    {
      "url": "http://www.myget.org/F/subpointsolutions-staging/api/v2"
    }
  ],
  "deployments": [
    {
      "TeamSite": {
        "url": "https://contoso.sharepoint.com/sites/intranet",
        "packages": [
          {
            "version": "1.0.0",
            "id": "contoso.metadata"
          },
          {
            "version": "1.2.0-beta1",
            "id": "contoso.teamsite"
          }
        ],
        "name": "team-site"
      }
    },
    {
      "LeaveRequestSite": {
        "url": "https://contoso.sharepoint.com/sites/intranet/leave-request",
        "packages": [
          {
            "id": "contoso.leaverequest"
          },
          {
            "version": "1.2.0-beta1",
            "id": "contoso.teamsite"
          }
        ],
        "name": "leave-request-site"
      }
    }
  ]
}

Credential managements is to be define later but here are a few options:

• credentials section within yaml config
• other ways to reference credentials within deployments section in a secure way

Such configuration would enable orchestration of a complex provision over multiple packages, SharePoint sites and tenants. Configuration is a pure draft, to be refined and enhanced as we go.

The other bit of the configuration is "metapackfile.lock" file.
The aim is to lock down versions of the packages per SharePoint sites. That should be done by similar yaml/json config in the following way:

locks:
    global:
        packages:
              - id: contoso.metadata
                version: 1.0.0
              - id: contoso.metadata.fields
                version: 1.2.0     
    local:
        - url: https://contoso.sharepoint.com/sites/intranet
          packages:
              - id: contoso.metadata
                version: 1.0.0
              - id: contoso.metadata.fields
                version: 1.2.0     
        - url: https://contoso.sharepoint.com
          packages:
              - id: contoso.intranet
                version: 1.2.0
              - id: contoso.metadata.fields
                version: 1.2.0                    

Global locks - all packages across the board.
Local locks - packages per particular site url.

The aim is to lock down versions which would (can be) deployed to target urls by default. MetaPack would check information in "metapackfile.lock" finding particular versions of the packages before deployment.

MetaPack.SPMeta2 should support incremental provision as per Implement 'incremental' provision support in SubPointSolutions/spmeta2#553

Should be an additional parameter for the MetaPack.SPMeta2 tool.

MetaPack CLI should support 'install' command to install packages. Similar to NuGet, the following spec should be followed:

metapack install

• [+] prerelease
• [-] NonInteractive
• [-] verbose
• [-] debug
• [+] url
• [+] username
• [+] userpassword
• [+] spversion
• [+] source
• [+] id
• [+] version

Examples:

Write-Host "Testing list command on local SharePoint" -fore green
.\metapack install --url 'http://dev13:31415' --source 'https://www.myget.org/F/subpointsolutions-ci/api/v2' --id 'SPMeta2.CI' --version '1.2016.1015.0557'

Write-Host "Testing list command on O365" -fore green
.\metapack install --url 'https://contoso.sharepoint.com/sites/intranet' --source 'https://www.myget.org/F/subpointsolutions-ci/api/v2' --id 'SPMeta2.CI' --version '1.2016.1015.0557' --username 'meme@meme' --userpassword 'meme' --spversion 'o365'

Similar to other projects, docs should be placed within the project and merged with the main repo every build.

Metapack should support custom model packaging and deployment workflow.

API should allow to inject custom model packaging provider to accomplish the following scenarios:

• Pack / unpack model with your own way (for instance, secure the model and add licensing)
• Add 3rd part model support

Implementation should work as NuGet based plugin similar to Cakebuild.

Regression testing has to address the following areas across both API and CLI:

API (all providers)

• packing solution
• unpacking solution
• pushing solution to NuGet

API provision via MetaPack.SPMeta2 provider

• SP2013 CSOM
• SP2013 SSOM
• O365 CSOM

API provision via MetaPack.SharePointPnP provider

• [unsupported yet] - SP2013 CSOM
• [unsupported yet] - SP2016 CSOM
• O365 CSOM

CLI, basic operations

• empty run
• version
• list (O365)
• list (SP2013)
• install (all providers and APIs)
• install -force (all providers and APIs)
• upgrade -force (all providers and APIs)

Needs to be covered by regression testing - both API and CLI.

Metapack should be able to work with multiple NuGet Galleries.

-source parameter should support ';' to provide multiple NuGet galleries
app.config should provide ability to configure multiple NuGet galleries as well

Basic API needs to be covered by the regression tests:

• Packing models into NuGet package
• Pushing models into NuGet Gallery
• Finding models from NuGet Gallery
• Unpacking models from NuGet Gannery / NuGet package
• Installing models via API
• Installing models via MetaPack API to SharePoint
• Querying SharePoint via MetaPack API - installed models and versions
• Dependency management - packing models with dependencies

"Action-CLI-Regression" should be enabled under CI build and O365 profile.
We should be sure that MetaPack CLI actually work and can provision at least O365 based packages.

Metapack should support both CSOM/SSOM provision.

CSOM should be default implementation for provision.
Additional parameter should be added to the command line to suggests which runtime to use: SSOM or CSOM.

Implementation should work as NuGet based plugin similar to Cakebuild.

Logging should be improved to meet the following criteria:

• [DONE] make sense for the end users.
• [DONE] trace to the file
• [DONE] options to enable verbose logging for both API and CLI

Release CLI packages should be pushed to Chocolatey Gallery in an automated manner.

• CLI should configure metapack manager with the right set of creds, api, runtimes
• params from CLI should be passed further to metapack manager

• Test execution
• NuGet packaging
• NuGet publishing

Model packaging should be simplified for both SPMeta2 and PnP providers. API should hide all complexity of adding models, configuring the order of provisioning and unique ID management.

The following should be implemented:

• Simple API to package SPMeta2/PnP models
• Simple API to set order of model provisioning

All reflection based operations should be implemented within a dedicated service. Refactoring and clean up is required.

Metapack should have documentation covering the following areas:

Overview

• General idea on metapack - why, what

API

• Packing models into NuGet package
• Pushing models into NuGet Gallery
• Finding models from NuGet Gallery
• Unpacking models from NuGet Gannery / NuGet package
• Installing models via API
• Installing models via MetaPack API to SharePoint
• Querying SharePoint via MetaPack API - installed models and versions
• Dependency management - packing models with dependencies

CLI usage

• Introduce MetaPack CLI and basic scenarios
• Installing models from NuGet Gallery to SharePoint
• Updating models from NuGet Gallery to SharePoint
• Showing list of the installed models in SharePoint

metapack version should show current version of the client.

It should be possible to deploy only selected models from the whole package.

MetaPack should provide model container ID, so that one or more ID would be provided at the deployment time and only selected models will be deployed.

Options with key "_metapack.model.id" it to be used.

So that no trace is provided at all.

Cake build needs to be migrated to the latest SubPointSolutions.CakeBuild package.

Investigate and document guides on running metapack API under Azure functions.

MetaPack CLI should support 'update' command to install packages. Similar to NuGet, the following spec should be followed:

metapack update

• [+] prerelease
• [-] NonInteractive
• [-] verbose
• [-] debug
• [+] url
• [+] username
• [+] userpassword
• [+] spversion
• [+] source
• [+] id
• [+] version

Examples:

Write-Host "Testing list command on local SharePoint" -fore green
.\metapack update --url 'http://dev13:31415' --source 'https://www.myget.org/F/subpointsolutions-ci/api/v2' --id 'SPMeta2.CI' --version '1.2016.1015.0557'

Write-Host "Testing list command on O365" -fore green
.\metapack update --url 'https://contoso.sharepoint.com/sites/intranet' --source 'https://www.myget.org/F/subpointsolutions-ci/api/v2' --id 'SPMeta2.CI' --version '1.2016.1015.0557' --username 'meme@meme' --userpassword 'meme' --spversion 'o365'

metapack trace and logs should also be available in JSON format.

API should have additional trace service for that, and CLI should have additional flag to indicate machine-readable trace mode. Flag as the followig will do:

--output json

Such feature is needed by 3rd part API integrations, CLI and desktop client so that they would be able to capture trace in platform/API independent way.

A public NuGet Gallery for MetaPack needs to be introduced

• Setup NuGet Gallery
• Basic scenarios and documentation

MetaPack GUI client should support "brandability" scenario so that it can be used as "my product installer". The following scope is to be considered:

• configuration to enable/disable metapack gui tabs
• configuration to inject your own custom landing page (start page control)
• configuration to setup default NuGet Galleries with your packages
• ability to create a ZIP package our of pre-configured MetaPack GUI client

MetaPack Desktop client needs to be migrated under GitHub repository.

• move initial source code to GitHub
• enhance CI build to create both zip/chocolatey packages

So that we can indicate file name for logging.

Metapack CLI should be delivered via chocolatey, the easiest deployment and updates channel available.

Regression testing should be reengineered to meet the following criteria:

• [Done] "CI.Core" category should always be run under CI build
• [Done] "Metapack.Core.Packaging" category - pack, unpack, push operations
• [Done] "Metapack.Client.Commands" - metapack manager operations - install, list, upgrade
• [Done] Configurable environment (nuget galleries, local repositories)
• [Done] Configurable environment for SharePoint - SP2013 CSOM/SSOM, O365
• [Done] Console app should be regression tested

MetaPack should allow to pick up the tool version while provisioning models.
Both tool version and provision runtime version (m2/pnp version) should be configurable.

Install scenario sometimes throws Microsoft.SharePoint.Client.Runtime {"File Not Found."} exception. Needs to be investigated and covered with regression tests.

Microsoft.SharePoint.Client.Runtime {"File Not Found."}

   at Microsoft.SharePoint.Client.ClientRequest.ProcessResponseStream(Stream responseStream)
   at Microsoft.SharePoint.Client.ClientRequest.ProcessResponse()
   at Microsoft.SharePoint.Client.ClientRequest.ExecuteQueryToServer(ChunkStringBuilder sb)
   at Microsoft.SharePoint.Client.ClientRequest.ExecuteQuery()
   at Microsoft.SharePoint.Client.ClientRuntimeContext.ExecuteQuery()
   at Microsoft.SharePoint.Client.ClientContext.ExecuteQuery()
   at MetaPack.NuGet.Services.SharePointCSOMFileSystem.AddFileToSharePoint(Folder rootFolder, String filePath, Stream fileStream)
   at MetaPack.NuGet.Services.SharePointCSOMFileSystem.AddFiles(IEnumerable`1 files, String rootDir)
   at NuGet.PackageManager.ExpandFiles(IPackage package)
   at NuGet.PackageManager.OnExpandFiles(PackageOperationEventArgs e)
   at NuGet.PackageManager.ExecuteInstall(IPackage package)
   at NuGet.PackageManager.Execute(PackageOperation operation)
   at NuGet.PackageManager.Execute(IPackage package, IPackageOperationResolver resolver)
   at NuGet.PackageManager.InstallPackage(IPackage package, FrameworkName targetFramework, Boolean ignoreDependencies, Boolean allowPrereleaseVersions, Boolean ignoreWalkInfo)
   at NuGet.PackageManager.InstallPackage(IPackage package, Boolean ignoreDependencies, Boolean allowPrereleaseVersions)
   at MetaPack.Client.Common.Commands.DefaultInstallCommand.<Execute>b__15_0(ClientContext context)
   at MetaPack.Client.Common.Services.SharePointService.WithSharePointContext(String url, String userName, String userPassword, Boolean isSharePointOnline, Action`1 action)
   at MetaPack.Client.Common.Commands.DefaultInstallCommand.Execute()

As described, all commands should have a simple trace/progress callback so that external API can benefit from it.

MetaPack CLI should support 'list' command to install packages. Similar to NuGet, the following spec should be followed:

metapack list

• [-] prerelease
• [-] NonInteractive
• [-] verbose
• [-] debug
• [+] url
• [+] username
• [+] userpassword
• [+] spversion

Examples:

cls

Write-Host "Testing list command on local SharePoint" -fore green
.\metapack list --url 'http://dev13:31415'

Write-Host "Testing list command on O365" -fore green
.\metapack list --url 'https://contoso.sharepoint.com/sites/intranet' --username '[email protected]' --userpassword 'meme' --spversion 'o365'

Newly build chocolatey package needs to be regressed. Pester is to be used potentially integrated with Cakebuild pipeline.

The following scenarios are to be tested:

• [done] can run (just default metapack run)
• [done] can get version (just default metapack version)
• [done] can deploy O365 solution (install, update, --force flags)
• [todo] can deploy SP2013 CSOM/SSOM solution (install, update, --force flags)

SP2013 CSOM/SSOM might be delayed but core testing is a must.

--force flag needs to be introduced to enforce already deployed provision.

MetaPack should support digital signing of a package with certificates. Most likely, a content of the NuGet packages is to be signed with a certificate. Model packaging workflow should support that case providing an easy API for signing and verifying the certificate.

Such feature would add value to the platform enabling secure packages distribution.

GUI shouldn't crash on any exception happening.

Currently, metapack seems to fetch all installed versions of a target package. Tha results in a huge performance degradation.

The following needs to be implemented:

• fetch only latest versions of the installed packages
• avoid opening file if possible (avoid pushing stream back and forth)

Log trace looks as following indicating fetch of all packages:

[Verbose]: Fetching files for path with filter: *.nupkg : MetaPack.SPMeta2.CI.1.2017.0311.2349
[Verbose]: Fetched files for path with filter: *.nupkg : metapack.spmeta2.ci.1.2017.0311.2349
[Verbose]: Opening file for path: metapack.spmeta2.ci.1.2017.0311.2349\MetaPack.SPMeta2.CI.1.2017.0311.2349.nupkg
[Verbose]: Opened file for path: metapack.spmeta2.ci.1.2017.0311.2349\MetaPack.SPMeta2.CI.1.2017.0311.2349.nupkg
[Verbose]: Fetching files for path with filter: *.nupkg : MetaPack.SPMeta2.CI.1.2017.0305.0748
[Verbose]: Fetched files for path with filter: *.nupkg : metapack.spmeta2.ci.1.2017.0305.0748
[Verbose]: Opening file for path: metapack.spmeta2.ci.1.2017.0305.0748\MetaPack.SPMeta2.CI.1.2017.0305.0748.nupkg
[Verbose]: Opened file for path: metapack.spmeta2.ci.1.2017.0305.0748\MetaPack.SPMeta2.CI.1.2017.0305.0748.nupkg
[Verbose]: Fetching files for path with filter: *.nupkg : MetaPack.SharePointPnP.CI.1.2017.0305.0803
[Verbose]: Fetched files for path with filter: *.nupkg : metapack.sharepointpnp.ci.1.2017.0305.0803
[Verbose]: Opening file for path: metapack.sharepointpnp.ci.1.2017.0305.0803\MetaPack.SharePointPnP.CI.1.2017.0305.0803.nupkg
[Verbose]: Opened file for path: metapack.sharepointpnp.ci.1.2017.0305.0803\MetaPack.SharePointPnP.CI.1.2017.0305.0803.nupkg
[Verbose]: Fetching files for path with filter: *.nupkg : MetaPack.SPMeta2.CI.1.2017.0311.2331
[Verbose]: Fetched files for path with filter: *.nupkg : metapack.spmeta2.ci.1.2017.0311.2331
[Verbose]: Opening file for path: metapack.spmeta2.ci.1.2017.0311.2331\MetaPack.SPMeta2.CI.1.2017.0311.2331.nupkg
[Verbose]: Opened file for path: metapack.spmeta2.ci.1.2017.0311.2331\MetaPack.SPMeta2.CI.1.2017.0311.2331.nupkg
[Verbose]: Fetching files for path with filter: *.nupkg : MetaPack.SPMeta2.CI.1.2017.0305.0707
[Verbose]: Fetched files for path with filter: *.nupkg : metapack.spmeta2.ci.1.2017.0305.0707
[Verbose]: Opening file for path: metapack.spmeta2.ci.1.2017.0305.0707\MetaPack.SPMeta2.CI.1.2017.0305.0707.nupkg
[Verbose]: Opened file for path: metapack.spmeta2.ci.1.2017.0305.0707\MetaPack.SPMeta2.CI.1.2017.0305.0707.nupkg
[Verbose]: Fetching files for path with filter: *.nupkg : MetaPack.SPMeta2.CI.1.2017.0311.2306
[Verbose]: Fetched files for path with filter: *.nupkg : metapack.spmeta2.ci.1.2017.0311.2306
[Verbose]: Opening file for path: metapack.spmeta2.ci.1.2017.0311.2306\MetaPack.SPMeta2.CI.1.2017.0311.2306.nupkg
[Verbose]: Opened file for path: metapack.spmeta2.ci.1.2017.0311.2306\MetaPack.SPMeta2.CI.1.2017.0311.2306.nupkg
[Verbose]: Fetching files for path with filter: *.nupkg : MetaPack.SPMeta2.CI.1.2017.0305.0751
[Verbose]: Fetched files for path with filter: *.nupkg : metapack.spmeta2.ci.1.2017.0305.0751
[Verbose]: Opening file for path: metapack.spmeta2.ci.1.2017.0305.0751\MetaPack.SPMeta2.CI.1.2017.0305.0751.nupkg
[Verbose]: Opened file for path: metapack.spmeta2.ci.1.2017.0305.0751\MetaPack.SPMeta2.CI.1.2017.0305.0751.nupkg
[Verbose]: Fetching files for path with filter: *.nupkg : MetaPack.SPMeta2.CI.1.2017.0312.0020
[Verbose]: Fetched files for path with filter: *.nupkg : metapack.spmeta2.ci.1.2017.0312.0020
[Verbose]: Opening file for path: metapack.spmeta2.ci.1.2017.0312.0020\MetaPack.SPMeta2.CI.1.2017.0312.0020.nupkg
[Verbose]: Opened file for path: metapack.spmeta2.ci.1.2017.0312.0020\MetaPack.SPMeta2.CI.1.2017.0312.0020.nupkg

All commands should be able to switch default output. By default, Console.Out should be used.

Once metapack works with multiple NuGet galleries ( Make metapack work with multiple NuGet Galleries #20), a local NuGet Gallery needs to be added.

The goal is to have a special local folder within metapack. This "local" gallery will be housing ISVs specific packages. The idea is that you can get metapack, put your packages into a folder, and then the final distribution would be able to see these packages. Such scenario is extremely useful for ISVs who provide pre-packaged solutions for customers.

Investigate the following options:

• Shared folders
• Local IIS setup
• Docker image
• Azure / AWS based deployment of NuGet Gallery

The following activities are to be completed for the first beta:

• MetaPack chocolatey package
• MetaPack zip package
• C# examples on how to pack SPMeta2 models
• C# examples on how to pack PnP models
• bat\ps examples on metapack installation
• bat\ps examples on metapack package provision (O365 CSOM / SP2013 CSOM)
• Initial write up on how to use metapack

Currently, NuGetListCommand only outputs all packages to console trace. Command should have additional property with collection of fetched packages.

MetaPack CLI should support 'push' command to push a package to NuGet gallery. Similar to NuGet, the following spec should be followed:

metapack push

• [-] prerelease
• [-] NonInteractive
• [-] verbose
• [-] debug
• [-] source
• [-] api key

CI build should produce correct CLI package:

• [done] a zip package with MD5 hash
• [done] chocolatey package