microsoft / cella

Project Cella

vcpkg-ce Project

In this case however, it's a tool for installing development tools, compilers, libraries, frameworks and board support profiles.

---

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

cella's Issues

Install Artifact

Install an artifact to the home folder structure

• resolve recursive set of Demands
• resolve references via matching versions
• generate the flattened package graph
• CLI: ce install <pkgname>
• CLI: ce uninstall <pkgname>
• write package info into target after install

Equivalent of "pragma detect_mismatch"

We probably need a mechanism to explain to the user that they attempted to activate mutually exclusive packages. For example:

cella add microsoft/crts/desktop
cella add microsoft/crts/onecore
cella activate --target:x86

main.ts has localizable string concatenation

cella/cli/main.ts

Line 88 in ef6dea0

log(green.dim(i`Use \`${cli} ${help.command}\` ${green.dim(i`to get help`)}`));

"Use" and "to get help" are part of the same sentence but localized separately. This may not work in all languages. For example, in German, sentences like "Um Hilfe zu kriegen," "benutzen" with the word order reversed are possible. In general, avoid concatenation of localizable strings.

add support for `apply:` in the manifest

• apply: allows us to pull files from the artifact folder and insert them into the project. no effect on ce use consumption of package
  • Q: possible other names: [apply-changes:,insert:,transform:,modify:,project:]?
  • show warnings if the files exist and are not the same as the source file? (would happen if the files are there from a previous activation and modified.)

    # the 'apply' section copies files from the artifact folder to the project folder.
    # apply:
    #  <KIND>:  ## [c/header/assembler/linker/etc] -- the kind may let us update/modify project files in the longer term
    #    <TARGET-PATH>: <SOURCE-PATH>
    #      -- TARGET-PATH is relative to the folder containing environment.yaml 
    #      -- SOURCE-PATH is relative to the artifact folder after being unpacked.
  
  
    apply:  # this section copies files 
      
      # target: source
      c: 
        src/board/main.c: project_template/board.c   
        src/board/main.cpp: project_template/board.cpp
  
      header: 
        src/include/board.h: project_template/board.h
      
      assembler: 
        src/asm/startup.s: Device/ARM/ARMCM3/Source/IAR/startup_ARMCM3.s
  
      linker: 
        src/linker/gcc_arm.ld: Device/ARM/ARMCM3/Source/GCC/gcc_arm.ld
  
      misc:
        docs/ : project_template/docs/** # support copying with wildcard/globbing
  
  NOT exists:src/board.c:  # maybe we should have a mediaquery for 'exists'? - then we could have an apply section active only when a file doesn't exist in the project.
    apply:
  
  

command.ts has markdown markup in localizable strings

Markup in localizable strings should be avoided because they require technical knowledge from localizers and it is difficult to catch localizations which break the syntax.

command.ts:

i`### \`${this.title}\``,

Should be something like:

`### \`` + i`${this.title}` + `\``,

Azure/IOT artifacts

Manually populate the artifact repository

• Identify Azure/IOT artifacts to include in repository
• Create/publish AMF metadata files for artifacts into the repo

Embedded Tools Artifacts

Manually populate the artifact repository

• Identify Embedded tools to include in repository
• Create/publish AMF metadata files for artifacts into the repo

Basic CLI Commands

• version
• help

Create Bug Report

Create bug report, new issue in repository

• CLI ce --bug-report ...
• capture project profile
• capture response messages
• capture command line
• capture user error description

Create CI Script to validate PRs

Create a CI Script(s) for the metadata repository to validate the PR

• properly formatted
• verify checksums

Add/Insert into project

• CLI ce add <artifact>- add artifact to project

• CLI ce insert <artifact> - insert artifact into project

  • copy files
  • or git submodule add

CE/VCPKG Merge Task: Environment Changes

• change CE_HOME to $VCPKG_ROOT (defaulting to ~/.vcpkg if not set)
  • change download(cache) folder to be $VCPKG_DOWNLOADS (defaults to $VCPKG_ROOT/downloads)
  • use vcpkg-configuration.json instead of environment.yaml file

Load/parse AMF files

Load and Parse AMF files into OM

• methods to maipulate OM (preserve comments, etc)
• write AMF file (support FS abstraction)

Find Artifact

Find Artifact

• query index for artifacts (find)
• CLI: ce find <pkgname>

Archive Handling [gitrepo]

Handle git repositories as an 'archive' for installation purposes

• git clone repository
• support tag
• support full
• support recurse

Artifact Mirroring

• configuration: where to mirror/what

• how to detect if a file is in a (self-managed) mirror?

  • detect when we have permission to mirror artifacts, update artifact metadata to allow that.
    • based on license? explicit permission?

Configuration Support

Add configuration file support

• load global config
  • telemetry opt in/out
  • (Deferred until we need it) sources
• find/load project profile file (json/yaml)
  • starting from the current directory, walk up until we can find the cella.yaml/yml/json file

MSVC Artifacts

Manually populate the artifact repository

• Identify MSVC artifacts to include in repository
• Create/publish AMF metadata files for artifacts into the repo

Flattening

• CLI: ce flatten - create flattend graph
• --update - write flattened graph to project profile

"ce help <command>" doesn't work

For example, using
"ce help new"
just returns the output of "ce help" rather than the correct output that
"ce new --help"
outputs.

ce install fails in PowerShell when $ErrorActionPreference = 'Stop'

Archive Handling [.zip/.nupkg]

Unpack zip files into target location

• support strip (remove top directories during unpack)
• support transform (transform filenames during unpack)
• preserve modification tomes on disk

ce activate fails when run detached from terminal

This is a blocking bug for CI scenarios because this is how ce activate would run as part of a GH workflow or ADO pipeline.

From powershell.exe on Windows

Steps:

1. iex (iwr -useb aka.ms/install-ce.ps1)
2. ce --accept-eula
3. ce activate 2>&1 | write-host <-- redirect to detach streams from terminal

Expected: ce activation succeeds

Actual:

vcpkg-ce command line utility [0.4.169]

Updating repository data from https://aka.ms/vcpkg-ce-default
Artifact                 Version    Status        Dependency  Summary
 tools/microsoft/openocd  0.11.0     will install  *           Free and open on-chip debugging
 tools/ninja-build/ninja  1.10.2     will install  *           Ninja is a small build system with a focus on speed.
 tools/kitware/cmake      3.20.1     will install  *           Kitware's cmake tool
 compilers/arm/gcc        2020.10.0  will install  *           GCC compiler for ARM CPUs.
C:\Users\alleu\.ce\node_modules\@microsoft\vcpkg-ce\cli\dist\lib\artifacts.js:88
                            dl.update(percent);
                               ^
System.Management.Automation.RemoteException
TypeError: Cannot read property 'update' of undefined
    at download (C:\Users\alleu\.ce\node_modules\@microsoft\vcpkg-ce\cli\dist\lib\artifacts.js:88:32)
    at C:\Users\alleu\.ce\node_modules\@microsoft\vcpkg-ce\common\temp\node_modules\.pnpm\[email protected][email protected]\node_modules\ee-ts\dist\ee.js:69:28
    at Set.forEach (<anonymous>)
    at ExtendedEmitter._emit (C:\Users\alleu\.ce\node_modules\@microsoft\vcpkg-ce\common\temp\node_modules\.pnpm\[email protected][email protected]\node_modules\ee-ts\dist\ee.js:69:12)
    at ExtendedEmitter.emit (C:\Users\alleu\.ce\node_modules\@microsoft\vcpkg-ce\common\temp\node_modules\.pnpm\[email protected][email protected]\node_modules\ee-ts\dist\ee.js:39:10)
    at ProgressTrackingStream.<anonymous> (C:\Users\alleu\.ce\node_modules\@microsoft\vcpkg-ce\core\dist\lib\acquire.js:170:62)
    at ProgressTrackingStream.emit (events.js:376:20)
    at ProgressTrackingStream._transform (C:\Users\alleu\.ce\node_modules\@microsoft\vcpkg-ce\core\dist\lib\streams.js:24:14)
    at ProgressTrackingStream.Transform._read (internal/streams/transform.js:205:10)
    at ProgressTrackingStream.Transform._write (internal/streams/transform.js:193:12)

From bash on Linux

embedded@ubuntu:~/proj/embedded/tests/integration$ . <(curl aka.ms/install-ce.sh -L)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   138  100   138    0     0    321      0 --:--:-- --:--:-- --:--:--   321
100   610  100   610    0     0   1032      0 --:--:-- --:--:-- --:--:--  1032
100 12379  100 12379    0     0  15590      0 --:--:-- --:--:-- --:--:-- 15590
Installing vcpkg-ce in /home/embedded/.ce
embedded@ubuntu:~/proj/embedded/tests/integration$ ce --accept-eula
vcpkg-ce command line utility [0.4.169]

You are accepting the end-user license agreement available at https://aka.ms/vcpkg-ce-eula.txt


Use ce help to get help
embedded@ubuntu:~/proj/embedded/tests/integration$ ce activate 2>&1 | cat
vcpkg-ce command line utility [0.4.169]

Updating repository data from https://aka.ms/vcpkg-ce-default
Artifact                 Version    Status        Dependency  Summary                                              
 tools/microsoft/openocd  0.11.0     will install  *           Free and open on-chip debugging                      
 tools/ninja-build/ninja  1.10.2     will install  *           Ninja is a small build system with a focus on speed. 
 tools/kitware/cmake      3.20.1     will install  *           Kitware's cmake tool                                 
 compilers/arm/gcc        2020.10.0  will install  *           GCC compiler for ARM CPUs.
/home/embedded/.ce/node_modules/@microsoft/vcpkg-ce/cli/dist/lib/artifacts.js:88
                            dl.update(percent);
                               ^

TypeError: Cannot read property 'update' of undefined
    at download (/home/embedded/.ce/node_modules/@microsoft/vcpkg-ce/cli/dist/lib/artifacts.js:88:32)
    at /home/embedded/.ce/node_modules/@microsoft/vcpkg-ce/common/temp/node_modules/.pnpm/[email protected][email protected]/node_modules/ee-ts/dist/ee.js:69:28
    at Set.forEach (<anonymous>)
    at ExtendedEmitter._emit (/home/embedded/.ce/node_modules/@microsoft/vcpkg-ce/common/temp/node_modules/.pnpm/[email protected][email protected]/node_modules/ee-ts/dist/ee.js:69:12)
    at ExtendedEmitter.emit (/home/embedded/.ce/node_modules/@microsoft/vcpkg-ce/common/temp/node_modules/.pnpm/[email protected][email protected]/node_modules/ee-ts/dist/ee.js:39:10)
    at ProgressTrackingStream.<anonymous> (/home/embedded/.ce/node_modules/@microsoft/vcpkg-ce/core/dist/lib/acquire.js:170:62)
    at ProgressTrackingStream.emit (events.js:375:28)
    at ProgressTrackingStream._transform (/home/embedded/.ce/node_modules/@microsoft/vcpkg-ce/core/dist/lib/streams.js:24:14)
    at ProgressTrackingStream.Transform._read (internal/streams/transform.js:205:10)
    at ProgressTrackingStream.Transform._write (internal/streams/transform.js:193:12)

From Node on Windows

PS C:\proj\embedded\tests\integration> node
Welcome to Node.js v14.17.0.
Type ".help" for more information.
> require("child_process").exec(`${process.env.userprofile}/.ce/ce activate`, (err,stdout,stderr) => console.log(stderr))
...
> C:\Users\alleu\.ce\node_modules\@microsoft\vcpkg-ce\cli\dist\lib\artifacts.js:88
                            dl.update(percent);
                               ^

TypeError: Cannot read property 'update' of undefined
    at download (C:\Users\alleu\.ce\node_modules\@microsoft\vcpkg-ce\cli\dist\lib\artifacts.js:88:32)
    at C:\Users\alleu\.ce\node_modules\@microsoft\vcpkg-ce\common\temp\node_modules\.pnpm\[email protected][email protected]\node_modules\ee-ts\dist\ee.js:69:28
    at Set.forEach (<anonymous>)
    at ExtendedEmitter._emit (C:\Users\alleu\.ce\node_modules\@microsoft\vcpkg-ce\common\temp\node_modules\.pnpm\[email protected][email protected]\node_modules\ee-ts\dist\ee.js:69:12)
    at ExtendedEmitter.emit (C:\Users\alleu\.ce\node_modules\@microsoft\vcpkg-ce\common\temp\node_modules\.pnpm\[email protected][email protected]\node_modules\ee-ts\dist\ee.js:39:10)
    at ProgressTrackingStream.<anonymous> (C:\Users\alleu\.ce\node_modules\@microsoft\vcpkg-ce\core\dist\lib\acquire.js:170:62)
    at ProgressTrackingStream.emit (events.js:376:20)
    at ProgressTrackingStream._transform (C:\Users\alleu\.ce\node_modules\@microsoft\vcpkg-ce\core\dist\lib\streams.js:24:14)
    at ProgressTrackingStream.Transform._read (internal/streams/transform.js:205:10)
    at ProgressTrackingStream.Transform._write (internal/streams/transform.js:193:12)

Project Activation [cmake]

• generate (cmakeuserpresets) json file

Archive Handling [.tar/.tgz/.tbz2]

Unpack tar/tgz/tbz2/xz (lzma) files into target location

• support strip (remove top directories during unpack)
• support transform (transform filenames during unpack )

Project Activation [basic]

• CLI: ce activate
• collect tools
• generate PATH
• build compiler/linker
• build environment variables
• combine defines

Build Shell Scripts

PowerShell and bash shell scripts for installation and running.

• create CLI bootstrap/shell scripts,
• register aka.ms URLs
• create delta for environment and apply (via tmpfile)

Handle Multiple Registries

add section to AMF to support registries

• global

• project-level

• CLI: ce add-source

• CLI: ce remove-source

• CLI: ce list sources

Follow-up items

• need check for multiple host options passed are errors (--osx --windows etc)
• disallow MD5 checksums
• add a bunch of validation rules for AMF files:
  • bad settings/paths (not valid, wrong category, etc)
  • environment variable may only have letters, numbers, underscore

HTTPS artifact source

The HTTPS artifact source allows artifacts to be referred to by uri/location:
- ce add https://github.com/foo/someproj
- works for direct link to a AMF file or inferrs the correct name
- detect git repositories? ( see git https smart protocol docs )
- looks for [uri]/environment.yaml
- treats uri as a zipfile with an environment.yaml in it.
- this is actually done by making a catalog implementation that's named https :D

CI Script to rebuild repo index on commit

• Should rebuild the index.yaml file on commit.

Default implementation of debug() should not output anything if --debug switch is not supplied

cella/cli/lib/styling.ts

Line 60 in 1e06889

export let debug: (message?: any, ...optionalParams: Array<any>) => void = (text) => { console.log(`${cyan.bold('debug: ')}${text}`); };

The concern is that there is a window of time between process start and the call to initStyling where debug output may leak even if --debug has not been specified an the command line.

Two options have been discussed:

1. Make the default implementation output nothing. It only starts emitting output after initialization in initStyling.
2. Make the default implementation inspect argv for the --debug switch.

Parse CLI command line

Parse CLI command line

• commands and dispatch
• link help support to command

Investigation

A bit of investigation to identify if any changes to CMake are desired to handle functionality required.

• Identify if there are any feature gaps
• Create tracking items for feature tasks and connect to dependent features

New Project Profile

CLI: ce new - create a new cella.yaml file

• --id
• --summary
• --description
• --version
• [ ]--clone // this would make it easier to clone an artifact and update the version

Project Activation [makefile]

• Support whatever Makefiles need

ce activate provides no actionable message when run in a directory without a manifest

Fundamentals/Abstraction

Create reusable fundementals for core library and CLI utility projects.

Should include

• Filesystem Virtualization/Abstraction (ensure that we can account for VSCode FS later)
• I18N abstraction layer. (msg tables? library?)
  • longer term: translations?
  • longer term: propogation when plugging into VS?
• Message/Warning/Error streams w/colorization
  • able to capture messages for logging and bug reporting
  • facility for logging all writes/reads?

CE/VCPKG Merge Task: output JSON instead of YAML if when the manifest is saved as .json instead of .yaml

Indexing Sources

• read feed list (default: cella's builting repo) - DEFERRED until multiple feeds are done.
• create short-name index (requires all feeds loaded)
• read each source/feed
  • create package id index
  • read the source feed and get all the package ids
  • create index.yaml representation (can be cached) -- needed by repo CI script

Host Query/selection

• prepare host data
• parse host query
• handle host query
• filter on host query

Project Activation [vcxproj]

• support vcxproj projects

Extracting downloaded files should only emit a single file open call

I was thinking that too. I'm not sure I want to do anything about this yet; I'm still wincing thinking about how this will work out when I've got the code running under vscode where I don't control the fs implementation.

Sadly, the only reason I even need the size, is for progress. I added it in because there was a perceptible pause while it validated very large archives.

I'm going to defer fixing this until I spin up a test extension for vscode and see how I'm going to accomplish that there.

Originally posted by @fearthecowboy in #56 (comment)

Add 'documentation' section in settings

  documentation: # lists places docs can be launched from. https:// required if not in the pack.
    Sample Documentation: CMSIS/Documentation/NN/html/index.html

Timelines [SWAG]

Timeline Planning

Assumptions

• Two active developers
• Sprints are approximately Two weeks
• TShirt sizes are
  • [L] - 1 person-sprint
  • [M] - 1/2 person-sprint
  • [S] - 1/3 person-sprint

Milestones

1. Dev POC
   Minimal Viable Product
   Five sprints
   Approximately 10 weeks

2. PM Repo Population - Concurrent with MVP
   Bootstrapping the artifact repository
   Four sprints

3. Dev Project/Activation
   Adding ability to 'activate' a project, generate build config
   Two Sprints
   Approximately 4 weeks

4. Dev Additional Scenarios
   Additional Features (Mirroring,Sources,Flattening)
   Approximately 4 weeks

Inventory

• update local artifact index from git repo (git pull)
• generate list of installed packages/versions
• CLI ce list installed
• CLI ce list (project)

Individual Tool Activation

Activate an artifact that is installed (but not in a project)

• modify environment
• CLI: ce use <artifact> [--version ver]

Remote File Acquisition

Create Functionality to acquire and verify files from remote locations
(implement HttpFileSystem)

• Downloading URLs
  • progress
  • uses fs virtualization (no direct writes)
  • checksum verification
  • supports mirrors
  • checks local cache
• Nuget Package Download (via HTTP)
• Write to local cache

Telemetry Support

Create Telemetry Support hooks

• Select telemetry library (vscode-telemetry pkg? )
• Create appropriate hooks for code to easily call into
• Setup queries/??? to access telemetry data

Preparation

• setup CI scripts to build repo, run tests on PR
• create artifact metadata repository
• setup project TS linter, rules
• pick NPM dependencies
  • typescript @4.2
  • Yaml Parser: [email protected]
  • linting: eslint, @typescript-eslint/parser, @typescript-eslint/eslint-plugin
  • unit testing: mocha, @testdeck/mocha,
  • mediaquery parsing: css-mediaquery
  • semver

These dependencies to be decided closer to implementation:

• git : ? ( borrow code from gitlens https://github.com/eamodio/vscode-gitlens/tree/main/src/git ? )
• tar/gz/bz2/zip handling: ? (ideally, this would work with our fs abstraction)
• npm pkg handling: yarn
• telemetry: https://github.com/microsoft/vscode-extension-telemetry
• URL downloading with progress? Abortable? failover? (this is almost what I want: https://github.com/andresusanto/easydl )
• i18n ???? https://github.com/Microsoft/vscode-nls + https://github.com/Microsoft/vscode-nls-dev -> transifex?
  https://www.npmjs.com/package/es2015-i18n-tag (I think a bit nicer.)