fjudith / spinalker

spinalker

This project aims to contenairize the Spinalcom Building operating system to improve the appliation portability and security.

• Overview of Cloud Native Security
• Pod security policy standards
• CIS Kubernetes Security Benchmark
• Dockerfile Best Practices
• Building a trusted software supply chain

About Git submodules

Git submodules are only referenced to track the various SpinalCom repository, they are not actively used in container builds.

Guidelines

AKS Deployment

The following guide describes the steps to acheive the Spinal-Core deployment in Azure Kubernetes Service.

• Run the following commands to create the Kubernetes namespace and populate shared persitent volumes with the example dataset.

# Namespace
kubectl apply -f deploy/aks/00_spinalcom-ns.yaml
# Persistent volumes
kubectl apply -f deploy/aks/01_azurefile-csi-nfs-sc.yaml
kubectl apply -f deploy/aks/01_azurefile-csi-smb-sc.yaml
kubectl apply -f deploy/aks/02_spinal-core-hub-pvc.yaml
# Dataset provisionner
kubectl apply -f deploy/aks/dataset-provisioner-sec.yaml
kubectl apply -f deploy/aks/dataset-provisioner-job.yaml
kubectl -n spinalcom wait --for=condition=complete --timeout=5m job/dataset-provisioner

• One the dataset provisioning completed, run the following command to deploy Spinal-Core's application services.

kubectl apply -f deploy/aks/

Dataset restore

In certain failure conditions, the memory/dump.db file can get deleted by the core-hub pod. Run the following command to restart the dataset-provisioner job.

kubectl -n spinalcom get job "dataset-provisioner" -o json | jq 'del(.spec.selector)' | jq 'del(.spec.template.metadata.labels)' | kubectl replace --force -f -

Remove submodules

To remove a submodule you need to:

1. Remove the relevant section from the .gitmodules file.
2. Stage the .gitmodules changes git add .gitmodules
3. Delete the relevant section from .git/config.
4. Run git rm --cached path_to_submodule (no trailing slash).
5. Run rm -rf .git/modules/path_to_submodule (no trailing slash).
6. Commit git commit -m "Removed submodule"
7. Delete the now untracked submodule files rm -rf path_to_submodule

spinalker's Issues

[spinal-organ-forge] Replace the storage folder symlink by a static one

Is your feature request related to a problem? Please describe.
As dicussed with @Diclah , the spinal-organ-forge shares persistent data with the spinal-core-hub; or maybe the spinal-http-server (see. #2).

Describe the solution you'd like
At the spinal-organ-forge container build time:

1. create the `./viewerForgeFiles/ in the app work directory
2. Update the docker-compose deployment as such the spinal-core-hub and spinal-organ-forge shares the viewerForgeFiles folder using a docker volume.

Describe alternatives you've considered
It is planned in the future to use a shared folder.

Additional context
n/a

[Security] Leverage Secrets for sensitive environment variables

Is your feature request related to a problem? Please describe.
spinal-core-hub credentials are currently passed as hardcoded strings in deployments

Describe the solution you'd like
Leverage Kubernetes secrets for spinal-core-hub clients credentials:

• spinal-organ-api-gateway
• spinal-organ-forge
• spinal-organ-network_sample

Describe alternatives you've considered
n/a

Additional context
n/a

Diagramme d'architecture v0.0.1

Is your feature request related to a problem? Please describe.
n/a

Describe the solution you'd like
We must deliver an architecture diagram of the containerized solution to close the 0.0.1 milestone.
It will the planning of the next one, that will focus on data segregation and deduplication.

the diagram will be store in the ./docs/diagrams folder located in this repo.

Describe alternatives you've considered
n/a

Additional context
Add any other context or screenshots about the feature request here.
n/a

rapport de vulnérabilité

@fjudith ci-après le rapport de vulnérabilité.

19_04_2021_17_h_02_m_rapportRecencementVulnerabilite.txt

[spinal-core-hub] Simplify container build

Is your feature request related to a problem? Please describe.
Relates to the acheivement of #2

The spinal-core-hub container image is based on the node:12-alpine because the spinalhub binary installation is acheived using the https://github.com/spinalcom/spinal-core-hub repo.
As discussed with @Diclah, this deployment type is required only if running the pm2 binary based deployment method.

Describe the solution you'd like

1. Replace the node:12-alpine by the alpine:3 base image
2. Replace the npm install instruction by a spinalhub binary download
3. Add a sha256 checksum of the download binary
4. Update symbolic links if required
5. Build and test locally using the ./build/spinal-core-hub.sh and ./hack/spinal-core-hub.sh scripts

Describe alternatives you've considered
n/a

Additional context
n/a

[kubernetes] Add manifests for the containers deployment in k8s

Is your feature request related to a problem? Please describe.
The end goal is to run the application components in a loosly coupled manner

Describe the solution you'd like
Since the local docker-compose deployment validated, create resources manifests for the deployment of the application in the Kubernetes distributed operating system.

Describe alternatives you've considered
n/a

Additional context
n/a

[Security] Unify system user in container

Is your feature request related to a problem? Please describe.
Currently containers are using a different user accounts, which leads to file permission issue for those who are sharing data.

Describe the solution you'd like
Configure images to leverage a similar user account:

Affected images

• spinal-core-hub
• spinal-organ-dump_manager
• spinal-organ-forge
• spinal-core-hub-dataset

User/Group name: spinalcom, User/Group ID: 1001

Describe alternatives you've considered
n/a

Additional context
n/a

[spinal-core-hub] Are we using the right binary version ?

Describe the bug
We are currently using the spinalhub Freenium version published on the https://github.com/spinalcom/spinal-core-hub repository.

We are neither able to login to the web ui or interact with it using the SDK

To Reproduce
Steps to reproduce the behavior:

1. Go to deploy/docker
2. Run docker-compose up
3. Browse to 'https://localhost:8889/admin'
4. Try to login.
5. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS] Ubuntu 20.04
• Browser [e.g. chrome, safari] vivaldi (chromium) and chrome
• Version [e.g. 22] latest

Additional context
spinal-core-hub logs shows the following message .

stdout: days_elapsed -> -18701

[spinal-organ-network_sample] The pm2 process crashes after 3 attemps

Describe the bug
The spinal-organ-network_sample container build seems to be successful. But the pm2 process manager goes into failure and stops working after few seconds.

To Reproduce
Steps to reproduce the behavior:

1. Go to deploy/docker
2. Run command docker-compose up
3. Watch logs of the spinal-organ-dump-manager container
4. See error

spinal-organ-network_sample_1  | 2021-03-18T23:38:01: PM2 log: App [spinal-organ-network_sample-8888:0] starting in -fork mode-
spinal-organ-network_sample_1  | 2021-03-18T23:38:01: PM2 log: App [spinal-organ-network_sample-8888:0] online
spinal-organ-network_sample_1  | Disconnected from the server.
spinal-organ-network_sample_1  | 2021-03-18T23:38:02: PM2 log: App [spinal-organ-network_sample-8888:0] exited with code [0] via signal [SIGINT]
spinal-organ-network_sample_1  | 2021-03-18T23:38:02: PM2 log: 0 application online, retry = 1
spinal-organ-network_sample_1  | 2021-03-18T23:38:04: PM2 log: 0 application online, retry = 0
spinal-organ-network_sample_1  | 2021-03-18T23:38:04: PM2 log: Stopping app:spinal-organ-network_sample-8888 id:0
spinal-organ-network_sample_1  | 2021-03-18T23:38:04: PM2 log: pid=63 msg=process killed
spinal-organ-network_sample_1  | 2021-03-18T23:38:04: PM2 log: PM2 successfully stopped

Expected behavior
Container should not crash

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS] Ubuntu 20.04
• Browser [e.g. chrome, safari] n/a
• Version [e.g. 22] _

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Logs from the spinal-core-hub container

spinal-core-hub_1              |
spinal-core-hub_1              | Welcome to SpinalHub!
spinal-core-hub_1              |
spinal-core-hub_1              | License
spinal-core-hub_1              | =======
spinal-core-hub_1              |
spinal-core-hub_1              | Copyright 2018 SpinalCom - http://www.spinalcom.com
spinal-core-hub_1              |
spinal-core-hub_1              | Please read all of the following terms and conditions of the Free Software license Agreement ("Agreement") carefully. This Agreement is a legally binding contract between the Licensee (as defined below) and SpinalCom that sets forth the terms and conditions that govern your use of the Program. By installing and/or using the Program, you agree to abide by all the terms and conditions stated or referenced herein. If you do not agree to abide by these terms and conditions, do not demonstrate your acceptance and do not install or use the Program.
spinal-core-hub_1              |
spinal-core-hub_1              | You should have received the license along with this file. If not, see <http://resources.spinalcom.com/licenses.pdf>.
spinal-core-hub_1              | WARNING: This trial version is available for a 30 days period. After that, all your data will be lost!
spinal-core-hub_1              |
spinal-core-hub_1              |   days_elapsed -> -18705
spinal-core-hub_1              |
spinal-core-hub_1              | System information
spinal-core-hub_1              | ==================
spinal-core-hub_1              |
spinal-core-hub_1              | Version: 3.1.0 (freemium)
spinal-core-hub_1              | JS Port: 8888
spinal-core-hub_1              | C++ Port: 8890
spinal-core-hub_1              | Monitor Port: 8889
spinal-core-hub_1              | Web root folder: html
spinal-core-hub_1              | Database file: memory/dump.db
spinal-core-hub_1              | Database bulk folder: memory/data.db
spinal-core-hub_1              |
spinal-core-hub_1              | admin User ID: 168
spinal-core-hub_1              | admin User Password: JHGgcz45JKilmzknzelf65ddDadggftIO98P
spinal-core-hub_1              | Read/Write User ID: 644
spinal-core-hub_1              | Read/Write User Password: 4YCSeYUzsDG8XSrjqXgkDPrdmJ3fQqHs
spinal-core-hub_1              | Read User ID: 1657
spinal-core-hub_1              | Read User Password: LQv2nm9G2rqMerk23Tav2ufeuRM2K5RG
spinal-core-hub_1              |
spinal-core-hub_1              | Your hub is now running...

[spinal-http-server] Move spinal-core-hub static content to spinal-http-server

Is your feature request related to a problem? Please describe.
Currently the spinal-core-hub image contains the static content generated by the https://github.com/spinalcom *-browser-*. As discussed with @Diclah this content should move from the spinal-core-hubto thespinal-http-server` container

Describe the solution you'd like
Move the FROM ... as ... and COPY --from=...instruction to thespinal-http-servercontainer. Update the ./src/spinal-http-server/index.js so that it access the static content from the.html` directory.

Describe alternatives you've considered
From a general point of view, the static content should be stored in an object storage service (s3) or content delivery network (cnd). but for the sake of speed an reproducibility of issues from the pm2 based deployment, the static must be stored in the web server container as of now.

[spinal-http-server] Populate the templates folder

Is your feature request related to a problem? Please describe.
As discussed with @Diclah , the nerve-center/templates directory contains code generated by the https://github.com/spinalcom *-template-* projects.

According to the #2 issue, we must manage to populate the templates directory in the spinal-http-server container.
Only the following projects are currently populating the directory:

• https://github.com/spinalcom/spinal-env-admin-panel-hub-status
• https://github.com/spinalcom/spinal-env-admin-panel-user-manager
• https://github.com/spinalcom/spinal-env-drive-panel-logs
• https://github.com/spinalcom/spinal-env-drive-plugin-base
• https://github.com/spinalcom/spinal-env-drive-plugin-digital_twin
• https://github.com/spinalcom/spinal-env-drive-plugin-version_manager
• https://github.com/spinalcom/spinal-env-drive-plugin-visa

Describe the solution you'd like
For each of the previously mentioned projects:

1. Add the git submodule `git submodule add src/
2. Create the build script ./build/<repo name>.sh
3. Create the container build folder ./containers/<repo name> and create the Dockerfile.

• Populate the .env file from the version filed of the project's package.json

4. Test the build locally using build scripts
5. Only If the build local build is successful, add the <repo name> to the image matrix located in ./.github/workflows/common_container_images.yaml action file

Once all images are available

Add the FROM ... as ... and COPY --from instructions to populate the html/lib directory of the spinal-http-server container with *-env-* generated artifacts.

Describe alternatives you've considered
Download *-env-* from the projet's build directory in the lib folder of the spinal-http-server container.

Additional context
n/a

[spinal-http-server] Populate the lib folder

[UPDATE 1: project dependency]

Is your feature request related to a problem? Please describe.
As discussed with @Diclah , the nerve-center/lib directory contains code generated by the https://github.com/spinalcom *-env* projects.

According to the #2 issue, we must manage to populate the lib directory in the spinal-http-server container.
Only the following projects are currently populating the directory:

• https://github.com/spinalcom/spinal-browser-admin
• https://github.com/spinalcom/spinal-browser-appstore
• https://github.com/spinalcom/spinal-browser-viewer

Describe the solution you'd like
For each project:

1. Add the git submodule `git submodule add src/
2. Create the build script ./build/<repo name>.sh
3. Create the container build folder ./containers/<repo name> and create the Dockerfile.

• Populate the .env file from the version filed of the project's package.json

4. Test the build locally using build scripts
5. Only If the build local build is successful, add the <repo name> to the image matrix located in ./.github/workflows/common_container_images.yaml action file

Once all images are available

Add the FROM ... as ... and COPY --from instructions to populate the html/lib directory of the spinal-http-server container with *-env-* generated artifacts.

Describe alternatives you've considered
Download *-env-* from the projet's build directory in the lib folder of the spinal-http-server container.

Additional context
n/a

[CI optimisation] Segregate build workflows per category

Is your feature request related to a problem? Please describe.
The common workflow matrix builds many different Spinalcom's projects.
The goad is to spread those project in dedicated project category workflows (i.e. browser, env, etc.)

This will help to prevent the rebuild of un modified code each push or PR, and increase the build speed

Describe the solution you'd like
Create dedicated workflows building following project categories:

• browser: build browser related static content browser_container_images.yaml
• env-admin: build admin panels related static content envadmin_container_images.yaml
• env-drive: build drive panels related static content envdrive_container_images.yaml
• env-viewer: build viewer related static content envviewer_container_images.yaml

What will remain in existing common and static workflows ?

• common: build executable code that can be retreived from https://github.com/spinalcom
• static: build executable code that cannot be retreived from https://github.com/spinalcom and built from the src directory.

Describe alternatives you've considered
n/a

Additional context
n/a

[spinal-dataset-provisioner] Deliver pedifined dataset to spinal-core-hub

Is your feature request related to a problem? Please describe.
The v0.0.1 milestone delivered only an empty instance.
The goal is now to use an example dataset to play with the application

Describe the solution you'd like
The spinal-dataset-provisioner is an exclusive feature of the containerized deployement.
The goal is to being able to

1. Download a tarball archive stored in a protected online object store,
2. Decompress the archive to populate persistent volumes
3. Apply appropriate permissions to enable sharing with spinal-organ-forge and spinal-organ-dump_manager containers.

Describe alternatives you've considered
n/a

Additional context
n/a

[spinal-http-server] Cannot open `viewer/models/Resource/3D View/{3D} 341878/`

Describe the bug
``

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[spinal-organ-network_sample] build fails

I'm getting the following error message when building the spinal-organ-network_sample

#11 [builder 5/5] RUN ["npm", "run", "build"]
#11 sha256:33b09aa338cb676883160d2b40d060056b1f5273c5b90a3899cafcacd6c6b917
#11 0.339 
#11 0.339 > [email protected] build /usr/share/spinal-organ-network_sample
#11 0.339 > rm -rf dist && tsc
#11 0.339 
#11 3.851 src/modules/InputData/InputData.ts(174,26): error TS2339: Property 'type' does not exist on type 'never'.
#11 3.853 src/modules/InputData/InputData.ts(175,17): error TS2339: Property 'currentValue' does not exist on type 'never'.
#11 3.853 src/modules/InputData/InputData.ts(176,26): error TS2339: Property 'type' does not exist on type 'never'.
#11 3.853 src/modules/InputData/InputData.ts(179,19): error TS2339: Property 'currentValue' does not exist on type 'never'.
#11 3.853 src/modules/InputData/InputData.ts(181,19): error TS2339: Property 'currentValue' does not exist on type 'never'.
#11 3.869 npm ERR! code ELIFECYCLE
#11 3.870 npm ERR! errno 2
#11 3.875 npm ERR! [email protected] build: `rm -rf dist && tsc`
#11 3.875 npm ERR! Exit status 2
#11 3.875 npm ERR! 
#11 3.875 npm ERR! Failed at the [email protected] build script.
#11 3.875 npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
#11 3.883 
#11 3.884 npm ERR! A complete log of this run can be found in:
#11 3.884 npm ERR!     /root/.npm/_logs/2021-03-15T02_54_27_543Z-debug.log
#11 ERROR: executor failed running [npm run build]: exit code: 2
------
 > [builder 5/5] RUN ["npm", "run", "build"]:
------
Dockerfile:31
--------------------
  29 |     WORKDIR ${APP_PATH}/spinal-organ-network_sample
  30 |     
  31 | >>> RUN ["npm", "run", "build"]
  32 |     
  33 |     # -----------------------------------------------
--------------------

Source repo: https://github.com/spinalcom/spinal-organ-network_sample

[spinal-http-server] Replace express.js web server

Is your feature request related to a problem? Please describe.
spinal-http-server is a node.js application running the express and express-http-proxy packages which is not easy to observe or scale.

Describe the solution you'd like
Update the spinal-http-server to run either an Nginx or Caddy web server to serve the static content and reverse-proxy the trafic to the spinal-core-hub instance.

List of routes to proxy

  '/sceen',
  '/get_user_id',
  '/get_admin_id',
  '/get_new_account',
  '/get_confirm_new_account',
  '/get_resend_confirmation',
  '/get_new_password',
  '/get_change_user_password',
  '/get_delete_account',
  '/get_change_user_password_by_admin',
  '/get_delete_account_by_admin',
  '/get_change_account_rights_by_admin'

Describe alternatives you've considered
n/a

Additional context
n/a

[spinal-organ-forge] The pm2 process crashes after 3 attemps

Describe the bug
The spinal-organ-forge container build seems to be successful. But the pm2 process manager goes into failure and stops working after few seconds.

To Reproduce
Steps to reproduce the behavior:

1. Go to deploy/docker
2. Run command docker-compose up
3. Watch logs of the spinal-organ-dump-manager container
4. See error

spinal-organ-forge_1           | Disconnected from the server.
spinal-organ-forge_1           | 2021-03-18T23:38:06: PM2 log: App [spinal-organ-forge:0] exited with code [0] via signal [SIGINT]
spinal-organ-forge_1           | 2021-03-18T23:38:06: PM2 log: 0 application online, retry = 3
spinal-organ-forge_1           | 2021-03-18T23:38:08: PM2 log: 0 application online, retry = 2
spinal-organ-forge_1           | 2021-03-18T23:38:09: PM2 log: App [spinal-organ-forge:0] starting in -fork mode-
spinal-organ-forge_1           | 2021-03-18T23:38:09: PM2 log: App [spinal-organ-forge:0] online
spinal-organ-forge_1           | [class FileVersionModel extends Model]

Expected behavior
Container should not crash

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS] Ubuntu 20.04
• Browser [e.g. chrome, safari] n/a
• Version [e.g. 22] _

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Logs from the spinal-core-hub container

spinal-core-hub_1              |
spinal-core-hub_1              | Welcome to SpinalHub!
spinal-core-hub_1              |
spinal-core-hub_1              | License
spinal-core-hub_1              | =======
spinal-core-hub_1              |
spinal-core-hub_1              | Copyright 2018 SpinalCom - http://www.spinalcom.com
spinal-core-hub_1              |
spinal-core-hub_1              | Please read all of the following terms and conditions of the Free Software license Agreement ("Agreement") carefully. This Agreement is a legally binding contract between the Licensee (as defined below) and SpinalCom that sets forth the terms and conditions that govern your use of the Program. By installing and/or using the Program, you agree to abide by all the terms and conditions stated or referenced herein. If you do not agree to abide by these terms and conditions, do not demonstrate your acceptance and do not install or use the Program.
spinal-core-hub_1              |
spinal-core-hub_1              | You should have received the license along with this file. If not, see <http://resources.spinalcom.com/licenses.pdf>.
spinal-core-hub_1              | WARNING: This trial version is available for a 30 days period. After that, all your data will be lost!
spinal-core-hub_1              |
spinal-core-hub_1              |   days_elapsed -> -18705
spinal-core-hub_1              |
spinal-core-hub_1              | System information
spinal-core-hub_1              | ==================
spinal-core-hub_1              |
spinal-core-hub_1              | Version: 3.1.0 (freemium)
spinal-core-hub_1              | JS Port: 8888
spinal-core-hub_1              | C++ Port: 8890
spinal-core-hub_1              | Monitor Port: 8889
spinal-core-hub_1              | Web root folder: html
spinal-core-hub_1              | Database file: memory/dump.db
spinal-core-hub_1              | Database bulk folder: memory/data.db
spinal-core-hub_1              |
spinal-core-hub_1              | admin User ID: 168
spinal-core-hub_1              | admin User Password: JHGgcz45JKilmzknzelf65ddDadggftIO98P
spinal-core-hub_1              | Read/Write User ID: 644
spinal-core-hub_1              | Read/Write User Password: 4YCSeYUzsDG8XSrjqXgkDPrdmJ3fQqHs
spinal-core-hub_1              | Read User ID: 1657
spinal-core-hub_1              | Read User Password: LQv2nm9G2rqMerk23Tav2ufeuRM2K5RG
spinal-core-hub_1              |
spinal-core-hub_1              | Your hub is now running...