unicef / magasin

Cloud native open-source end-to-end data / AI / ML platform

magasin : cloud native end-to-end open-source data platform

magasin enables organizations to perform of automatic data ingestion, storage, analysis, ML/AI compute and visualization at scale.

Learn more about why magasin and its architecture.

Get started

🚀 Go to get started

In the get started you will install magasin within your local machine or in a kubernetes cluster, then you will perform an end-to-end data analysis that includes:

• exploratory analysis of a data source,
• create a pipeline to automate data ingestion, and
• create a dashboard to present your findings.

Documentation

All the documentation, features, architecture, get started, advanced installation, deployment, contributing...

📚 Go to documentation

Contributing

Magasin follows an open approach towards accepting contributions.

🇺🇳 Learn about contributing

License

Apache License Version 2.0 UNICEF

magasin's Issues

Installer declare -A invalid option on MacOS

On MacOs running ./install-magasin.sh in zsh the following is displayed:

-----------
./install-magasin.sh: line 254: declare: -A: invalid option
declare: usage: declare [-afFirtx] [-p] [name[=value] ...]

Automate magasin changes in the daskhub helm chart

By default daskhub helm chart sets a public proxy service in LoadBalancer mode.
f that is installed in a regular kubernetes cloud service, it exposes a jupyterhub interface to any user without password and therefore allows running arbitrary code in that instance.

To prevent that this change is added in the values.yaml file within the helm chart.

jupyterhub:
  proxy:
    service:
      type: ClusterIP

However, if the helm chart is updated (dev-scripts/update-helm-charts.sh) this change will be overwritten.

Get started / Step 2 - Sections 3.2, 3.3 and summary uncomplete

In Get started step 2 sections

• 3.2 adding a scheduler https://unicef.github.io/magasin/get-started/automate-data-ingestion.html#adding-a-scheduler
• Section 3.3 deploy pipeline in the cluster https://unicef.github.io/magasin/get-started/automate-data-ingestion.html#deploy-the-pipeline-in-the-cluster
• Summary

are uncomplete.

Add in documentation a reference to troubleshooting when installing superset in an ARM64 architecture.

Installer does not work when missing dependencies using curl piping

In the installer, when it is run through curl piping and there are dependencies missing (which is almost always as mag is usually not installed), the installation fails when it reaches the point in which it asks the user if he wants to install the dependencies.

The reason for this is that the shell launched with curl piping is not interactive.

Solution:
Detect if the shell is interactive, if so => allow the question otherwise automatically install the dependencies.

Add expected output in the Manual Installation instructions

Under the Step 1. Install tools on your computer in the Manual installation page, include expected output following execution of command.

This should be, for instance:

Kubernetes control plane is running at https://127.0.0.1:6443 CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Run magsh in Windows

magsh was not tested on windows when released for the First time.

After testing it on windows it should be something like:

docker run -ti -P -v "$env:USERPROFILE\.kube\config:/kube/config" -v "$env:USERPROFILE/.mc:/root/.mc" -v "$env:USERPROFILE/magsh:/shared" merlos/magsh:latest

Where $env:USERPROFILE will be replaced by C:\Users\currentUserName\

Improve narrative and formatting of why magasin

Why magasin is a key document as is the current introduction on why magasin and why it is needed.

• Improve the formatting (too many headers)
• Improve the narrative itself.

Uninstall chart tenant on uninstaller not working properly

[ i ] helm uninstall tenant --namespace magasin-tenant
Error: failed to delete release: tenant
[ ✗ ] Could not uninstall magasin/tenant in the namespace magasin-tenant
namespace "magasin" deleted

May be related with uninstalling operator first.

Install dagster within the installer

Dagster is a very agile product and the newer versions may not be compatible with the older ones. To ensure that the correct version of dagster is installed this can be somehow managed by the setup.

The version can be obtained by running

helm list --all-namespaces | grep magasin-dagster | awk '{print $10}'

Where magasin-dagster is the namespace for the dagster component within the realm magasin

Then the package can be setup using

pip install dagster==<version>

Alternatively, dagster could be a dependency within the setup.py of the CLI. So that by installing the mag client it is also installed dagster.

Cannot create main namespace for only suffix realm

Realms with only a suffix start with - (f.i., '-dev') . During the installation, the installer tries to create a namespace with the name "-" (f.i., '-dev'). However given the rules for creating a namespace this is not allowed.

This namespace is created for future use, such as identification of realms, or magasin workloads.

The conditions are restricted by label names RFC 1123 (kubernetes creates a label when a new namespace is created)
https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#dns-label-names

• metadata.name: Invalid value: "_dev": a lowercase RFC 1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name', or '123-abc', regex used for validation is 'a-z0-9?')
• metadata.labels: Invalid value: "dev": a valid label must be an empty string or consist of alphanumeric characters, '-', '' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')

Options:

1. Do not allow the suffix only realms
2. Add some kind of prefix in the main realm name such as "m--".

Enhancements in Contributing and support Documentation

Cotributing

• Ways of contributing: till now it is focused on developers, but there are other ways of contributing such as filing an issue, writing about magasin, sharing in social networks.
• How to report an issue + Issue template
• Code of conduct

Support

• include what are the ways of getting support from the community and what kind of support is available.
• Include internal support within UNICEF.

Advanced installation typos

In Advanced installation, in the section customizing the setup of each component

• Superset default values link is not working
• Missing links to minio value components operator and tenant.

Add an example that uses Azure Blob to store the data

In the tutorial we use fsspec to save into MinIO, however providing an example of the same for azure is useful.

Helm command for adding repo breaking

Helm command for adding repo manually as below seems to be breaking

helm repo add https://unicef.github.io/magasin/

Should be:

helm repo add magasin https://unicef.github.io/magasin/

Add administrator guides

There are some tasks that are commonly done in the different components such as adding a new store in drill or adding a new datasource in superset.

This needs to be broken down and expanded.

Incorrect format in Step 2 what's next

Include trufflehog as part of the security processes

More on:
https://github.com/trufflesecurity/trufflehog

minio alias creation and management

Currently alias creation is manual. Ideally, this should be managed automatically during setup and/or in the mag CLI.

Unformatted output in troubleshooting FAQ

Web publishing Github action does not include install and uninstall script

Quarto publish action wipes out the gh-pages.

As a result, when a new version is deployed anything that is not managed by quarto is removed.
The current version does manage to keep the index.yaml file, but does not do the same with install and unistall scripts.

If a new version of the web is released it breaks the curl piped install and uninstall as the files are removed.

curl -X https://unicef.github.io/magasin/install-magasin.sh | bash

Automate changes in superset values.yaml

There are three items in values than need to be updated from the default values of the superset helm chart

• bootstrapScript: |
  pip installs sqlalchemy-drill
• extraSecretEnv SUPERSET_SECRET_KEY: 'LNS78tGfwCYyiIggQxLBviTT83itPTgbho822Wr9IWjo/cNcojL/CZK5'
• initImage:
  repository: apache/superset
  tag: ea6cbcef3e980e42d3a19b8fc5928f973fd7be4a-dockerize-linux-amd64-3.9-slim-bookworm
  pullPolicy: IfNotPresent (No longer needed, multi-arch image added when fixing #50)

Enable consent popup on public website

A popup or something is displayed requesting consent for cookies. If rejected, then do not enable matomo on the browser.

Add security.txt to publish-web deployment

Security.txt provides a machine-readable file which defines core attributes or your VDP, and is designed to be hosted on websites.

The security.txt file should be placed under the /.well-known/ path (/.well-known/security.txt) on websites. It can also be placed in the root directory (/security.txt) of a website, especially if the /.well-known/ directory cannot be used for technical reasons or as a fallback. The file can be placed in both locations of a website at the same time.

For more information visit https://securitytxt.org/ and the associated RFC8615.

We have the /docs/security.txt, It has to be added to the publish-web github action.

Add references to why magasin and architecture in the Get started intro

If someone needs more background on the app, it needs some kind of intro references.

Error: Unable to create magasin-dagster namespace using manual installation

Running the helm command below to create a magasin-dagster namespace returns error using manual installation.

helm install dagster magasin/dagster --namespace magasin-dagster --create-namespace

Error returned:

Error: INSTALLATION FAILED: template: dagster/templates/deployment-webserver.yaml:3:4: executing "dagster/templates/deployment-webserver.yaml" at <include "deployment-webserver" $data>: error calling include: template: dagster/templates/helpers/_deployment-webserver.tpl:36:38: executing "deployment-webserver" at <include (print $.Template.BasePath "/configmap-instance.yaml") .>: error calling include: template: dagster/templates/configmap-instance.yaml:16:12: executing "dagster/templates/configmap-instance.yaml" at <include "dagsterYaml.scheduler.daemon" .>: error calling include: template: no template "dagsterYaml.scheduler.daemon" associated with template "gotpl"

Additional context:
kubectl version
Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.4

Install mentions twice helm working

[ i ] Verifying commands are working...
[ ✓ ] kubectl is working
[ ✓ ] helm is working
[ ✓ ] helm is working
[ i ] magasin helm repository already exists. Resetting it...
[ i ] Running: helm repo remove magasin; helm repo add magasin http://unicef.github.io/magasin 

Change minio tenant to use ssl

Enable ssl in minio tenant by default

Memory allocation prevents install magasin in docker desktop

If there is not enough memory allocated in docker desktop magasin cannot be installed successfully.
This also affects minikube.

This needs to be documented.

mag-cli support for address in port forward

When magsh was introduced (PR #58), in order to support port forwarding in the Docker image to allow opening the UI in localhost, it was needed to set the --address 0.0.0.0 in the kubectl command of mag <component> ui.
Adding --address 0.0.0.0 makes kubectl open the port in all interfaces. In the case of Docker, it can be done through

 [Kubernetes cluster service] <------> [docker-image (runs kubectl port-forward) docker-ip ]<----->[docker host: localhost] 

Before this change kubectl only listened to localhost.

Opening the port in all interfaces is not an issue in the docker image, but running mag-cli in a computer, it will open the ports in all the interfaces, which may create a attack vector.

Potential solutions

1. Enable --address parameter in mag <component> ui and any command that forwards a port so that by default it opens localhost but there is an option for launching the ui kubectl listening to all ports.

2. In docker run test using ---network=host when launching magsh.

3. Enable a config setting for enforcing the default behaviour. That way it can be enforced through config to set the address to listen to by default.

4. Make the mag client savier. If it cannot listen to localhost (which happens in the image) try to listen to the (hostname -i) address.

References:

• How to double port forward kubernetes - docker - localhost
  https://serverfault.com/questions/1065922/double-port-forwarding-kubernetes-docker

Enable stats during the release of the public website

Display a consent popup & enable matomo